consider extending sort field to all fields and/or changing HTML stripping
- Dominant language
- Rust
- Stars
- 30.5k
- Forks
- 3.2k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 60
Description
Currently the sort field stores the contents of a single field with HTML stripped. This enables faster sorting, and allows the user to find text that has embedded HTML in it, if that text occurs in the sort field.
There are some downsides to the current approach:
- if the user tries to search for text with embedded HTML that is not in the sort field, it is not found
- substring matches require a full table scan
- the current HTML stripping replaces tags with an empty string, so `foo
bar` becomes `foobar`. Because the lack of spaces is hard to read, the browse screen currently loads the full field content and uses a different HTML stripping routine that replaces the tags with a space, which is more expensive.
If instead of inserting just a single field into the sort field, we inserted sort_field + all_other_fields, using the standard field separator, it would solve the searching issue, at the cost of extra disk space.
A more comprehensive alternative would be to use sqlite's FTS, which is something I haven't really looked into yet. I gather it does quite well with space-separated text, but it may not be as useful for languages like Japanese? I'm also a bit concerned by the previous security issues it had, which could be triggered with malformed databases.
Contributor guide
Assessment
This issue has not been assessed yet.