"every doc needs a text field/property" policy
- Vorherrschende Sprache
- Python
- Sterne
- 391
- Forks
- 58
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
**Is your feature request related to a problem? Please describe.**
@andrewyates points out these two use cases when working with docs:
1. do something sane for the situation where you want to swap out the datasets without specializing your model to one (e.g., treat cord section headers as part of the paragraph)
2. tailor your approach to collection-specific markup, like cord section headers, extra WaPo fields, maybe dates in robust04, etc
Right now the design favours (2) -- trying to remain as true to the original dataset as possible. But the (1) use case is still pretty valuable, and right now it can be tricky -- especially when some fields contain lists of sub-elements. (Without this, it's not so hard to always have the user specify a list of fields to concatenate.)
**Describe the solution you'd like**
Enforce a new policy that all document types need a `text` field that concatenates all "valuable" text (judgment call here), without markup. To avoid excessive redundancy/memory/etc., this could be in the form of a property of the named tuple, like so:
```
>>> from typing import NamedTuple
>>> class Doc(NamedTuple):
... doc_id: str
... title: str
... body: str
... @property
... def text(self):
... return f'{self.title} {self.body}'
...
>>> Doc('1', 'title', 'body').text
'title body'
```
**Describe alternatives you've considered**
As mentioned above, the original design was pushing the concatenation work on to the user. But this gets much more challenging for list-based fields. This technique can still be used, even if there's this new policy.
**Additional context**
We already have a policy where all documents need a `doc_id`.
Do we need a similar policy for queries? Probably not, as the query fields tend to be distinct representations of the information need---which probably shouldn't be combined, as that would not really reflect a real-life situation.
Since properties are not represented in the NamedTuple fields, how is this incorporated into the documentation?
Does any code that relies on `_fields` or `__annotations__` need to be updated to reflect the new `text` property?
Does this have an effect on pickled content (i.e., in built docstores)? Will we need to include migration logic for a ton of datasets (uggh) or will the new property flow through automatically in new versions?
Is there a complication with the trend for approaches to passage the document body and prepend the title to every passage (e.g., in ColBERT and others, I think)? This would mean that the first passage may have the title repeated, for instance.
How much is too much work to be done in the property? Could HTML parsing be done there? Should it be cached once processed?
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.