optimize `pget_db_query` by allowing field filter
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
## User Story
In order to reduce data requested from the h20 db, datagov wants to update the `pget_db_query` db interface function to support specific fields
## Acceptance Criteria
[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development#Behavioral_specifications).]
- [ ] GIVEN the [pget_db_query](https://github.com/GSA/datagov-harvester/blob/358100ad7a5f2a703c094f1e489ead6d71b84b99/database/interface.py#L683) function \
WHEN a list of field strings is used as input \
THEN only those fields from the model will be returned in the output
## Background
- working on [#5358](https://github.com/GSA/data.gov/issues/5358) I discovered that `pget_db_query` returns all model fields which we don't need when we're clearing out records (we only need record identifiers and the harvest source)
## Security Considerations ([required](https://nvd.nist.gov/800-53/Rev4/control/CM-4))
[comment]: # "Our SSP says 'The Data.gov team ensures security implications are considered as part of the agile requirements refinement process by including a section in the issue template used as a basis for new work.' so please don't remove this section without care."
[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]
## Sketch
- i got ahead and made this solution for [#5358](https://github.com/GSA/data.gov/issues/5358) but it isn't needed. here's how i did it...
```python
def get_model_fields_by_filter(self, model, fields_filter=None):
"""
return all Columns from the model or just those in fields_filter
"""
if fields_filter is None:
field_names = [field.name for field in model.__table__.columns]
else:
field_names = [
field.name
for field in model.__table__.columns
if field.name in fields_filter
]
return [getattr(model, field) for field in field_names]
#...
def pget_db_query(self, model=None, facets="", order_by="asc", **kwargs):
#...
model_data = self.get_model_fields_by_filter(
model_name, kwargs.get("fields_filter")
)
return (
self.db.query(model_name)
.options(load_only(*model_data))
.filter(text(facet_string))
.order_by(order_by_val)
)
```
[Notes or a checklist reflecting our understanding of the selected approach]
Contributor guide
Assessment
This issue has not been assessed yet.