GSA / GSA/data.gov

optimize `pget_db_query` by allowing field filter

Open
#5,372 0 comments 0 reactions 0 assignees View on GitHub
Harvest-DB
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

## User Story

In order to reduce data requested from the h20 db, datagov wants to update the `pget_db_query` db interface function to support specific fields

## Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development#Behavioral_specifications).]

- [ ] GIVEN the [pget_db_query](https://github.com/GSA/datagov-harvester/blob/358100ad7a5f2a703c094f1e489ead6d71b84b99/database/interface.py#L683) function \
WHEN a list of field strings is used as input \
THEN only those fields from the model will be returned in the output

## Background
- working on [#5358](https://github.com/GSA/data.gov/issues/5358) I discovered that `pget_db_query` returns all model fields which we don't need when we're clearing out records (we only need record identifiers and the harvest source)

## Security Considerations ([required](https://nvd.nist.gov/800-53/Rev4/control/CM-4))

[comment]: # "Our SSP says 'The Data.gov team ensures security implications are considered as part of the agile requirements refinement process by including a section in the issue template used as a basis for new work.' so please don't remove this section without care."
[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

## Sketch
- i got ahead and made this solution for [#5358](https://github.com/GSA/data.gov/issues/5358) but it isn't needed. here's how i did it...

```python
def get_model_fields_by_filter(self, model, fields_filter=None):
"""
return all Columns from the model or just those in fields_filter
"""
if fields_filter is None:
field_names = [field.name for field in model.__table__.columns]
else:
field_names = [
field.name
for field in model.__table__.columns
if field.name in fields_filter
]

return [getattr(model, field) for field in field_names]

#...

def pget_db_query(self, model=None, facets="", order_by="asc", **kwargs):
#...
model_data = self.get_model_fields_by_filter(
model_name, kwargs.get("fields_filter")
)

return (
self.db.query(model_name)
.options(load_only(*model_data))
.filter(text(facet_string))
.order_by(order_by_val)
)

```

[Notes or a checklist reflecting our understanding of the selected approach]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.