Check if the schema contains timestamp field if timestamp_field is not None
- Dominant language
- Python
- Stars
- 350
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
Currently, we can construct a Source with timestamp_field set but not contained in the schema like the following test. We should raise an exception in Feathub instead of failing until submitting to a processor.
```
def test_get_output_fields(self):
field_names = ["id", "val1", "val2", "val3"]
source = DataGenSource(
name="source_1",
schema=Schema(field_names, [Int64, Int64, Int64, Int64]),
timestamp_field="lpep_dropoff_datetime",
timestamp_format="%Y-%m-%d %H:%M:%S",
keys=["id"],
)
feature_view_2 = DerivedFeatureView(
name="feature_view_2",
source=source,
features=["val2", "val3"],
keep_source_fields=False,
)
built_feature_view_2 = feature_view_2.build(self.registry)
if isinstance(built_feature_view_2, FeatureView):
self.assertEqual(
["lpep_dropoff_datetime", "id", "val2", "val3"],
built_feature_view_2.get_output_fields(field_names),
)
else:
raise Exception("FeatureView is expected.")
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the test_get_output_fields case and the DataGenSource construction shown in the issue, then trace where timestamp_field and schema are validated before processor submission. Done means an invalid timestamp_field raises an exception in Feathub during source or feature-view setup, with coverage for the missing schema field.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100