alibaba / alibaba/feathub

Check if the schema contains timestamp field if timestamp_field is not None

Open
#103 0 comments 0 reactions 0 assignees View on GitHub
type:improvement
Dominant language
Python
Stars
350
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Currently, we can construct a Source with timestamp_field set but not contained in the schema like the following test. We should raise an exception in Feathub instead of failing until submitting to a processor.
```
def test_get_output_fields(self):
field_names = ["id", "val1", "val2", "val3"]
source = DataGenSource(
name="source_1",
schema=Schema(field_names, [Int64, Int64, Int64, Int64]),
timestamp_field="lpep_dropoff_datetime",
timestamp_format="%Y-%m-%d %H:%M:%S",
keys=["id"],
)

feature_view_2 = DerivedFeatureView(
name="feature_view_2",
source=source,
features=["val2", "val3"],
keep_source_fields=False,
)

built_feature_view_2 = feature_view_2.build(self.registry)

if isinstance(built_feature_view_2, FeatureView):
self.assertEqual(
["lpep_dropoff_datetime", "id", "val2", "val3"],
built_feature_view_2.get_output_fields(field_names),
)
else:
raise Exception("FeatureView is expected.")

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the test_get_output_fields case and the DataGenSource construction shown in the issue, then trace where timestamp_field and schema are validated before processor submission. Done means an invalid timestamp_field raises an exception in Feathub during source or feature-view setup, with coverage for the missing schema field.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.