feast-dev / feast-dev/feast

s3_staging_location of AthenaOfflineStoreConfig causes materialization issues if work group specifies output location

Open
#5,518 0 comments 0 reactions 1 assignee Claimed by @haoxu0 View on GitHub
kind/bug priority/p2
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

Raising again this issue first seen in https://github.com/feast-dev/feast/issues/3326.

Currently both `s3_staging_location` and `workgroup` are mandatory arguments for the `AthenaOfflineStoreConfig`.

My config looks the following way:
```python
def get_feature_store_config(
s3_bucket: str, project: str, athena_workgroup: str, region: str = "eu-central-1"
return RepoConfig(
project=project,
registry=RegistryConfig(path=f"s3://{s3_bucket}/{project}/registry.db"),
provider="aws",
offline_store=AthenaOfflineStoreConfig(
type="athena",
data_source="AwsDataCatalog",
region=region,
database=f"feature_store_{project}",
workgroup=athena_workgroup,
s3_staging_location=f"s3://{s3_bucket}/{project}/athena-staging",
),
online_store=DynamoDBOnlineStoreConfig(region=region),
entity_key_serialization_version=3,
)

```

if the workgroup I specify defines an output location, running this code:
```python
fs = FeatureStore(
config=get_feature_store_config(
s3_bucket_fs,
feature_store_name,
f"wg_re-data-services-feature-store-stk-{env}",
)
)
fs.materialize(
start_date=result_df["timestamp"].min(),
end_date=datetime.fromisoformat(data_interval_end),
feature_views=json.loads(feature_views),
)
```

results in the following error:

[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/feast/infra/utils/aws_utils.py:766, in execute_athena_query_async(athena_data_client, data_source, database, workgroup, query)
764 try:
765 # return athena_data_client.execute_statement(
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - --> 766 return athena_data_client.start_query_execution(
767 QueryString=query,
768 QueryExecutionContext={"Database": database, "Catalog": data_source},
769 WorkGroup=workgroup,
770 )
772 except ClientError as e:
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/botocore/client.py:595, in ClientCreator._create_api_method.._api_call(self, *args, **kwargs)
594 # The "self" in this scope is referring to the BaseClient.
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - --> 595 return self._make_api_call(operation_name, kwargs)
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/botocore/context.py:123, in with_current_context..decorator..wrapper(*args, **kwargs)
122 hook()
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - --> 123 return func(*args, **kwargs)
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/botocore/client.py:1058, in BaseClient._make_api_call(self, operation_name, api_params)
1057 error_class = self.exceptions.from_code(error_code)
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - -> 1058 raise error_class(parsed_response, operation_name)
1059 else:
[2025-07-15, 15:28:44 UTC] {logging_mixin.py:188} INFO - InvalidRequestException: An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: The Create Table As Select query failed because it was submitted with an 'external_location' property to an Athena Workgroup that enforces a centralized output location for all queries. Please remove the 'external_location' property and resubmit the query.

If I remove the output location from the work group, I get this:
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/feast/infra/utils/aws_utils.py:766, in execute_athena_query_async(athena_data_client, data_source, database, workgroup, query)
764 try:
765 # return athena_data_client.execute_statement(
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - --> 766 return athena_data_client.start_query_execution(
767 QueryString=query,
768 QueryExecutionContext={"Database": database, "Catalog": data_source},
769 WorkGroup=workgroup,
770 )
772 except ClientError as e:
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/botocore/client.py:595, in ClientCreator._create_api_method.._api_call(self, *args, **kwargs)
594 # The "self" in this scope is referring to the BaseClient.
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - --> 595 return self._make_api_call(operation_name, kwargs)
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/botocore/context.py:123, in with_current_context..decorator..wrapper(*args, **kwargs)
122 hook()
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - --> 123 return func(*args, **kwargs)
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - File /usr/local/lib/python3.11/site-packages/botocore/client.py:1058, in BaseClient._make_api_call(self, operation_name, api_params)
1057 error_class = self.exceptions.from_code(error_code)
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - -> 1058 raise error_class(parsed_response, operation_name)
1059 else:
[2025-07-16, 09:11:20 UTC] {logging_mixin.py:188} INFO - InvalidRequestException: An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: No output location provided. An output location is required either through the Workgroup result configuration setting or as an API input.

It seems there is a mismatch of expectations here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.