aws / aws/aws-sdk-cpp

AWS Glue - Null values in partition column

Open
#3,165 4 comments 0 reactions 0 assignees View on GitHub
bug glue p3 service-api
Dominant language
C++
Stars
2.2k
Forks
1.2k
Avg merge
3d 14h
Merged PRs (30d)
12

Description

Original discussion: https://github.com/aws/aws-sdk-cpp/discussions/2803

It seems like glue isn't handling the getpartitions API correctly where a partition column has null value.
Example below, I am using the aws cli for simiplicity , which gives the same output as the SDK

My table data is structured as below in S3
```
s3://example-bucket/example_table/
├── int_partition_col=null/
│ ├── string_partition_col=null/
│ │ └── data-part-00001.csv
├── int_partition_col=1/
│ ├── string_partition_col=A/
│ │ └── data-part-00002.csv
└── int_partition_col=2/
├── string_partition_col=B/
│ └── data-part-00003.csv

```
```
> aws glue get-partitions --database-name example_db --table-name example_table --expression "(int_partition_col >= 0)" ->
An error occurred (InvalidStateException) when calling the GetPartitions operation: For input string: "null" is not an integer.

> aws glue get-partitions --database-name example_db --table-name example_table --expression "(string_partition_col is null)" -> Returns empty

> aws glue get-partitions --database-name example_db --table-name example_table --expression "(string_partition_col = 'null')"-> works correctly

```
So it seems like the null value is being considered as a string literal? But from the documentation [here](https://docs.aws.amazon.com/glue/latest/webapi/API_GetPartitions.html), it seems IS NULL etc are supported?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.