aws-samples / aws-samples/aws-glue-samples
writing data to s3 using spark and updating catalog
Open
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 834
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 1
Description
I'm working in a glue job and I want to reflect my appended data when I query into Athena what are the options considering I'm writing like :
df.write.mode("append").partitionBy(*cols).parquet(path)?
I would be glad with some help here
Contributor guide
Research direction
The issue mentions a Glue job writing partitioned Parquet with df.write(...).parquet(path), then querying the appended data through Athena. Start by reviewing the job's S3 write path and the catalog behavior for new partitions; done means identifying the applicable options for making appended data visible in Athena.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, spark
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100