[Feature] Improve reporting of errors when syncing Parquet files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 7h 27m
- Merged PRs (30d)
- 222
Description
Feature request
Is your feature request related to a problem?
When loading a parquet file in data warehouse from a self managed source, if the parquet file doesn't have field types defined (for example, because it's all nulls), we get a generic "Create table failed: Could not get columns" which is not very informative.
Describe the solution you'd like
We should report a clearer error message on the logs, like "failed to determine types from parquet file" or something.
Describe alternatives you've considered
N/A
Additional context
N/A
Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named in the issue. Start by tracing the Parquet loading and sync path for self-managed sources, then inspect where "Could not get columns" is logged; done means files with missing field types produce a clearer error explaining that types could not be determined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100