MagicStack / MagicStack/asyncpg
copy_from_query drops headers if any postgis NOTICE/LOCATION messages produced
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 468
- PR merge metrics
- No merged PRs in 30d
Description
when using `Connection.copy_from_query` I had a statement which eventually resolved to something like:
```sql
COPY (
select 1
from field_year_crop_variety_layer fycvl
cross join filters
join field_meta fm on
fm.field_id = fycvl.field_id and
(filters.filter_geometry is null or st_intersects(filters.filter_geometry, fm.centroid))
order by fycvl.field_id, fycvl.harvest_year, fycvl.layer_id, fycvl.crop, fycvl.variety
limit '100000'::int
) TO STDOUT (FORMAT 'csv', HEADER True)
```
where `filter_geometry` is a `geometry`, and `fm.centroid` is a `geography`.
and no matter what I did asyncpg would not return the headers. It wasn't until I broke it down and ran it from psql that I noticed the output:
```
field_id,harvest_year,layer_id,crop,variety,is_precision_data,date_period,avg_date,data_is_missing,missing_data_message,crop_variety_layer_area_m2,geometry,computed_at
NOTICE: 00000: Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY
LOCATION: geography_from_geometry, geography_inout.c:510
NOTICE: 00000: Coordinate values were coerced into range [-180 -90, 180 90] for GEOGRAPHY
LOCATION: geography_from_geometry, geography_inout.c:510
143011,2017,0,corn,unknown,t,"[2017-04-25,2017-04-26)",2017-04-24,f,,7240.833,,1752130305.616610
```
sure enough, if I fixed the bug (casting fm.centroid to a geometry) the headers appeared. This seems like a bug in asyncpg when interpreting the data returned.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Connection.copy_from_query and reproduce the provided COPY query against PostgreSQL with the PostGIS NOTICE/LOCATION messages enabled. Trace how those messages are handled alongside COPY output; done means the returned data retains its CSV header when such notices are produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100