OvertureMaps / OvertureMaps/data

String contains null byte (Places Data)

Open
#357 2 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
47
Avg merge
3h 42m
Merged PRs (30d)
1

Description

This is possibly more of a question than an issue, but I find that when trying to download the places dataset with duckdb and materializing it (in either R or Python) I get a "String contains null byte" error from a number of columns, e.g. addresses[1].freeform. I've not been able to pin it down whether this is becaus it actually contains \0 or whether the freeform field is missing here. In any case, I don't know how to deal with this in DuckDB, I've tried e.g. REPLACE(addresses[1].freeform, '\0', ''), COALESCE(addresses[1].freeform, ''), and CAST(addresses[1].freeform AS VARCHAR). My query is:

select 
        id, 
        sources[1].dataset as source, 
        strptime(sources[1].update_time, '%Y-%m-%dT%H:%M:%S.%fZ') as update_time, 
        names.primary as name, 
        categories.primary as category,
        categories.alternate as category_alt,
        brand.names.primary as brand,
        brand.wikidata as brand_wikidata,
        confidence, 
        addresses[1].freeform as address, 
        addresses[1].locality as locality, 
        addresses[1].postcode as postcode,
        addresses[1].region as region, 
        addresses[1].country as country, 
        websites[1] as website, 
        emails[1] as email, 
        phones[1] as tel,
        ST_X(geometry) as lon, 
        ST_Y(geometry) as lat
    from
       read_parquet('s3://overturemaps-us-west-2/release/2025-03-19.0/theme=places/*/*')

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.