HydrologicEngineeringCenter / HydrologicEngineeringCenter/cwms-python

JSON data from fetch can't be directly used to store again

Open
#273 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
16
Forks
11
Avg merge
4d 6h
Merged PRs (30d)
8

Description

At least for the /locations endpoint, the json data from the fetch result can't be passed again into the appropriate store endpoint. CDA reports a serialization error.

Purpose:
I need to load all the locations from our production instance into cwbi-dev, so I wanted to use cwms-python to dump all the locations and insert them into cwbi.

Here's the script I used:

import os
import cwms

CWBI_API_KEY="apikey"
CWBI_API_ROOT="https://water.dev.cwbi.us/cwms-data"

cwms.api.init_session(api_root=os.environ.get("CDA_APT_ROOT", None), api_key=os.environ.get("CDA_APT_KEY", None))
locations = cwms.get_locations_catalog(office_id='SPK', unit_system='EN')

cwms.api.init_session(api_root=CWBI_API_ROOT, api_key=CWBI_API_KEY)

for location in locations.json['entries']:
	print(location)
	cwms.store_location(data=location, fail_if_exists=False)

The JSON has different key names than CDA returns, or expects in the POST.
For example:
office-id -> office
timezone-name -> time-zone

and many more.

Expectation:
I would expect the JSON data from the get_ method to match what CDA would return, especially since I have to supply JSON data to the relevant store_ methods in cwms-python.

Additional info:
Here's the JSON returned by CDA:

{
    "office-id": "SPK",
    "name": "Truckee R-Vista",
    "latitude": 0,
    "longitude": 0,
    "active": true,
    "public-name": "Truckee River At Vista, NV",
    "timezone-name": "US/Pacific",
    "location-kind": "SITE",
    "nation": "US",
    "state-initial": "NV",
    "county-name": "Washoe",
    "nearest-city": "Sparks",
    "horizontal-datum": "NAD27",
    "published-longitude": 119.7,
    "published-latitude": 39.520555555556,
    "vertical-datum": "NAVD88",
    "elevation": 0,
    "bounding-office-id": "UNK",
    "elevation-units": "ft"
}

Here's the JSON returned by cwms-python:

{
	"office": "SPK",
	"name": "Truckee R-Vista",
	"nearest-city": "Sparks",
	"public-name": "Truckee River At Vista, NV",
	"kind": "SITE",
	"time-zone": "US/Pacific",
	"latitude": 0.0,
	"longitude": 0.0,
	"published-latitude": 39.520555555556,
	"published-longitude": 119.7,
	"horizontal-datum": "NAD27",
	"elevation": 0.0,
	"unit": "ft",
	"vertical-datum": "NAVD88",
	"nation": "United States",
	"state": "NV",
	"county": "Washoe",
	"bounding-office": "UNK",
	"active": true,
	"aliases": [
		{"name": "Agency Aliases-USGS Station Name", "value": "Truckee River At Vista, NV"},
		{"name": "Agency Aliases-USGS Station Number", "value": "10350000"}
	]
}

The "aliases" key is acceptable for storing data, so I'm not sure why it doesn't show up on CDA when fetching via swagger.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing cwms.get_locations_catalog and cwms.store_location for the /locations endpoint, then compare their serialization and deserialization mappings with the CDA JSON shown in the issue. Verify the location object returned by the fetch path can be supplied to the store path, including aliases, and confirm the round trip no longer produces a serialization error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.