pathwaycom / pathwaycom/llm-app
Pathway's connector issue in Jsonlines
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 58.9k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
Flatten data structures in a Jsonline file can not be mapped to structured schemas automatically.
For example, list_price and current_price mapping to the scheme is failing:
{"position": 1, "link": "https://www.amazon.com/Avia-Resistant-Restaurant-Service-Sneakers/dp/B0BJY1FN8F", "asin": "B0BJXSKK9L", "is_lightning_deal": false, "deal_type": "BEST_DEAL", "is_prime_exclusive": false, "starts_at": "2023-08-14T07:00:08.270Z", "ends_at": "2023-08-21T06:45:08.270Z", "type": "multi_item", "title": "Avia Anchor SR Mesh Slip On Black Non Slip Shoes for Women, Comfortable Water Resistant Womens Food Service Sneakers - Black, Blue, or White Med or Wide Restaurant, Slip Resistant Work Shoes Women", "image": "https://m.media-amazon.com/images/I/3195IpEIRpL._SY500_.jpg", "deal_price": 39.98, "list_price": {"value": 59.98, "currency": "USD", "symbol": "$", "raw": "59.98", "name": "List Price"}, "current_price": {"value": 39.98, "currency": "USD", "symbol": "$", "raw": "39.98", "name": "Current Price"}, "merchant_name": "Galaxy Active", "free_shipping": false, "is_prime": true, "is_map": false, "deal_id": "34f3da97", "seller_id": "A3GMJQO0HY62S", "description": "Avia Anchor SR Mesh Slip On Black Non Slip Shoes for Women, Comfortable Water Resistant Womens Food Service Sneakers - Black, Blue, or White Med or Wide Restaurant, Slip Resistant Work Shoes Women", "rating": 4.16, "ratings_total": 1148, "old_price": 59.98, "currency": "USD"}
In this data schema:
class Price(pw.Schema):
value: float
currency: str
symbol: str
raw: str
name: str
class DealResult(pw.Schema):
position: int
link: str
asin: str
is_lightning_deal: bool
deal_type: str
is_prime_exclusive: bool
starts_at: str
ends_at: str
type: str
title: str
image: str
deal_price: Price
list_price: Price
current_price: Price
merchant_name: str
free_shipping: bool
is_prime: bool
is_map: bool
The error I got:
Read data parsed unsuccessfully. field deal_price with no JsonPointer path specified is absent in
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Jsonlines connector failure with the nested Price and DealResult schemas shown in the issue. Inspect how the connector maps flattened JSON structures to nested schema fields; done when list_price and current_price, along with the other nested fields, map successfully without the missing-field error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100