alibaba / alibaba/DataX

does ossreader suppport regex?

Open
#563 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

I have csvfile under OSS, that i want to write to OTS. I cannot use default delimiter for separating columns as each row has column-wise JSON string, is there anyway i can make it work. I guess i can split column-wise JSON, if i can match by regex.

sample data:

f0000743-3135-4fac-92d8-2265dbcf5d88,12,{"EventType":"article:created","RawData":"{\"meta\":{\"user\":\"5ceba48db7dfbf000177b27e\",\"created\":\"2010-01-28T16:00:00.000Z\",\"inputted\":\"2010-01-28T20:02:12.000Z\"}}","Data":null,"Timestamp":"2019-12-11T15:29:28.84297294Z","AggregateType":"article","AggregateID":"f0000743-3135-4fac-92d8-2265dbcf5d88","Version":1,"Branch":"current"},{"EventType":"article:provider_updated","RawData":"{\"meta\":{\"user\":\"5ceba48db7dfbf000177b27e\",\"updated\":\"2010-01-28T16:00:00.000Z\",\"provider\":\"SCMP\"}}","Data":null,"Timestamp":"2019-12-11T15:29:28.842973852Z","AggregateType":"article","AggregateID":"f0000743-3135-4fac-92d8-2265dbcf5d88","Version":2,"Branch":"current"},{"EventType":"article:publication_type_updated","RawData":"{\"meta\":{\"user\":\"5ceba48db7dfbf000177b27e\",\"updated\":\"2010-01-28T16:00:00.000Z\",\"publicationType\":\"print\"}}","Data":null,"Timestamp":"2019-12-11T15:29:28.842974592Z","AggregateType":"article","AggregateID":"f0000743-3135-4fac-92d8-2265dbcf5d88","Version":3,"Branch":"current"},{"EventType":"article:merlin_id_updated","RawData":"{\"meta\":{\"user\":\"5ceba48db7dfbf000177b27e\",\"updated\":\"2010-01-28T20:02:12.000Z\",\"merlinID\":13571301}}","Data":null,"Timestamp":"2019-12-11T15:29:28.886009117Z","AggregateType":"article","AggregateID":"f0000743-3135-4fac-92d8-2265dbcf5d88","Version":4,"Branch":"current"},{"EventType":"article:has_copyright_updated","RawData":"{\"meta\":{\"user\":\"5ceba48db7dfbf000177b27e\",\"updated\":\"2010-01-28T20:02:12.000Z\",\"hasCopyright\":true}}","Data":null,"Timestamp":"2019-12-11T15:29:28.886010459Z","AggregateType":"article","AggregateID":"f0000743-3135-4fac-92d8-2265dbcf5d88","Version":5,"Branch":"current"},{"EventType":"article:creator_categories_set","RawData":"{\"meta\":{\"user\":\"5ceba48db7dfbf000177b27e\",\"updated\":\"2010-01-28T20:02:12.000Z\",\"creatorCategories\":[\"staff\"]}}","Data":null,"Timestamp":"2019-12-11T15:29:28.886012933Z","AggregateType":"article","AggregateID":"f0000743-3135-4fac-92d8-2265dbcf5d88","Version":6,"Branch":"current"}

Above sample has total 8 columns, 6 out of which are in JSON, need to match it and write under OTS

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names the OSS reader and OTS writer and provides a sample CSV row; start by checking the reader’s delimiter and parsing entry point and how the writer receives columns. Define whether regex or another custom parsing mode is supported, then verify the sample produces eight columns with six JSON values before writing to OTS.

Written by the indexing model from the issue text.

Assessment

Tech stack
json
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.