`CSV` adapter difficult to use and not very flexible
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 441
- Forks
- 90
- Avg merge
- 1d 12m
- Merged PRs (30d)
- 5
Description
Discovered during the hackathon, the current CSV adapter is not great. It was difficult to map multiple columns as datetimes, it basically presupposes a "symbol column", and it doesnt allow for returning just e.g. a dict of values in the row. Here is a naive alternative built for the hackathon to read Citibike historical CSV data:
import csv as pycsv
from datetime import datetime
from csp import ts
from csp.impl.pulladapter import PullInputAdapter
from csp.impl.wiring import py_pull_adapter_def
class CSVAdapterImpl(PullInputAdapter):
def __init__(self, filename: str, datetime_columns: list = None):
if not datetime_columns:
raise Exception("Must provide at least one datetime column")
self._filename = filename
self._datetime_columns = datetime_columns
self._csv_reader = None
self._first_row = None
super().__init__()
def start(self, starttime, endtime):
super().start(starttime, endtime)
self._csv_reader = pycsv.DictReader(open(self._filename, "r"))
# fast forward to first record
while True:
try:
row = next(self._csv_reader)
time = datetime.strptime(
row[self._datetime_columns[0]], "%Y-%m-%d %H:%M:%S"
)
if time < starttime:
continue
for dtc in self._datetime_columns:
row[dtc] = datetime.strptime(row[dtc], "%Y-%m-%d %H:%M:%S")
self._first_row = row
break
except StopIteration:
return
def stop(self):
self._csv_reader = None
def next(self):
if self._first_row is not None:
ret = self._first_row[self._datetime_columns[0]], self._first_row
self._first_row = None
try:
row = next(self._csv_reader)
time = datetime.strptime(
row[self._datetime_columns[0]], "%Y-%m-%d %H:%M:%S"
)
for dtc in self._datetime_columns:
row[dtc] = datetime.strptime(row[dtc], "%Y-%m-%d %H:%M:%S")
return time, row
except StopIteration:
return None
CSVAdapter = py_pull_adapter_def(
"CSVAdapter", CSVAdapterImpl, ts[dict], filename=str, datetime_columns=list
)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the current CSV adapter alongside csp.impl.pulladapter.PullInputAdapter and py_pull_adapter_def. Use the provided CSVAdapterImpl example as the reference for supporting multiple datetime columns, avoiding a required symbol column, and returning row dictionaries. Done means the adapter API and behavior cover these use cases without losing time-based reading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100