`dcpy.lifecycle.*` Documentation

TODOs

• License (Fvk)
• Initial README in dcpy (DM)
• Ingest rough roadmap
• Package and dist rough roadmap
• Top-level diagram of lifecycle
• Finn's notebook sketches
• Doc Research about a similar product. (AR)
  - check DuckDB docs
  - doc'ing python API vs CLI.

Research / Questions

• Where do docs live. README in dcpy (Damon to take an initial stab)
• what would be the most impactful docs to have ready for the presentations
• how usable are Ingest / Package / Dist in their current state?
• what help could we use from third parties / contributors
• Where/how should interested folks contact us
• Roadmap! (with explicit warning about our level of support)
• Determine Learning Objectives for presentations.

Supporting Docs To Generate/Reuse

• Diagram of e2e lifecycle
• Reference Product for docs (template-db?)
• Ingest - reference examples for Socrata or other source. Concrete examples. (or even emailed data)

Nice to have

• A LOGO?!

Ingest

Talk focuses on "what is this thing".

Problems Solved

• inherited tool that worked really well, but was inflexible. Need fine-grained control of ingested datasets.
• Not having to write new code to pull in a dataset (all/most config)
• enables basic cleaning as we pull in new datasets.
• validation built into the tool. (validation for columns already exists)

Package and Dist

Problems Solved

• don't want to manually upload files, to different types/destinations
  • didn't want something to JUST dist to Socrata
  • something scalable to many, many datasets. (Don't be bottlenecked by # of datasets)
  • single source of truth for info about datasets. And accommodating historical quirks (misnamed columns etc.)

Start by reviewing the README in dcpy and the open TODOs for the lifecycle, roadmap, and presentation goals. Research the DuckDB documentation as a reference, then define the needed lifecycle diagram and concrete Ingest examples for Socrata or emailed data. Done means the README and supporting docs explain the Ingest, Package, and Dist lifecycle and provide the requested roadmap and examples.