ODW: `package.assembly` steps

Acceptance Criteria:

• Assembly steps can be defined in the product-metadata repo at the product level. Those steps should be sufficient to assemble a distributable package from a source url
• Assembly should be executable via CLI. dcpy.cli lifecycle package assemble [target-name] -v [version] should be sufficient for now. We can think about additional args later, ideally.
• add the above CLI command as a step in the bytes->socrata GHA
• Refactor the XSLX OTI Data diction generation to be executed only when listed as an assembly step.
• Refactor Shapefile dis-assembly to be executed only when listed as an assembly step.
• generalize lifecycle.package. (ie. grep -r bytes ./dcpy/lifecycle/package/ should return nothing.)

Prototype of steps here

Not In Scope

• Distribute scripts.

Questions

• Include assembly at dataset level here?
• Overlap with Ingest Steps?

Solution

Models

In models.lifecycle.assemble define models for

• Assembly.Script
• Assembly.Script.Step

Script.Steps should have a

• type - ie pull_file, disassemble, generate_artifact. (start with enum)
• source - the thing to assemble from,
• targets - references to files to generate, e.g. special_purpose_districts.oti_data_dictionary

Add those models to models.product.metadata.ProductMetadataFile under a new lifecycle model, which will have an attribute for assemble.

Lifecycle functions / CLI

• refactor lifecycle.package.assemble.assemble_dataset_from_bytes -> lifecycle.package.assemble.run which should take an argument for product, optional dataset, and script name. This should execute all defined steps, and produce a distributable package.

Nice to have:

move files around in lifecycle.package so that they're contained in either

• lifecycle.package._assemble
• lifecycle.package._qa

and expose those via an API in:

• lifecycle.package.__init__

so that we have nice clean calls e.g.
package.assemble(...)
package.qa(...)