catalog access / catalog transformations

Use Case

• security assertions on the catalog
• global transmutations of the catalog
• generating component resources from native providers (e.g. concat -> file)
• agent-side catalog operations (e.g. deferred functions)

Describe the Solution You Would Like

Add new compilation stages before the catalog is transmitted from the server and after it has been received at the agent. Allow modules to inject transformation functions that can mutate the catalog before further processing. Each of the transformation functions would take (part of) the catalog and be able to add/change/delete resources as it sees fit.

To help operators understand the impact of transformation functions, they should provide metadata of which types the function looks at and which types get emitted by them. This information can be used to ease implementation (only pass in requested resources, reducing filtering requirements) and safety (check that only expected types are returned) of those functions. At the same time this information can be used to ascertain a safe application order of the transforms:

• two or more transforms requesting different type inputs are always safe to run and do not require a specific order
  • given f(A): B and g(C): D, f and g have no order requirement
• two or more transforms requesting the same input types are always safe to run and do not require a specific order
  • given f(A): B and g(A): C, f and g have no order requirement
• a transform has to run after all other transforms that output required types for this transform
  • given f(A): B and g(B): C, f needs to be applied before g
• a cycle in transform type requirements is illegal
  • given f(A): B and g(B): A, f and g are incompatible
  • except for the special case of a single transform f(A): A, which can be safely applied

Should this turn out to be too restrictive for useful configurations, we can consider allowing folks to provide a partial priority override for transforms that conflict. Caveat emptor that in those cases the operator needs to take on additional responsibilities to understand and sequence those transforms and the consequences that'll have on the catalog.

Describe Alternatives You've Considered

Exposing the current mechanisms for doing these kinds of transformations would make implementing the Resource API outside of puppet a lot harder, while at the same time perpetuating the brittleness and opaqueness of the generate approach.

Additional Context

Originally tracked in https://tickets.puppetlabs.com/browse/PDK-533 , https://github.com/puppetlabs/puppet-specifications/blob/master/language/resource-api/README.md#catalog-access

Read the linked resource-api README and this issue first; no repository files or tests are identified. The design needs agreement on the transformation API, metadata, ordering, and cycle handling before implementation can be scoped. Done means the agreed catalog transformations work safely with coverage for their ordering rules.