pyinfra-dev / pyinfra-dev/pyinfra

Merging of data structures in data hierarchy

Open
#468 5 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

API spec
Dominant language
Python
Stars
6k
Forks
548
Avg merge
7d 17h
Merged PRs (30d)
13

Description

Describe the solution you'd like
In the existing implementation of the data hierarchy, there is a good way for handling overriding of flat values, but for the case where you might want to use nested structures (e.g. dictionaries, lists, sets) it might be useful to allow for merging of those structures as you traverse the hierarchy. This could be made configurable using the config functionality so that it can be set on a per-module basis.

For instance, if I have a deploy where I would like to install and configure a MySQL database, I can set the default set of configuration options in all.py as a dictionary of key/value pairs. In a staging.py file that targets hosts in a pre-production environment I can then specify a dictionary that overrides a subset of those values.

This can currently be achieved by flattening the structure and specifying all of the keys at the top level and prefixing them with a namespace for the deploy, but using a more rich data structure it provides implicit namespacing. It also provides the advantage of being able to render out configuration files that are intended to be JSON or YAML by simply serializing the merged result of the data object and writing that as the contents of the file.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing data hierarchy and config functionality to understand how flat values are currently overridden as modules are traversed. Define the merge behavior for nested dictionaries, lists, and sets, including how it is configured per module, then verify that merged results can be serialized for JSON or YAML output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.