canonical / canonical/cloud-init

[docs]: Incomplete and misleading guidance on merging vendor-data

Open
#6,268 3 comments 4 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
3.8k
Forks
1.1k
Avg merge
2d 23h
Merged PRs (30d)
18

Description

Issue #2979 described a problem where vendor-data and user-data are not merged. I've run into the same confusion, but didn't want to necro a closed, 2 year old issue, plus I have an additional request for clarification that wasn't in that issue.

### Misleading documentation

In the [Vendor-data](https://cloudinit.readthedocs.io/en/latest/explanation/vendordata.html) page, it states:

> User-supplied cloud-config is merged over cloud-config from vendor-data.

To me, the phrase "merged over" here implies a recursive merge, with user-data taking precedence over vendor-data. Eg., given

```yaml
# user-data
map_a:
key_a: scalar from user-data
key_b:
subkey_u: hello user

list_a:
- item from user-data

top_level_scalar: foobar

# vendor-data
map_a:
key_a: scalar from vendor-data
key_b:
subkey_v: hello vendor

list_a:
- item from vendor-data

top_level_scalar: raboof
```

The combined config I expected would be

```yaml
map_a:
key_a: scalar from user-data
key_b:
subkey_u: hello user
subkey_v: hello vendor

list_a:
- item from user-data
- item from vendor-data

top_level_scalar: foobar
```

It would be helpful if the documentation was more explicit that this is not how it works:

> User-supplied cloud-config is merged, non-recursively, over cloud-config from vendor-data. When a key contains a collection (map or list) and is present in both, only the user-data collection is retained.

It might be helpful to further clarify that vendor-data is not suitable for platform-supplied defaults, but only for a platform-defined "bare" setup. The hint, "if the use of vendor-data is required for the instance to run, then vendor-data should not be used," doesn't quite land right if the use-case doesn't involve configuration that would prevent the instance from running if missing, eg. using `write_file` to create an `/etc/motd` with the vendor's ASCII art logo.

I can imagine a good reason for it to work the way that it does so I wouldn't call this a bug, but that only came after losing several hours wondering why it wasn't working the way I expected.

### Incomplete information

While searching on the topic, I came across this post: https://discuss.linuxcontainers.org/t/how-to-merge-profiles-user-vendor-data/774/17:

> Still don’t know in detail how all magic works, but when I add the `merge_how` section (https://cloudinit.readthedocs.io/en/latest/topics/merging.html) it works – without the `merge_how`, one file wins!

Notably, the [Merging page](https://cloudinit.readthedocs.io/en/latest/reference/merging.html) makes no mention at all that `merge_how` acts on vendor-data.

I took a leap of faith that an old comment on an LXD forum might be onto something. So, now I'm testing `merge_how`, following the information in the ["How to activate" section](https://cloudinit.readthedocs.io/en/latest/reference/merging.html#how-to-activate).

What I've observed it that some setting combinations have no effect, and (at least) one actually _inverts_ the behavior of user-data overriding vendor-data, so the final, combined config is only vendor-data. At least that proves that `merge_how` _does_ apply to vendor-data 🤷

This configuration causes the inversion:

```yaml
# user-data
merge_how:
- name: list
settings: [append]

apt: ...

write_files:
- {file_u1: {...}}
- ...

# vendor-data
merge_how:
- name: list
settings: [append]
# There doesn't seem to be any difference if merge_how is given in both, one, or the other.

write_files:
- {file_v1: {...}}

# combined-config
write_files:
- {file_v1: {...}}

# The apt block and all write_files items from user-data are gone
```

Since there's no documentation saying whether `merge_how` applies to vendor-data in the first place, I can't say if this is expected behavior or a bug. It feels like a bug to me, though.

This configuration does absolutely nothing, the end result being identical to not setting `merge_how` at all:

```yaml
# user-data
merge_how:
- name: list
settings: [append, recurse_list, recurse_dict]
- name: dict
settings: [no_replace, recurse_list, recurse_dict]

apt: ...

write_files:
-
- ...

# vendor-data
merge_how:
- name: list
settings: [append, recurse_list, recurse_dict]
- name: dict
settings: [no_replace, recurse_list, recurse_dict]
# Again, it doesn't seem to matter if this is in either one or both

write_files:
-

# combined-config
apt: ...

write_files:
-
- ... (but NOT file_v1)
```

I haven't gone through the whole matrix of possible merge setting combinations, but if I do happen to brute-force one that does what I want (or causes more weird behavior, like inversion) I'll be sure to update. I plan next to try varying the replace/no_replace options, since it's not clear if they mean replacing the entire collection or replacing conflicting items within the collection.

Any tips on getting the `write_files` list to merge would be appreciated! (Especially if they get added to the docs!)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.