esphome / esphome/feature-requests

Adding some programmability capabilities to yaml configs

Open
#805 11 comments 12 reactions 0 assignees View on GitHub
Core
Dominant language
No language data
Stars
450
Forks
29
PR merge metrics
No merged PRs in 30d

Description

**Describe the problem you have/What new integration you would like**

It would be nice to extend yaml configuration with some programmability features which would allow modifying configuration without direct changes to the yaml code defining components. Instead it will be achievable by setting configuration options.

By programmability I mean things like:

* **conditions** - e.g. include or exclude platform component from the config based on the value of substitution.
* **loops** - e.g. create multiple similar components basing on the config defined once.
* **expressions** - e.g. extend substitutions capabilities to support expressions. This will allow us to inject information available in build time and will extend templating capabilities.

The proposed solution will be to add a term *operator* which is essentially a key in component definition prefixed with some character e.g. `_`. This key is transparent for component configuration but might modify config by adding, excluding, or transforming existing definitions. See the examples below.

**Please describe your use case for this integration and alternatives you've tried:**

#### Scenario 1. Different hardware versions of the same device require slightly different configuration (conditions use case).

Let's say we have a device that could exist in a few variations. To be more specific let it be some universal air quality sensor designed in the way when PCB has slots for 5 different sensors. On the assembly stage, you decide which sensors to use.

**The problem**: For each combination of sensors you have to create a separate config file. So a) you have to manage a bunch of very similar files b) you will need some naming convention to give reasonable names

**Proposed solution**:

Simplistic approach (if we don't have expressions):
```yaml
substitutions:
disable_bme280: false
disable_mhz19: true
sensors:
- platform: mhz19
_exclude: $disable_mhz19
co2:
name: "CO2"
temperature:
name: "Temperature"
update_interval: 60s
- platform: bme280
_exclude: $disable_bme280
temperature:
name: "BME280 Temperature"
id: bme280_temperature
pressure:
name: "BME280 Pressure"
id: bme280_pressure
address: 0x77
update_interval: 15s
```

In this case by changing substitution value OR which is more likely by passing command line argument we could include\exclude sensors. This solution is based on `_exclude` operator which excludes a piece of configuration if the value is true.

The other and probably a more intuitive option would be `_if` operator which does the same but expects the opposite value.

#### Scenario 2. One device might require the bunch of similar set's of components (loops use case)

Consider the following example. We have a PCB for a device that controls floor heating in the house. It could manage up to 6 zones. Each zone needs a temperature sensor + relay to control the valve. The PCB is universal so it has a place for components for 6 zones, but again you could solder as just 4 if you don't need more.

**The Problem**: a) You have to copy and paste a lot of code. For each zone you will need at least sensor, output, switch, bangbang controller.
b) when you need multiple devices like this, again you have to manage
a bunch of very similar files
c) regardless of home many physical sensors\relays you have on board if in fact, you need just 2 of 6 available you don't want to home assistannt to show and track excessive entities that are not in use.

**Proposed Solution**

We have `_count` operator which duplicates config as many times as we need. It also exposes the special variable loop.counter` which will be available inside the definition.

See example:

```yaml
substitutions:
zones_count: 3
not_expose_valves: true
sensor:
- platform: dallas
_count: $zones_count
address: 0x1c0000031edd2a28
name: "Zone ${loop.counter} temperature"
id: "temperature_zone_${loop.counter}"
outputs:
- platform: gpio
count: $zones_count
id: zone${loop.counter}_output
pin:
number: ${loop.counter}
pcf8574: io_hub0
switch
- platform: output
internal: "${not_expose_valves}"
name: "Zone ${loop.counter} valve"
id: "_valve${loop.counter}"
output: ${loop.counter}_output
climate:
- platform: bang_bang
id: "zone${loop.counter}_controller"
name: "Zone ${loop.counter}"
sensor: "zone${zone1_id}_temperature"
heat_action:
- switch.turn_on: "zone${loop.counter}_valve"
idle_action:
- switch.turn_off: "zone${loop.counter}_valve"
```

### Scenario 3. Adding information available in build time into config (use case for expressions)

It might be very useful to expose information about config which is currently running especially when you have a lot of devices and have a sort of CI.

**Proposed Solution**: Allow to evaluate some template language expressions e.g. jinja2 which might replace substitutions in future and will give much more flexibility.

```yaml
text_sensor:
- platform: template
name: "Firmware Build Date"
lambda: |-
return {"{{ date(now)|format(yyyymmdd) }}"};
- platform: template
name: "Firmware Config"
lambda: |-
return {"{{ runtime.config_txt|sha256 }}"};
```

Also, this will resolve FR #804.

### Scenario 4. More flexibility with conditions and loops (use case for expressions)

To make it defining loops and conditions more comfortable simple substitutions will be not enough. There are multiple examples where you might need expressions. Just a few ones:

* Simple math to defined correct values in the loops. Let's say for example below numbering of pins pcf8574 starts not from 0:
```yaml
- platform: gpio
count: $zones_count
id: zone${loop.counter}_output
pin:
number: ${loop.counter + 2}
pcf8574: io_hub0
```
* Using text functions to follow naming convention and do not repeat
yourself:
```yaml
- platform: gpio
count: $zones_count
id: {{zones[loop.counter].name|camelcase}}
name: {{zones[loop.counter].name|capitalize}}
```
* Using more complex conditions:
```yaml
- platform: gpio
_if: {{ myVar == '123' and anotherVar == '456' }}
id: {{zones[loop.counter].name|camelcase}}
name: {{zones[loop.counter].name|capitalize}}
```

### Scenario 5. Going a little bit further with Packages feature

While initially [packages](https://github.com/esphome/esphome/pull/1052) were built mainly to help better organize the code implementing programmability features might become a good background to create universal, sharable, reusable configurations and a kind of open storage with ready to use packages. Basically the similar to [Ansible Galaxy](https://galaxy.ansible.com/) or
[Terraform Registry](https://registry.terraform.io/)

**Additional context**

Now I manage around 15 devices running ESPHome, so the problem I'm raising will be relevant rather for ppl who reached the state when it's time to think about build and deployment automation to keep things organized then for the average hobbyist.

The proposed solution was inspired by the approach implemented in Terraform, a tool I frequently use to define cloud infrastructure as a code and manage infrastructure changes, deployments, and other staff. Basically they met pretty similar issue some time ago and the solution proved its viability.

### Some other considerations

For simplicity proposes in my examples I used substitutions but in fact in the way they are implemented right now, it won't work. We will have to either re-implement the way they are processed and treated (which is a risk in terms of backward compatibility) or implement separate the mechanism which will be processed in another way.

The goal of this long read is to ask the community for feedback on the concept. Right now it is rather in the status of idea and a lot of details need to be clarified and designed. I'm ready to help with implementation (while it will not be fast) but before I move on I'd like
to see if this finds any support.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing how substitutions are currently processed and the packages approach referenced by PR #1052. Compare the proposed conditions, loops, and expressions with the existing configuration model; the issue is complete only after the community agrees on a scoped design and implementation plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.