platformio / platformio/platformio-core
Remove environment name from `build_cache_dir` cache keys
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9.5k
- Forks
- 905
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 2
Description
What kind of issue is this?
- Feature Request.
Start by telling us what problem you’re trying to solve. Often a solution
already exists! Don’t send pull requests to implement new features without first getting our
support. Sometimes we leave features out on purpose to keep the project small.
Configuration
Operating system: docker/linux, host is Debian 10, image is ghcr.io/esphome/esphome-hassio:2023.3.0
PlatformIO Version (platformio --version): PlatformIO Core, version 6.1.6
Description of problem
I'm using esphome to manage tens of mostly identical devices with identical sets of source code sans a few items such as their individual name and some encryption keys. When I use esphome to compile the firmwares for these devices, they create a separate environment in platformio for each device, which is good for isolation. It does take a long time to compile the mostly identical source code though, for all these devices, when updates are available. I'd like to alleviate that.
In another issue, I've been digging in to what can be done, and I've experimented with the build_cache_dir setting. It's not currently used by esphome. This setting works wonders for a single device: if I recompile after cleaning the output, the build is instantaneous (as can be), as it uses the cache.
The cached output from one device is not used in another, and I've found this topic (https://community.platformio.org/t/build-cache-dir-will-not-share-object-files-between-envs/10011/8) on your community forums that provides a crucial hint. It seems the env:name is part of the cache key, and as each device in esphome is a separate environment, I have my issue.
I hope this issue can lead to the env:name being removed from the cache key, to allow the build cache to be reused across environments. It's my impression that the rest of the build commandline includes all the details that are in environments, such as boards, platforms and libraries.
Steps to Reproduce
- Create two environments of identical board, platform, dependencies, code
- Set
build_cache_dirto some directory - Build the two environments
- Observe that the identical code is compiled twice
Actual Results
The cache is not reused between identical builds, if they have two environments.
Expected Results
Environment name should not affect the cache key. It's my impression that the rest of the build commandline (which is part of the key) is enough to distinguish one output from another.
If problems with PlatformIO Build System:
The content of platformio.ini:
This file is modified by me, locally, to include the cache_build_dir. It is identical to other devices I have, except for the environment name.
; Auto generated code by esphome
[common]
lib_deps =
build_flags =
upload_flags =
; ========== AUTO GENERATED CODE BEGIN ===========
[platformio]
description = ESPHome 2023.3.0
build_cache_dir = /esphome_cache/
[env:light-extra01]
board = esp8285
board_build.flash_mode = dout
board_build.ldscript = eagle.flash.1m.ld
build_flags =
-DESPHOME_LOG_LEVEL=ESPHOME_LOG_LEVEL_INFO
-DNEW_OOM_ABORT
-DPIO_FRAMEWORK_ARDUINO_LWIP2_HIGHER_BANDWIDTH_LOW_FLASH
-DUSE_ARDUINO
-DUSE_ESP8266
-DUSE_ESP8266_FRAMEWORK_ARDUINO
-DUSE_STORE_LOG_STR_IN_FLASH
-Wno-nonnull-compare
-Wno-sign-compare
-Wno-unused-but-set-variable
-Wno-unused-variable
-fno-exceptions
extra_scripts =
post:post_build.py
framework = arduino
lib_deps =
ottowinter/ESPAsyncTCP-esphome@1.2.3
esphome/ESPAsyncWebServer-esphome@2.1.0
DNSServer
ESP8266WiFi
ESP8266mDNS
esphome/noise-c@0.1.4
bblanchon/ArduinoJson@6.18.5
${common.lib_deps}
lib_ldf_mode = off
platform = platformio/espressif8266 @ 3.2.0
platform_packages =
platformio/framework-arduinoespressif8266 @ ~3.30002.0
; =========== AUTO GENERATED CODE END ============
Additional info
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the generated platformio.ini and its build_cache_dir setting, then trace how PlatformIO forms cache keys for two otherwise identical environments. Reproduce the two-environment build described in the issue and verify that identical outputs reuse the cache without mixing incompatible builds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- arduino, docker, python
- Domain
- build-system, embedded-iot, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100