platformio / platformio/platformio-core

Remove environment name from `build_cache_dir` cache keys

Open
#4,574 41 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

build system known issue
Dominant language
Python
Stars
9.5k
Forks
905
Avg merge
2d 13h
Merged PRs (30d)
2

Description

What kind of issue is this?

  • Feature Request.
    Start by telling us what problem you’re trying to solve. Often a solution
    already exists! Don’t send pull requests to implement new features without first getting our
    support. Sometimes we leave features out on purpose to keep the project small.

Configuration

Operating system: docker/linux, host is Debian 10, image is ghcr.io/esphome/esphome-hassio:2023.3.0
PlatformIO Version (platformio --version): PlatformIO Core, version 6.1.6

Description of problem

I'm using esphome to manage tens of mostly identical devices with identical sets of source code sans a few items such as their individual name and some encryption keys. When I use esphome to compile the firmwares for these devices, they create a separate environment in platformio for each device, which is good for isolation. It does take a long time to compile the mostly identical source code though, for all these devices, when updates are available. I'd like to alleviate that.

In another issue, I've been digging in to what can be done, and I've experimented with the build_cache_dir setting. It's not currently used by esphome. This setting works wonders for a single device: if I recompile after cleaning the output, the build is instantaneous (as can be), as it uses the cache.

The cached output from one device is not used in another, and I've found this topic (https://community.platformio.org/t/build-cache-dir-will-not-share-object-files-between-envs/10011/8) on your community forums that provides a crucial hint. It seems the env:name is part of the cache key, and as each device in esphome is a separate environment, I have my issue.

I hope this issue can lead to the env:name being removed from the cache key, to allow the build cache to be reused across environments. It's my impression that the rest of the build commandline includes all the details that are in environments, such as boards, platforms and libraries.

Steps to Reproduce
  1. Create two environments of identical board, platform, dependencies, code
  2. Set build_cache_dir to some directory
  3. Build the two environments
  4. Observe that the identical code is compiled twice
Actual Results

The cache is not reused between identical builds, if they have two environments.

Expected Results

Environment name should not affect the cache key. It's my impression that the rest of the build commandline (which is part of the key) is enough to distinguish one output from another.

If problems with PlatformIO Build System:

The content of platformio.ini:
This file is modified by me, locally, to include the cache_build_dir. It is identical to other devices I have, except for the environment name.

; Auto generated code by esphome

[common]
lib_deps =
build_flags =
upload_flags =

; ========== AUTO GENERATED CODE BEGIN ===========
[platformio]
description = ESPHome 2023.3.0
build_cache_dir = /esphome_cache/
[env:light-extra01]
board = esp8285
board_build.flash_mode = dout
board_build.ldscript = eagle.flash.1m.ld
build_flags =
    -DESPHOME_LOG_LEVEL=ESPHOME_LOG_LEVEL_INFO
    -DNEW_OOM_ABORT
    -DPIO_FRAMEWORK_ARDUINO_LWIP2_HIGHER_BANDWIDTH_LOW_FLASH
    -DUSE_ARDUINO
    -DUSE_ESP8266
    -DUSE_ESP8266_FRAMEWORK_ARDUINO
    -DUSE_STORE_LOG_STR_IN_FLASH
    -Wno-nonnull-compare
    -Wno-sign-compare
    -Wno-unused-but-set-variable
    -Wno-unused-variable
    -fno-exceptions
extra_scripts =
    post:post_build.py
framework = arduino
lib_deps =
    ottowinter/ESPAsyncTCP-esphome@1.2.3
    esphome/ESPAsyncWebServer-esphome@2.1.0
    DNSServer
    ESP8266WiFi
    ESP8266mDNS
    esphome/noise-c@0.1.4
    bblanchon/ArduinoJson@6.18.5
    ${common.lib_deps}
lib_ldf_mode = off
platform = platformio/espressif8266 @ 3.2.0
platform_packages =
    platformio/framework-arduinoespressif8266 @ ~3.30002.0
; =========== AUTO GENERATED CODE END ============
Additional info

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the generated platformio.ini and its build_cache_dir setting, then trace how PlatformIO forms cache keys for two otherwise identical environments. Reproduce the two-environment build described in the issue and verify that identical outputs reuse the cache without mixing incompatible builds.

Written by the indexing model from the issue text.

Assessment

Tech stack
arduino, docker, python
Domain
build-system, embedded-iot, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.