platformio / platformio/platformio-core

Libraries not installed from registry cannot be detected as already installed dependencies of other libraries

Open
#5,463 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement package management
Dominant language
Python
Stars
9.5k
Forks
905
Avg merge
2d 13h
Merged PRs (30d)
2

Description

When a library is either installed from anywhere other than the main package registry, the owner field for the spec is always empty because owner isn't a supported field that is read from either the library.json or a library.properties field. But owner is a supported field for a library dependency in a library.json file. So if a library has detailed dependency list including library owners and a user installs both that library and it's dependencies from any source other than the registry, the dependency finder will re-install all of the dependencies because they are cannot be matched because of the missing owner field.

Detailed Example:
Manually install low-level library
  • User installs SensorModbusMaster directly from GitHub using the command pio pkg install --library https://github.com/EnviroDIY/SensorModbusMaster/
  • The SensorModbusMaster library is installed in the lib_deps folder as expected.
  • The SensorModbusMaster library.json does not support an owner field; the package manager has no way of knowing the package owner.
Manually install library depending on low-level library
  • User installs YosemitechModbus directly from GitHub using the command pio pkg install --library https://github.com/EnviroDIY/YosemitechModbus
  • The YosemitechModbus library is installed in the lib_deps folder as expected.
  • The package manager scans the library.json manifest of the YosemitechModbus library and sees the dependency named SensorModbusMaster with the owner envirodiy.
  • The package manager attempts to match the dependency with already installed libraries. It cannot be matched to the already installed SensorModbusMaster because the owner of the installed library is unknown.
    • The failing test is in the test_pkg_spec function at line 297
  • The package manager installs a second version of the SensorModbusMaster library from the PlatformIO registry.
Result
  • The user now has two possibly different versions of the same library installed which is unlikely to be what was desired.
Possible Solutions
  • support and read an owner field for the library.json file
    • requires modification of the spec and validation of library.json files in addition to the machinery that reads the spec
    • requires library authors to add the owner field
    • requires validation on submission to the registry that the owner field in the manifest is aligned with the user submitting the library
  • remove check for matching owners when checking if a library is already installed.
    • only requires removing a few lines of code
    • users may end up with mis-matched libraries and other packages that do not work together in cases where the same package has been submitted by multiple owners or forked libraries are used
  • Where possible, infer the library owner from the installation location if a library is installed from outside of the registry
    • This is probably only possible for istallations from sources like GitHub where the repo owner could be used
    • This would only help in cases where the repo owner exactly matches the registry owner.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in platformio/package/manager/base.py, especially test_pkg_spec around lines 276-306 and the failing comparison near line 297. Reproduce the SensorModbusMaster and YosemitechModbus installation sequence, then trace how owners are obtained for non-registry libraries and matched against dependency owners. Done means an already installed external dependency is not duplicated while owner mismatches remain safe.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.