platformio / platformio/platformio-core
Libraries not installed from registry cannot be detected as already installed dependencies of other libraries
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9.5k
- Forks
- 905
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 2
Description
When a library is either installed from anywhere other than the main package registry, the owner field for the spec is always empty because owner isn't a supported field that is read from either the library.json or a library.properties field. But owner is a supported field for a library dependency in a library.json file. So if a library has detailed dependency list including library owners and a user installs both that library and it's dependencies from any source other than the registry, the dependency finder will re-install all of the dependencies because they are cannot be matched because of the missing owner field.
Detailed Example:
Manually install low-level library
- User installs SensorModbusMaster directly from GitHub using the command
pio pkg install --library https://github.com/EnviroDIY/SensorModbusMaster/ - The SensorModbusMaster library is installed in the lib_deps folder as expected.
- The SensorModbusMaster library.json does not support an owner field; the package manager has no way of knowing the package owner.
Manually install library depending on low-level library
- User installs YosemitechModbus directly from GitHub using the command
pio pkg install --library https://github.com/EnviroDIY/YosemitechModbus - The YosemitechModbus library is installed in the lib_deps folder as expected.
- The package manager scans the library.json manifest of the YosemitechModbus library and sees the dependency named
SensorModbusMasterwith the ownerenvirodiy. - The package manager attempts to match the dependency with already installed libraries. It cannot be matched to the already installed SensorModbusMaster because the owner of the installed library is unknown.
- The failing test is in the test_pkg_spec function at line 297
- The package manager installs a second version of the SensorModbusMaster library from the PlatformIO registry.
Result
- The user now has two possibly different versions of the same library installed which is unlikely to be what was desired.
Possible Solutions
- support and read an owner field for the library.json file
- requires modification of the spec and validation of library.json files in addition to the machinery that reads the spec
- requires library authors to add the owner field
- requires validation on submission to the registry that the owner field in the manifest is aligned with the user submitting the library
- remove check for matching owners when checking if a library is already installed.
- only requires removing a few lines of code
- users may end up with mis-matched libraries and other packages that do not work together in cases where the same package has been submitted by multiple owners or forked libraries are used
- Where possible, infer the library owner from the installation location if a library is installed from outside of the registry
- This is probably only possible for istallations from sources like GitHub where the repo owner could be used
- This would only help in cases where the repo owner exactly matches the registry owner.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in platformio/package/manager/base.py, especially test_pkg_spec around lines 276-306 and the failing comparison near line 297. Reproduce the SensorModbusMaster and YosemitechModbus installation sequence, then trace how owners are obtained for non-registry libraries and matched against dependency owners. Done means an already installed external dependency is not duplicated while owner mismatches remain safe.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100