galaxyproject / galaxyproject/idc
DM loading in the build instance is a hack
- Dominant language
- Shell
- Stars
- 10
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
On the builder instance `database_connection` is Main's DB since we want DM builds to be persistent on Main. This means that config watchers are also Main's config watchers, and so if we install DMs on the build instance, the install process never completes because Main's config watcher doesn't see the change to `shed_data_managers_conf.xml`.
As a workaround, we preinstall DMs properly on Main in [galaxyproject/usegalaxy-tools](https://github.com/galaxyproject/usegalaxy-tools/). Ideally we could just load `/cvmfs/main.galaxyproject.org/config/shed_data_manager_conf.xml`, but this not possible since the `shed_tool_conf.xml` referenced in the `shed_data_manager_conf.xml` has to be loaded:
```
[pN:idc-build,p:19389,tN:MainThread] '/cvmfs/main.galaxyproject.org/config/shed_tool_conf.xml' not among installable tool config files (/srv/galaxy/var/config/shed_tool_conf.xml, /srv/galaxy/var/config/migrated_tools_conf.xml)
```
As a result, we manually add to and deploy [a non-shed `data_manager_conf.xml` in the playbook](ansible/files/idc/data_manager_conf.xml). This is an extra step any time you need to install or update a DM, and also has the downside of making our DMs unversionable (we install shed versioned DMs but load them in an "local" unversioned way).
The proper solution is probably a Galaxy fix to make it possible to load `shed_data_manager_conf.xml` without the corresponding `shed_tool_conf.xml`.
xref [full discussion on matrix](https://matrix.to/#/!dLvLUwzoHbvrrUfQEZ:gitter.im/$yWx9zOCxtbFTQ6WSwBPtbrvpWryiG8RJNlPRClNfi8I?via=gitter.im&via=matrix.org)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ansible/files/idc/data_manager_conf.xml and the build-instance configuration described in the issue. Reproduce or trace DM installation while comparing shed_data_manager_conf.xml with shed_tool_conf.xml, using the reported installable-tool-config error as the starting point. Done means shed data managers can be loaded and installed without manually maintaining a non-shed configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, shell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100