con / con/duct

Provide file(s) and env var pointing to it (them) for underlying tools to populate

Open
#445 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
12
Forks
6
Avg merge
11d 11h
Merged PRs (30d)
1

Description

Somewhat of an alternative way for

- #31

I was thinking on if we would like to make `duct` collected metadata expandable, we might want actually underlying tools to become duct aware and populate whatever info they want into duct info record, may be under `"external"` key . To make it non competing between tools (as multiple might decide to write into it at the same time), I think we should

- set for the child process the `CON_DUCT_EXTERNAL_PREFIX` env var which would be the prefix to all the other files with only `_external-` suffix.
- Then any external tool could write into `{CON_DUCT_EXTERNAL_PREFIX}{name}-{pid}.json` a dict with key `"{name}"` (or multiple)
- Upon finalization `duct` would collect all the json files and populate with their content the `"external"` attribute within `_info.json`.

This way we can e.g. `datalad` to provide the entire `"datalad-wtf"` or alike (may be all the sections separately under `"datalad:{section}"` keys) within there if datalad configured to do so on the machine (or in a given dataset)!

The question would be what `con-duct ls` etc should do to enable their listing. But it could be quite cool to be able to interrogate for internal tool specific details across runs!

Contributor guide

Open the contributing guide

Research direction

Start by locating the metadata finalization path that writes _info.json and the con-duct ls entry point; this issue names no files or tests. Clarify the external-file format, collection lifecycle, and how listing should expose the new data before implementation. Done means the agreed behavior is implemented and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.