Support merging with existing dataset_description.json when converting to existing BIDS directory
- Dominant language
- Python
- Stars
- 5
- Forks
- 6
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 1
Description
## Background
Currently, when `nwb2bids` converts NWB files to an existing BIDS directory, it completely overwrites the existing `dataset_description.json` file. This means any metadata in the existing file (including custom `GeneratedBy` entries from previous processing pipelines) is lost.
## Current Behavior
From `_dataset_converter.py`:
```python
def write_dataset_description(self) -> None:
"""Write the `dataset_description.json` file."""
dataset_description_dictionary = self.dataset_description.model_dump()
dataset_description_file_path = self.run_config.bids_directory / "dataset_description.json"
with dataset_description_file_path.open(mode="w") as file_stream:
json.dump(obj=dataset_description_dictionary, fp=file_stream, indent=4)
```
The converter:
1. Only reads dataset_description from `additional_metadata_file_path` (if provided)
2. Does NOT read existing `dataset_description.json` from `bids_directory`
3. Completely overwrites the file when writing
## Expected Behavior
When converting to an existing BIDS directory that already has a `dataset_description.json`:
1. Read the existing `dataset_description.json`
2. Merge with any metadata from `additional_metadata_file_path`
3. Ensure nwb2bids GeneratedBy entry is appended (not replacing existing entries)
4. Write the merged result
### Example Scenario
**Existing dataset_description.json:**
```json
{
"Name": "My Dataset",
"BIDSVersion": "1.10",
"Description": "Previously processed data",
"GeneratedBy": [
{
"Name": "custom-pipeline",
"Version": "1.0.0",
"Description": "Initial processing",
"CodeURL": "https://github.com/example/pipeline"
}
]
}
```
**After nwb2bids conversion (desired):**
```json
{
"Name": "My Dataset",
"BIDSVersion": "1.10",
"Description": "Previously processed data",
"GeneratedBy": [
{
"Name": "custom-pipeline",
"Version": "1.0.0",
"Description": "Initial processing",
"CodeURL": "https://github.com/example/pipeline"
},
{
"Name": "nwb2bids",
"Version": "0.5.0",
"Description": "Tool to reorganize NWB files into a BIDS directory layout.",
"CodeURL": "https://github.com/con/nwb2bids"
}
]
}
```
## Implementation Considerations
### Merge Strategy
1. **GeneratedBy field**: Append nwb2bids entry to existing entries
2. **Other fields**: Precedence order (highest to lowest):
- `additional_metadata_file_path` (user explicitly provided)
- Existing `dataset_description.json` (preserve existing)
- Defaults from NWB metadata extraction
### Edge Cases
1. What if existing GeneratedBy already contains an nwb2bids entry?
- Current validator enforces exactly one nwb2bids entry
- Should we update the existing entry or throw an error?
2. What if `additional_metadata_file_path` also contains GeneratedBy?
- Should merge all three sources: existing + additional_metadata + nwb2bids
3. Conflicting field values?
- Need clear precedence rules for each field type
- @CodyCBakerPhD can you verify precedence should be nwb2bids > additional_metadata > existing?
## Testing Requirements
New test cases needed:
1. **Test: Existing BIDS with GeneratedBy** - Verify nwb2bids appends to existing GeneratedBy
2. **Test: Existing + additional_metadata both with GeneratedBy** - Verify all entries are preserved
3. **Test: Existing with conflicting metadata** - Verify precedence rules work correctly
4. **Test: Empty existing dataset_description.json** - Verify current behavior is maintained
## Related
- PR #170: Adds GeneratedBy field with auto-injection of nwb2bids
- Review comment: https://github.com/con/nwb2bids/pull/170#discussion_r2495552530
Contributor guide
Research direction
Start in _dataset_converter.py, especially write_dataset_description and the existing handling of additional_metadata_file_path. Define the precedence and duplicate-GeneratedBy behavior with the maintainer, then add coverage for existing metadata, combined GeneratedBy entries, conflicts, and an empty dataset_description.json. Done means existing metadata is preserved and the agreed merge rules are verified by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100