con / con/nwb2bids

Support merging with existing dataset_description.json when converting to existing BIDS directory

Open
#189 0 comments 0 reactions 0 assignees View on GitHub
enhancement future
Dominant language
Python
Stars
5
Forks
6
Avg merge
4h 6m
Merged PRs (30d)
1

Description

## Background

Currently, when `nwb2bids` converts NWB files to an existing BIDS directory, it completely overwrites the existing `dataset_description.json` file. This means any metadata in the existing file (including custom `GeneratedBy` entries from previous processing pipelines) is lost.

## Current Behavior

From `_dataset_converter.py`:

```python
def write_dataset_description(self) -> None:
"""Write the `dataset_description.json` file."""
dataset_description_dictionary = self.dataset_description.model_dump()

dataset_description_file_path = self.run_config.bids_directory / "dataset_description.json"
with dataset_description_file_path.open(mode="w") as file_stream:
json.dump(obj=dataset_description_dictionary, fp=file_stream, indent=4)
```

The converter:
1. Only reads dataset_description from `additional_metadata_file_path` (if provided)
2. Does NOT read existing `dataset_description.json` from `bids_directory`
3. Completely overwrites the file when writing

## Expected Behavior

When converting to an existing BIDS directory that already has a `dataset_description.json`:

1. Read the existing `dataset_description.json`
2. Merge with any metadata from `additional_metadata_file_path`
3. Ensure nwb2bids GeneratedBy entry is appended (not replacing existing entries)
4. Write the merged result

### Example Scenario

**Existing dataset_description.json:**
```json
{
"Name": "My Dataset",
"BIDSVersion": "1.10",
"Description": "Previously processed data",
"GeneratedBy": [
{
"Name": "custom-pipeline",
"Version": "1.0.0",
"Description": "Initial processing",
"CodeURL": "https://github.com/example/pipeline"
}
]
}
```

**After nwb2bids conversion (desired):**
```json
{
"Name": "My Dataset",
"BIDSVersion": "1.10",
"Description": "Previously processed data",
"GeneratedBy": [
{
"Name": "custom-pipeline",
"Version": "1.0.0",
"Description": "Initial processing",
"CodeURL": "https://github.com/example/pipeline"
},
{
"Name": "nwb2bids",
"Version": "0.5.0",
"Description": "Tool to reorganize NWB files into a BIDS directory layout.",
"CodeURL": "https://github.com/con/nwb2bids"
}
]
}
```

## Implementation Considerations

### Merge Strategy

1. **GeneratedBy field**: Append nwb2bids entry to existing entries
2. **Other fields**: Precedence order (highest to lowest):
- `additional_metadata_file_path` (user explicitly provided)
- Existing `dataset_description.json` (preserve existing)
- Defaults from NWB metadata extraction

### Edge Cases

1. What if existing GeneratedBy already contains an nwb2bids entry?
- Current validator enforces exactly one nwb2bids entry
- Should we update the existing entry or throw an error?

2. What if `additional_metadata_file_path` also contains GeneratedBy?
- Should merge all three sources: existing + additional_metadata + nwb2bids

3. Conflicting field values?
- Need clear precedence rules for each field type
- @CodyCBakerPhD can you verify precedence should be nwb2bids > additional_metadata > existing?

## Testing Requirements

New test cases needed:

1. **Test: Existing BIDS with GeneratedBy** - Verify nwb2bids appends to existing GeneratedBy
2. **Test: Existing + additional_metadata both with GeneratedBy** - Verify all entries are preserved
3. **Test: Existing with conflicting metadata** - Verify precedence rules work correctly
4. **Test: Empty existing dataset_description.json** - Verify current behavior is maintained

## Related

- PR #170: Adds GeneratedBy field with auto-injection of nwb2bids
- Review comment: https://github.com/con/nwb2bids/pull/170#discussion_r2495552530

Contributor guide

Open the contributing guide

Research direction

Start in _dataset_converter.py, especially write_dataset_description and the existing handling of additional_metadata_file_path. Define the precedence and duplicate-GeneratedBy behavior with the maintainer, then add coverage for existing metadata, combined GeneratedBy entries, conflicts, and an empty dataset_description.json. Done means existing metadata is preserved and the agreed merge rules are verified by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.