mapbox / mapbox/robosat

Utility for syncing training, validation, and evaluation data.

Open
#188 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.1k
Forks
385
PR merge metrics
No merged PRs in 30d

Description

On most of the datasets I'm putting together, there is not always a 1-1 matching of masks to tiles. At the very least there should be clarification that the trainer needs a directory where all files are in sync. Even better would be to provide a simple pre-processing script for syncing the masks/tiles or in `rs_trainer` provide an option to ignore or remove un-synced masks/tiles.

Currently I just use a simple python script to sync the directory:

```python
import os
import argparse

def dir_dict(dir: str) -> dict:
dd = {}

for subdir, dirs, files in os.walk(dir):
for file in files:
f = '/'.join(os.path.join(subdir, file).split("/")[-3:])
dd[f] = os.path.join(subdir, file)

return dd

def main():
parser = argparse.ArgumentParser()
parser.add_argument('dir1', type=str)
parser.add_argument('dir2', type=str)
args = parser.parse_args()

removed = []

dir1_dict = dir_dict(args.dir1)
dir2_dict = dir_dict(args.dir2)

for k, v in dir1_dict.items():
if not dir2_dict.get(k):
removed.append(v)

for k, v in dir2_dict.items():
if not dir1_dict.get(k):
removed.append(v)

for file in removed:
os.remove(file)

return len(removed)


if __name__ == "__main__":
print ( f"removed {main()} un-synced files" )
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the rs_trainer data-loading entry point and inspect how training, validation, and evaluation directories are matched. Decide whether the scope is a standalone preprocessing utility, documentation, or an option to ignore or remove un-synced masks and tiles; done means the chosen behavior is defined and verified for mismatched directories.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.