mapbox / mapbox/robosat

rs_subset should accept multiple file inputs

Open
#189 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.1k
Forks
385
PR merge metrics
No merged PRs in 30d

Description

Similar to #187

With the introduction of the batching feature extraction (#148), the inputs to rs_subset is likely going to be multiple `.tiles` files. My current workaround to combine all of these files prior to running `rs_subset`

```python
import os
import csv
import argparse
from random import shuffle

def main():
parser = argparse.ArgumentParser()
parser.add_argument('dir', type=str)
parser.add_argument('out', type=str)
parser.add_argument('--ext', type=str, default='tiles')
args = parser.parse_args()

os.makedirs(args.out, exist_ok=True)

all, train, validation, evaluation = [], [], [], []

# combine tile csv files
for file in os.listdir(args.dir):
if file.endswith(args.ext):
csv_file = os.path.join(args.dir, file)

print(f"processing file: {csv_file}")

with open(csv_file, newline='') as csvfile:
spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
for row in spamreader:
all.append(row)

# shuffle
shuffle(all)

_80 = int(len(all) * 0.80)
_10 = int(len(all) * 0.10)

train = all[0:_80]
validation = all[_80: _80 + _10]
evaluation = all[_80 + _10:]

# write new csv files
for k, v in {"train.tiles": train, "validation.tiles": validation, "evaluation.tiles": evaluation}.items():
with open(os.path.join(args.out, k), 'w', newline='') as f:
wr = csv.writer(f, delimiter=' ', escapechar=' ', quoting=csv.QUOTE_NONE)
for line in v:
wr.writerow(line)

if __name__ == "__main__":
main()
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the rs_subset command and read the related issues #187 and #148 for the expected batching behavior. The change is done when rs_subset accepts multiple .tiles inputs and produces the train, validation, and evaluation outputs without requiring users to combine files first.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.