MLMI2-CSSI / MLMI2-CSSI/foundry

Automate publishing matminer and deepchem datasets

Open
#188 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

publish data
Dominant language
Python
Stars
88
Forks
17
PR merge metrics
No merged PRs in 30d

Description

write a script you can run on your local machine (like a .py script you can run in PyCharm) that automates publication of datasets.

The following are some general things to think about, but I suggest breaking them out into your own tasks in this story

Some steps to investigate:

  • whether or not you can pull the download links to the datasets programmatically, by webscraping with BeautifulSoup or a similar Python package
  • how to write the metadata to .json files (use the json package)
  • how to read in the metadata files and data files from your local machine programmatically (I suggest keeping them all in a folder that you walk through using os or something similar)
  • how to map the metadata files to the data files
  • how to pass arguments to the script so you can run it easily from the commandline (I suggest argparse) -- an example of an argument you might want to pass in would be the path to the directory containing the data
  • The ultimate goal is to be able to just run the script and publish these datasets with as little human labor as possible. So if there's something you can code to make less human work in the future, do that something! :)

if there does not appear to be a way to read in the title and related information from the metadata, then create a new issue to add that to foundry and amend publish(). But first, see if you can include it in the metadata

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing publish() flow and how metadata and data files are currently handled. Investigate the matminer and deepchem dataset sources, then define the script's inputs, metadata mapping, and command-line behavior; done means local execution publishes both datasets with minimal manual work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.