Allow custom drivers and pin types
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 59
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
I'm looking at using pins to version various model assets, but one of the pain points I'm running into is custom drivers for reading and writing. Many of our model artifacts are serialized to YAML or ONNX or a custom serialization, but there's no driver for those. There's friction too if we wanted, say a polars driver to read a csv or parquet directly into polars.
This leaves pin_upload and pin_download, which while workable, doesn't provide the same level of integration as pin_read and pin_write. If you use a user-defined helper with pin_download to read a custom type, then users have to know in advance to use that helper, versus just board.pin_read(name) for officially supported types.
For example, to show the difference in experience between a csv file and yaml:
import pandas as pd
import yaml
from pins import board_temp
board = board_temp()
data = pd.DataFrame({"a": [1,2,3]})
board.pin_write(data, "pandas_data", type="csv")
with open("data.yml", "w") as f:
yaml.dump({"a": [1,2,3]}, f)
board.pin_upload("data.yml", name="yaml_data")
# Reading
pandas_data_read = board.pin_read("pandas_data")
yaml_path = board.pin_download("yaml_data")[0]
with open(yaml_path) as f:
yaml_data_read = yaml.safe_load(f)
# Using helper
def pin_read_yaml(board, *args, **kwargs):
yaml_path = board.pin_download(*args, **kwargs)[0]
with open(yaml_path) as f:
return yaml.safe_load(f)
yaml_helper_read = pin_read_yaml(board, "yaml_data") # need to know in advance to use yaml, not just pin_read(name)
Optimally, I want to be able to specify custom types that are associated with custom drivers. The custom type would be written to data.txt, which would allow for dispatching to the associated driver on pin_read like is done for officially supported types, without needing to know in advance to use a user-defined helper.
I'm also seeing similarities between this problem and "dynamic artifact handling" in another library we work on (lazyscribe/lazyscribe#56), where we used handler classes and entrypoints to allow users to define custom artifact serializers/deserializers.
Anyway, please let me know if I'm missing anything here and what you think!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing pin_read, pin_write, pin_upload, and pin_download entry points and the officially supported driver handling. Compare how types and metadata are currently dispatched, then review the linked lazyscribe handler and entrypoint example for relevant constraints. Done means custom drivers can be associated with custom types and used through the pin_read and pin_write interfaces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100