cucapra / cucapra/pollen

Expose GAF-lookup functionality through the FlatGFA Python API

Open
#205 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
45
Forks
3
PR merge metrics
No merged PRs in 30d

Description

In general, we would like to expose lots of existing FlatGFA functionality through the Python bindings instead of just through the command-line interface (see #179). In specific, I think it would be really cool to take our recent work on GAF lookup and produce Python bindings for that—with the medium-term goal of making it possible to implement a kind of "model" of an ongoing GWAS project within the Panorama umbrella that needs this as one component.

Here is a plan, divided into phases.

## 1: Eager API

Let's expose our GAF parser and stuff through an "eager" API that produces a bunch of complete Python-native data structures up front. @mt-xing has already done this! It's in #203.

We should merge this and get happy with some tests for the end-to-end functionality.

## 2: Lazy API

The problem with the previous milestone is that it doesn't scale well to big GAF files. So let's expose an iterator-based API that attempts to parse only one line at a time.

@JerryJi0730 is already well on his way to making this work. Let's make this version pass the same Python-side tests as the eager implementation from phase 1.

Let's merge this and replace the eager version from phase 1. Then, this would be a good time to benchmark the Python-wrapped implementation against the `fgfa gaf` pure-Rust command-line tools. We expect it to be slower—the question is, how much slower?

## 3: Strawperson Configurability, and Start a Conversation

The problem with both of the APIs above is that they require processing GAF files with a sequential Python `for` loop. Even with the iterator-based API, just doing `for line in gpa: ...` is extremely costly, because of (1) the Python loop overhead and (2) the loss of parallelism.

We have a vague vision for a nicer approach that would move the looping to Rust, making it faster and parallel. Broadly, the idea is to have the Python program build up a description of _what to do with each line_ and then hand it off to the Rust side. Then, the Rust-side loop runs and carries out those orders.

To keep things simple, let's start with a really simple definition of "what to do with each line": we will offer just 3 possible options, mapping to the 3 options that exist on the command line (do nothing, print the nucleotide sequence, and print information about the segment offsets). So the API could look really basic from the Python perspective; maybe something like this:

```py
gaf = flatgfa.load_gaf("foo.gaf")
gaf_lines = gaf.lookup("bar.flatgfa")
gaf_lines.print_seqs.run()
```

This is not meant to be useful by itself—it's just to have _some_ kind of configurability!

The real next step here is to start a conversation with our actual bio collaborators to see what they need. We would schedule a meeting, show off this super basic functionality, and try to understand what the next step is in this rumored GWAS pipeline.

## 4: A More Sophisticated Embedded DSL

Based on that conversation, we start an open-ended exploration into designing a Python DSL that can support the kind of composition necessary to drive the GWAS pipeline.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Python bindings and the work referenced in #203, then compare the existing `fgfa gaf` command-line functionality with the `flatgfa.load_gaf` and `gaf.lookup` entry points described here. Review the eager and lazy API phases, their Python-side tests, and the proposed configurability before deciding what remains; done is not defined because later phases depend on collaborator feedback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.