opensafely-core / opensafely-core/opensafely-cli

Consider making a new command to make .arrow output files more easily viewable

Open
#324 1 comment 0 reactions 1 assignee View on GitHub

@mikerkelly is already working on this.

Since May 31, 2026.

arrow deck-scrubbing enhancement
Dominant language
Python
Stars
2
Forks
5
Avg merge
2d 1h
Merged PRs (30d)
4

Description

The recommended output format for ehrQL = is arrow. However, if users run an opensafely run action locally, the arrow output file is not easily viewable. (There are a couple of VS code extensions in the marketplace, but nothing official).

The research template currently still defaults to using the output format csv.gz; this is because we have a way for users to unzip and view a csv.gz output (with the opensafely unzip command). It would be better if we could update the template to use arrow, because users are likely to use whatever we put in the starting template, irrespective of what the docs say.

We'd like to consider adding a command to either view and arrow file or (probably more useful) convert an arrow file to csv. If done in python (with pandas), this would be very simple, essentially just:

df = pd.read_feather("dataset.arrow")
df.to_csv("dataset.csv")

It should be possible to make this a wrapper around an opensafely exec python command to run the actual code using the python image, which already has the necessary pandas/pyarrow dependencies.

More details in slack thread

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.