astropy / astropy/pyvo

Overcoming the limitation of http get-based DALResults from_result_url method

Open
#601 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
88
Forks
64
Avg merge
38m
Merged PRs (30d)
1

Description

At ESO we have a datalink service that associates calibration files to science raw files.
For efficiency, it is better (much better) to pass to such service a big number of science raw files IDs (up to several thousands), instead of splitting the request into many chunks. Passing so many IDs is not possible using an http get, http post must be used instead.

The from_result_url() method is therefore of no use here, as it uses a "session.get", hence limiting to few hundreds the max number of IDs passed at once.

The gain in efficiency depends on the instrument and mode, I have seen cases whereby a POST returns in few minutes while the equivalent set of chunked GET requests takes several hours. On top, after those several hours, the software has still to merge the results of each chunked GET.

I'm aware of the existing DatalinkResultsMixin() method; that would (partially) remove the management of merging results, but it cannot overcome the mentioned efficiency issue, as, using GET, it cannot make use of the optimised internal logic of the calibration service when used on a big number of IDs.

Could you implement a pyvo method that would implement the equivalent of from_result_url using POST instead of GET?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the from_result_url() method and the DatalinkResultsMixin() implementation, focusing on how the current session.get request builds and retrieves results. Define the equivalent POST request for large sets of science raw-file IDs while preserving the existing result behavior. Done means callers can submit thousands of IDs through a pyvo method without chunked GET requests or manual result merging.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.