astropy / astropy/astroquery

Gaia.load_data increase Maximum number of source Ids.

Open
#2,843 4 comments 0 reactions 0 assignees View on GitHub
Documentation gaia
Dominant language
Python
Stars
791
Forks
451
Avg merge
1d 3h
Merged PRs (30d)
4

Description

I would like to use Gaia.load_data to grab epoch photometry for ~500,000 sources. I can successfully do this for a small number of sources, but run into a limit when I scale up. Error message is below. Is it possible to increase the allowed maximum number of source Ids? I'd vote for 500,000 as the limit just because it is my use case, but could see reasons to go higher. I don't know what the true limitations are. I'd also be open to other suggestions for how I should access the epochal photometry for these sources.

Background: I can get the 500,000 source id's using a table upload and a JOIN inside of a query to Gaia.launch_job_async. This is lightning fast and fantastic.

Here is my actual code:
```python
## Some Definitions
retrieval_type = 'EPOCH_PHOTOMETRY'
data_structure = 'INDIVIDUAL'
data_release = 'Gaia DR3'

## Get the files
datalink = Gaia.load_data(ids=ids,
data_release = data_release,
retrieval_type=retrieval_type,
data_structure = data_structure, verbose = False, output_file = None , overwrite_output_file=True)

```

And the error message I am getting (on testing with fewer sources):
```
Maximum number of Source Ids reached (max: 5000, found: 22577)

Cannot process request: 'https://gea.esac.esa.int/data-server/data' (req: Reqid: anonymous1695744305998, retrieval access: DIRECT, retrieval type: EPOCH_PHOTOMETRY, compression: null), for user: UwsJobOwner{id='anonymous', name='null', mail='null', authUsername='null', authGroups=[], pseudo='anonymous', session='ABF99E52685E5059A90EECD530692465', ip='54.82.184.137', roles=0, parameters=Owner parameters: 6}, due to: Maximum number of Source Ids reached (max: 5000, found: 22577)

```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Gaia.load_data reproduction and compare it with the Gaia.launch_job_async table-upload query. Trace where the 5,000-source limit is enforced and determine whether it belongs to astroquery or the remote Gaia data service. Done means establishing a supported bulk-retrieval path or documenting the service limitation and an alternative.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.