Gaia.load_data increase Maximum number of source Ids.
- Dominant language
- Python
- Stars
- 791
- Forks
- 451
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
I would like to use Gaia.load_data to grab epoch photometry for ~500,000 sources. I can successfully do this for a small number of sources, but run into a limit when I scale up. Error message is below. Is it possible to increase the allowed maximum number of source Ids? I'd vote for 500,000 as the limit just because it is my use case, but could see reasons to go higher. I don't know what the true limitations are. I'd also be open to other suggestions for how I should access the epochal photometry for these sources.
Background: I can get the 500,000 source id's using a table upload and a JOIN inside of a query to Gaia.launch_job_async. This is lightning fast and fantastic.
Here is my actual code:
```python
## Some Definitions
retrieval_type = 'EPOCH_PHOTOMETRY'
data_structure = 'INDIVIDUAL'
data_release = 'Gaia DR3'
## Get the files
datalink = Gaia.load_data(ids=ids,
data_release = data_release,
retrieval_type=retrieval_type,
data_structure = data_structure, verbose = False, output_file = None , overwrite_output_file=True)
```
And the error message I am getting (on testing with fewer sources):
```
Maximum number of Source Ids reached (max: 5000, found: 22577)
Cannot process request: 'https://gea.esac.esa.int/data-server/data' (req: Reqid: anonymous1695744305998, retrieval access: DIRECT, retrieval type: EPOCH_PHOTOMETRY, compression: null), for user: UwsJobOwner{id='anonymous', name='null', mail='null', authUsername='null', authGroups=[], pseudo='anonymous', session='ABF99E52685E5059A90EECD530692465', ip='54.82.184.137', roles=0, parameters=Owner parameters: 6}, due to: Maximum number of Source Ids reached (max: 5000, found: 22577)
```
Contributor guide
Research direction
Start with the supplied Gaia.load_data reproduction and compare it with the Gaia.launch_job_async table-upload query. Trace where the 5,000-source limit is enforced and determine whether it belongs to astroquery or the remote Gaia data service. Done means establishing a supported bulk-retrieval path or documenting the service limitation and an alternative.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100