Suggestion/Request/Contribution: Create a TNS service to access the Transient Name Server
- Dominant language
- Python
- Stars
- 791
- Forks
- 451
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
TNS is the IAU approved catalog for all transient data.
Currently programmatic TNS access is locked behind a TNS account, creating a BOT key, and using the TNS API, and requires scientific credentials, none of which is good for open source or enabling access. (Although there is a public UI website to do all of this of course).
I propose that astroquery works with TNS to enable making TNS queries that follow the standard astroquery API and client-side caching. The problem is that TNS does not want to leak any API keys to the public, nor overload the searches. It is also not ideal to have to scrape the website, as breakages can happen and this can put strain on TNS servers.
The usecase I have in mind is two fold, one short term and one longer term and more involved:
1. **Short**: Query TNS for basic transient information such as name, ra & dec, type, redshift, discovery date. (Small json formatted payload of textual data). Pros: Low load, can be cached, query complexity can be tightly controlled. Cons: Do not get access to all TNS data.
2. **Long**: Query TNS for additional metadata for the transient, including things such as uploaded spectra. Search TNS for combinations of parameters. Pros: Do everything the TNS API can do, (and actually more). Cons: High load, easily abusable, cached results can be significantly wrong our outdated, query complexity can grow quickly.
In a research group, I personally have a bot for doing TNS searches to obtain new transients, and have an API wrapper to accomplish the second set of goals. But this cannot scale to the whole public or easily be prevented from abuse.
However, number 1 IS easy. Currently, I self-host on Google Cloud the `tnsquery` API: https://tnsquery.darkdata.ml/api/docs
Which basically allows for doing limited size queries that return things like the name redshift, coordinates, etc., without leaking any API credentials, and allowing access to the public. See: https://github.com/emirkmo/tnsquery. The queries are cached locally at `tnsquery` API level in a performant and scalable DB so that repeat queries never even hit the TNS server, unless explicitly requested via a `force_tns = Ttue` parameter.
I propose that astroquery uses a service like this, and we add a way to publicly query for basic transient information. In fact, it is totally fine to use `tnsquery` itself, until a longer term situation is figured out with TNS runners, as even querying the entire TNS db non-stop would not be a big cost, and it's possible to rate limit way before that. The cached DB and the rate limit of the `tnsquery` API key already secures 99% of what's needed.
Existing alternatives that for example use SIMBAD do not work as they do not have the latest transients. Nor can everyone use the TNS API since access requires scientific credentials.
Additionally, there is a benefit that `Astropy` itself can use this service to create `SkyCoord` objects from transient names, which is a hit or miss feature currently that requires the transient to be on Simbad or some other non-TNS source. I will open a related issue there.
I am willing to help contribute the code if the basics are agreed upon, and am willing to handle contact with the TNS runners if that is not something `astroquery` already has in the network. I did write to them and they agreed that it would be good to give programmatic access to the public somehow, similar to other astronomical databases. I can even likely secure funding or resources to run the `tnsquery` sub-service. But only if it will be used in some official capacity by `astroquery/astropy`.
Cheers,
Emir
Contributor guide
Research direction
Start by reviewing astroquery's existing service patterns and the linked tnsquery API documentation and repository. Clarify with TNS maintainers whether the short-term basic-information service is in scope, then define the public API, caching, rate limits, and acceptance criteria before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100