lincc-frameworks / lincc-frameworks/notebooks_lf

Tool to populate TAP server DB from existing TAP server

Open
#48 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Overview
Implement a script or tool to automatically query an external TAP server (such as the Caltech IRSA TAP service) for schema metadata of a specified catalog (e.g., Gaia DR3), parse the metadata, and insert it into our local TAP schema SQLite database.

## Requirements
- Accept TAP server URL and catalog name/table name as arguments or configuration.
- Query the remote TAP server for schema and column metadata (e.g., via ADQL queries to `tap_schema.tables` and `tap_schema.columns`).
- Parse the returned metadata (fields such as schema name, table name, column name, data type, units, UCD, description, etc.).
- Insert the parsed metadata into the corresponding tables (`schemas`, `tables`, `columns`) in our local `tap_schema.db` SQLite database, using the established schema (see Issue #45).
- Handle errors gracefully (invalid catalog/table, network issues, incomplete metadata, etc.).
- Provide simple logging or reporting of what was imported and any issues encountered.
- Document usage and provide example invocation (e.g., for Gaia DR3 at Caltech).

## Example Workflow
1. User runs: `python import_tap_schema.py --url https://irsa.ipac.caltech.edu/TAP --catalog gaiadr3`.
2. Script queries the TAP server for all relevant metadata for the specified catalog.
3. Script parses the result and inserts metadata into local database.
4. User can inspect the populated database or query it using the TAP service.

## Deliverables
- Python script/module to perform the import
- Example usage instructions
- Integration guidance for the TAP prototype server

## References
- [IVOA TAP 1.1 Specification](https://www.ivoa.net/documents/TAP/20181024/PR-TAP-1.1-20181024.html)
- [IRSA Caltech TAP Service](https://irsa.ipac.caltech.edu/TAP)
- [Gaia DR3 TAP Service](https://gea.esac.esa.int/tap-server/tap)

---
This task will enable the service to be populated with real astronomical catalog metadata, supporting true TAP schema responses for users.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the proposed import_tap_schema.py entry point and the established tap_schema.db schema from Issue #45, focusing on the schemas, tables, and columns tables. Verify the TAP metadata queries and expected fields before implementing the import flow. Done means a catalog can be selected, metadata is stored locally, errors are reported, and example usage is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql, sqlite
Domain
backend, database, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.