anomaly / anomaly/gallagher

Provide a SQL (preferably a SQLAlchemy `dialect`) interface to query the REST API

Open
#31 4 comments 0 reactions 1 assignee Claimed by @devraj View on GitHub
Dominant language
Python
Stars
17
Forks
2
PR merge metrics
No merged PRs in 30d

Description

As our ambition around the library grows, the sync feature #9 looks to be a lot more useful than I previously imagined. One of my thoughts was to treat this as if they were two databases sources and use a layer like SQLAlchemy to keep them in sync.

Sync is a difficult problem (see also [SQL Sync for Schema with SQLAlchemy](https://www.dolthub.com/blog/2020-08-24-schema-support-in-sql-sync/)
[python-sync-db](https://github.com/bintlabs/python-sync-db)) at the best of times. My initial research lead me down the path of writing a [SQLAlchemy dialect](https://docs.sqlalchemy.org/en/20/dialects/#external-dialects) primarily thinking about if it's possible to wrap a REST service as a SQL source.

I found a number of articles and discussions which leads to believe this is a possible way forward:
- [How to add support for a new SQLAlchemy dialect](https://legacy.docs.greatexpectations.io/en/latest/guides/how_to_guides/miscellaneous/how_to_add_and_test_a_new_sqlalchemydataset_class.html)
- Discussion on Stackoverflow on [How to write my own dialect in sqlalchemy to adapt HTTP API?](https://stackoverflow.com/questions/43929132/how-to-write-my-own-dialect-in-sqlalchemy-to-adapt-http-api)

Digging through the list of [external dialects](https://docs.sqlalchemy.org/en/20/dialects/#external-dialects) I found [betodealmeida](https://github.com/betodealmeida) / [gsheets-db-api](https://github.com/betodealmeida/gsheets-db-api) which provides that we can converse with a REST API via a SQLAlchemy dialect. This project links to [Shillelagh](https://github.com/betodealmeida/shillelagh/?tab=readme-ov-file) which is a library is an implementation of the [Python DB API 2.0](https://www.python.org/dev/peps/pep-0249/) based on [SQLite](https://sqlite.org/index.html) (using the [APSW](https://rogerbinns.github.io/apsw/) library)

Other resources:
- [Dolt](https://www.dolthub.com) version controlled MySQL database

The requirement is thus to research the above resources and outline the possibility of using a SQLALchemy dialect to write the sync module with the view of being able to use the REST client to communicate with the Gallagher proxy.

## Why SQL?

One of the major questions around this is `Why SQL?`, we could head down the route of using an object database. The question to consider is if the end user / customer would benefit from the data being available in an object database?

With a foundation of writing to SQL backend we also stand the advantage of writing to / reading from corporate database backends.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.