☂ Enhance OpenAPI experience
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
DataFrame has the ability to import Data Schemas from the type definitions of OpenAPI files: https://kotlin.github.io/dataframe/schemasimportopenapigradle.html, https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html.
This works, but I keep feeling we're missing something:
#### Just Types?
OpenAPI files contain both the API endpoints (aka urls, parameters etc.) as well as the type schema's of potentially returned types of these endpoints. Currently, DataFrame ignores endpoints and just extracts the types to convert them to DataSchema's. It's up to the user to call the API itself using `DataFrame.readJson("url")` and convert the result to the generated type (or they call `GeneratedType.readJson("url")`), however, all of this breaks down when the API has any authentication or requires a PUT instead of a GET request.
At the moment, even with our OpenAPI support, it's not trivial to use DataFrame with a Json API.
Also, OpenAPI with "just types" has a name, it's [JSON Schema](https://json-schema.org/specification), but that we don't support right now: https://github.com/Kotlin/dataframe/issues/683
#### Issues with implementation
Our implementation also has a couple issues:
- We don't support OpenAPI 3.1 https://github.com/Kotlin/dataframe/issues/683
- Bugs: https://github.com/Kotlin/dataframe/issues/288, https://github.com/Kotlin/dataframe/issues/255
- Annoying logs: https://github.com/Kotlin/dataframe/issues/785
- No JSON Schema support: https://github.com/Kotlin/dataframe/issues/683
- While we have plenty of tests for our DataSchema generation, we know it's not perfect and there are probably many breaking cases to be found for valid OpenAPI schemas.
- https://github.com/Kotlin/dataframe/issues/459
#### How forward
I have a couple of ideas of how to remedy this situation, but feedback would be welcome.
First of all, we need to take a look at [other tools](https://github.com/Kotlin/dataframe/issues/369) to handle the code generation. An (apparently) industry leading project appears to be [OpenAPITools OpenAPI Generator](https://github.com/OpenAPITools/openapi-generator). It can for instance generate a ktor project automatically and with a lot of options. This project can then be used as a dependency in a notebook (even now!) or in another project. I found it works wonders together with our new compiler plugin in this [example project](https://github.com/Jolanrensen/df-openapi-generator-plugin-demo). We can use any JSON parser the generator provides, as converting Kotlin objects to DataFrames is easy :) (except perhaps for [maps](https://github.com/Kotlin/dataframe/issues/896)).
However, for this to take off, I'd imagine a situation where a user would just have to import an openAPI file with a single statement and everything would be generated, imported, and ready to go. Especially in notebooks, this would be awesome.
Next, we should probably support "just" Json Schemas to generate DataSchemas. A user could still feed it an openapi file, since they're now a superset of json schema, but it would make it easier to market for sure.
What do you think?
Contributor guide
Research direction
Review the existing OpenAPI schema-import documentation for Gradle and Jupyter, then read the linked issues covering OpenAPI 3.1, bugs, logging, JSON Schema, and maps. Compare the proposed OpenAPI Generator and compiler-plugin example to identify a concrete, bounded first change. Done requires an agreed scope rather than the broad set of improvements described here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin, openapi
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100