Write a guide about embedding Conduit
- Dominant language
- Go
- Stars
- 610
- Forks
- 63
- Avg merge
- 12h 28m
- Merged PRs (30d)
- 57
Description
We should write a guide and example on how to embed Conduit in your own Go application. Here's part of a conversation with a user who played around with embedding Conduit, the guide should be more detailed.
---
You should start with [conduit.NewRuntime](https://github.com/ConduitIO/conduit/blob/6d322761970ff1668d7ad44207dba781e1d4f761/pkg/conduit/runtime.go#L92), it will give you an object you can use to work with Conduit. Specifically the [Orchestrator and ProvisioningService](https://github.com/ConduitIO/conduit/blob/6d322761970ff1668d7ad44207dba781e1d4f761/pkg/conduit/runtime.go#L78-L79) expose all important functionality. To get an idea of how to configure the runtime, you can have a look at the [entrypoint](https://github.com/ConduitIO/conduit/blob/6d322761970ff1668d7ad44207dba781e1d4f761/pkg/conduit/entrypoint.go#L42) used by the Conduit CLI. This should already be a good starting point. If you then want to provision a pipeline, you can either pass a [configuration file](https://conduit.io/docs/pipeline-configuration-files/getting-started) to the provisioning service or use the orchestrator to manually create the pipeline, connectors and processors.
The database is used to store both things you mentioned, the pipeline configuration and its state. All Conduit needs is a key-value store. Out of the box we provide a way to store this data in a Postgres instance, locally in a badger DB or just in memory (but that gets wiped between restarts, so that's just useful for development purposes). You can also implement your own storage mechanism, it needs to implement the interface [database.DB](https://github.com/ConduitIO/conduit/blob/main/pkg/foundation/database/database.go#L31) and pass the [acceptance test](https://github.com/ConduitIO/conduit/blob/main/pkg/foundation/database/acceptance_testing.go#L38), then you can provide it when configuring conduit as the [DB driver](https://github.com/ConduitIO/conduit/blob/31e283db4884003e4427e8bedbd598f3a3e250e7/pkg/conduit/config.go#L39).
Contributor guide
Assessment
This issue has not been assessed yet.