googleapis / googleapis/librarian

sidekick: automatically detect dependencies between packages

Open
#7,300 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

sidekick
Dominant language
Go
Stars
45
Forks
62
Avg merge
18h 32m
Merged PRs (30d)
175

Description

Sidekick-based languages use an explicit configuration to detect "dependencies" between packages, for example, Swift has this configuration:

https://github.com/googleapis/librarian/blob/01e160eba49a8e46334c2d5ff9925c75662ed696/internal/config/swift.go#L19-L20

Which is manually curated:

https://github.com/googleapis/google-cloud-swift/blob/92151d196557a6a616ec61adb9303361e8aa9efa/librarian.yaml#L33-L52

Curating this dependency list is tedious. It also disrupts any automation to update the service definitions: whenever API A grows a dependency on API B, we need to add the library for API B to this dependency. Most often the dependency may already be there, but cases of "B is a dependency for the first time" do happen several times a year.

It would be desirable if `sidekick` could figure out where the dependencies are automatically. It is at least within reach as:

- All the libraries are listed in `librarian.yaml`, any new dependency must already exist there.
- When `generate` runs `sidekick` knows what namespace / Protobuf package is used.

Note that the generated code needs information from these dependencies. It needs to generate additional `import` statements, and additional information in the package metadata (e.g. `Cargo.toml` or `Package.swift` files). So somehow this "global" information needs to be collected and provided to each package.

For this to work, `generate --all` would need to run in two phases:

- First collect the existing APIs, their library info (package name and version), but not generate any code
- Second, use that information to generate each library in parallel

Something like `generate google-cloud-secretmanager-v1` would still need to run in two phases too, but the second phase would be faster.

I think the first phase would require parsing the protos / discovery doc. This is not the slowest thing, but it is not trivial. A benchmark seems appropriate.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with internal/config/swift.go and the dependency configuration in librarian.yaml, then trace the generate and generate --all entry points in sidekick. Review how protos or discovery documents are parsed and benchmark the proposed first phase. Done means dependencies are detected automatically and generation can use the collected library metadata in its second phase.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust, swift
Domain
build-system, cli, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.