digitalocean / digitalocean/godo
Provide convenience methods for paging over collections
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 389
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 15
Description
A number of API responses return collections (i.e., a slice of objects as opposed to just a single object) that callers need to [page through](https://developers.digitalocean.com/documentation/v2/#links) in order to visit exhaustively. Examples are droplets, volumes, and Kubernetes clusters.
While [the implementation](https://github.com/digitalocean/godo#pagination) isn't necessarily complex, the boilerplate adds up quickly when multiple collection object types are involved or iteration needs to happen repeatedly in client code. What's usually happening in my code then is that abstractions emerge that allow me to reuse paging. However, that approach doesn't work across repositories.
Consequently, I'd like to kick off a discussion on the possibility of providing convenience methods for iterating over collections in godo; essentially, my ask is to move the kind of abstraction I talked about above into godo.
Assuming for a second that this is an idea we can get onboard with in general, the interesting question is how it could be implemented, which in turn depends on what we imagine the signature to look like. One challenge here may be that different collections return different types, and Go's lack of generics does not allow us to (easily) provide a single implementation.
My basic thinking is that we offer a method which accepts a function to handle a given (paged over) object, with possibilities to indicate if paging should stop prematurely as well as error propagation. The scaffolding of increasing the page variable and handling paging-related errors should be done by the outer implementation, not the passed in custom function from the user.
With that in mind, here's my initial proposal (using Volumes as an example):
We'd provide
```go
type VolumeHandler func(context.Context, Volume, *Response) (done bool, err error)
```
and extend the `StorageService` interface by a new method:
```go
PageVolumes(context.Context, VolumeHandler, *ListVolumeParams) error
```
The implementation of `PageVolumes` could then look something like this:
```go
func (svc *StorageServiceOp) PageVolumes(ctx context.Context, handler VolumeHandler, params *ListVolumeParams) error {
for {
volumes, resp, err := svc.Client.ListVolumes(ctx, params)
if err != nil {
return err
}
// handle each volume by the caller's handler function
for _, vol := range volumes {
done, err := handler(ctx, vol, resp)
if err != nil {
return err
}
if done {
return nil
}
}
// if we are at the last page, break out the for loop
if resp.Links == nil || resp.Links.IsLastPage() {
break
}
page, err := resp.Links.CurrentPage()
if err != nil {
return err
}
// set the page we want for the next request
params.ListOptions.Page = page + 1
}
return nil
}
```
I realize the proposal may be far from (our) ideal; and there are likely detailed design aspects we'd have to discuss (e.g., whether we should pass in the `*Response` into the handler function or not). More importantly though, there's the question of whether this is the right approach on a higher level: Should this be a new method on each interface? Should we consider code generation to avoid repeating lots of code? Or is there perhaps a chance to centralize the implementation by using `interface{}`s, at the price of less type safety?
My primary goal with this issue is to start a discussion, so I'm very open to exploring any directions you may feel are more appropriate.
Thanks!
Contributor guide
Research direction
Start by reading the existing StorageService interface, ListVolumes method, pagination links, and the proposed VolumeHandler signature. Compare how collection types such as droplets, volumes, and Kubernetes clusters are represented before deciding whether the design should use per-interface methods, code generation, or a shared abstraction. Done means the API design is agreed, implemented consistently, and its pagination, early-stop, and error behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100