[Feature Request]: Retry policies for ParDo

### What would you like to happen?

I have a `ParDo` in a streaming pipeline that needs to be given some retry policy because it depends on an external service.

I understand this doesn't [meet desired "no external side effects"](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#hdr-Fault_Tolerance) behavior/usage that is suggested.

If I introduce the retry policy as sleeping in the `DoFn` all sort of issues seems to occur (principally that stage of the pipeline stalling in GCP dataflow). I would like to instead define some retry function elsewhere to prevent retrying records from causing degradations for other messages being processed through the pipeline.

Is there an accepted interface to use to orchestrate retries with some backoff policy on the Beam level?

### Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [X] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner