Improve metrics in Celery instrumentation

What problem do you want to solve?

Our instrumented Celery applications often are deployed along with Celery Flower to monitor a few metrics on tasks and workers. But those metrics can also be made available through auto-instrumentation. I am sure for those whose usage of Flower is limited having it all in a single instrumentation would be beneficial.

Describe the solution you'd like

• Add metrics from Celery Flower, i.e. counts for executing and prefetched tasks as well as prefetch and execution duration of a task.
• Rename those metrics according to OTel semantic conventions
• Fix a memory leak in current metrics generation (this entry never gets removed)

Describe alternatives you've considered

No response

Additional Context

I got some parts working in my local environment but testing has been a challenge. I don't think I ever had all unit tests passing even in clean setup. I have no idea if it's something in my own environment as Celery tests appear to be generally green in PRs.

Would you like to implement a fix?

Yes

Start in opentelemetry-instrumentation-celery/src/opentelemetry/instrumentation/celery/init.py, around the metrics-generation entry cited in the issue. Review the existing Celery instrumentation and its unit tests, then verify the requested Flower metrics, OpenTelemetry naming, and removal of retained entries so the memory leak is fixed.