Add a cloud_detection switch and probe cloud metadata concurrently
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 323
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 12
Description
Feature proposal.
Problem
Every online EmissionsTracker probes the cloud instance metadata service (IMDS) at startup to find out whether it runs on AWS, Azure or GCP. get_env_cloud_details() in codecarbon/core/cloud.py walks CLOUD_METADATA_MAPPING sequentially, with a 1 s timeout per provider. All three entries live on the same link-local address 169.254.169.254, so on a laptop, an on-prem server or any machine where that address is simply not routed, we pay up to three consecutive timeouts before concluding "not on a cloud". That is a guaranteed-failing, guaranteed-slow startup cost for every non-cloud user, and it grows linearly with each provider we might add later.
Some users cannot pay it at all: in air-gapped or tightly egress-filtered environments the probe is not just slow, it is noise in security monitoring, and there is currently no way to turn it off. The only escape today is to use OfflineEmissionsTracker, which gives up much more than cloud detection.
Proposed design
Two small, independent changes to the same code path:
-
cloud_detectionswitch. A new tracker option, following the existing configuration conventions, so it works as a constructor argument, a.codecarbon.configkey and an environment variable:[codecarbon] cloud_detection = falseEmissionsTracker(cloud_detection=False)CODECARBON_CLOUD_DETECTION=falseDefault stays
true, so behaviour is unchanged for everyone who does not set it. When disabled,EmissionsTracker._get_cloud_metadata()returns an emptyCloudMetadatawithout issuing any HTTP request, and the tracker takes the normal non-cloud geolocation path. -
Concurrent IMDS probing. Replace the sequential loop in
get_env_cloud_details()with a boundedconcurrent.futures.ThreadPoolExecutorfan-out over the same mapping. Total detection wall time becomes roughly one timeout instead of one per provider, and stays constant if the mapping grows. Provider selection stays deterministic: results are resolved inCLOUD_METADATA_MAPPINGorder, not in completion order, so a machine that somehow answers on two entries always reports the same provider it does today.
Neither change alters the public return shape of get_env_cloud_details() or the CloudMetadata dataclass, and no new dependency is involved — concurrent.futures is stdlib.
Why it fits the existing extension points
CLOUD_METADATA_MAPPING is already the single source of truth for what gets probed and how; the concurrency change is confined to the loop that consumes it, and the per-entry postprocess_function hook (used by GCP to strip attributes, which carries Kubernetes config and secrets) is preserved unchanged. The configuration switch uses _set_from_conf, the same mechanism as every other tracker option, so it inherits config-file and environment-variable support for free.
Scope boundary
This proposal deliberately covers only the two items above. Explicitly out of scope, and better argued separately:
- Support for additional providers (OVH, Scaleway, Hetzner, OCI). Adding entries is cheap, but each needs a payload parser and a verification predicate — several vendors answer on the same link-local address, and the generic OpenStack metadata path answers on any OpenStack private cloud, so a naive entry risks mislabelling a machine's region. A confidently wrong region is worse than no region.
- Region-level carbon intensity in
codecarbon/data/cloud/impact.csv. The schema already supports it; the real cost is data curation and provenance (location-based vs market-based figures, sourcing, staleness), which deserves its own discussion. - A
carbon_intensity_sourcefield onEmissionsDatarecording which fallback rung produced the intensity.
Happy to open follow-ups for those once the plumbing here is in place.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with get_env_cloud_details() and CLOUD_METADATA_MAPPING in codecarbon/core/cloud.py, then trace EmissionsTracker._get_cloud_metadata() and the _set_from_conf configuration path. Implement the opt-out behavior and bounded concurrent probing while preserving mapping-order provider selection and the existing CloudMetadata return shape. Done means constructor, config-file, and environment-variable settings work, disabled detection makes no HTTP request, and cloud probing no longer waits sequentially.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100