pytorch / pytorch/rl

[BUG] SACTrainer and CQLTrainer unnecessarily query collector environments

Open
#4,376 1 comment 0 reactions 1 assignee View on GitHub

@aswanth-07 is already working on this.

Since Sep 14, 2026.

Dominant language
Python
Stars
3.6k
Forks
487
Avg merge
1d 1h
Merged PRs (30d)
207

Description

Problem

SACTrainer and CQLTrainer read full_action_spec_unbatched from the collector whenever the loss has no explicit action_spec. That lookup is unnecessary when a numeric target_entropy already determines the entropy target or when the actor already carries the action spec.

Collectors that do not expose getattr_env therefore cannot construct either trainer even though the loss has all information it needs.

This is the SAC/CQL follow-up requested in the review of #4368.

Reproduction

A loss with an explicit target works independently:

loss = SACLoss(actor, qvalue, target_entropy=0.0)
assert loss.target_entropy == 0.0

Passing the same loss to SACTrainer with a collector that implements the ordinary trainer collector methods but has no getattr_env currently fails during construction:

AttributeError: 'Collector' object has no attribute 'getattr_env'

CQLTrainer fails at the equivalent lookup. The same failure occurs with target_entropy="auto" when the actor already has a valid action spec.

Expected behavior

Both trainers should use an already-resolved numeric target or the actor's action spec without querying the collector environment. The existing collector lookup should remain available when SAC uses automatic target entropy and neither the loss nor actor has an action spec.

Proposed scope

  • Restrict the collector action-spec lookup to unresolved automatic target entropy.
  • Cover numeric and actor-spec resolution for both trainers through their public constructors.
  • Preserve and test SAC's collector fallback.

Checklist

  • I checked open and closed issues and pull requests for overlapping SAC/CQL trainer, target entropy, action-spec, and getattr_env changes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.