facebookresearch / facebookresearch/ReAgent
edp.logged_propensities always have a value 1.
- Dominant language
- Python
- Stars
- 3.7k
- Forks
- 529
- PR merge metrics
- No merged PRs in 30d
Description
I have a question about cpe calculation.
The value of propensity in log data needed to calculate ips disappears at preprocessing phase.
In more detail, a value of `action_probability` is needed in calculation of ips. - (1)
But a field `action_probability` of raw data changed into a field `propensity` at preprocessing phase. - (2)
So, all value of `action_probability` have a value 1 because a field `action_probability` is not in training_data. - (3)
I want to know whether this is intentional or not.
Thank you.
------------------------------
(1) A field `logged_propensitities` is used in ips calculation.
Horizon/ml/rl/evaluation/doubly_robust_estimator.py
```
importance_weight = (
target_propensity_for_action / edp.logged_propensities
).float()
```
(2) A field `action_probability` changed into a field name `propensity` at preprocessing level.
Horizon/preprocessing/src/main/scala/com/facebook/spark/rl/Query.scala
```
var query = """
SELECT
mdp_id,
sequence_number,
action_probability as propensity,
state_features,
CASE action
"""
```
(3) Since, a field `action_probability` is not in training_data, propensities always have a value 1.
Horizon/ml/rl/workflow/preprocess_handler.py
```
if "action_probability" in batch:
propensities = torch.tensor(
batch["action_probability"], dtype=torch.float32
).reshape(-1, 1)
else:
propensities = torch.ones(rewards.shape, dtype=torch.float32)
```
Contributor guide
Assessment
This issue has not been assessed yet.