AI4Finance-Foundation / AI4Finance-Foundation/FinRL
[Bug] processor_ccxt.py: Local timezone conversion causes index drift in CCXTEngineer.data_fetch
- Ngôn ngữ chính
- Jupyter Notebook
- Star
- 16.3k
- Fork
- 3.5k
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
### Summary of the Defect
In `finrl/meta/data_processors/processor_ccxt.py`, `CCXTEngineer.data_fetch` converts exchange millisecond timestamps into DataFrame datetime indices using naive `datetime.fromtimestamp()`:
```python
# finrl/meta/data_processors/processor_ccxt.py lines 49-51
df["time"] = [
datetime.fromtimestamp(float(time) / 1000) for time in df["time"]
]
```
Because `datetime.fromtimestamp()` without an explicit timezone parameter defaults to the local OS timezone, executing this data processor on machines configured in non-UTC regions (e.g. UTC+5:30, UTC-5, UTC+8) produces local naive datetimes.
### Inconsistency Breakdown
1. **Exchange Contract Violation:** The CCXT exchange protocol returns standard Unix epoch millisecond timestamps in UTC.
2. **Query Window vs Index Mismatch:**
- Lines 66–67 use `calendar.timegm(start_dt.utctimetuple())` which treats the naive string input as UTC.
- Lines 70 and 75 generate window slices using `datetime.utcfromtimestamp()` (which is also deprecated in Python 3.12+).
- But lines 49–51 parse the fetched rows using local time.
3. **Downstream Quant / RL Impact:**
- **Multi-Pair Misalignment:** When aligning crypto pairs or combining CCXT data with other data sources (`YahooFinanceProcessor`, `AlpacaProcessor`), timestamps are offset by the local machine's UTC offset, producing corrupted cross-asset matrices.
- **Feature Drift:** Technical indicator calculations (RSI, MACD) risk time-shift errors when joined against UTC-indexed macro feeds.
### Minimal Standalone Reproduction (Zero Network Dependency)
```python
import os
import time
from datetime import datetime, timezone
# 1704067200000 ms is exactly 2024-01-01 00:00:00 UTC
exchange_ms = 1704067200000
# Current FinRL implementation:
current_parsed = datetime.fromtimestamp(float(exchange_ms) / 1000)
# Expected UTC implementation:
expected_utc = datetime.fromtimestamp(float(exchange_ms) / 1000, tz=timezone.utc)
print(f"Raw Exchange Timestamp: {exchange_ms}")
print(f"Current processor output (Local): {current_parsed}")
print(f"Expected canonical output (UTC) : {expected_utc}")
# On any non-UTC system, this assertion fails:
# assert current_parsed.hour == 0
```
### Proposed Fix
Replace naive conversions and deprecated `utcfromtimestamp` calls with `datetime.fromtimestamp(..., tz=timezone.utc)`:
```diff
--- a/finrl/meta/data_processors/processor_ccxt.py
+++ b/finrl/meta/data_processors/processor_ccxt.py
@@ -1,7 +1,7 @@
from __future__ import annotations
import calendar
-from datetime import datetime
+from datetime import datetime, timezone
import ccxt
import numpy as np
@@ -48,7 +48,7 @@ class CCXTEngineer:
df = pd.DataFrame(
ohlcv, columns=["time", "open", "high", "low", "close", "volume"]
)
- df["time"] = [
- datetime.fromtimestamp(float(time) / 1000) for time in df["time"]
+ df["time"] = [
+ datetime.fromtimestamp(float(time) / 1000, tz=timezone.utc) for time in df["time"]
]
df["open"] = df["open"].astype(np.float64)
@@ -69,12 +69,12 @@ class CCXTEngineer:
if period == "1m":
date_list = [
- datetime.utcfromtimestamp(float(time))
+ datetime.fromtimestamp(float(time), tz=timezone.utc)
for time in range(start_timestamp, end_timestamp, 60 * 720)
]
else:
date_list = [
- datetime.utcfromtimestamp(float(time))
+ datetime.fromtimestamp(float(time), tz=timezone.utc)
for time in range(start_timestamp, end_timestamp, 60 * 1440)
]
```
### Environment
- **OS:** Windows / Linux / macOS
- **Python:** 3.10 / 3.11 / 3.12+
- **FinRL:** `master`
Happy to open a PR with unit tests covering this fix if the maintainers agree with this direction.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.