AI4Finance-Foundation / AI4Finance-Foundation/FinRL

[Bug] processor_ccxt.py: Local timezone conversion causes index drift in CCXTEngineer.data_fetch

Đang mở
#1,440 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Jupyter Notebook
Star
16.3k
Fork
3.5k
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

### Summary of the Defect

In `finrl/meta/data_processors/processor_ccxt.py`, `CCXTEngineer.data_fetch` converts exchange millisecond timestamps into DataFrame datetime indices using naive `datetime.fromtimestamp()`:

```python
# finrl/meta/data_processors/processor_ccxt.py lines 49-51
df["time"] = [
datetime.fromtimestamp(float(time) / 1000) for time in df["time"]
]
```

Because `datetime.fromtimestamp()` without an explicit timezone parameter defaults to the local OS timezone, executing this data processor on machines configured in non-UTC regions (e.g. UTC+5:30, UTC-5, UTC+8) produces local naive datetimes.

### Inconsistency Breakdown

1. **Exchange Contract Violation:** The CCXT exchange protocol returns standard Unix epoch millisecond timestamps in UTC.
2. **Query Window vs Index Mismatch:**
- Lines 66–67 use `calendar.timegm(start_dt.utctimetuple())` which treats the naive string input as UTC.
- Lines 70 and 75 generate window slices using `datetime.utcfromtimestamp()` (which is also deprecated in Python 3.12+).
- But lines 49–51 parse the fetched rows using local time.
3. **Downstream Quant / RL Impact:**
- **Multi-Pair Misalignment:** When aligning crypto pairs or combining CCXT data with other data sources (`YahooFinanceProcessor`, `AlpacaProcessor`), timestamps are offset by the local machine's UTC offset, producing corrupted cross-asset matrices.
- **Feature Drift:** Technical indicator calculations (RSI, MACD) risk time-shift errors when joined against UTC-indexed macro feeds.

### Minimal Standalone Reproduction (Zero Network Dependency)

```python
import os
import time
from datetime import datetime, timezone

# 1704067200000 ms is exactly 2024-01-01 00:00:00 UTC
exchange_ms = 1704067200000

# Current FinRL implementation:
current_parsed = datetime.fromtimestamp(float(exchange_ms) / 1000)

# Expected UTC implementation:
expected_utc = datetime.fromtimestamp(float(exchange_ms) / 1000, tz=timezone.utc)

print(f"Raw Exchange Timestamp: {exchange_ms}")
print(f"Current processor output (Local): {current_parsed}")
print(f"Expected canonical output (UTC) : {expected_utc}")

# On any non-UTC system, this assertion fails:
# assert current_parsed.hour == 0
```

### Proposed Fix

Replace naive conversions and deprecated `utcfromtimestamp` calls with `datetime.fromtimestamp(..., tz=timezone.utc)`:

```diff
--- a/finrl/meta/data_processors/processor_ccxt.py
+++ b/finrl/meta/data_processors/processor_ccxt.py
@@ -1,7 +1,7 @@
from __future__ import annotations

import calendar
-from datetime import datetime
+from datetime import datetime, timezone

import ccxt
import numpy as np
@@ -48,7 +48,7 @@ class CCXTEngineer:
df = pd.DataFrame(
ohlcv, columns=["time", "open", "high", "low", "close", "volume"]
)
- df["time"] = [
- datetime.fromtimestamp(float(time) / 1000) for time in df["time"]
+ df["time"] = [
+ datetime.fromtimestamp(float(time) / 1000, tz=timezone.utc) for time in df["time"]
]
df["open"] = df["open"].astype(np.float64)
@@ -69,12 +69,12 @@ class CCXTEngineer:
if period == "1m":
date_list = [
- datetime.utcfromtimestamp(float(time))
+ datetime.fromtimestamp(float(time), tz=timezone.utc)
for time in range(start_timestamp, end_timestamp, 60 * 720)
]
else:
date_list = [
- datetime.utcfromtimestamp(float(time))
+ datetime.fromtimestamp(float(time), tz=timezone.utc)
for time in range(start_timestamp, end_timestamp, 60 * 1440)
]
```

### Environment
- **OS:** Windows / Linux / macOS
- **Python:** 3.10 / 3.11 / 3.12+
- **FinRL:** `master`

Happy to open a PR with unit tests covering this fix if the maintainers agree with this direction.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.