alpacahq / alpacahq/alpaca-py

Client should *not* un-paginate large results. Should return a `generator` that does this for you.

Đang mở
#433 1 bình luận 2 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
1.5k
Fork
398
Merge trung bình
1 ngày 16 giờ
Pull request đã merge (30 ngày)
6

Mô tả

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Is your feature request related to a problem? Please describe.

When fetching historical data, even simple queries (fetching all trades / quotes for a single symbol on a single day) can have very large result sets which are paginated by the API. The data client attempts to un-paginate these and load them all into a single return structure. This is *very* slow (probably the main cause for #204). It also means that the client consumer has no choice but to allow this process to run (single-threaded) until it completes, or potentially fails with an OOM or similar.

### Describe the solution you'd like.

The client should return a data structure that gives easy access to the paginated results, without actually loading them. The consumer can then decide how to access these results - possibly by looping through them in a single-threaded manner, but potentially also by parallelizing this data-loading to make it more efficient. It would also give the consumer the option of serializing each page of results and so avoid the OOM issue of building very large data structures in memory.

A Python generator seems a natural way to provide this functionality. The client can return an object that contains a generator which will (when accessed) fetch the appropriate pages of data from the API. This might look something like:

```
client = StockHistoricalDataClient(...)

trades_request = StockTradesRequest(symbol_or_symbols='NVDA')
trades_resultset = client.get_stock_trades_resultset(trades_request)

for(page_data in trades_resultset):
# do something with the data (summarize it, serialize it, etc.)
```
*note* here I'm assuming that `trades_resultset` is a generator

### Describe an alternate solution.

Another way to address this is to provide a client method for fetching an individual page of results, something like:

```
client = StockHistoricalDataClient(...)

trades_request = StockTradesRequest(symbol_or_symbols='NVDA')
trades_resultset = client.get_stock_trades_resultset(trades_request)

for(page in trades_resultset.pages):
page_data = client.get_stock_trades_data(trades_request, page)
# do something with the data (summarize it, serialize it, etc.)
```
*note* in this example I'm assuming that `trades_resultset` is an object that contains a reference to an iterator over page symbols.

### Anything else? (Additional Context)

Give the user the option of how to handle fetching large data. Don't force them to wait on a single-threaded and potentially failure-bound process.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.