kernc / kernc/backtesting.py

Backtest/Plot are not thread safe ?

Open
#125 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug Hacktoberfest help wanted
Dominant language
Python
Stars
9k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

Expected Behavior

Run backtests and plots with multiple threads without concurrency problems.

Actual Behavior

Multithreaded execution results in data inconsistency. When plotting. (and backtesting maybe?)

Steps to Reproduce
  1. Run this code.
import os
import threading
from uuid import uuid4

from backtesting import Backtest, Strategy
from backtesting.lib import crossover
from backtesting.test import GOOG, SMA
from flask import Flask
from waitress import serve

app = Flask(__name__)


class SmaCross(Strategy):
    n1 = 15
    n2 = 30

    def init(self):
        setattr(self, "sma1", self.I(SMA, self.data.Close, self.n1))
        setattr(self, "sma2", self.I(SMA, self.data.Close, self.n2))

    def next(self):
        if crossover(getattr(self, "sma1"), getattr(self, "sma2")):
            self.buy()
        elif crossover(getattr(self, "sma2"), getattr(self, "sma1")):
            self.sell()


class Backtesting:
    def compute(self):
        bt = Backtest(GOOG, SmaCross, cash=10000, commission=.002, exclusive_orders=True)
        bt.run()
        file_uuid = str(uuid4())
        filename = "/tmp/backtest_plot_" + file_uuid + ".html"
        print("thread:", threading.get_ident(), "start creating file:", filename)
        bt.plot(open_browser=False, filename=filename)
        try:
            f = open(filename)
            some_raw_data = f.read()
            f.close()
            os.remove(filename)
        except FileNotFoundError:
            some_raw_data = ""
            print("thread:", threading.get_ident(), "file:", filename, "not found!")
        print("thread:", threading.get_ident(), "end creating file:", filename)
        return ""


@app.route('/', methods=['GET'])
def index():
    return Backtesting().compute()


if __name__ == '__main__':
    serve(app, host='0.0.0.0', port=9090, threads=10)
  1. Open two terminals and run on both (or use jmeter or something like that):
    while true ; do curl localhost:9090 ; done

  2. Check the logs

thread: 140514310731520 start creating file: /tmp/backtest_plot_bd84f04f-6c98-4180-9d4a-e353d917b4c3.html
thread: 140514302338816 start creating file: /tmp/backtest_plot_46abb441-d70a-472b-b67e-de446dd4d8c1.html
thread: 140514310731520 file: /tmp/backtest_plot_bd84f04f-6c98-4180-9d4a-e353d917b4c3.html not found!
thread: 140514310731520 end creating file: /tmp/backtest_plot_bd84f04f-6c98-4180-9d4a-e353d917b4c3.html
thread: 140514302338816 end creating file: /tmp/backtest_plot_46abb441-d70a-472b-b67e-de446dd4d8c1.html

simplifying...

thread: 1 start creating file:  A
thread: 2 start creating file:  B
thread: 1 file:                 A not found!
thread: 1 end creating file:    A
thread: 2 end creating file:    B

The "thread 1" starts creating the "A" file, then the "thread 2" starts creating the "B" file.
As we can see, there is a inconsistency due the threads usage (sorry for the flask example).

I suspect, the problem is the "SmaCross" and "Strategy" classes.

@kernc would you have suggestions, how we can fix this ?

Additional info
  • Backtesting version: 0.2.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the supplied Flask and Waitress example, then trace the Backtest.run() and Backtest.plot() entry points while running concurrent requests. Check the Strategy and plotting state involved in the missing output file; done means concurrent backtests and plots complete without data inconsistency or missing files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.