When using class based tests, setUp is not called for each hypothesis test

I was just writing some test for a custom data structure and had a class based test with a setUp function that initialized a fresh instance for each test (so I don't have to copy the code in each test). Some test would fail randomly most but not all executions. After investigating this I found that setUp was simply not called for every hypothesis test which resulted in a "dirty" data structure and in term made some tests failing if the values came in the "wrong" order.

Here is an example:

class TestHypothesis(unittest.TestCase):

    def setUp(self):
        super(TestHypothesis, self).setUp()
        self.test_set = set()
        print "setUp called"

    @given(unicode)
    def test_example(self, text):
        chars = [c for c in text]
        for c in chars:
            assert c not in self.test_set
        self.test_set.update(chars)
        print "test called with", text

If I run this, I get the following output:

setUp called
test called with 
test called with \U0001bf50
test called with \U0004ac1e
test called with \U000d5c8f\U0002fb61\U00051be8\U000d5c8f\U0002fb61\U0002fb61\U00051be8\U000d5c8f\U00095c18\U0010a11f\U000d5c8f\U00051be8\U00095c18\U0002fb61\U000361af\U000d5c8f\U00019548\U000361af\U000d5c8f\U0010a11f\U000361af\U000d5c8f\U0002fb61\U000361af\U0010a11f\U0010a11f\U00095c18\U000361af\U000361af\U0010a11f\U0002fb61\U0010a11f\U000361af\U00095c18\U00019548\U000d5c8f\U000d5c8f\U00019548\U0002fb61\U0010a11f\U000361af\U00019548\U0010a11f\U00095c18\U000361af
test called with 0

[lots of other lines]

test called with \U00095c0f
Falsifying example: test_example(self=TestHypothesis(methodName='test_example'), text='\U00095c18')

The reason for this is obivous: Since setUp was only called once the data structure got "dirty".

I'm not really sure if this is something that needs to be fixed but I think it needs to be documented that you should not use setUp in this way (which in my opinion is perfectly fine) when using hypothesis.

Start by reproducing the class-based TestHypothesis example with setUp and the test_example method decorated with @given(unicode). Trace how Hypothesis invokes the unittest test method across generated examples. Done means either setUp runs for each generated test as expected or the supported behavior and required usage are documented.