SynchronousTestCase should fail when returning a Deferred from a test method

	@lvh reported
Trac ID	trac#6834
Type	enhancement
Created	2013-11-14 10:46:26Z

It seems to me (and jonathanj, and idnar, and jml and undoubtedly others) that if you're returning a deferred from a SynchronousTestCase, you're doing it wrong, and you should almost certainly be seeing a test failure (preferably one pointing you to thinks like successResultOf/failureResultOf) instead of silently passing the test.

Following IRC conversation:

10:27 < jonathanj> is there a reason that SynchronousTestCase doesn't fail if you
          return a Deferred from a test method?
10:34 < lvh> jonathanj: My guess is backwards compat? Althoguh I guess that's why
          we have SynchronousTestCase in the first place
10:35 < lvh> jonathanj: Can't find any tickets. Sounds like a bug to me (although
          maybe it's too late to change the behavior)
10:35 < jonathanj> it would be kind of convenient if returning a Deferred from a
          SynchronousTestCase method actually failed the test instead of silently
          passing it regardless of the result
10:35 < lvh> then again I'm pretty sure returning a Deferred from a
          SynchronousTestCase should always fail :)
10:39 < jml> I thought the point of a synchronoustestcase was a deferred-returning
          test case that doesn't use the reactor.
10:39 < jml> but icbw
10:41 < lvh> jml: Part of the point was to decouple the reactor at least
10:41 < lvh> but if you're not using the reactor who will fire those deferreds
          you're returning
10:41 < lvh> (and what's the point of self.successResultOf/self.failureResultOf)
10:41 < lifeless> the postactor
10:41 < jonathanj> jml: well the source code for _runTestsAndFixturesBlahBlah
          disregards the result of runWithWarningsSuppressed entirely
10:41 < jonathanj> jml: so i don't think that's it
10:41 < lvh> lifeless: the what
10:41 < jonathanj> sorry, just _run
10:42 < idnar> lvh: the point of successResultOf/failureResultOf is to avoid
          returning a Deferred from the test
10:42 < lifeless> lvh: I'm very sorry, but - YHBT, HAND, HTH.
10:42 < lvh> idnar: so does that mean returning a deferred from the test method is
          always wrong
10:42 < idnar> lvh: or having the reactor involved at all
10:42 < jonathanj> YHBT?
10:42 < jonathanj> You Have Been Terminated?
10:42 < lvh> idnar: You have been trolled
10:42 < jonathanj> oh
10:42 < idnar> lvh: uhm
10:42 < idnar> lvh: yes, I would say so
10:43 < lvh> idnar: that's what it looks like to me
10:43 < lvh> idnar: given what I thought sRO and fRO were for
10:43 < idnar> lvh: ie. you should always use SynchronousTestCase instead of TestCase,
          at least when writing a unit test
10:43 < lvh> and what the entire point was of SynchronousTestCase
10:43 < idnar> (people also use trial to write things that are not unit tests, though,
          so...)

trac-id__6834 6834
type__enhancement enhancement
reporter__lvh lvh
priority__normal normal
milestone__ 
branch__ 
branch_author__ 
status__new new
resolution__None None
component__trial trial
keywords__ 
time__1384425986000000 1384425986000000
changetime__1384691571000000 1384691571000000
version__None None
owner__ 
cc__jml cc__mithrandi

Start by locating SynchronousTestCase and its _run path in the trial component, then review the existing successResultOf/failureResultOf tests and behavior. Add a regression test for a test method returning a Deferred; done means that case fails clearly rather than silently passing and points users toward the synchronous result helpers.