[FSDP] issue with different ways of doing activation checkpointing

unit test are being added here: https://github.com/facebookresearch/fairscale/pull/476

but I don't have a big picture of what's needed to be fixed yet. Some observations:

1. we know that if root FSDP is empty with zero params and checkpoint is used, backward can finish too early
2. I have seen issues around checkpoint(FSDP()) vs. FSDP(checkpoint()). It is only the former work in vissl.
3. I have seen issue that if we checkpoint the AnyStage and let the inner blocks be FSDP wrapped in vissl, we also crash.
4. In the past, we have seen issues around for-loop in the forward pass doesn't play well with checkpointing. (i.e. the same module forwarded multiple times in the forward pass.)

cc: @prigoyal @myleott

I am going to document a list of issues we found so far for tracking purpose.