Add support of removing a CRD from the dynamic informer factory

Currently, the client-go dynamic informer factory does not support removing an informer. It does support a stop on the informer but we only used one for all the informers. So we could at least
1. separate out the informer factory stop channel into each informer so we can at least stop them.
- We need to make sure that the infomer can be joined again.
2. Add the remove function in the dynamic informer
3. Propose a change on the go-client project and try to upstream this change back.