Make sure all trait listeners get unhoooked

ref: #870

There are some cases where we set up trait listeners but never ultimately unhook them (i.e. we never call `.observe(_____, remove=True)`). In many of these cases the use of the code is transient (e.g. examples / tests), but in any case it is best to always be safe. Any code which isn't transient this is very important as it could lead to strange bugs.