Review unsafe scopes

In general the use of unsafe should be limited to blocks where unsafe code is used , with the exception that public functions that dereference raw pointer arguments should be marked as unsafe see [here](https://rust-lang.github.io/rust-clippy/master/index.html#/not_unsafe_ptr_arg_deref).

We need to review the code base to use the smallest possible scopes.

This can be split into multiple smaller issues to address specific parts of the codebase.