Reduce binary size

The current binary size for client (17-18 mb) and server (25 mb) isn't a huge problem. However, I do think we should explore ways to optimize the binary size. One clear way would be to go over the dependencies, remove any unused deps and also disable any flags that are not currently in use. For instance, if a crate has a `default` flag that installs extras deps automatically, but we only use a few of the deps, we should try and remove the `default` flag an only enable the deps, which we use.