Large amount of data render the cli unusable

#### Version information:

```
go-ipfs version: 0.4.22-
Repo version: 7
System version: amd64/windows
Golang version: go1.12.7
```

#### Description:

I've added 624 GB (670.271.987.712 bytes, 1.974 Files, 12 Folders) of data using `ipfs add --recursive --pin=true --progress`, and now the client is extremely slow to unusable.
~The desktop client (which uses the javascript implementation) is unusable and cannot handle that data at all~ (apparently fixed it self, I don't know how) and the go client (the command line tool) takes ages (5h 45m 59s) to perform a `ipfs pin ls`.

But despite taking for ages everything else seam to be fine. Maybe some form of caching could help here?