Performance test - Add/Cat on a single IPFS instance

Version/Platform/Processor information (from `ipfs version --all`):
go-ipfs version: 0.4.3-rc3-685cd28
Repo version: 4
System version: amd64/linux
Golang version: go1.6.3

Type (bug, feature, meta, test failure, question): test failure
Area (api, commands, daemon, fuse, etc): performance
Priority (from P0: functioning, to P4: operations on fire): P3

Description:

I am running a small performance test on a single IPFS node without peers.
Steps:
1. Add N unique strings, B bytes each, save produced hashes
2. Cat each hash from step 1.
3. Repeat steps 1,2 R times

The test is implemented in golang, using https://github.com/ipfs/go-ipfs-api.

Add/Cat performance is measured by starting timing just before sh.Add/sh.Cat
and stopping just after adding/reading from the stream.

The test is single-threaded, all ADD/CAT operations are sequential.

IPFS configuration is here:
[config.txt](https://github.com/ipfs/go-ipfs/files/439915/config.txt)

Test results are further below.

We see three issues:
1. IPFS "freezes" - At some point, ADD just crawls, and IPFS has to be restarted.
2. Enormous disk space usage. The test stored about 18 thousand items, 1K each,
and the disk usage went up by 5.7GB. I assume this is not expected?
3. CAT performance drops significantly as we ADD more items. As you can see,
when the storage is empty, ADD takes about 13ms, and CAT takes about 20ms
(average for the first 1000 items). As we add data, CAT performance
drops quickly. After adding about ten thousand 1K items it goes from 20ms to 200ms
on average, and after another 5-6 thousand it goes over 300ms, up to 836ms just
before I stopped the test.

I am using the latest release candidate, but we observed the performance
of CAT going down (as we add more data) with 0.4.2 too.

I have read claims that IPFS can be optimized for handling small data chunks at low
latency (https://www.reddit.com/r/ethereum/comments/3hbqbv/ipfs_vs_swarm/),
but I don't know if the person who put it out there is associated with IPFS.
So, is this possible?

Thanks

---------- results (start) -------------------
[root@localhost ipfs-test]# df .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 19040256 5011776 14028480 27% /

Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 13.667732ms Max: 47.800133ms Min: 11.235873ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 20.596004ms Max: 47.210248ms Min: 18.017563ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 15.117807ms Max: 37.270977ms Min: 12.291369ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 35.278503ms Max: 69.662723ms Min: 30.177991ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 17.508281ms Max: 44.217647ms Min: 14.533264ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 49.930275ms Max: 84.347783ms Min: 42.924832ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 19.69559ms Max: 44.971566ms Min: 16.335943ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 65.615005ms Max: 97.144663ms Min: 55.737121ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 22.436205ms Max: 51.080924ms Min: 18.401863ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 81.402915ms Max: 139.940411ms Min: 68.97214ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 24.644547ms Max: 75.510189ms Min: 20.558274ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 95.622227ms Max: 170.023458ms Min: 81.537579ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 27.226879ms Max: 86.281592ms Min: 22.393129ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 112.275953ms Max: 232.695093ms Min: 93.186463ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 29.171508ms Max: 97.310115ms Min: 24.212177ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 128.163756ms Max: 248.911895ms Min: 105.169842ms
Adding to w1 1000 X 1024 bytes: 19 %

*****\* IPFS "freezed" at this point, and I had to restart it. It didn't really freeze, just
the performance of ADD slowed down to the point of crawling - it took few minutes to
add 1% (10 items).

After IPFS restart:

Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 14.250202ms Max: 50.711397ms Min: 11.737141ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 187.384534ms Max: 295.008167ms Min: 172.695922ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 16.326336ms Max: 78.536979ms Min: 13.28502ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 208.796685ms Max: 332.815333ms Min: 188.348064ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 18.595265ms Max: 59.661409ms Min: 15.453035ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 220.769552ms Max: 302.246728ms Min: 203.131583ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 20.731586ms Max: 91.863613ms Min: 17.253222ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 240.298857ms Max: 383.348867ms Min: 217.021817ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 23.018644ms Max: 55.909878ms Min: 18.936139ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 260.894209ms Max: 522.349637ms Min: 234.09257ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 25.623337ms Max: 99.908365ms Min: 20.95587ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 275.623074ms Max: 390.414318ms Min: 250.79436ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 28.111817ms Max: 93.7706ms Min: 22.917624ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 296.518033ms Max: 447.364316ms Min: 267.358562ms
Adding to w1 1000 X 1024 bytes: 94 %

*****\* IPFS "freezed" again, same as the first time, it took several minutes to go
from 94% to 95%.

After IPFS restart:

Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 15.353887ms Max: 126.766604ms Min: 11.851121ms
Getting from w1 1000 items: 100 %
Agent: w1 Operation Get 1000 X 1024 bytes. Average: 836.060083ms Max: 6.196891317s Min: 788.782009ms
Adding to w1 1000 X 1024 bytes: 100 %
Agent: w1 Operation Add 1000 X 1024 bytes. Average: 17.793028ms Max: 135.421942ms Min: 13.608686ms
Getting from w1 1000 items: 7 %

*****\* Stopped the test.

[root@localhost ipfs-test]# df .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 19040256 10826124 8214132 57% /

[root@localhost 1]# ll
total 48
-rw-r--r--. 1 root root 23 Aug 26 13:56 api
drwxr-xr-x. 514 root root 12288 Aug 26 12:59 blocks
-rw-rw----. 1 root root 2913 Aug 26 12:58 config
drwxr-xr-x. 2 root root 4096 Aug 26 13:56 datastore
-rw-r--r--. 1 root root 0 Aug 26 13:56 repo.lock
-rw-r--r--. 1 root root 2 Aug 26 12:58 version
[root@localhost 1]# du . --summarize
5746360 .
## ---------- results (end) ---------------------

Start by reviewing the linked config.txt and reproduce the single-node sequential Add/Cat workload described, using the Go test and go-ipfs-api setup. Compare latency and disk usage as items accumulate and capture where progress degrades; done means the freeze, storage growth, and Cat slowdown have an identified cause and a verified resolution or clearly documented limitation.

Written by the indexing model from the issue text.