Incorrectly formed object can settle in the container

Overview

In current implementation, when storage node forms object within trusted ObjectService.Put request with session, it saves prepared replicas locally (if local node matches storage policy). In this case local save is not preceded by a format check, because it is assumed that the node always forms the object correctly. Other storage candidates will most likely deny the replica due to invalid format.

Later node will detect shortage of object replicas in the container according to storage policy, and will try to replicate the object. Of course, as previously, all candidates will reject the replica.

Possible solutions

Pre-check before local saving will decrease probability of incorrect replica storage. But this slows down PUT operation and won't fix cases when replica is corrupted within local storage.

Lets now consider case when incorrect replica is settled in the node storage. In order to prevent attempts to distribute "bad" object over the network, node should self-check each replica and try heal problem ones (if there are any other holders).

Related areas

Lets consider malicious (buggy) node which holds single replica of the "bad" object. How user can detect the problem after reading the object? Of course, the only way is to verify each recevied replica on the client side. This decreases performance, but seems like there is nothing else client can do.

Start by tracing the trusted ObjectService.Put path and the replica distribution and validation behavior described in the issue. Determine how incorrectly formed or corrupted replicas can be detected and healed without redistributing bad data. Done means the chosen handling prevents invalid replicas from spreading and defines how a client can detect an invalid sole replica.