Lamination and consistency model

This week, I've been working on the new metadata implementation (I will just call margotree) which will replace our current mdhim. And, it looks like we need to establish our consistency model clearly.

A notable difference in margotree metadata from using mdhim is on how the metadata is distributed across compute nodes. With mdhim, the UnifyFS server does not have any dedicated data structures for the file metadata but simply relies on mdhim to store or retrieve them. Therefore, when the UnifyFS client library sends the metadata (upon sync()), the server simply put the records onto the kvstore (mdhim). However, this single step process becomes two step process in the margotree metadata. Specifically, in the margotree metadata, the UnifyFS server itself has a new data structure, i.e., unifyfs_inode, which keeps file metadata including stat and extent information. So, when a server daemon receives the indexing metadata records (or extent map) from clients, it first stores them on its local inode structure (step 1). Then, we have have to decide when we have to share the extent tree with other server daemons (step 2). In the current margotree implementation, both step 1 and 2 happen with sync(), which is actually triggered when an application calls fsync(). However, according to our documentation, this doesn't seem to be a correct way.

Our documentation says that application file operations should happen in two strictly-divided write and read phases. Also, an explicit lamination needs to be placed between those two phases. A typical example would be:

fd = open(file, O_WRONLY);
write(fd, data, len);
fsync(fd);
close(fd);

chmod(file, 0444);  // laminate the file, no further changes

fd = open(file, O_RDONLY);
read(file, data, len);
close(fd);

Currently, UnifyFS disregard the file descriptor (fd) with fsync() and syncs all file indexing metadata buffered in the client to the metadata storage (mdhim). The documentation says that fsync() sends file indexing information from the client library to the server daemon, but it does not say that any subsequent read operation should correctly fetch remote data. If this is not true, we would need to update the documentation accordingly. And, if the documentation is correct as it is now, fsync() feels unnecessary in the above example because the buffered indexing information is expected to be pushed anyway to the server on close(). The documentation also says that:

If the application process group fails before a file has been laminated, UnifyFS may delete the file.

This implies that fsync() also has nothing to do with persisting the file data.

So, back to the original problem of the margotree, I am wondering what is the expected way for the UnifyFS server daemon to handle extent records from clients. Assuming that we will have a proper fsync() implementation, one reasonable option would be:

• fsync(): send buffered extent records to the server daemon, which will keep in its local inode data structure (step 1)
• close(): synchronize extent information between all servers (step 2)

or , it's also possible to be like:

• fsync(): do nothing
• close(): send buffered extent records to server (step 1)
• laminate/chmod(): synchronize between servers (step 2)

I do not have any particular opinion here, but it would be more likely for application developers to miss the lamination/chmod() call than missing close(). But, there would be also cases where the explicit lamination is necessary, for instance, when an application opens a file multiple times just for writing. We may also think of having a custom open flags for lamination, e.g., O_LAMINATE_ON_CLOSE or O_NO_LAMINATE_ON_CLOSE, if this makes sense.

Anyway, I thought it was a good time for discussing this before moving forward. Please share your thoughts.

Start by reviewing the documented semantics for fsync(), close(), chmod()/lamination, and the current margotree and mdhim metadata paths. Compare the proposed ordering of client buffering, local inode updates, and cross-server extent synchronization. Done means the project has an agreed consistency model and documentation that clearly defines these operations.