SendChannel.send() doesn't have a defined ordering

I hope I've understood the nuances of checkpoints correctly here. I had thought I spotted a race condition in my Trio-using code. I concluded that there's not currently any way to avoid this in Trio's API as defined at the moment, but Trio's current implementation means that the race won't occur. I wonder if the API could be defined to be more deterministic to match the code, to make it possible to reliably avoid my race?

In my code the order that values get sent into a memory channel are important. Trio guarantees that a call to send() includes a checkpoint. But I don't think it defines whether or not such a checkpoint will occur before values are "serialized" into the channel, thus defining their order.

What I mean by "serialized": if two tasks write to a channel "concurrently", then the final ordering may go either way depending on the order in which tasks are scheduled, but at some point that order becomes determined.

If send() were to do this serialization before any checkpoint, then the caller can, with some care, precisely control the resulting ordering by managing its own checkpoints. If send() were not to guarantee this, then I think this level of control becomes impossible. I might try to work around this by coalescing the values myself, but the obvious way to do so would be to use a memory channel, at which point I'd be back to where I started with the same problem.

Looking at the implementation, I think it does happen to be the case that if the call is not cancelled, no checkpoint will occur before serialization. However, if this changes it will break me. So could this be defined to be the case in the API, please?

For example, consider the following contrived code:

import trio

send_channel, receive_channel = trio.open_memory_channel(0)

last_value_sent = None
all_values_received = []

async def receive_values():
    for i in range(2):
        value = await receive_channel.receive()
        all_values_received.append(value)

async def send_value(value):
    global last_value_sent
    last_value_sent = value
    await send_channel.send(value)

async def test():
    async with trio.open_nursery() as nursery:
        nursery.start_soon(receive_values)
        nursery.start_soon(send_value, 1)
        nursery.start_soon(send_value, 2)

    assert last_value_sent == all_values_received[-1]

trio.run(test)

The assertion passes in practice, I think due to Trio's current implementation, but I don't think that Trio currently guarantees that this won't change. If the send_channel.send() call were to checkpoint immediately when it is called, then it's possible that the values would come out of receive_channel in the opposite order from which I sent them, depending on the order in which the two tasks resumed following their respective checkpoints. Then last_value_sent would mismatch the order in which the values come out of the channel.

In my real world use case, I am sending values into a channel, but also processing those values immediately. I would like to ensure that the ordering of the values coming out of the channel is the same as the order in which I processed them. I can do this by carefully considering checkpoints - ensuring that there is no checkpoint between processing a value and sending it into the channel. However, this depends on Trio's currently undefined behaviour of the guaranteed checkpoint never occuring before the ordering.

My request is therefore:

1. Document that values will come out of the channel in the same order that send() calls are awaited (which implies that the guaranteed checkpoint will occur only after the ordering is recorded).

2. Consider it API breakage if this changes in the future.

Thank you for reading this far to consider this!

Start with the SendChannel.send API and memory-channel behavior described in the issue, then reproduce the two-sender example under trio.run. Review the 16-comment discussion for an accepted position on ordering and checkpoints. Done means the contract is either documented and covered by a test, or the request is explicitly resolved.