[mqtt - enhancement] setting up sock read() chunk size

Im encountering an issue, while transmitting large data (eg. files, ~50kb) over MQTT. The first transmission works flawlessly, but then the ESP has problems to allocate enough heap for further large transmissions in receiving. Blog posts depicts that the problem could be, that the heap is fragmentet after some allocations.

My solution here was to setting up a max chunk size for the read() command, e.g. 1024 byte (multiple of four, to retain the possibility to decode base64 encoded data).

max_size = 1024
left_size = sz
while left_size > 0:
    r = min(max_size, left_size)
    msg = self.sock.read(r)
    left_size -= r
    self.cb(topic, msg, (r<max_size))

The additional bool argument for the callback indicates the completeness of the transmission.

Peter Hinch already mentioned to me (in general) a better solution: send multiple short messages or use an device with SPIRAM. Im not entirely sure if this would fit in my situation, where i publish files via MQTT to update these in the filesystem. So at least I wanted to share my results here once again.

Inadvertent i've initially opened this issue under the mqtt repo of @peterhinch (whoops, that was probably a case of mental derangement, im sorry for that). Here was the intended place.

Start by locating the MQTT implementation and the socket read path used for incoming messages. Review how message chunks are delivered to the callback and how large payloads are handled. Done means agreeing on and implementing a bounded read size without breaking callback semantics or complete MQTT payload delivery.