Hang when application does not return from MessageArrived promptly

migrated from Bugzilla [#476036](https://bugs.eclipse.org/bugs/show_bug.cgi?id=476036)
status NEW severity _normal_ in component _MQTT-Java_ for _1.2_
Reported in version _1.1_ on platform _PC_
Assigned to: Ian Craggs

On 2015-08-27 08:21:16 -0400, Ian Craggs wrote:

> Reported by: Jason Blakely. blakely@us.ibm.com
>
> So I did more looking and some debugging and tracing since - even you acks solution seems to hang.
>
> So using the ver 1.0.0 paho code (since hangs still happen) I did some tcpdump tracing using wireshark and paho tracing and some print statements in paho code itself. I still think the hang is because of the ACK/RST TCP received by Paho and the handling of the error.
>
> This is what I see when the hang occurs:
> -- CommsSender.java handleRunException method calls ClientComms.java shutdownConnection method
> -- ClientComms.java shutdownConnection method calls CommsReceiver stop method
> -- CommsReceiver stop method does a join on the receive thread and hangs
>
> I am unsure who is suppose to stop the receiving thread so the join does NOT hang.
> I did try putting an argument in the join (see code below) to only wait a few seconds and not hang - and it seems to work ok (as far as i can tell ) and continue receiving messages. Although the connectionLost metthod in my MqttCallback message handler class is never called. Also lots of duplicate messages are received as I have mentioned in my previous note.
>
> if (!Thread.currentThread().equals(recThread)) {
> try {
> // Wait for the thread to finish.
> recThread.join(3000);
> }
> catch (InterruptedException ex) {
> }
> }
>
> here is the wireshark screen capture of the traffic to server on a linux client.
>
> this is the corresponding paho trace : [attachment "paho0.log.0.linux.wireshark" deleted by Ian Craggs/UK/IBM]
>
> here is the paho trace on windows: [attachment "paho0.log.0.windows1" deleted by Ian Craggs/UK/IBM]
> ---
>
> Now on the duplicate messages received when a connections closes: seems like most of the messages should have been acked already ?
> here is my client log of messages received (this is with my change with the receive thread join) - [attachment "switchbox.log.windows2" deleted by Ian Craggs/UK/IBM]
>
> below is a sample line from this file -- first field is my seq num of each msg handled, ignore t=0, Evt=nnnn where nnnn the seq num of the publish order i put this in the message during publish to help with dups. Notice after each tcp conn reset the pub order seq dups
>
> 298 t=0 Evt=1254_files.file.comment.updated id=5e1c30fe-5760-4574-9f8e-47ccc6588293 actor=_PublishTestActor orgID=20001050 ts[2015-08-11 06:29:12.310] now[2015-08-12 09:55:57.506]
>
> Note: a little while back - i put in a hack in my client to restart MqttClient if the hang happens. I supplied my own logger to Paho
> LoggerFactory.setLogger("com.ibm.switchbox.test.SwitchboxTest$SwitchboxLogger");
> which looked for the particular Timed out as no activity severe log entry and restart MqttClient -- when i did this only two messages were dups !!!

On 2015-08-27 08:52:51 -0400, Ian Craggs wrote:

> Notes:
>
> 1) It was the design intention of the MessageArrived callback that it be returned from promptly. It was never intended to work correctly if the application did spend a significant amount of time in it.
>
> 2) I made an enhancement to the API to notionally allow MessageArrived processing to take a long time in this bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=472172. This actually means that MessageArrived must be returned from quickly but the acks are delayed until another method is called.
>
> Jason, you say this: "even you acks solution seems to hang." Yes this is true if you still spend a relatively "long" time in MessageArrived. Did you change your application to not spend any time in MessageArrived? Because if you didn't, the fix to bug 472172 won't fix your problem. The key question: is the approach outlined in bug 472172 a solution that would be satisfactory for you, if it did work? I mean from the external API point of view. Are you happy to not spend a long time in MessageArrived, and allow the ack to be sent by calling a separate method.
>
> Is the real core of your request to be able to spend as long as you want in the MessageArrived callback?
>
> I really need to focus on the external behavior rather than the internal call sequence of the client library, because a fix for one use case might not be appropriate for all circumstances. I want to understand the problem from the API/application point of view so we make sure we fix all cases that might occur, not just one.

On 2015-08-27 10:34:38 -0400, jason blakely wrote:

> the hang seems to have nothing to do with the amount of time spent in MessageArrived. The problem is the handling of the connection reset. Paho handling of the reset eventually tries to do a join of a thread that does not end!

On 2015-09-08 20:38:57 -0400, jason blakely wrote:

> Have not seen any update on this. A fix is needed for the hang. No product should just hang and not have any way to let the client know there is a hang. There must be a logic error in the way Paho handles the connection reset to cause this hang. Please investigate more to discover the error.