Mosh fails with cell phone service throttled to 64KB/s

An IRC user has had problems with Mosh on a throttled Internet connection via a phone that's gone over quota for its account. ddrown on irc worked with him and did some hacks to determine that the service seems to drop UDP packets when they come in too fast (>900-bytes packets at >144KB/s), rather than buffering them and rate-limiting their output.

Mosh's protocol layers aren't very well prepared to handle this currently; SSH over TCP would probably handle this better. Solving this problem in Mosh might be pretty similar to solving the more general problems we have with MTU issues.

09:15 --> dadada (~pagovor@x2f7ff0c.dyn.telefonica.de) has joined #mosh
09:15 <dadada> hi
09:16 <dadada> when using mosh I often get the "Last contact" away message, and it often takes around one minute to continue normal operations
09:17 <dadada> I've a low bandwith internet connection (64kbits), but shouldn't that be enough to still keep a constant stable connection open with mosh?
09:17 <dadada> I can download videos from youtube with youtube-dl with stable 6-7 kByte/s
09:18 <dadada> I'm using the latest mosh from git now
09:18 <dadada> I'v used the mosh version from fedora 27 before, with the same issue
09:19 <@achin> mosh can be kinda chatty (senidng packets frequently, even on idle links), so it doesn't always work well on ultra-low bandwidth links.  but to be honest, i don't know exactly the minimum required bandwidth
09:20 <@achin> there is a 'lowbandwidth' branch in it that might be interesting to try (it might need to be rebased on top of latest master)

15:40 <dadada> KeithW: I'm using a slow internet connection (64kbit/s), and mosh loses contact all the time
15:41 <dadada> it's waiting for contact for minutes, although I'm not doing anything else with my connection
15:41 <dadada> and I know the server is fast
15:42 <dadada> using latest mosh from git
15:43 <ddrown> doing some quick tests, mosh should be using around 300 bits per second when idle on average
15:43 <ddrown> does your connection have any packet loss? is it wireless?
15:44 <dadada> yeah, it's 3G
15:44 <dadada> I don't know about the packet loss
15:45 <ddrown> do pings get through when mosh shows no contact?
15:45 <dadada> however I know I can get a constant download speed of 7kBytes/s
15:46 <dadada> yes, I'm connecting to my VPS for example, and I can ping the server and use other mosh or telnet or ssh sessions at the same time
15:47 <ddrown> mosh sessions to the same server? different server on the same remote network? different server, different network?
15:47 <dadada> same server and different server both
15:48 <dadada> I'm making all connections through 3G
15:48 <dadada> but it's throttled to 64kbits
15:49 <dadada> both up and down
15:51 <ddrown> something must be dropping the packets, the question is what
15:51 <ddrown> it'd be nice to get a packet capture on both sides while this is happening
15:51 <dadada> what would I have to do for it?
15:52 <dadada> btw. telnet connections are pretty stable
15:52 <dadada> albeit slow
15:52 <ddrown> I know some providers rate limit udp to very low limits
15:52 <dadada> would make sense
15:53 <ddrown> to the point that google blacklists them from using quic
15:53 <dadada> in that case, it would be nice to have an alternative mosh mode where it's using tcp
15:53 <ddrown> but that should affect all your mosh sessions equally
15:54 <ddrown> it could be a udp session timeout, but that should clear pretty quickly
15:54 <dadada> well, that's not happening equally, 2 sessions may work fine while the third is stuck for mintues
15:56 <ddrown> but to go back to packet capture: tcpdump -pni $int udp and port $port
15:56 <dadada> what's int?
15:56 <ddrown> where $int is your network interface and $port is your mosh port
15:56 <ddrown> needs to run as root
15:57 <dadada> ok, and what do I do with the data?
15:57 <dadada> the default mosh port is?
15:57 <ddrown> it starts at 60001
15:57 <ddrown> netstat -nap | grep mosh
15:57 <ddrown> will tell you which one
15:57 <dadada> ok
15:58 <ddrown> if you run a capture on both sides, you'll be able to see the server sending packets
15:58 <ddrown> and the client will probably not get them
15:58 <ddrown> and when it starts working, it's useful to see if the client's port has changed
15:59 <dadada> let's say it did, what would that mean
15:59 <ddrown> udp session timed out, possibly packetloss between you and your provider's CGNAT hardware
16:00 <dadada> seems likely
16:00 <dadada> how could that be averted?
16:00 <ddrown> sometimes it can't
16:01 <ddrown> but it'd be good to confirm what is happening
16:01 <dadada> what I mean is, say I know that my providers upd support can't be trusted
16:01 <dadada> idea: couldn't mosh open a second udp connection proactively once contact is lost
16:02 <dadada> and directly switch?
16:02 <dadada> or even before contact is lost
16:02 <ddrown> I think it does that
16:02 <dadada> and switch once it is lost
16:02 <ddrown> there was a thing about that previously
16:03 <dadada> my guess is that my provider tries to prevent p2p piracy and therefore limits udp usefulness
16:04 <dadada> ddrown: there may be a timeout regarding udp switching in the code... maybe this timeout should be like 2 seconds instead of whatever it is now
16:04 <dadada> :-)
16:06 <ddrown> have a meeting now, I can go look for the code after that
16:06 <dadada> ddrown: a more positive note, I kept mosh connections open for over a week, with my laptop shut down etc., and it session was always reconnecting, although I sometimes had to wait
16:07 <dadada> ddrown: thank, you please do so
16:08 <dadada> would be nice if timeouts like that could be set from the command line or env variable
16:37 <ddrown> dadada: ok, this is the change for port hopping - https://github.com/mobile-shell/mosh/commit/c0092a6e7ee481fc897661ff107e3fa940547574#diff-476dc4978a171cc05dc0e617b6149bdb
16:38 <ddrown> https://github.com/mobile-shell/mosh/blob/befdb96ad1f11053e2bb5f44d62d5d27f97e2693/src/network/network.cc - Connection::hop_port
16:39 <ddrown> and PORT_HOP_INTERVAL define
16:40 <dadada> MAX_OLD_SOCKET_AGE         = 60000;
16:41 <dadada> what about that?
16:41 <ddrown> that's in Connection::prune_sockets
16:41 <dadada> PORT_HOP_INTERVAL          = 10000; is lower than the timeouts I get
16:41 <dadada> assuming 10000 microsecods = 10 seconds
16:41 <ddrown> it'll have multiple ports opened, and close inactive ones older than MAX_OLD_SOCKET_AGE
16:42 <dadada> I suggest letting advanced user play around with all those variables through command line options or env variables
16:42 <ddrown> those numbers are in milliseconds, and 10000ms = 10s
16:42 <dadada> I'll then be able to tell you what helps and what doesn't
16:43 <dadada>  CONGESTION_TIMESTAMP_PENALTY = 500; ? what's that?
16:44 <ddrown> slowdown amount on congestion
16:45 <ddrown> and that's just the ECN flag
16:45 <ddrown> I don't think most networks set the ECN flag
16:50 <dadada> ddrown: what about my thoughts on allowing users to set some of those variables
16:51 <ddrown> seems reasonable
16:57 --> dadada (~pagovor@x2f7fe78.dyn.telefonica.de) has joined #mosh
16:57 -- dadada is now known as Guest72289
17:11 <ddrown> heh
17:11 <ddrown> well, nobody look too closely at this hack: https://github.com/ddrown/mosh/commit/004adf28f384f3816b6698fb510e511c7a19e31c
17:11 <Guest72289> what's there to laugh
17:11 <ddrown> I used macro programming and I hate macro programming
17:12 <ddrown> I haven't tested it, but it should allow you to change the client settings
17:12 <Guest72289> ddrown: you think this patch will be accepted
17:13 <ddrown> Guest72289: unlikely
17:13 <ddrown> but feel free to test it out
17:23 <Guest72289> ddrown: git cloned your repo, but the envset branch isn't there
17:24 <ddrown> $ git clone https://github.com/ddrown/mosh mosh2
17:25 <Guest72289> pardon me, dumb mistake
17:25 <ddrown> $ cd mosh2 ; $ git branch -a | grep envset
17:25 <ddrown>   remotes/origin/envset
17:25 <ddrown> k
17:27 <Guest72289> I'll compile
17:41 <Guest72289> ddrown: I set MOSH_SERVER_ASSOCIATION_TIMEOUT,
17:41 <Guest72289>     MOSH_PORT_HOP_INTERVAL, MOSH_MAX_OLD_SOCKET_AGE
17:41 <Guest72289> all to 5000 and it's still losing contact and not recovering
17:41 <Guest72289> "last contact 1:43 ago" now
17:41 <Guest72289> now over 2 minutes :D
17:41 <Guest72289> approaching 4 minutes
17:41 <ddrown> ok, setting the same MOSH_PORT_HOP_INTERVAL and MOSH_MAX_OLD_SOCKET_AGE is probably bad
17:42 <ddrown> can you get a tcpdump of both sides of the connection?
17:42 <Guest72289> I'll try to, it's my tcp server after all
17:42 <Guest72289> I mean VPS server
17:44 <Guest72289> ddrown: I could give you access to the server in order to see if there's a problem there
17:45 <Guest72289> can you artificially throttle the speed of your connection to simulate my situation?
17:45 <ddrown> I can, but I don't think it's a speed issue
17:46 <Guest72289> it's an openVZ based server that is running a (to put it mildly) outdated 2.6 kerne
17:46 <Guest72289> kernel
17:46 <Guest72289> maybe OpenVZ just sucks
17:48 <ddrown> my personal experience with openvz has been negative
17:48 <Guest72289> I'
17:49 <Guest72289> m testing this server since a few days, since it is so cheap
17:49 <Guest72289> I'm paying 1.4€ for a month, and there's no contract, so by not paying it ends after this month
17:49 <ddrown> that's super cheap
17:49 <Guest72289> I'm trying to see what I can so with it as long as I have it
17:49 <ddrown> which is what openvz is good at, being super cheap
17:49 <Guest72289> sure
17:50 <Guest72289> if only the kernel wasn't old as ****
17:50 <Guest72289> then I could use something newer than debian jessie :D
17:51 <ddrown> how's the filesystem latency?
17:51 <Guest72289> not sure, but seems okay
18:15 <@cgull> In my experience, when mosh continually loses and regains contact, it's because something is blocking UDP sessions after a few packets, Mosh hops to a new port, and this cycle repeats endlessly.  Guest72289's problem doesn't sound quite like that, I suspect something is blocking packets a little more effectively.
18:15 <ddrown> took it to /msg and it looks like packets over ~900 bytes are dropped
18:16 <@cgull> OK.
18:16 <@cgull> Mosh does need a neuron or two to handle MTU issues.
02:42 <Guest57868> ddrown: hi, good morning, was sleeping, /msg ed you
02:42 <Guest57868> the problem isn't solved, but I've an idea what my provider is doing now
06:39 --> dadada (~pagovor@x2f7fe78.dyn.telefonica.de) has joined #mosh
06:39 -- dadada is now known as Guest72159
07:06 <Guest72159> ddrown: hi, ping me when you read this
07:45 <Guest72159> if I could throttle the speed at which the mosh-server sends packets to my client
07:45 <Guest72159> this might solve my issue
07:45 <Guest72159> seems like my 3G ISP drops UDP packages when they're coming in fast and many at a time
18:42 <ddrown> new hack, rate limit mosh-server to max of 900 bytes every 50ms - https://github.com/ddrown/mosh/commit/990b13234b826299c41627b2e84d35c4aefb079d
18:53 <Guest64526> I can confirm that ddrown hack works
06:43 --> k_sze (~Kira@unaffiliated/kira) has joined #mosh
06:44 <k_sze> There's a bit of difference between SSH and mosh that I would like to fix.
06:51 <k_sze> When I ssh into my Ubuntu server, I get the motd, but I don't get it when I mosh into my Ubuntu server.
06:57 <@andersk> k_sze: That should have been addressed in Mosh 1.3.2: https://github.com/mobile-shell/mosh/commit/32b1e6ee4be0706e4047d6927cf73e6333c01774
06:57 <k_sze> I guess it's just Ubuntu 16.04 having an outdated mosh.
07:01 <k_sze> Looks like there's a PPA
07:03 <k_sze> yay, I get motd with the PPA
19:09 <@cgull> Guest15786: This might be worth opening a PR for, so we remember which provider is involved.  What ISP are you using there?  And is it via a phone, or some other method?
19:10 <Guest15786> via phone, it's cell phone ISP Telefonica/O2 in Germany...
19:10 <Guest15786> however I'm testing this with the throttled 64kbit/s bandwith
19:11 <Guest15786> don't know about unthrottled behaviour, but guess it is much better
19:11 <@cgull> Is it throttled because you've gone over a quota, or for some other reason?
19:11 <Guest15786> usb-tethering to my linux box
19:11 <Guest15786> quota
19:13 <@cgull> Hmm, my phone on T-Mobile USA should get throttled to I think 128KB/s, but I think I've only had it happen once.
19:14 <@cgull> That's a situation where SSH over TCP might actually be better.
19:15 <Guest15786> the plans for 3G/4G internet are terrible here, providers milk their users
19:15 <Guest15786> 64kbits is among the better rates

Start with src/network/network.cc, especially Connection::hop_port, Connection::prune_sockets, PORT_HOP_INTERVAL, and MAX_OLD_SOCKET_AGE. Review the referenced port-hopping and rate-limiting commits, then reproduce the reported packet-size and throttling behavior. Done is not defined in the issue and would require an agreed scope and acceptance criteria.