As you saw in my previous post I was able to get the same mqttc example from NuttX running on Linux.

Unfortunately the NuttX application is not working, then I decided to run the WireShark to compare what the Linux application was doing different from the NuttX application.

It should be easier to sniffer the network if it was Ethernet instead WiFi, so I used the ESP32-Ethernet-Kit board to test with NuttX and a RaspberryPI 3 to test with Linux (I tried to use other computer with Linux, but there are so many packages from Linux services that it was also impossible to compare).

Initially I connected both boards to my WiFi router RJ45 ports, but the router seems to isolate each port, so it is working like an Ethernet Switch instead of working like an Ethernet HUB.

Then I decided to use my laptop to share Internet with those board (one by time). I remember that I did it almost 15 years ago and documented it here (sorry in Portuguese).

NuttX:
3	0.050060828	192.168.1.100	75.2.83.130	TCP	60	4097 → 1883 [SYN] Seq=0 Win=5488 Len=0 MSS=536

Linux:
7	1.359660410	192.168.1.100	99.83.172.119	TCP	74	40264 → 1883 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2409185516 TSecr=0 WS=128

Maybe these differences could explain why NuttX was not working, but unfortunately increasing the MSS to 1460 didn’t help, I still receiving a Ack=0 from the server:

6 34.353497310 192.168.1.100 75.2.83.130 TCP 60 4097 → 1883 [SYN] Seq=0 Win=5488 Len=0 MSS=1446
7 34.496251127 75.2.83.130 192.168.1.100 TCP 58 1883 → 4097 [SYN, ACK] Seq=0 Ack=0 Win=65535 Len=0 MSS=0

So, let to test other ideas, I also noticed that the Linux application was using a big Window (Win=64240), the “Selective Acknowledgements” (SACK_PERM=1), time stamp (TSval=2409185516 TSecr=0) and Window Size Scaling (WS=128)

Increasing the Window to 65KB doesn’t sound like a good idea, let’s to modify the other Linux TCP stack parameters to become similar to NuttX:

$ sudo sysctl -w net.ipv4.tcp_sack=0
$ sudo sysctl -w net.ipv4.tcp_timestamps=0
$ sudo sysctl -w net.ipv4.tcp_window_scaling=0

These modifications also didn’t solve the issue, then I also tried to reduce the Win to 4KB:

# echo "4096 4096 4096" > /proc/sys/net/ipv4/tcp_rmem
# echo "4096 4096 4096" > /proc/sys/net/ipv4/tcp_wmem

It also didn’t make the issue show up on Linux, so there is some other detail, see complete Linux log:

1	0.000000000	192.168.1.100	8.8.8.8	DNS	72	Standard query 0x6055 A mqtt.tago.io
2	0.041538839	8.8.8.8	192.168.1.100	DNS	104	Standard query response 0x6055 A mqtt.tago.io A 99.83.172.119 A 75.2.83.130
3	0.043238286	192.168.1.100	99.83.172.119	TCP	60	59880 → 1883 [SYN] Seq=0 Win=1460 Len=0 MSS=1460
4	0.190940161	99.83.172.119	192.168.1.100	TCP	58	1883 → 59880 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1416
5	0.191451199	192.168.1.100	99.83.172.119	TCP	60	59880 → 1883 [ACK] Seq=1 Ack=1 Win=1460 Len=0
6	0.192109946	192.168.1.100	99.83.172.119	MQTT	117	Connect Command
7	0.371321172	99.83.172.119	192.168.1.100	TCP	54	1883 → 59880 [ACK] Seq=1 Ack=64 Win=65535 Len=0
8	0.371805531	192.168.1.100	99.83.172.119	MQTT	111	Publish Message [tago/data/post]
9	0.467783276	99.83.172.119	192.168.1.100	MQTT	58	Connect Ack
10	0.468174062	192.168.1.100	99.83.172.119	TCP	60	59880 → 1883 [ACK] Seq=121 Ack=5 Win=1456 Len=0
11	0.515294572	99.83.172.119	192.168.1.100	TCP	54	1883 → 59880 [ACK] Seq=5 Ack=121 Win=65535 Len=0
12	5.098640506	20:7b:d2:0d:a8:d0	Raspberr_65:2a:1a	ARP	42	Who has 192.168.1.100? Tell 192.168.1.1
13	5.099071927	Raspberr_65:2a:1a	20:7b:d2:0d:a8:d0	ARP	60	192.168.1.100 is at b8:27:eb:65:2a:1a
14	5.193192947	192.168.1.100	99.83.172.119	MQTT	60	Disconnect Req
15	5.193699389	192.168.1.100	99.83.172.119	TCP	60	59880 → 1883 [FIN, ACK] Seq=123 Ack=5 Win=1456 Len=0
16	5.338709513	99.83.172.119	192.168.1.100	TCP	54	1883 → 59880 [ACK] Seq=5 Ack=123 Win=65535 Len=0
17	5.338714704	99.83.172.119	192.168.1.100	TCP	54	1883 → 59880 [FIN, ACK] Seq=5 Ack=124 Win=65535 Len=0
18	5.339267333	192.168.1.100	99.83.172.119	TCP	60	59880 → 1883 [ACK] Seq=124 Ack=6 Win=1455 Len=0