Summary
Troubleshooting Internet applications using network sniffers revealed the shortcomings of the underlying MAC protocol spec. This led to a patent application.
Context
Working for the German automotive industry my job was to verify the network security of internet applications like POP3 and SMTP in cars. The TCP/IP protocol stack was available as asynchronous channel within the cyclic MOST protocol. Media access protocol was MAMAC48 (think of 802.11 or WiFi). On the back end mobile wireless communication enabled internet access.
Preparing The Network Stack For a Test Scenario
While soft and hardware was brand new I realised that I needed to test functionality first. There were many edges to hone. We were a few developers around the globe communicating over IM/video links.
On my business public web servers I set up a mock POP3/SMTP system so that I could emulate real data transfer scenarios. Right from the beginning we had mysterious data corruption and interrupted data transfer.
On the embedded system there was not much functionality available but I could dump out network conversation data. I also could request debugging statements in the network drivers from the hardware supplier. This was all helpful to debug, improve, and stabilise the mobile subscriber line protocols but the big data transfer problem remained.
In order to reproduce these problems and to make sure that they were not caused by our test system I emulated the internet and subscriber line environment locally. I forced access to the local TCP/IP network (on the MOST bus) which ran the connection between internet access system and email application. This was achieved by separating the two instances, access system and application by spoofing the media access address (think of ARP spoofing or layer 2 routing).
Then, triggering a fake subscriber line login and keeping the line open with ICMP packets, the spoofed system could take over. We played with POP3 and SMTP between the systems in order to test large data transfers over TCP with a local workstation.
In later versions externally connected mobile phones were used for opening the subscriber lines and accessing the internet. This was even more convenient to emulate with both, wired and Bluetooth. Some PERL scripts did the job.
Debugging The Network Stack
Once I had the workstation running data transfers for testing, the process was straight forward. I could produce long data transfers in either direction as large as needed. I used sniffers, mostly Wireshark (or Ethereal as it was called then) in order to analyse transmission data.
In particular the TCP analysing flow and timing tools came in handy. It showed that we had some random packet loss at some point in all large data transfers.
At this point I needed to synchronise the TCP/IP data packets with the underlying MAC protocol (asynchronous MAMAC48 was layered cyclic MOST). I could find the spot where things got weird. Still I was not clear where the MAMAC48 system to deviated from the specification. With the help of the hardware supplier we could debug the firmware driver. It turned out that the hardware was sometimes longer blocked after receiving a (48 bytes) data slice than the cyclic hardware bus “waited” to provide the next input for the driver.
But the real problem was that this situation was not covered in the protocol specification and some slices in the MAMAC48 layer slipped through unacknowledged. So the TCP protocol lost packets systematically which could not be fixed by the TCP protocol. TCP is good only with sporadic errors but in this case connections stalled and finally broke mainly due to exponentially increasing waiting times (which is TCP inherent).
Stabilising The Protocol
Being able to reproduce micro and macro behaviour I could state the boundary conditions needed in order to become the whole system stable. This basically meant (in this particular case) that I had to make sure that the TCP-ACK packets of one data stream did not destroy the data packets of another one.
Additionally I defined a CSMA/CA inspired protocol fix for the MAMAC48 protocol which lead to a patent application Local transmission system for a vehicle.