Background
Mid of the 90’s I ran a company specialised in protocol gateways for SCADA systems (in particular a predecessor of MicroSCADA). These gateways were basically PCs with many AT-slots where each one was a fully embedded sub-system with a outbound serial ports. The sub-system CPUs were Z80 comand set compatible NEC micro processors. I delivered the firmware for some of these systems. Communication took place over dual-ported RAM.
A Real-Time Software Stack
The sub-systems ran on multi-threaded operating systems. Outbound, these systems understood one of the bespoke serial or cyclic industry protocols and inbound they translated to an internal dual-ported RAM based protocol.
In theory the serial outbound protocols were mostly master-slave although this mode was not always honoured. I had to deal with interleaved protocol sessions which meant that protocol data had to be cached and processed not before a master-slave session could be fully completed. Then protocol session data could be passed on and the cache freed.
All this was highly time critical for medium voltage switching (about 1k amperes). Processing power and memory was scarce - relatively to the fact that the gateway always needed to be able to process incoming data.
Messaging And Ageing Memory
I decided not to use techniques like GC (or mark and sweep) in order to get the memory management predictable. Instead I divided the available heap into data blocks of equal size and called them messages (loosely inspired by playing with Smalltalk and micro kernel architectures). So a single message was able to handle a complete master-slave protocol session and could be compared to a circulating form where each instance fills in a part, only.
Messages were the only argument a task or a function could be passed. Upon return the message was sent to the next instance when due. A message could be accessed by one instance only. As a consequence, memory management and multi threaded synchronisation logic became almost trivial.
At the top level, a master-slave protocol reception handler would try to find a message relating to a partially completed session or take appropriate action (e.g. dropping the message) if there was none.
Defining an ageing and dying process for the messages allowed to make it predictable when or which messages were dropped latest. It also allowed to the gateways to always recover. There was no permanent deadlock possible by design.