The advantage of using SCSI interfaces to encapsulate IP is the large bandwidth possible using SCSI interfaces, that can be up to 80 MByte/s with SCSI-2 and 160 MByte/s with SCSI-3 (Note: with fast ethernet only 10.7 MByte/s can be achieved). It can be used in local networks or Beowulf clusters where a high throughput is necessary for large messages (and a small latency for small messages in e.g. reduction loops...) in MPI applications (see last part of this page).
There is a patch to the 2.2.14 Linux kernel that was developed by Marc Schaefer.
Interest of UALG Vision Lab: MPI-over-SCSI
The Ping-Pong Beowulf (two dual-PIII nodes at 450 MHz with fast ethernet between them) outperforms the SGI Origin 200, hence in the future we will work more with Beowulfs. The throughput between the two nodes is 10.7 Mbyte/s (86 Mbit/s of the 100 max); between two CPUs in one node it is only a factor of 2.66 more: 28 Mbyte/s. These throughputs have been measured for messages from 10,000 REALs to 1,000,000 REALs (one REAL is 4 bytes). The latencies for small messages are 0.35 and 0.15 ms.
The measured factor of 2.66 implies that if the ethernet wire is replaced with a SCSI cable, and the TCP/IP protocol is still used, the throughput between nodes will increase from 10.7 to 28 Mbyte/s, but NOT to the SCSI throughput...
Apart from buying gigabit ethernet (Myrinet), which doubles the price of a cluster, we want
to develop a cheap but fast alternative: MPI-over-SCSI in which the MPI and SCSI
protocols are directly linked and in which MPI uses a routing table to get the best
communication performance. More about this follows later.
Hans du Buf
Luis Sismeiro, 27 March 2000