Solaris Was Real-time Before Real-time Was Cool
By Jim Connors 23 Feb 2007

In the financial services market, there is a general trend to move key systems close to, or even right inside the exchanges themselves -- the idea being that the nearer you are to the source, the less network infrastructure and latency you'll experience.  With this advantage firms can potentially take on additional transaction loads at higher transaction rates.  These systems typically use the latest Intel or AMD processors and run a commercial distribution of Linux.¹

[1] Thank you Eric Bruno for your brief description, and for unknowingly letting me (slightly) plagiarize your comments.

Indeed these co-located systems perform as expected almost all the time.  But there are periodic intervals where the latency increases by several orders of magnitude, the ramifications of which could be financially disastrous.  After eliminating other components, the street seems to be focusing its wrath on commercial Linux distributions and their lack of real-time capabilities.

The linux community is actively working to include underpinnings to support real-time, but as of yet these capabilities are not part of a standard major (i.e. Red Hat, SuSE) distribution.  Instead, an alternate version of linux with real-time extensions is offered.  These extensions are in effect separate non-standard OS releases, and have not had the soak time required by many institutions.

Back in the early 90's, I volunteered to move over to Sun's newly formed SunSoft business unit.  One of it's main charters was to push the concept of running Solaris on alternate, i.e. Intel, platforms.  (Don't get me started here, can you imagine where Solaris would be right now if Sun had actually taken this initiative seriously back then?)  As part of that transition, I had the opportunity to take a Solaris Internals course, and couldn't help but notice the effort put in architecturally to address short latencies.  I still have the course notebook; it is dated September 1993.

The point is Solaris already has the real-time capabilities claimed by these add-on linux extensions.  It is built into the operating system, has been for quite some time, is rock solid and doesn't require any additional components.  A partial list of features include:

• Real-time Scheduling Class - In order to allow equal opportunity to system resources, traditional Unix schedulers transparently change process priorities to give competing processes a fair chance.  Although well suited for timesharing systems, this is unacceptable real-time behavior.  Since its outset, Solaris through its SVR4 roots, introduced the concept of alternate scheduling classes.  It includes a real-time scheduling class, which furnishes fixed-priority process scheduling at the highest priority levels in the system.
  

• Fine-Grained Processor Control / Processor Sets - Solaris allows threads and applications to be bound to specific individual processors. In addition, processors within a system can be grouped together as a processor set and dedicated to real-time tasks.²  Here's a nice article describing processor sets. Dated June 2001,  processor sets have been a part of Solaris since release 2.6.

• Interrupt Sheltering - Available since Solaris 7, this feature enables CPUs to be sheltered from unbound interrupts.  It can be used in conjunction with processor sets to shelter all CPUs in a processor set.  Note: At least one processor in the system must be kept unsheltered.
  

• Priority Inheritance - Priority inversion occurs when a high-priority thread blocks on a resource that is held by a lower-priority thread. A runnable thread with a priority between the high and low-priority threads creates a priority inversion because it can receive processor resources ahead of the high-priority thread. 

To avoid priority inversions with kernel synchronization primitives, Solaris employs a transient priority inheritance protocol. The protocol enables the low-priority thread holding the resource to “inherit” the higher priority of the blocked high-priority thread. This approach gives the blocking low-priority thread the CPU resources it needs to complete its task as soon as possible so that it can release the synchronization primitive. Upon completion, all threads are returned to their respective priorities by the kernel.³

• High Resolution Timers - Solaris 8 introduces the cyclic subsystem; this allows for timers of much better granularity -- in the microsecond and nanosecond range -- without burdening the system with a high interrupt rate.

• Memory Locking - The paging in and out of data from disk to memory may be considered normal behavior for virtual memory systems, but it is unacceptable for real-time applications.  Solaris addresses this problem by allowing the locking down of a process' pages into memory, using mlock(3C) or mlockall(3C) system calls.
  

• Early Binding - By default, linking of dynamic libraries in the Solaris is done on an as-needed basis. The runtime binding for a dynamically linked function isn't determined until its first invocation. Though flexible, this behavior can induce indeterminism and unpredictable jitter in the timing of a real-time application. To avoid jitter, the Solaris provides for early binding of dynamic libraries. By setting the LD_BIND_NOW environment variable to "1", libraries are bound at application startup time. Using early binding together with memory locking is a very effective approach to avoiding jitter.⁴

[2,3,4] Shameful plagiarism from Scalable Real-Time Computing in the Solaris ��� Operating Environment. A Technical White Paper. To further prove the maturity of Solaris' real-time features, this document was written in the Solaris 8 time frame.  It was copyrighted in 2000.

So why not give Solaris more consideration?  It's way more mature.  And in the unlikely event (chcukle, chuckle) that a lower-latency OS might not solve all your performance problems, I'd put my money on Solaris and DTrace over anything Linux could offer in finding the real problem.