Overhead in Increasing the Solaris System Clock Rate
By Jim Connors 14 January 2009

In a previous entry entitled Real-Time Java and High Resolution Timers, we discussed how Sun's Java Real-Time System requires access to timers with a resolution greater than the default 10ms to do anything really interesting.   It was also stated that most modern processors have an APIC or Advanced Programmable Interrupt Controller which supports much finer-grained clock tick rates.

Unfortunately there are many instances where a system does indeed contain an APIC, but it is not exposed by the BIOS.  Furthermore, we've found that some of the embedded, low-power x86-based processors do not contain an APIC at all.  For an example, take a look at the AMD Geode LX 800 based fit-PC Slim.

So if you wanted to utilize higher resolution timers for this class of system, you'd have to resort to alternative methods.  Solaris and OpenSolaris provide two /etc/system parameters called hires_tick and hires_hz to facilitate increasing your default system clock tick.  By adding the following line to /etc/system, you'll increase the system clock tick rate from the default of 100 per second to 1,000 per second, effectively changing the clock resolution from 10ms to 1ms.

   set hires_tick=1

If you want to further increase the clock resolution, you can do so via the hires_hz system tunable parameter.  Although formally unsupported, it does work.   In order to, for example, increase the clock tick rate to 10,000, add this to /etc/system:

    set hires_tick=1
    set hires_hz=10000

To achieve the desired effect above, you must include both the hires_tick assignment in addition to setting the hires_hz parameter.

These modifications do not come without side-effects, and depending upon the hardware in question and the granularity of the desired timer resolution, they could be significant.  In short, it takes additional CPU cycles to field all those timer interrupts.  So I thought it'd be interesting to see what effect changing the clock tick rate had on two separate systems.   Here they are:

The measuring tool used for this simple experiment is vmstat(1m).  Solaris aficionados will likely point out that there are much more accurate alternatives, but I think vmstat(1m) gives a decent feel for what's going on without having to expend a whole lot of extra energy.  In particular we'll look at the following fields returned by issuing a 'vmstat 5', and picking one of the interim samples:

• in - interrupts per second
• us - percentage usage of CPU time in user space
  
• sy - percentage usage of CPU time in system space
  
• id - percentage usage of CPU time idling

The sum of (us + sy + id) should approximate 100%.  The table below shows sample vmstat output on various clock tick settings for our two hardware platforms.

 none (default)

 set hires_tick=1

set hires_tick=1
set hires_hz=10000

set hires_tick=1
set hires_hz=100000

 in: 201
 us: 0
 sy: 1
 id: 99

 in: 2001
 us: 0
 sy: 5
 id: 95

 in: 19831
 us: 0
 sy: 43
 id: 57 

 in: 471
 us: 0
 sy: 0
 id: 99

 in: 2278
 us: 0
 sy: 1
 id: 99

 in: 20299
 us: 0
 sy: 5
 id: 95

 in: 200307
 us: 0
 sy: 21
 id: 79

Notes/Conclusions:

• The vmstat(5) was let run for about a minute.  The output above shows one of the typical 5 second samples.  The other 5 second samples are almost identical.
  
• The user (us) CPU time numbers give us a reasonable idea that these systems were predominantly in an idle state when being sampled.
• The number of interrupts serviced per second is directly proportional to the clock tick rate.
• And of course, the larger the number of interrupts, the more system CPU time is required.
• The amount of overhead taken up by increasing the clock rate is a function of system capability.  The CF-30 not only has a much faster processor, it also has two cores to share the interrupt load.  As such it could accommodate a much higher clock tick rate.  For the fit-PC, performance is impacted profoundly even at modest clock tick rates.