Useful stats from a CM system (part 2) the push

Hi all:

Here are some more musings on stats that may tell you how your CM
system is working.

DACS uses a push model for distribution, but I think the stats are
applicable to push and pull models.

From the distribution system we can get metrics on the number of files
pushed and which files are being pushed.

   1. The "(number of files pushed)/(machine in service)" metric
      should be rising or steady as it is an indication of how many
      changes are occurring. If it drops, and the number of tickets
      being closed remains the same it's an indication that people
      aren't using the CM system to manage new systems. Does anybody
      have any numbers on this for their sites?

      For my company it's running approx 50 files/month-machine
      (I.E. approx 100 machines, 5000 files updated per month). Note
      the number of machines includes any system type that is
      supported by DACS, but isn't managed via DACS. So Cyclades and
      Opengear terminal servers aren't included, but that Centos 5
      virtual machine that isn't updated via DACS is included.

   2. The number of changes/week, month etc. Where a change is a file
      pushed to an end system. We average somewhere around 5000 file
      changes per month. This includes password files, service
      configuration and firewalls changes.

      For reference, the number of files changed per month for our
      systems from November 2007 (partial month) through July 31 2008
      is: 2378, 4481, 4996, 5413, 5595, 5179, 5115, 5141, 5283.

      This has a mean of 5150.38 and median of 5160 with a standard
      deviation of 329.134 (6.3%).  Does anybody have any info on what
      this metric is at your site and how variable this metric?

      One thing that this metric does not mean is the number of files
      changed/edited by a person in DACS. The problem with trying to
      correlate the number of input (svn) changes to this metric is
      that a single file change in DACS (e.g. assigning a new service
      to a host) can result in dozens of files being changed. Even in
      systems where there is no file generation, this number depends
      on the number of systems receiving a particular file and not on
      the number of changed files.

   3. I claim the ratio of changed files (in svn) to pushed files
      (generated via DACS) is useful as a measure of a multiplier
      effect. But I am not collecting info on that (nor do I really
      have any idea yet on how to do that). Is anybody watching this
      metric? Does anybody have a similar metric they would like to
      share?

   4. Does anybody track the most distributed files, least distributed
      files? Are those really a useful metric?

One other metric that I would like to obtain is the number of
intentional vs. non-intentional pushes of files to systems. But am
stumped on how to obtain it. As an example:

  Deploy a new service, change a bunch of files to implement the
  service. They are all intentional changes.

  Deploy an rpm that overwrites a file (say /etc/TZ) and DACS has to
  update the timezone file. That is an unintentional change.

This is made more useful/difficult because of our way of making local
changes to vendor supplied RPMS. When we want to fix/enhance an RPM
package, especially when it's vendor supplied, we just push changes
from DACS rather than rebuilding the RPM and having to track the new
releases of the upstream RPM and rebuild them. Having a way to
automatically differentiate between files that are pushed because of
an internal (intentional) event in DACS (a checked in change of some
sort) versus an push due to external forces (unintentional change):

   * RPM "upgrade"

   * cowboy admin editing files on a system outside of CM procedures
     etc.

Note that the latter would not include debugging/testing in place
which we encourage and which will ultimately be fixed as an
intentional change.

But more on that in the next installment.

--

-- 
				-- rouilj

John Rouillard
System Administrator
Renesys Corporation
603-244-9084 (cell)
603-643-9300 x 111