Re: Useful stats from a CM system

On 23 Sep 2008, at 17:25, John Rouillard <rouilj@...> wrote:

> In my first musings I talked about using a version control system to
> determine how efficient your CM system is ....

Hi John ...

This is all very interesting indeed. I'll try and summarised the  
issues that it
raised for me ...

(*) I'm really interested in metrics. I'd certainly be interested in  
sharing some information and
doing some joint analysis on this if it looked like it would be  
useful. I have 5-10 years worth
of configuration change data (in CVS) for 1000+ machines. What I  
don't have is a good
correlation between this and the "why" (tickets). In theory the CVS  
commit comment should
say something useful, but in practice, it often doesn't. I'd be  
interested in a very simple classification
of the reasons - eg. software upgrade, hardware upgrade,  
configuration bug fix, etc. etc.

(*) A "file" is not a good granularity for me - our config system  
(based on LCFG) deals with "resources"
which are a slightly higher level and a single resource change may  
affect multiple files. I'm not sure what
is a good level for a useful metric - but I don't think "file" is the  
right level - that feels like trying to measure the
changes to source code by comparing the object code ... (but I guess  
it depends on exactly what you want
to use the metric for).

(*) Our system is fully "prescriptive" - except in extreme  
emergencies, or during development, we don't
have anomalies  The config system monitors compliance and anything  
which is out indicates
an error. But I know that other people don't have the luxury of being  
able to enforce this kind of policy.

(*) "How do you tell whether your config management system is making  
things better?". It would be very
interesting to do some kind of formal study on this - but I think it  
would be long-term project. At present, I
think this is just empirical - everyone who has real experience of a  
good tool in a large environment
seems to agree - Narayan's simple "could you handle the volume  
otherwise?" question is an obvious
starting point, but actually, I think the other metrics are just as  
important - and people (management?)
are not so likely to be aware of these if they are not familiar with  
the area ....

(*) "Is there a common set of metrics?". I think there is certainly a  
common set of "dimensions". Whether
we could agree on how to put actual numbers to them is another  
matter ... people tend to emphasise
simple direct efficiency, but for me, there are range of other things  
which are just as important - eg.
correctness (security), reliability (pre-checks), uniformity,  
etc .... All of these things have an indirect
effect on the efficiency. The SAGE system config book says quite a  
bit about this & I think it makes
a good argument to present to management ... (the OS vendor tools  
don't really cut it here).

(*) I like the comments from someone who was saying that sysadmins  
didn't want to change
configurations. A good system *does* help with this - if you can make  
large changes easily
and be reasonably confident that they aren't going to break anything,  
then you are going to
be a lot more receptive to doing it! The ability to *continuously*  
change configurations is
vital to meeting business requirements in most cases.

(*) The ability for ("power") users to control (parts of) the  
configurations of their own machines
seems to be important. This is tough one - in general, having them  
make manual changes
which potentially conflict with some CM change is a recipe for  
disaster. We have been
experimenting with allowing users to make changes to their own  
machines via the CM
system - but this involves learning/understanding more about the  
system than most users
want (should be expected?) to do.

   Paul

PS. Will you be at LISA? Would be nice to have a face-to-face  
discussion ....

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.