I just bashed out this powershell script to query the build farm status using the CC.NET server report XML page:

# get-buildstatus.ps1
$client = new-object system.net.webClient
$client.Headers.add("user-agent", "PowerShell")
$data = $client.openRead( 'http://builddashboard/ccnet/XmlServerReport.aspx' )
$reader = new-object system.io.streamReader $data
$s = $reader.readToEnd()
$data.close()
$reader.close()  
([xml]$s).CruiseControl.Projects.Project | ft;

Wicked simple - the WebClient class is used to connect to the build dashboard, and a stream reader object pulls the farm data.  The data is XML, which PowerShell considers warm butter and happily pivots and formats as a pretty table:

> get-buildstatus
name           category       activity       lastBuildStatu lastBuildLabel lastBuildTime  nextBuildTime  webUrl
                                             s
----           --------       --------       -------------- -------------- -------------  -------------  ------
SuperDuo 3....                Pending        Success        SuperDuo_14890 2009-07-21T... 2009-07-21T... http://buil...
SuperDuo Sa...                Pending        Success        SuperDuo_14725 2009-06-16T... 2009-07-21T... http://buil...
SuperDuo 2....                Pending        Success        SuperDuo_14706 2009-06-09T... 2009-07-21T... http://buil...
SuperDuo 2.2                  Pending        Success        SuperDuo_14888 2009-07-21T... 2009-07-21T... http://buil...
...

Of course, if you have the PowerShell Community Extensions installed, the get-url cmdlet reduces the script to a one-liner:

# get-buildstatus.ps1
([xml](get-url 'http://builddashboard/ccnet/XmlServerReport.aspx')).CruiseControl.Projects.Project | ft;

I think I'll push this into a PowerGUI powerpack...

Super happy fun time deluxe!  (Enjoy!)

Our CI system isn't terribly complicated, thankfully.  It evolved from a batch file on a single machine to a farm of VMs running CruiseControl.NET and a single Subversion repository.  The triggering mechanism hasn't changed during this transition though: it's a custom beast, consisting of a post-commit hook on our repository logs revisions into a database, and a custom sourcecontrol task is used to poll the database for revisions that haven't been built yet.  It works fine and naturally creates a balanced build farm: several VMs can be configured to build the same project, and the database sourcecontrol task prevents more than one VM from building the same revision.

As well as it works, it has some problems.  First, the post-commit hook relies heavily on SVN properties, which are a pain to maintain and make it impossible to simply "add a project to the build" without going through a lot of unnecessary configuration.  Moreover, the hook is starting to seriously hold up commits, sometimes as long as 20 seconds.  

Second, and more irritating, the system builds every individual commit.  By that, I mean it builds each and every revision committed to the repository - which may not sound like a bad thing, except that it includes every revision, even those that occurred before the project was added to the CI farm - all the way back to the revision that created the project or branch.  I have to manually fudge the database, adding fake build results to prevent the builds from occurring.  It's not hard, but it is a pain in the ass.  And with the team's overuse of branching I'm finding myself having to fudge more and more. 

I'm really trying to move the system towards what "everyone else does," by which I mean trigger builds by polling source control for changes.  No more database, no more post-commit hook, no more SVN property configuration, just Subversion and CruiseControl.NET.  It would be easy enough to do - simply change our CC.NET project configurations to use the standard SVN source control task.  The problem is that without the database, the farm is no longer automagically load-balancing - every VM in the farm would end up building the same source, which defeats the purpose of the farm.

I figured that I could recoup the load-balancing if I had an "edge" server between Subversion and the build farm.  This server could monitor source control and when necessary trigger a build on one of the VMs in the build farm.  So instead of each farm server determining when a build should occur, there is a single edge server making that decision.

CC.NET ships with the ability to split the build across machines - that is, for a build on one machine (like the edge server) to trigger a build on another machine (like a farm server); however, there is no load-balancing logic available.  So I made some of my own...

Edge Server CC.NET Plugin

The edge server plugin operates on a very simple algorithm:

1. Get a list of farm servers from configuration;
2. Determine which of farm servers are not currently building the project;
3. Fail the build if no farm server is available for the project;
4. Force a build of the project on the first available farm server.
   

If all you want is the project source code, here it is: ccnet.edgeserver.zip (607.87 kb)

Take a look at the configuration of the plugin; I think it will make the code easier to digest.

Edge Server Configuration

The edge server consists of little more than a source control trigger and a list of farm servers:

<cruisecontrol>
  <project name="MyProject">
    <triggers>
      <intervalTrigger seconds="10" />
    </triggers>
    <sourcecontrol type="svn">
      <trunkUrl>svn://sourcecontrol/Trunk</trunkUrl>
      <autoGetSource>false</autoGetSource>
    </sourcecontrol>
    <labeller type="lastChangeLabeller" prefix="MyProject_"/>
    <tasks>
      <farmBuild>
        <farmServers>
          <farmServer priority="1" uri="tcp://build-vm-1:21234/CruiseManager.rem" />
          <farmServer priority="2" uri="tcp://build-vm-2:21234/CruiseManager.rem" />
          <farmServer priority="3" uri="tcp://build-vm-3:21234/CruiseManager.rem" />
        </farmServers>
      </farmBuild>
    </tasks>
    <publishers>
      <nullTask />
    </publishers>
  </project>
</cruisecontrol>

When CC.NET is run with this configuration, it will monitor the subversion repository for changes to the "MyProject" trunk (lines 6-9); note that since autoGetSource is false, no checkout will occur.  The edge server will never have a working copy of the source.

The load-balancing is configured in lines 12-18; in this example, three farm servers are configured in the farm for "MyProject", with build-vm-1 having the highest priority for the build (meaning it will be used first when all three servers are available).  When a change is committed to the repository, the edge server will choose one of these servers based on its availability and priority, and then force it to build the project.

Farm Server Configuration

The farm server is configured just as a normal CC.NET build, except for two key differences: first, it is configured with no trigger; second, a remoteProjectLabeller is used to label the build.  Here's a sample configuration, with mundane build tasks omitted for brevity:

<cruisecontrol>
  <project name="MyProject">
    
    <triggers/>
    <sourcecontrol type="svn">
      <trunkUrl>svn://sourcecontrol/MyProject/Trunk</trunkUrl>
      <autoGetSource>true</autoGetSource>
    </sourcecontrol>
    <labeller type="remoteProjectLabeller">
      <project>MyProject</project>
      <serverUri>tcp://edgeServer:21234/CruiseManager.rem</serverUri>
    </labeller>
    <tasks>
      <!--
              ... 
      -->
    </tasks>
    <publishers>
      <!--
              ... 
      -->
    </publishers>
  </project>
</cruisecontrol> 

Details to note here are:

• the labeller points to the edge server to obtain the build label; this is necessary because labels are generated during the build trigger, which on the farm server is always forced and won't include any source revision information;
• the project name on the farm server matches exactly the project name on the edge server; this is a convention assumed by the plugin.

Source Code Overview

I need a FarmServer type to support the CC.NET configuration layer:

using System;
using System.Collections.Generic;
using System.Text;
using ThoughtWorks.CruiseControl.Core;
using ThoughtWorks.CruiseControl.Remote;
using Exortech.NetReflector;
using ThoughtWorks.CruiseControl.Core.Publishers;
using System.Collections;
using ThoughtWorks.CruiseControl.Core.Util;
namespace CCNET.EdgeServer
{
    [ReflectorType( "farmServer" )]
    public class FarmServer
    {
        [ReflectorProperty( "uri" )]
        public string Uri;
 
        [ReflectorProperty( "priority" )]
        public int Priority;
    }
}

No real surprises here.  Each FarmServer instance holds a URI to a CC.NET farm server and it's priority in the balance algorithm. 

The real meat is in the FarmPublisher class:

using System;
using System.Collections.Generic;
using System.Text;
using ThoughtWorks.CruiseControl.Core;
using ThoughtWorks.CruiseControl.Remote;
using Exortech.NetReflector;
using ThoughtWorks.CruiseControl.Core.Publishers;
using System.Collections;
using ThoughtWorks.CruiseControl.Core.Util;
namespace CCNET.EdgeServer
{
    [ReflectorType( "farmBuild" )]
    public class FarmPublisher : ITask
    {
        ICruiseManagerFactory factory;
 
        [ReflectorProperty( "Name", Required = false )]
        public string EnforcerName;
 
        [ReflectorHash( "farmServers", "uri", Required = true )]
        public Hashtable FarmServers;
 
        public FarmPublisher() : this( new RemoteCruiseManagerFactory() ) { }
 
        public FarmPublisher( ICruiseManagerFactory factory )
        {
            this.factory = factory;
            this.EnforcerName = Environment.MachineName;
        }
 
        public void Run( IIntegrationResult result )
        {
            // build a list of available farm servers
            //  based off of the plugin configuration
            Dictionary<int, ICruiseManager> servers = new Dictionary<int, ICruiseManager>();
            FindAvailableFarmServers( result, servers );
 
            if( 0 == servers.Count )
            {
                Log.Info( "No servers are available for this project at this time" );
                result.Status = IntegrationStatus.Failure;
                return;
            }
 
            // sort the available servers by priority
            List<int> keys = new List<int>( servers.Keys );
            keys.Sort();
 
            // force a build on the server with the highest 
            //  priority
            ICruiseManager availableServer = servers[ keys[ 0 ] ];
            Log.Info( "forcing build on server ..." );
            availableServer.ForceBuild( result.ProjectName, EnforcerName );
        }
        ...

FarmPublisher is configured with a list of FarmServer objects (lines 20-21).  The Run method (starting on line 31) implements the simple load-balancing algorithm:

1. a list of farm servers which are available to build the project is constructed (lines 33-36);
2. if no server is available to build the project, the edge server reports a build failure (line 38-43);
3. the list of available farm servers is sorted by priority (lines 45-47);
4. the project build is started on the farm server configured with the highest priority (line 53).

Determining a list of available farm servers is pretty straightforward:

void FindAvailableFarmServers( IIntegrationResult result, IDictionary<int, ICruiseManager> servers )
{
    // predicate to locate a server that isn't actively building 
    // the current project
    Predicate<ProjectStatus> predicate = delegate( ProjectStatus prj )
    {
        return IsServerAvailableToBuildProject( result, prj );
    };
 
    // check the status of each configured farm server
    foreach( FarmServer server in FarmServers.Values )
    {
        ICruiseManager manager = null;
        try
        {
            manager = ( ICruiseManager )factory.GetCruiseManager( server.Uri );
 
            // get a local copy of server's current project status snapshot
            List<ProjectStatus> projects = new List<ProjectStatus>( manager.GetProjectStatus() );
            if( null != projects.Find( predicate ) )
            {
                // add the farm server to the list of available servers, 
                //  keyed by its configured priority
                servers[ server.Priority ] = manager;
            }
        }
        catch( Exception e )
        {
            Log.Warning( e );
        }
    }
}

Available servers are saved in the servers dictionary, keyed by their configured priority.  The availability of each farm server listed in the task configuration is checked by obtaining the status of the farm server's projects, and passing them to the IsServerAvaialbleToBuildProject method:

bool IsServerAvailableToBuildProject( IIntegrationResult result, ProjectStatus prj )
{
    if( null == prj || null == result )
    {
        return false;
    }
    bool status = (          
        // project name must match
        StringComparer.InvariantCultureIgnoreCase.Equals( result.ProjectName, prj.Name ) &&
 
        // integrator status must be "running"
        prj.Status == ProjectIntegratorState.Running &&                
 
        // build activity must be "sleeping"
        prj.Activity.IsSleeping()
    );
    return status;
} 

which simply returns true when:

• the farm server configuration contains the project,
  
• the project is currently running, and
  
• the project isn't currently building.

Download

This code is basically spike-quality at this point.  I fully expect to throw this away in favor of something better (or get my manager to splurge for TeamCity).  There's still a lot of stuff to do.  E.g., the algorithm assumes that a farm server is capable of building more than one project at a time - that is, if a farm server is busy building one project, it can still be available to build another concurrently.  My assumption is that I'll manage this with the farm server priority configuration.  I'd like to leverage the queuing features available in CC.NET; however, I see no way of querying the queue status of a farm server in the CC.NET API.  But at least I can start disabling the post-commit hook on our repository.

The project contains the code, a test/demo, and just a few basic unit tests; I'll update the download if/when the project matures.  If you use it, or have any suggestions, please let me know in the comments of this post.  Enjoy!

ccnet.edgeserver.zip (607.87 kb)

You can tell from my recent blog posts that I'm spending a good deal of time in my SCM role at work.  I've recently been looking at using powershell scripts to replace some binary hooks that have become dated, and to expand our repository's integration with our bug tracking system.  All in all it was pretty simple, but a few gotchas held me up:

First, SVN hooks have to be directly executable (e.g., .exe, .bat., .com, .cmd, or windows script files); powershell scripts are not directly executable for security reasons, so I had to use a batch script as a proxy.  

Second, SVN invokes the hooks with an empty environment (again for security purposes); without an active PATH variable specified, the batch and powershell scripts need to specify full paths to all files.

Finally, some hooks, such as the post-revprop-change hook, read data from STDIN, but the proxy batch file will not forward its STDIN to the powershell process without some intervention.  

Here is the batch proxy for the post-revprop-change hook:

rem   =====================================================================
rem    %  Argument     Description
rem   --- ------------ ----------------------------------------------------
rem   [1] REPOS-PATH   (the path to this repository)
rem   [2] REV          (the revision that was tweaked)
rem   [3] USER         (the username of the person tweaking the property)
rem   [4] PROPNAME     (the property that was changed)
rem   [5] ACTION       (the property was 'A'dded, 'M'odified, or 'D'eleted)
rem
rem   [STDIN] PROPVAL  ** the old property value is passed via STDIN.
rem   ---------------------------------------------------------------------

c:\windows\system32\windowspowershell\v1.0\powershell.exe -command "$input | d:\repository\hooks\postrevprop.ps1" -args "%1" "%2" "%3" "%4" "%5"

A few things to note:

• I'm using full paths to the powershell executable and the powershell script; this is necessary because the batch file will be run with an empty environment and no PATH to search.
• I'm not really executing the powershell script, I'm executing a command string that passes the implicit $input variable to the powershell script.  This $input variable resolves to the STDIN for the batch proxy, and enables the powershell script to read it.
• Because my command string invokes a script file, I need to make sure that the powershell execution policy is set to RemoteSigned or Unrestricted on the SVN server.    

Here is the powershell script invoked by the batch file proxy:

param( $repo, $rev, $user, $propname, $action );
@($repo, $rev, $user, $propname, $action) + $input
    | out-file "d:\repository\hooks\log.txt" -append

The script doesn't do much at the moment - just logs the hook parameters and input to a file so I can verify that the script is executing correctly.  Now that I have ironed out the major wrinkles, I can start porting our hooks to powershell!

We've been using CruiseControl.NET at my shop for a few years.  One of the things I've never really enjoyed about the CC.NET dashboard is the lack of a recent build history for the entire build farm.  When someone reports an issue with a build, I have use the CC.NET dashboard to drill through every VM in the farm until I figure out which server built the revision I'm looking for.  It's a waste of time.  Moreover, the default farm view is really really boring - don't get me wrong, it's vital to communicate the current state of the builds, but I want more.  I want to be able to show my boss how much work is being done, how much coordination is taking place on the team, and how badly we need another VM host in the farm.  

So I spent a little spare time on a pet project: a CC.NET dashboard plugin that visualizes the build farm activity using the SIMILE Timeline project.  Before I dig into the grit, take a quick look at this sample timeline; it's a snapshot of about 4 hours of activity from one of our build VMs:

The plugin only took an hour or so to pull together; of course, that was after an evening of digging through the CC.NET source code to figure out NVelocity and the Objection IoC container voodoo.  The project is still very raw - there is a lot of missing error checking and no unit tests.  I'm offering up the source code for download anyway in order to get some feedback: ccnet.timeline.plugin.zip (6.69 kb)

Of course, it comes as-is with no commitment of support or fitness of purpose, etc.  There are no special build requirements, outside of VS 2005 or later and the CC.NET source code.

Installation

To install the plugin:

1. copy ccnet.timeline.plugin.dll from the project output folder into the CC.NET dashboard bin directory;
2. copy the timeline.vm template from the project template directory into the CC.NET dashboard template directory;

Configure the Dashboard

To configure the dashboard:

1. modify the list of of HttpHandlers in the CC.NET dashboard web.config to include the timeline.aspx handler shown below; make sure it appears first in the list:

   				<httpHandlers> 	  <add verb="*" path="timeline.aspx" type="ccnet.timeline.plugin.TimelineHttpHandler,ccnet.timeline.plugin"/> 	  <add verb="*" path="*.aspx" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard"/> 	  <add verb="*" path="*.xml" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard"/> 	</httpHandlers>
   	

2. add the farmTimelinePlugin element to the list of farm plugins in the dashboard.config file:

   				<farmPlugins> 	  <farmReportFarmPlugin /> 	  <farmTimelinePlugin /> 	  <cctrayDownloadPlugin /> 	</farmPlugins> 	
   	

This will result in a "Timeline" link in the farm view of the CC.NET dashboard that will present the farm timeline.

Configure the Projects

The timeline is driven by data from the statistics publisher, so any project you want to include on the timeline will need to publish statistics.  To enable this, just add a <statistics/> element to the project's list of publisher tasks, taking care that it follows all file merge tasks:

<publishers>
  <merge>
    <files>
      <file>c:\program files\cruisecontrol.net\server\build-server.xml</file>
      <file>tests\nunit-*-results.xml</file>
      <file>tests\coverage-*-results.xml</file>
    </files>
  </merge>
  <xmllogger />
  <statistics />
  ...

More Features

This initial spike is very promising.  Here are a couple of ideas I have to make the timeline better:

• the timeline uses color to indicate build success or failure, but it would be nice to propogate any build errors into the event on the timeline, so you could see the error by clicking on the event.
• operationalizing the event generation configuration - as it works now, there is no customization points, so what you see it what you get.
• create timeline server and project plugins.
• create a timeline view of a build report: e.g., display individual build tasks, tests, etc.
• the ability to mark public releases ( or any non-CC.NET event ) on the timeline.  This would help demonstrate the "mad dash" effect an approaching release date has on our team.
• display source control activity.
• parameterizing the timeline view - maybe controlling the projects displayed, hilighing certain projects, or adjusting the units in the timeline band to compact or expand the display.  

As the project matures and stabilizes, I'll post updates.  If there is enough interest, I'll start looking into making a full contribution to the CC.NET project.  Let me know if you find this useful, and if you have any ideas for improvements.