Continuous Integration Timeline

§ November 27, 2008 18:01 by beefarino |

We've been using CruiseControl.NET at my shop for a few years.  One of the things I've never really enjoyed about the CC.NET dashboard is the lack of a recent build history for the entire build farm.  When someone reports an issue with a build, I have use the CC.NET dashboard to drill through every VM in the farm until I figure out which server built the revision I'm looking for.  It's a waste of time.  Moreover, the default farm view is really really boring - don't get me wrong, it's vital to communicate the current state of the builds, but I want more.  I want to be able to show my boss how much work is being done, how much coordination is taking place on the team, and how badly we need another VM host in the farm.  

So I spent a little spare time on a pet project: a CC.NET dashboard plugin that visualizes the build farm activity using the SIMILE Timeline project.  Before I dig into the grit, take a quick look at this sample timeline; it's a snapshot of about 4 hours of activity from one of our build VMs:


The plugin only took an hour or so to pull together; of course, that was after an evening of digging through the CC.NET source code to figure out NVelocity and the Objection IoC container voodoo.  The project is still very raw - there is a lot of missing error checking and no unit tests.  I'm offering up the source code for download anyway in order to get some feedback: (6.69 kb)

Of course, it comes as-is with no commitment of support or fitness of purpose, etc.  There are no special build requirements, outside of VS 2005 or later and the CC.NET source code.


To install the plugin:

  1. copy ccnet.timeline.plugin.dll from the project output folder into the CC.NET dashboard bin directory;
  2. copy the timeline.vm template from the project template directory into the CC.NET dashboard template directory;

Configure the Dashboard

To configure the dashboard:

  1. modify the list of of HttpHandlers in the CC.NET dashboard web.config to include the timeline.aspx handler shown below; make sure it appears first in the list:
    				<httpHandlers> 	  <add verb="*" path="timeline.aspx" type="ccnet.timeline.plugin.TimelineHttpHandler,ccnet.timeline.plugin"/> 	  <add verb="*" path="*.aspx" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard"/> 	  <add verb="*" path="*.xml" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard"/> 	</httpHandlers>
  2. add the farmTimelinePlugin element to the list of farm plugins in the dashboard.config file:
    				<farmPlugins> 	  <farmReportFarmPlugin /> 	  <farmTimelinePlugin /> 	  <cctrayDownloadPlugin /> 	</farmPlugins> 	

This will result in a "Timeline" link in the farm view of the CC.NET dashboard that will present the farm timeline.

Configure the Projects

The timeline is driven by data from the statistics publisher, so any project you want to include on the timeline will need to publish statistics.  To enable this, just add a <statistics/> element to the project's list of publisher tasks, taking care that it follows all file merge tasks:

      <file>c:\program files\\server\build-server.xml</file>
  <xmllogger />
  <statistics />

More Features

This initial spike is very promising.  Here are a couple of ideas I have to make the timeline better:

  • the timeline uses color to indicate build success or failure, but it would be nice to propogate any build errors into the event on the timeline, so you could see the error by clicking on the event.
  • operationalizing the event generation configuration - as it works now, there is no customization points, so what you see it what you get.
  • create timeline server and project plugins.
  • create a timeline view of a build report: e.g., display individual build tasks, tests, etc.
  • the ability to mark public releases ( or any non-CC.NET event ) on the timeline.  This would help demonstrate the "mad dash" effect an approaching release date has on our team.
  • display source control activity.
  • parameterizing the timeline view - maybe controlling the projects displayed, hilighing certain projects, or adjusting the units in the timeline band to compact or expand the display.  

As the project matures and stabilizes, I'll post updates.  If there is enough interest, I'll start looking into making a full contribution to the CC.NET project.  Let me know if you find this useful, and if you have any ideas for improvements.  

Resolving Binary References in MSBuild

§ November 20, 2008 17:49 by beefarino |

An acquaintance of a friend tweeted about a problem he's having with MSBuild: it often fails to resolve nth-tier binary dependencies of a project:

The application references a binary directly, which makes the class library a first-tier dependency of the application.  The class library references another binary assembly, which becomes a second-tier dependency of the application.  

Here's a Visual Studio solution with the same dependency configuration: (31.73 kb)

Build the application (use the Debug configuration) and check out the application's output path.  You would expect to find application.exe, classlibrary.dll, and binaryassembly.dll, but binaryassembly.dll is missing.  However, if you scrutinize the MSBuild output during the application.csproj build (use the /v:d switch), you can see that it's trying like mad to find binaryassembly.dll:

Project "D:\Project\Dump\msbuldreferencedemo\application\application.csproj" on node 0 (Rebuild target(s)).
Dependency "BinaryAssembly, Version=, Culture=neutral, PublicKeyToken=null".
  Could not resolve this reference. Could not locate the assembly "BinaryAssembly, Version=, Culture=neutral, PublicKeyToken=null". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.
  For SearchPath "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug".
  Considered "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug\BinaryAssembly.exe", but it didn't exist.
  Considered "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug\BinaryAssembly.dll", but it didn't exist.
  For SearchPath "C:\Project\Dump\msbuldreferencedemo\BinaryAssembly\bin\Debug\".
  Considered "C:\Project\Dump\msbuldreferencedemo\BinaryAssembly\bin\Debug\BinaryAssembly.exe", but it didn't exist.
  Considered "C:\Program Files\Microsoft SQL Server\90\DTS\ForEachEnumerators\BinaryAssembly.exe", but it didn't exist.
  Considered "C:\Program Files\Microsoft SQL Server\90\DTS\ForEachEnumerators\BinaryAssembly.dll", but it didn't exist.
  For SearchPath "{GAC}".
  Considered "BinaryAssembly, Version=, Culture=neutral, PublicKeyToken=null", which was not found in the GAC.
  For SearchPath "bin\Debug\".
  Considered "bin\Debug\BinaryAssembly.exe", but it didn't exist.
  Considered "bin\Debug\BinaryAssembly.dll", but it didn't exist.
  Required by "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug\ClassLibrary.dll".

MSBuild doesn't know where to look for the file.  My team hit this problem after we started replacing projects in our source hive with prebuilt binaries; the builds would succeed, but the applications failed to run because none of the second-tier dependencies of a first-tier binary reference would show up in the output folder.  I did some digging and found one simple way and one complex way to resolve the issue.

Easy: Put All Binaries in One Place

The easiest way to address this problem is to place all binary dependencies into a single folder.  If you do reference a binary from this folder, and the folder contains the binary's dependencies, those second-tier dependencies will be found during the build.  In the project file, the first-tier binary includes a hint path so the compiler knows where to look for the file.  For example, open application.csproj and change the hint path for the classlibrary reference to point to the \lib folder (line 5):  

  <Reference Include="ClassLibrary, Version=, Culture=neutral, processorArchitecture=MSIL">
  <Reference Include="System" />

Rebuild the project and you'll find both classlibrary.dll and binaryassembly.dll in the output folder.  This hint path is used as one of the search paths when MSBuild attempts to resolve a dependency during the build, and since the hint path also contains binaryassembly.dll, MSBuild is able to find that second-tier dependency as well.

Hard: Modify the Assembly Search Path List

This is the solution we had to use on my team.  The choice was made to keep each binary in a unique folder, so interdependencies between the binaries could not be managed with just the hint paths. 

The magic starts in Microsoft.Common.targets, which you will find under %FRAMEWORKDIR%\v2.0.50727\.  In that file there is a series of targets aimed at resolving different kinds of references in your project, from project references to COM references to assembly references.  The target that holds the key is ResolveAssemblyReferences, which is nothing more than a wrapper around the ResolveAssemblyReference task.  This task accepts a parameter named SearchPaths that determines where looks for assemblies.  The value of this parameter is determined by the AssemblySearchPaths property, which is created earlier in Microsoft.Common.targets:

The SearchPaths property is set to find assemblies in the following order:

    (1) Files from current project - indicated by {CandidateAssemblyFiles}
    (2) $(ReferencePath) - the reference path property, which comes from the .USER file.
    (3) The hintpath from the referenced item itself, indicated by {HintPathFromItem}.
    (4) The directory of MSBuild's "target" runtime from GetFrameworkPath.
        The "target" runtime folder is the folder of the runtime that MSBuild is a part of.
    (5) Registered assembly folders, indicated by {Registry:*,*,*}
    (6) Legacy registered assembly folders, indicated by {AssemblyFolders}
    (7) Look in the application's output folder (like bin\debug)
    (8) Resolve to the GAC.
    (9) Treat the reference's Include as if it were a real file name.
  <AssemblySearchPaths Condition=" '$(AssemblySearchPaths)' == '' ">

The value of this property is a semicolon-delimited list of paths and search targets.  This list can be easily altered by redefining the property.  If we know the path where the binary reference can be found, we can add it explicitly to the search path by overriding the BeforeResolveReferences target.  Add this target to the end of application.csproj, adjusting the path in line 3 to your local machine (be sure to revert it if you tested the simple solution above):

<Target Name="BeforeResolveReferences">
    <Output TaskParameter="Value"
        PropertyName="AssemblySearchPaths" />

Rebuild application.csproj and you'll find both classlibrary.dll and binaryassembly.dll in the output folder.

Of course, hard-coding paths into the build is a Bad Thing, so choose a more general solution if you have to tread this path too.  I ended up using this little diddy of a target to determine the directories where all assemblies are located, and then prepending that list to the AssemblySearchPaths property:

  <BRSearchPathFiles Include="$(SolutionDir)..\**\*.dll" />

<Target Name="BeforeResolveReferences">
  <RemoveDuplicates Inputs="@(BRSearchPathFiles->'%(RootDir)%(Directory)')">    
    <Output TaskParameter="Filtered" ItemName="BRSearchPath" />

  <CreateProperty Value="@(BRSearchPath);$(AssemblySearchPaths)">
    <Output TaskParameter="Value"
        PropertyName="AssemblySearchPaths" />

Updated to 1.4.5

§ November 17, 2008 18:28 by beefarino |

I just finished updated the site - no major additions, just updated BlogEngine and added some niceties I've wanted for some time like addthis and twitter.  If you can read this, then I guess it's a good sign that things are working!


Effective Retrospectives

§ November 17, 2008 09:29 by beefarino |

Matt Grommes wrote another thought-provoker today on sprint retrospectives; my comments started wandering and found their way into this post...

The few retrospectives I've facilitated have taught me a few tricks.  Matt is correct when he says that a retrospective can turn into monster vetching session if allowed to do so.  In my opinion there are two keys to avoiding this:

  1. provide a structured way for the team to express their input;
  2. timebox everything. 

I've found the following retrospective structure to be effective in producing actionable feedback....


I show up to facilitate a retrospective armed with the following items:

  • a white board with working markers and an eraser;
  • green, pink, and yellow post-its;
  • pens;
  • a kitchen timer;

Sometimes I bring a yummy treat, but I've had that backfire when the sugar rush ebbs and the team starts to wander mentally.  I also try to schedule the retrospective as the last event in the workday for everybody, but early in the week.  This seems to keep the team at ease, but focused.

First: Establish Trust in the Process

Like confessions extracted under coercion, feedback is worthless when obtained from someone who doesn't feel secure enough to be honest.  Agile processes in general rely heavily on open communication, and I found that my non-agile team displayed a lot of mistrust in the retrospective's goals.  

I begin each retrospective with a technique described in Agile Retrospectives: Making Good Teams Great that is aimed at measuring the willingness of the group to communicate.  Each person writes on a post-it a number between one and five inclusive, indicating their willingness to participate in the discussion to follow:

  1. I will not speak;
  2. I will be quiet and let others talk;
  3. Some things I will discuss, some things I will not;
  4. I'll talk about almost anything;
  5. I'll talk about anything.

The anonymous post-its are collected, tallied, and discussed briefly.  Obviously the goal is to have all fives, maybe one or two fours in the mix.  The times I've seen anything else, it's either because a communication problem exists on the team, or the goal of the retrospective is in question (e.g., QA feels like they will be scapegoated for a missed deadline).  

I keep discussion to five minutes, reiterating the retrospective Prime Directive, making sure that everyone knows they are expected to keep the conversation productive, and that the goal is to make the team work together better.

Second: Discuss Good Things

Next I pass out the green post-its and pens, and ask everyone to write down as many things as they can think of that worked well during the sprint.  I ask them to include team and personal efforts,  new or existing processes, anything they can think of that made a positive contribution to the effort.  I use the timer to box the task to 5 minutes.  

I collect the anonymous post-its and start going through them one at a time, collating as necessary.  I read each Good Thing aloud and open the floor for discussion.  I try my best to categorize them on the whiteboard as we go, so the team can see common trends.  For example, several post-its may comment on positive effects realized in documentation and QA from adding a bit of structure to the developer's subversion commit notes.

This part of the retrospective is generally pretty easy.  There isn't a lot of gray area in terms of choosing to maintain a practice that has a positive effect on the team. In addition, I find that discussing the positives first helps the team open up more during the next task...

Third: Collect and Categorize Bad Things

I pass out the pink post-its, set the timer for 5 minutes, and have the team jot down anything they felt impeded their work.  I emphasize "anything" - I would rather someone write down 10 also-rans and 2 gems than to spend the entire time deliberating on what to write at all.  

Against the advice of most retrospective experts, I ask the team *not* to exclude personal remarks about other team members; however, I do remind them that the remarks should be constructive, and that the retrospective is not a performance review.  For example, I would consider this appropriate:

Jim would benefit from writing more unit tests, having regular peer reviews, and mentoring.

but this inappropriate:

Jim's code sucks and never works.

As usual, I collect the post-its anonymously (I like to use a coffee can with a slit in the lid, BTW), during which time I draw two columns on the whiteboard: "Under Our Control" and "Out of Our Control".  I read each Bad Thing aloud and ask the team a single question:

Do we have control over this?

Their answer is usually unanimous, and it dictates which column the post-it falls on the white board.  There is no discussion of the Bad Thing at the point - the purpose of this task is only to isolate whether the Bad Thing is something team can fix themselves. 

Finally: Discuss Solutions and Take Action

Once the team can see what they can control and what they can't, the discussion can begin.  I spend a few minutes on the "Out of Our Control" items, but only to alleviate fears and to keep the discussion solution-focused and positive; I then remove those items from the board.

Moving on to the remaining post-its classified as "Under Team Control," I align them down the left side of the board in no particular order and draw racing lanes across the board for each.  I then ask the team which Bad Thing they believe had the most negative impact on their work, and we start the discussion there.

This is the part where I find the role of facilitator to be most important. It is very easy for the team to drift off-topic, or for to get bogged down in complicated solutions to simple problems.  I find it helps to focus discussion by reiterating one simple question:

What can we do today to prevent the Bad Thing from happening tomorrow?

Most of the time the team will produce a viable solution.  If the team can't gel on a fix, I pass out the yellow post-its, we storm out fixes for a few minutes, collate them in the racing lane, and then discuss.  The point is to keep the conversation on-target and constantly moving forward.  Once the team settles on a solution, I jot down the actionable items in the racing lane.

I repeat this process for each Bad Thing, allowing the team to choose the ordering, spending at most 10 minutes on each one.  If we get through them all in the retrospective's timebox, I'll open the floor to general discussion; however, my experience is that there is little else to discuss at that point if I've done my job.

Offline: Summarize and Reiterate the Solutions

Once the retrospective is over, I write up a short summary.  I list all the Good Things, and all the Bad Things and their proposed solutions.  I send this out to everyone involved, both pigs and chickens.  I also keep a copy handy to remind myself and others of the specific commitments we made as a team to better ourselves.

So there you have it Matt.  In my limited experience, what I find makes an effective retrospective is lots of structure to focus the team on one thing at a time and curb the naturally vetching tendencies.