Using PowerShell Scripts as SVN Hooks

§ December 2, 2008 18:18 by beefarino |

You can tell from my recent blog posts that I'm spending a good deal of time in my SCM role at work.  I've recently been looking at using powershell scripts to replace some binary hooks that have become dated, and to expand our repository's integration with our bug tracking system.  All in all it was pretty simple, but a few gotchas held me up:

First, SVN hooks have to be directly executable (e.g., .exe, .bat., .com, .cmd, or windows script files); powershell scripts are not directly executable for security reasons, so I had to use a batch script as a proxy.  

Second, SVN invokes the hooks with an empty environment (again for security purposes); without an active PATH variable specified, the batch and powershell scripts need to specify full paths to all files.

Finally, some hooks, such as the post-revprop-change hook, read data from STDIN, but the proxy batch file will not forward its STDIN to the powershell process without some intervention.  

Here is the batch proxy for the post-revprop-change hook:

rem   =====================================================================
rem    %  Argument     Description
rem   --- ------------ ----------------------------------------------------
rem   [1] REPOS-PATH   (the path to this repository)
rem   [2] REV          (the revision that was tweaked)
rem   [3] USER         (the username of the person tweaking the property)
rem   [4] PROPNAME     (the property that was changed)
rem   [5] ACTION       (the property was 'A'dded, 'M'odified, or 'D'eleted)
rem   [STDIN] PROPVAL  ** the old property value is passed via STDIN.
rem   ---------------------------------------------------------------------

c:\windows\system32\windowspowershell\v1.0\powershell.exe -command "$input | d:\repository\hooks\postrevprop.ps1" -args "%1" "%2" "%3" "%4" "%5"

A few things to note:

  • I'm using full paths to the powershell executable and the powershell script; this is necessary because the batch file will be run with an empty environment and no PATH to search.
  • I'm not really executing the powershell script, I'm executing a command string that passes the implicit $input variable to the powershell script.  This $input variable resolves to the STDIN for the batch proxy, and enables the powershell script to read it.
  • Because my command string invokes a script file, I need to make sure that the powershell execution policy is set to RemoteSigned or Unrestricted on the SVN server.    

Here is the powershell script invoked by the batch file proxy:

param( $repo, $rev, $user, $propname, $action );
@($repo, $rev, $user, $propname, $action) + $input
    | out-file "d:\repository\hooks\log.txt" -append

The script doesn't do much at the moment - just logs the hook parameters and input to a file so I can verify that the script is executing correctly.  Now that I have ironed out the major wrinkles, I can start porting our hooks to powershell!

Continuous Integration Timeline

§ November 27, 2008 18:01 by beefarino |

We've been using CruiseControl.NET at my shop for a few years.  One of the things I've never really enjoyed about the CC.NET dashboard is the lack of a recent build history for the entire build farm.  When someone reports an issue with a build, I have use the CC.NET dashboard to drill through every VM in the farm until I figure out which server built the revision I'm looking for.  It's a waste of time.  Moreover, the default farm view is really really boring - don't get me wrong, it's vital to communicate the current state of the builds, but I want more.  I want to be able to show my boss how much work is being done, how much coordination is taking place on the team, and how badly we need another VM host in the farm.  

So I spent a little spare time on a pet project: a CC.NET dashboard plugin that visualizes the build farm activity using the SIMILE Timeline project.  Before I dig into the grit, take a quick look at this sample timeline; it's a snapshot of about 4 hours of activity from one of our build VMs:


The plugin only took an hour or so to pull together; of course, that was after an evening of digging through the CC.NET source code to figure out NVelocity and the Objection IoC container voodoo.  The project is still very raw - there is a lot of missing error checking and no unit tests.  I'm offering up the source code for download anyway in order to get some feedback: (6.69 kb)

Of course, it comes as-is with no commitment of support or fitness of purpose, etc.  There are no special build requirements, outside of VS 2005 or later and the CC.NET source code.


To install the plugin:

  1. copy ccnet.timeline.plugin.dll from the project output folder into the CC.NET dashboard bin directory;
  2. copy the timeline.vm template from the project template directory into the CC.NET dashboard template directory;

Configure the Dashboard

To configure the dashboard:

  1. modify the list of of HttpHandlers in the CC.NET dashboard web.config to include the timeline.aspx handler shown below; make sure it appears first in the list:
    				<httpHandlers> 	  <add verb="*" path="timeline.aspx" type="ccnet.timeline.plugin.TimelineHttpHandler,ccnet.timeline.plugin"/> 	  <add verb="*" path="*.aspx" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard"/> 	  <add verb="*" path="*.xml" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard"/> 	</httpHandlers>
  2. add the farmTimelinePlugin element to the list of farm plugins in the dashboard.config file:
    				<farmPlugins> 	  <farmReportFarmPlugin /> 	  <farmTimelinePlugin /> 	  <cctrayDownloadPlugin /> 	</farmPlugins> 	

This will result in a "Timeline" link in the farm view of the CC.NET dashboard that will present the farm timeline.

Configure the Projects

The timeline is driven by data from the statistics publisher, so any project you want to include on the timeline will need to publish statistics.  To enable this, just add a <statistics/> element to the project's list of publisher tasks, taking care that it follows all file merge tasks:

      <file>c:\program files\\server\build-server.xml</file>
  <xmllogger />
  <statistics />

More Features

This initial spike is very promising.  Here are a couple of ideas I have to make the timeline better:

  • the timeline uses color to indicate build success or failure, but it would be nice to propogate any build errors into the event on the timeline, so you could see the error by clicking on the event.
  • operationalizing the event generation configuration - as it works now, there is no customization points, so what you see it what you get.
  • create timeline server and project plugins.
  • create a timeline view of a build report: e.g., display individual build tasks, tests, etc.
  • the ability to mark public releases ( or any non-CC.NET event ) on the timeline.  This would help demonstrate the "mad dash" effect an approaching release date has on our team.
  • display source control activity.
  • parameterizing the timeline view - maybe controlling the projects displayed, hilighing certain projects, or adjusting the units in the timeline band to compact or expand the display.  

As the project matures and stabilizes, I'll post updates.  If there is enough interest, I'll start looking into making a full contribution to the CC.NET project.  Let me know if you find this useful, and if you have any ideas for improvements.  

Resolving Binary References in MSBuild

§ November 20, 2008 17:49 by beefarino |

An acquaintance of a friend tweeted about a problem he's having with MSBuild: it often fails to resolve nth-tier binary dependencies of a project:

The application references a binary directly, which makes the class library a first-tier dependency of the application.  The class library references another binary assembly, which becomes a second-tier dependency of the application.  

Here's a Visual Studio solution with the same dependency configuration: (31.73 kb)

Build the application (use the Debug configuration) and check out the application's output path.  You would expect to find application.exe, classlibrary.dll, and binaryassembly.dll, but binaryassembly.dll is missing.  However, if you scrutinize the MSBuild output during the application.csproj build (use the /v:d switch), you can see that it's trying like mad to find binaryassembly.dll:

Project "D:\Project\Dump\msbuldreferencedemo\application\application.csproj" on node 0 (Rebuild target(s)).
Dependency "BinaryAssembly, Version=, Culture=neutral, PublicKeyToken=null".
  Could not resolve this reference. Could not locate the assembly "BinaryAssembly, Version=, Culture=neutral, PublicKeyToken=null". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.
  For SearchPath "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug".
  Considered "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug\BinaryAssembly.exe", but it didn't exist.
  Considered "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug\BinaryAssembly.dll", but it didn't exist.
  For SearchPath "C:\Project\Dump\msbuldreferencedemo\BinaryAssembly\bin\Debug\".
  Considered "C:\Project\Dump\msbuldreferencedemo\BinaryAssembly\bin\Debug\BinaryAssembly.exe", but it didn't exist.
  Considered "C:\Program Files\Microsoft SQL Server\90\DTS\ForEachEnumerators\BinaryAssembly.exe", but it didn't exist.
  Considered "C:\Program Files\Microsoft SQL Server\90\DTS\ForEachEnumerators\BinaryAssembly.dll", but it didn't exist.
  For SearchPath "{GAC}".
  Considered "BinaryAssembly, Version=, Culture=neutral, PublicKeyToken=null", which was not found in the GAC.
  For SearchPath "bin\Debug\".
  Considered "bin\Debug\BinaryAssembly.exe", but it didn't exist.
  Considered "bin\Debug\BinaryAssembly.dll", but it didn't exist.
  Required by "D:\Project\Dump\msbuldreferencedemo\classlibrary\bin\Debug\ClassLibrary.dll".

MSBuild doesn't know where to look for the file.  My team hit this problem after we started replacing projects in our source hive with prebuilt binaries; the builds would succeed, but the applications failed to run because none of the second-tier dependencies of a first-tier binary reference would show up in the output folder.  I did some digging and found one simple way and one complex way to resolve the issue.

Easy: Put All Binaries in One Place

The easiest way to address this problem is to place all binary dependencies into a single folder.  If you do reference a binary from this folder, and the folder contains the binary's dependencies, those second-tier dependencies will be found during the build.  In the project file, the first-tier binary includes a hint path so the compiler knows where to look for the file.  For example, open application.csproj and change the hint path for the classlibrary reference to point to the \lib folder (line 5):  

  <Reference Include="ClassLibrary, Version=, Culture=neutral, processorArchitecture=MSIL">
  <Reference Include="System" />

Rebuild the project and you'll find both classlibrary.dll and binaryassembly.dll in the output folder.  This hint path is used as one of the search paths when MSBuild attempts to resolve a dependency during the build, and since the hint path also contains binaryassembly.dll, MSBuild is able to find that second-tier dependency as well.

Hard: Modify the Assembly Search Path List

This is the solution we had to use on my team.  The choice was made to keep each binary in a unique folder, so interdependencies between the binaries could not be managed with just the hint paths. 

The magic starts in Microsoft.Common.targets, which you will find under %FRAMEWORKDIR%\v2.0.50727\.  In that file there is a series of targets aimed at resolving different kinds of references in your project, from project references to COM references to assembly references.  The target that holds the key is ResolveAssemblyReferences, which is nothing more than a wrapper around the ResolveAssemblyReference task.  This task accepts a parameter named SearchPaths that determines where looks for assemblies.  The value of this parameter is determined by the AssemblySearchPaths property, which is created earlier in Microsoft.Common.targets:

The SearchPaths property is set to find assemblies in the following order:

    (1) Files from current project - indicated by {CandidateAssemblyFiles}
    (2) $(ReferencePath) - the reference path property, which comes from the .USER file.
    (3) The hintpath from the referenced item itself, indicated by {HintPathFromItem}.
    (4) The directory of MSBuild's "target" runtime from GetFrameworkPath.
        The "target" runtime folder is the folder of the runtime that MSBuild is a part of.
    (5) Registered assembly folders, indicated by {Registry:*,*,*}
    (6) Legacy registered assembly folders, indicated by {AssemblyFolders}
    (7) Look in the application's output folder (like bin\debug)
    (8) Resolve to the GAC.
    (9) Treat the reference's Include as if it were a real file name.
  <AssemblySearchPaths Condition=" '$(AssemblySearchPaths)' == '' ">

The value of this property is a semicolon-delimited list of paths and search targets.  This list can be easily altered by redefining the property.  If we know the path where the binary reference can be found, we can add it explicitly to the search path by overriding the BeforeResolveReferences target.  Add this target to the end of application.csproj, adjusting the path in line 3 to your local machine (be sure to revert it if you tested the simple solution above):

<Target Name="BeforeResolveReferences">
    <Output TaskParameter="Value"
        PropertyName="AssemblySearchPaths" />

Rebuild application.csproj and you'll find both classlibrary.dll and binaryassembly.dll in the output folder.

Of course, hard-coding paths into the build is a Bad Thing, so choose a more general solution if you have to tread this path too.  I ended up using this little diddy of a target to determine the directories where all assemblies are located, and then prepending that list to the AssemblySearchPaths property:

  <BRSearchPathFiles Include="$(SolutionDir)..\**\*.dll" />

<Target Name="BeforeResolveReferences">
  <RemoveDuplicates Inputs="@(BRSearchPathFiles->'%(RootDir)%(Directory)')">    
    <Output TaskParameter="Filtered" ItemName="BRSearchPath" />

  <CreateProperty Value="@(BRSearchPath);$(AssemblySearchPaths)">
    <Output TaskParameter="Value"
        PropertyName="AssemblySearchPaths" />

Updated to 1.4.5

§ November 17, 2008 18:28 by beefarino |

I just finished updated the site - no major additions, just updated BlogEngine and added some niceties I've wanted for some time like addthis and twitter.  If you can read this, then I guess it's a good sign that things are working!