So, funny story in honor of Ada Lovelace day.

My first job was pretty amazing.  I worked with a bunch of really smart people doing really smart things.  Space things.  And optics things.  Now my role on this team was fairly meager – I basically helped the documentation team try and capture all of the smart things the other smarter people were doing – but it was still amazing to be around that stuff.  Space telescopes, satellites, cell phones, automotive components, this team helped design all sorts of useful and interesting crap.

Anyway, one woman on the engineering team – we shall call her Mary although that’s not her name – had a very sharp sense of humor and keen hearing, both of which come into play in this tale.  One evening we were out celebrating …. something – a release perhaps?  maybe a project wrap-up?  There was drinking and nogoodnicking.  One of my outside-of-work friends from my previous stint in grad school was there – let’s call him Joe.  This evening Joe happened to be hitting on Mary.  Hard.  I watched from my bar stool several meters away while he strutted about her like a peacock with a spare ego.  Mary was unimpressed and Joe eventually took the hint.

So Joe politely adjourns from Mary and joins me at the bar.  I asked him, “So, what do you think of Mary?  She’s kind of awesome, right?”

Joe sighed and said, “Yeah, she’s cute.  But she’s no rocket scientist.”

I started laughing.  That loud obnoxious laugh you may know.  Joe wanted to know why.  So I turned around and shouted, “Hey Mary, what is it you do for a living?”

She finished her beer, slammed the stein on the table, stood up to look Joe square in the eye and as she was leaving, she said:

I’m a fucking rocket scientist.

And the best part?  It was true.  100% honest.  She did rockets for a living.  A literal rocket scientist that sends things into space.  Ego Joe never stood a chance.

Re-Entry

So originally this post was a bit different.  It wasn’t a story, it was a rant.  I didn’t like the rant so I wanted to turn it into something positive. 

Here’s the thing – Ada Lovelace is important, and I’ve told my girls about her because I want them to know that women have had a profound impact on society.  Having that history is vital for them I think – I do believe the technical contributions from women are under-celebrated and often deliberately downplayed.  So I want them to know about Ada and what she contributed, and that we can literally thank her for my career and our family’s happiness.

But I know that won’t do anything real for them.  I mean, why should I expect stories of Ada to have a significant impact on my girls, when similar stories about C.S. Babbage had fairly milquetoast effects on me when I was their age.  Interesting, perhaps necessary, but it won’t light their fuse.

Which is why I spend so much time finding Marys.  Living people my girls can see doing real and amazing things, people they can talk to about wonders and interests.  Sure I want them to know that “once upon a time there was Ada,” but I think knowing that there is Mary right here, right now, doing really amazing stuff, is far more important to their mindset.

My biggest fear for my daughters is that they will come to believe that they can’t do something because of their gender or sex.  My biggest tool to counteract that fear, is to show them someone who can – someone like Mary.

I’ve spent the last week slammed with unplanned work for a client.  No lie, it sucked.  In a nutshell, the system was asked to perform at a new scale it was simply not ready to handle.  Having read The Phoenix Project recently, I was especially cognizant of the loss of time caused by us flailing around reactively. 

As happy as I am to say that the system is now coping with the extra load, getting there was hell.  And on inspection the problem was obvious once you saw it at the proper scale.  That is, the problem didn’t become a problem until the load increased.

And that’s how performance problems tend to manifest.  Things are fine at one scale, but broken at another.  It reminds me of this magnet my mom used to keep on the refrigerator that read:

It’s not the mountain that wears you out, it’s the grain of sand in your shoe.

Which is so true: you don’t even notice that grain of sand until you’ve walked miles, cut up the bottom of your foot, and bloodied your shoe.  Then it’s an obvious thing – it’s painful and you can’t ignore it.

Thinking in Scales

Like anything, achieving performance is a balance.  You want the best possible performance for the least cost and effort.

The way I tend to think about scales – whether we’re talking about time, people, requests, whatever – is by powers of growth.  For instance, imagine planning a party for a group of people, and consider the challenges as the guest list grows to:

• 1 person
• 10 people
• 100 people
• 1000 people
• 10000 people
• 100000 people
• … and so forth

That is, planning a party for 200 people isn’t that much different than planning for 100, but planning for 1000 is WAY harder than planning for 100.

For time scales, I find it helpful to think in the following units:

• nanoseconds
• milliseconds
• seconds
• minutes
• hours
• days
• weeks
• months
• years
• decades
• centuries
• millennia
• geologic time

For example, during planning meetings you’ll frequently hear me ask “are we talking hours, days, weeks, or months?”

Now in the realm of scaling software, here is the money question:

At what scale is the system be expected to operate?

An application that needs to support 100 operations every second can be designed very differently than an application that needs to support 1000 operations every second.  The logic applies to bugs too: small performance issues that can be ignored at a small scale become debilitating as you scale up your effort. 

Just like that grain of sand in your shoe – it probably won’t be an issue while you’re fetching your mail, but you might want to address it before you head up Grandfather Mountain.

Those of you that know me well know I don’t have many strong opinions.  I tend to keep multiple perspectives and work in whatever limits are provided.  So when I do express a strong opinion, it tends to be backed up with experience and reason (or at times, alcohol).

With that out of the way, let me express the only feature I really desire in the next version of Visual Studio:

Replace the format of all project and solution files with PowerShell scripts.

I hear you groaning – just hear me out.  I have many reasons for wanting this – too many to list all but the highlights here.  In a nutshell they all boil down to the notion of simplicity.

Expressing Data and Logic

A build has two broad parts: data that defines WHAT to build, and logic that defines HOW to build it.  E.g., your typical C# project file contains a list of source files and project/assembly references, and a set of instructions for accomplishing specific things with that data – such as producing assemblies or deploying a website.

It’s simple to express data in PowerShell.  You can declare arrays and hashtables inline.  You can create complex object hierarchies if needed. 

It also simple to express data in XML.  After all, that’s what XML is for.

It’s simple to express logic in PowerShell; like data and XML, expressing logic is what a programming language is designed to do.  However, expressing logic in XML is … well, “cumbersome” is a generous word.  “Obtuse” is perhaps a better choice.

Customizing the Build

It gets worse when you require custom build steps.  Getting logic into the MSBuild XML schema requires a lot of ceremony that serves nothing outside of MSBuild.  Consider the process:

1. Implement your build task in C# (pulling in all the necessary bits to make MSBuild recognize your task)
2. Import the DLL into your project using the <Import /> element
3. Add the XML necessary to get your task working in the build
4. Reload the project in Visual Studio
5. Address any security warnings that pop up related to the new “unknown” task you added

At this point, the logic of your custom build task is about as far away from the build as it can get – inside of a binary dll that now must be packed around with the project file.  You have literally no inroad to the task in the same band as the build – all information about using the task (its name, dependencies, parameters, outputs, etc) must be communicated elsewhere.

Now, look at customization process for the PowerShell project file:

1. Modify the PowerShell build script
2. There is no step 2.  Again, simplicity.

Moreover, PowerShell is explicitly transparent – documentation is a get-help command away, and the built-in discovery mechanisms let you know what’s there.  Plus, the build logic stays with the build.

Running the Build

Here’s where my opinion really polarizes.  I get irate when a software project can only build inside Visual Studio.  Recent examples of my experiences here include Azure deployments and SCOM management packs.  It’s not that I mind the experience of pushing the Deploy button and having a magical process ensue that mystically transfers and configures an entire website and database – quite the contrary I want that experience.  The thing is, I want it everywhere, not just in Visual Studio.  What exactly constitutes “everywhere?”  For starters, I want the same build experience in:

• Visual Studio
• the shell
• my co-worker’s machine
• a fresh VM image
• the automated CI server or build farm

If the build was “just PowerShell,” this would be a piece of cake.  In fact, I’ve taken to using PSake to drive my builds these days for this very reason – so I can maintain a consistent expectation of the build from one location to the next. 

What it Might Look Like

I dunno.  Let’s see what I can do with a default CSharp class library project.  Here’s the original MSBuild project file:

   1: <?xml version="1.0" encoding="utf-8"?>

   2: <Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

   3:   <PropertyGroup>

   4:     <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>

   5:     <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>

   6:     <ProductVersion>8.0.30703</ProductVersion>

   7:     <SchemaVersion>2.0</SchemaVersion>

   8:     <ProjectGuid>{5C818DB6-C86B-4B05-AF13-A4CF46C8ACA9}</ProjectGuid>

   9:     <OutputType>Library</OutputType>

  10:     <AppDesignerFolder>Properties</AppDesignerFolder>

  11:     <RootNamespace>ClassLibrary1</RootNamespace>

  12:     <AssemblyName>ClassLibrary1</AssemblyName>

  13:     <TargetFrameworkVersion>v4.0</TargetFrameworkVersion>

  14:     <FileAlignment>512</FileAlignment>

  15:   </PropertyGroup>

  16:   <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">

  17:     <DebugSymbols>true</DebugSymbols>

  18:     <DebugType>full</DebugType>

  19:     <Optimize>false</Optimize>

  20:     <OutputPath>bin\Debug\</OutputPath>

  21:     <DefineConstants>DEBUG;TRACE</DefineConstants>

  22:     <ErrorReport>prompt</ErrorReport>

  23:     <WarningLevel>4</WarningLevel>

  24:   </PropertyGroup>

  25:   <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">

  26:     <DebugType>pdbonly</DebugType>

  27:     <Optimize>true</Optimize>

  28:     <OutputPath>bin\Release\</OutputPath>

  29:     <DefineConstants>TRACE</DefineConstants>

  30:     <ErrorReport>prompt</ErrorReport>

  31:     <WarningLevel>4</WarningLevel>

  32:   </PropertyGroup>

  33:   <ItemGroup>

  34:     <Reference Include="System" />

  35:     <Reference Include="System.Core" />

  36:     <Reference Include="System.Xml.Linq" />

  37:     <Reference Include="System.Data.DataSetExtensions" />

  38:     <Reference Include="Microsoft.CSharp" />

  39:     <Reference Include="System.Data" />

  40:     <Reference Include="System.Xml" />

  41:   </ItemGroup>

  42:   <ItemGroup>

  43:     <Compile Include="Class1.cs" />

  44:     <Compile Include="Properties\AssemblyInfo.cs" />

  45:   </ItemGroup>

  46:   <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />

  47:   <!-- To modify your build process, add your task inside one of the targets below and uncomment it. 

  48:        Other similar extension points exist, see Microsoft.Common.targets.

  49:   <Target Name="BeforeBuild">

  50:   </Target>

  51:   <Target Name="AfterBuild">

  52:   </Target>

  53:   -->

  54: </Project>

And here’s some PowerShell I hacked up to represent the same thing:

   1: param(

   2:     $configuration = 'debug',

   3:     $platform = 'anycpu'

   4: );

   5:  

   6: $outputType = 'library'

   7: $projectGuid = [guid]::newguid('5C818DB6-C86B-4B05-AF13-A4CF46C8ACA9');

   8: $targetFrameworkVersion = 'v4.0'

   9:  

  10: $errorReport = 'prompt';

  11: $warnLevel = 4;

  12:  

  13: switch ( "$configuration|$platform" ) {

  14:     'debug|anycpu' {

  15:         $optimize = $false;

  16:         $outputPath = 'bin\debug';

  17:         $defines = 'debug','trace';

  18:         

  19:         $debugSymbols = $true;

  20:         $debugType = 'full'; 

  21:     }

  22:     

  23:     'release|anycpu' {

  24:         $optimize = $true;

  25:         $outputPath = 'bin\release';

  26:         

  27:         $debugSymbols = $true;

  28:         $debugType = 'pdbonly'; 

  29:     }

  30:     

  31:     default {

  32:         throw "$buildFlavor is not a valid build configuration"

  33:     }

  34: };

  35:  

  36: $assemblyReferences = @(

  37:     "System", "System.Core", "System.Xml.Linq", 

  38:     "System.Data.DataSetExtensions", 

  39:     "Microsoft.CSharp", "System.Data", "System.Xml"

  40: );

  41:  

  42: $sourceFiles = @(

  43:     "Class1.cs",

  44:     "Properties\AssemblyInfo.cs"

  45: );

  46:   

  47: import-module psbuild;

  48: function invoke-BeforeBuild {}

  49: function invoke-AfterBuild {}

Granted, I’m not putting much thought into this conversion – but even as is, this script lends itself to many more possibilities than its MSBuild counterpart.  For example, if you wanted to run your own static analysis on the source code for the project, you could dot-source this file and reference the $sourceFiles variable in your own script…

Anyway, opinion expressed.  Back to work.

I’ve been pretty quiet lately and for good reason.  My company (Code Owls LLC) has just released their first commercial software product. 

The product is SeeShell, and it aims to fill a void that exists in the PowerShell space: visualizing data.

PowerShell gives you access to tons of data sources: WMI objects, performance counters, event logs, files, databases, system events, etc.  And with PowerShell remoting these data sources can be remote, multiplying the amount of data accessible by the size of your network.  Clearly getting the data is no longer a problem – PowerShell has evolved to fill this gap.

PowerShell is still limited in the ways it can show this data: tables and lists.  To be sure, these are very useful, but there are times when they are simply the wrong choice.

An Example of Awesomeness

Ever wanted to correlate an event log against a performance counter?  I’m not talking about a systems monitoring or Big Data situation – just a time when you wanted to snoop out a performance counter’s relationship to a set of events that appear in the event log.   Perhaps you have a hunch and just want to confirm it.  Maybe an app is consuming memory at specific events and never releasing it.

Think about how you might solve that problem on your own for a minute … no seriously, think about it for a minute …

… and then see how the problem is solved using SeeShell.  This script:

   1: import-module seeshell

   2:  

   3: {

   4:   get-counter '\memory\available bytes' -continuous | `

   5:     select -expand countersamples

   6: } | out-chart -name memory -type spline -plot CookedValue -by Timestamp

   7:  

   8: {

   9:   $i = 0;

  10:   while( $true ){

  11:     $e = get-eventlog -log application -source MyApp -newest 1 | `

  12:       where {$_.index -gt $i };

  13:  

  14:     if( $e ) {

  15:       $i = $e.index;

  16:       $e

  17:     }

  18:   }

  19: } | out-chart -name memory -type timeline -plot Message -by TimeGenerated

produces this visualization:

that shows a clear relationship between the Starting Operations event logged by the application and a massive consumption of memory (around .5 Gigabytes) that immediately follows it. 

The visualization overlays the events logged by MyApp on to the performance counter data values. The events and the counter share a common axis: the timestamp of the counter sample or event.  SeeShell is smart enough to see that the X axis of the counter and the X axis of the event series are both datetimes, and assumes that you want them plotted on a common scale.

This is one small example of what SeeShell can do.  If you think this is cool, head on over to the SeeShell product demos page and check out the videos there.  Your mind will explode with joy.

Rationale

I’m a visual thinker – I find it far easier to intuit the meaning of data when it is presented visually than numerically.  Gauging the reactions of those who’ve seen a SeeShell demo, I can see that there are many others in the same boat.

I’m also lazy – not the procrastinating or passive-aggressive kind of lazy, but the good kind of lazy that makes me want to have flexible tools I can bend to my current needs.  It’s the reason I’ve globbed on to PowerShell for most of my software tooling on Windows. 

PowerShell provides a platform where:

• tools are small and simple
• each tool does one thing very well
• simple tools can be composed to solve complex problems

SeeShell aims to be one of those simple tools, nothing more.