How we do Git branching

At Headfitted we need to solve typical challenges for our development cycle: versioning our source code, supporting a release/bugfixing cycle, and because we work with distributed development teams, we need a branching model that works well across physical locations and time zones. For this we have settled on Git and GitHub as our preferred version control tools. Taking inspiration from how big open source projects handle challenges much similar to ours and from the excellent A Successfull Git Branching Model post, we have developed a branching model and review workflow that works well for us. Let me share some of the details.

Main branches

Any project will start out with these branches:

These are our main branches. Current development goes into develop, and when we do a release, develop is merged into master. In our TeamCity continuous integration server, there will be a Continuous Integration build that feeds from develop branch and immediately does a build when something is pushed to this branch in the central repository (GitHub). Similarly there is a build that feeds from the master branch and uses our automated deployment system to deploy to production.

We do feature branching. Each feature under development gets it's own branch created, work is done here, and finally the feature branch is merged back into the develop branch using GitHub pull requests.


A feature branch is usually pushed to the central repository (GitHub), which enables more than one developer to work together on a feature, and also enables ad hoc peer-to-peer code sharing. Because switching branches is easy in Git, a developer can easily stash her current work, switch to the feature branch of her colleague, assist on any issue, and later switch back.

One of the challenges of feature branching is keeping in sync with the develop branch, because it will typically move forward in time, while any feature branch is based on the state of the develop branch at the time of the feature branch's birth. We are using our pull request-based review workflow to ensure that the developer in charge of a feature is also responsible for resolving any merge issues:

• Developer finishes feature, makes pull request
• Reviewer uses GitHub to do the review.
• GitHub will tell if the pull request cannot automatically be merged (i.e. cannot be merged back into develop branch without conflicts). In that case the reviewer will ask the developer to do a “reverse merge”.
• A reverse merge means: From the feature branch, merge latest develop into it. Resolve any merge conflicts. Finally push the reverse merged feature branch.

Release Management

Our release management process includes a release candidate branch (rc). When we are preparing the next release, we merge the development branch into rc.


Now any stabilizing of the upcoming release is done on rc. We typically let our build server feed from the rc branch to a build that also makes automated deployment to an rc area, where the project stakeholders can preview, test and verify the upcoming release.
We use the branching to rc to free the develop branch, so work on the next version can immediately start, without depending on the release to be finished.
Once rc is considered stable, we merge to master, which by definition is a release (again, our build system has a build that feeds from master and deploys to production). We also merge back any changes done on rc to the develop branch.

Hotfixing

Whenever a bug is discovered in production (yes it happens even to us), we do hotfixing directly on the master branch.


The hotfix is done just as we do feature branches, except it branches directly of the master branch. Once done and merged back into master, we also merge it into develop to keep things in sync.

Traceability

We couple our branching model with our issue tracking system, so feature branches and hotfixes are named after the issue tracker ids. We have branches with names like feature-2316, bug-4323 or hotfix-3784. Additionally we always refer issue tracker ids in our commit messages, which will automatically be picked up by the issue tracker (setting issues to “ready to test”, “done”, etc.), but that’s a story for another blog post…

Conclusion

Our branching model supports our software development cycle well, and gives us a consistent way of ensuring a frictionless workflow with good traceability and integration between our tools. If you have any comments, questions or experiences from your workflow to share, please don’t hesitate to use the comments.

How we do versioning

Keeping track of version numbers as a part of the release management process seems to be one of these things that should be simple, but is done in so many different ways. Here is the approach we use at Headfitted that works very well for us:

First, let's take a look at semver. This is an initiative of keeping a common standard for version numbers that should be supported whenever possible. In short a version number should look like this:

MAJOR.MINOR.PATCH

This is what we use.
In both our node/Javascript projects and in our .NET projects, we keep the version number at one central place in the source code. The tooling and build scripts do the rest: Making sure this version number is stamped into all compiled binaries and static script files.

Javascript/node
In node/Javscript we have a version.json file simply like this:

{ 'version' : '1.2.0' }

.NET
In .NET we need to do a few tricks to keep things simple. First, in all AssemblyInfo.cs files for our Visual Studio solution, we strip out all version number related stuff, and instead we create one SolutionInfo.cs file that looks like this:

#if DEBUG
[assembly: AssemblyConfiguration("Debug")]
#else
[assembly: AssemblyConfiguration("Release")]
#endif
[assembly: AssemblyVersion("1.2.0.0")]

We don't provide an AssemblyFileVersion, because by omitting this, it will take the same version as stated in AssemblyVersion. Why make it more complicated?

Finally we add a kind of symbolic link to SolutionInfo.cs from each of the projcets of our solution: Add -> Existing Item, and then click the arrow to be able to select Add As Link:

This way we only need to maintain the version number at one place for the entire Visual Studio Solution.

Another obstacle is that in .NET the AssemblyVersion is done like this: MAJOR.MINOR.BUILD.REVISION. So to make it easy we let Microsoft have it's will and let it name it "BUILD", when really it is Semver's "PATCH". Peace. We simply only use the first 3 parts, and always keep the REVISION at 0.

Incrementing
So when should the different parts of the version number be incremented? How and by whom? Semver gives us the answer to the first question:

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.

How and by whom? This is where you need to make some decisions on the practical aspects. In our case we made these:

• MAJOR and MINOR are maintained in the source code.
• PATCH is maintained by the build server.

This works because we run all projects through a build server. In the code, we typically bump the MINOR (or even the MAJOR) at the beginning of a sprint/development cycle. Our commit comments look like this:

Bumped minor version. We are now working on version 1.2.

Now each time our build server makes a build, it will extract the MAJOR.MINOR from the source code, and call a very simple homemade counter service. This service will take any key and maintain a counter for this key. So when we pass the project name and MAJOR.MINOR to it (eg "projectX-1.2"), we might get "1" back, and "2", "3", ... on each subsequent call. This is our PATCH, and now the build server has the complete version number to stamp into the source code before building.

This works beautifully because if we change MAJOR or MINOR in the source code, the key will be new, and PATCH numbering will automatically restart at 0. It also makes sure that each build gets an unique version number, even if we have multiple build configurations (eg. test, staging, release). In our build server (Teamcity), we echo back the version number to the server, so we get a nice overview like this:

We don't check the final version number back into version control. In our source code, version numbers will always have PATCH set to 0. The build server replaces this with the correct PATCH number, builds, and then discards the changed source files. When building locally on a developer machine, PATCH will always be 0. This way we never have version number merge conflicts, and we avoid polluting the commit history with patch increments.

Our approach works well with our branching model, as the MAJOR and MINOR always goes with the source code (and branch). In the above screen dump, you can see how our Release Candidate and master branch is on version 2.16, while the develop branch has progressed and is on 2.17.

The result
We have unique version numbers, we follow common Semver conventions, the version number is always present in our releases, and it works well with our branching model.
Whenever possible, we discretely display the version number on the screen, typically in the footer. This is a great help for our issue tracking, so we can track exactly at what version a given bug started appearing, and we can say from which version the bug was fixed. We also use the version number for update detection, especially in single-page Javascript apps, so we can notify the user if a new version was deployed and the page should be refreshed. With some clever appending the version number to Javascript file names, we can also avoid browser-caching-old-version issues.

Structuring tests using contexts and scenarios

One of the greatest challenges of writing good tests is to keep the tests short and readable. This can be achieved by composition - here is the way I use to structure AAA-style tests:

[Test]
public void ProductValidator_IllegalProductCode_Fails()
{
    // Arrange
    var context = new TestContextBuilder<ProductValidatorContext>()
        .WithScenario(AnonymousProduct)
        .WithScenario(ProductHasIllegalProductCode)
        .BuildContext();
    
    ProductValidator sut = context.CreateSubjectUnderTest();

    // Act
    bool res = sut.Validate();

    // Assert
    Assert.IsFalse(res);
}

The goal is to provide more readable tests and to encapsulate (and reuse) test setup code. As a bonus it will be easy to move between tests and these context/scenario-style tests and using a framework like StoryQ .

Using AAA syntax, the Arrange part of each test uses a specific test context, and aditionally applies one or more scenarios to this context. In the Act part the Subject Under Test is exercised in the context, and in the Assert part the assertions are made, typically on the Subject Under Test or on the context.

The test context:

The test context is the context needed to run a test on the subject under test. In other words, it is the fixed part that stays the same for each test in your test class. You want to keep the context at a minimal level, meaning that the context establishes just enough stuff to be able to create the subject under test. Also the context exposes the things you might want to modify from test to test or make assertions against. We often place the test context class as a private, nested class inside the test class itself.

public class ProductValidatorContext : TestContextBase
{
    public Product TheProduct { get; set; }

    public ProductValidator CreateSubjectUnderTest()
    {
        return new ProductValidator(TheProduct);
    }
}

The scenarios:

Each specific test will use a context builder to build the context. The context builder initializes the context and allows you specify one or more scenarios to apply to the context. Scenarios can be expressed using simple lambda expressíons, methods or classes:

Using lambda:

var context = new TestContextBuilder<ProductValidatorContext>()
    .WithScenario(AnonymousProduct)
    .WithScenario(x => x.TheProduct.Code = "Illegal product code")
    .BuildContext();

Using a method:

var context = new TestContextBuilder<ProductValidatorContext>()
    .WithScenario(AnonymousProduct)
    .WithScenario(ProductHasIllegalProductCode)
    .BuildContext();

...

private void ProductHasIllegalProductCode(ProductValidatorContext context)
{
    context.TheProduct.Code = "Some illegal product code";
}

Using a class:

var context = new TestContextBuilder<ProductValidatorContext>()
    .WithScenario(AnonymousProduct)
    .WithScenari<ProductHasIllegalProductCode>()
    .BuildContext();

...

private class ProductHasIllegalProductCode : TestScenarioBase<ProductValidatorContext>
{
    public override void Apply(ProductValidatorContext context)
    {
        context.TheProduct.Code = "Some illegal product code";
    }
}

Scenario classes can expose properties that are specific to the scenario in case you need to assert against these:

Assert.That(context.Scenario<SomeProductInEditing>.TheProduct.Name, Is.Not.Null);

So typically the context establishes mocks or test instances of all the dependencies of the subject under test. The scenario(s) set the behavior of these mocks or builds the test instances. Here I typically use frameworks like Moq and Autofixture.

I have put the context builder as well as base classes for the contexts and scenarios on Code Gallery.

Best practices for Assert statements in unit tests

Some may find this trivial, but I find myself going over this in almost every code review: You’ve got to pay attention to your unit test assert statements:

Don’t hide the comparison in your assert
Consider this assert statement:

Assert.IsTrue(myObj.Name == "Expected name");

Why is this bad? Take a look at the test runner output when the test fails:

NUnit.Framework.AssertionException:   Expected: True
But was:  False

The comparison is hidden, so the outcome of the test is always a non-informative true/false. Change the statement to

Assert.AreEqual("Expected name", myObj.Name);

This will give the following output when the test fails:

NUnit.Framework.AssertionException:   Expected string length 13 but was 12. Strings differ at index 0.
Expected: "Expected name"
But was:  "Another name"
-----------^

Much more informative, right?

I personally like the alternate fluent assert constructs you can do with NUnit, because it makes you write the assert statement right almost without thinking about it:

Assert.That(myObj.Name, Is.EqualTo("Expected name"));

The assert message should add information or be omitted

How about:

Assert.AreEqual(myObj.Name, "Peter", "Expected name to be Peter.");

The assert message is a waste of typing efforts, because the output of the test runner will tell you anyway. Use the assertion messages only to provide extra information, or omit them:

myObj.DoSomething();
Assert.That(myObj.HasError, Is.True, "Expected HasError to be set because DoSomething produces an error.");

Multiple assert statements

When completing a complex arrange part of a test, it is tempting to make a lot of assert statements (after all this trouble, it’s time to assert to heck out of it, right?) Well not really…

The assert statement is the test. When you look at a test that has multiple assert statements, the test will most likely violate the SRP principle (yes, it also applies to tests), meaning that the test is testing to many things at once. Why is this bad? Because the test will most likely be big and difficult to understand. What will you name the test if it tests 5 different things? Have you seen each of the assert statements fail? If you haven’t, how do you know that the test is working?

Resolution is simply to split it up. You can extract any complex arrange/setup part to a private method and reuse this from your split up tests.

The only case where I see myself diverting from this practice is when doing progressive assertions against the same object. Again it is about making the test informative. Here my expected outcome is that a person with the expected name is added to Persons:

Assert.That(myObj.Persons[0].Name, Is.EqualTo("Expected name"));

Will potentially throw a NullReferenceException or an IndexOutOfRangeException, which doesn’t give a clear clue to what the problem is. Here I would use:

Assert.That(myObj.Persons, Is.Not.Null);

Assert.That(myObj.Persons, Is.EqualTo(1));

Assert.That(myObj.Persons[0].Is.EqualTo("Expected name"));

This will be more informative in case the Persons list is not instantiated or populated.

Conclusion

The general issue here is, that when you are constructing your test, you know exactly what is going on, at that precise moment. It is very easy to be blinded by focusing only on making this test go green. But try to look ahead – in 3 months, this test may fail. Imagine your trusted coworker having to find out what went wrong – fast. That’s why your tests should be easy to understand and informative when failing. And that, my friend, is achieved by paying attention to the assert statement.

Code Coverage analysis for .NET projects in TeamCity 5

I personally don't use test code coverage as a goal in itself, but it is a great tool to find the blind spots - finding classes and methods that don't have coverage, but really should.
Speaking in numbers, I don't set a certain code coverage percentage as a strict required minimum, instead I look for classes or namespaces that falls below a certain treshold, like 60%.

My favorite build server of choice now makes it easier to keep track of the code test coverage. Team City Version 5.0+ has a nice little new part (Build Configuration –> Runner) :

Out of the box TeamCity supports NCover (commercial and community versions) and PartCover.  Let’s look at NCover – there is a free (however old) community edition. NCover provides much more progressed commercial versions if you want to do advanced coverage analysis and Visual Studio integration, but let’s start small with the community edition.

On the build server:
Download and install NCover Community 1.5.8 (you need to register, which is free).
Download and install NCover Explorer 1.4.0.7 (this is just a zip, unzip it to an “Explorer” sub folder to the NCover install location).
Now you just need to set it up from the TeamCity Build Configuration –> Runner page:

A few things to note:
Don’t count on the autodetection of the NCover install files, especially if you are on a 64-bit server. Put those paths in manually.
Enter the assemblies to analyze – without extension, and one on each line.

Now you are ready to go. Run a build and check out the code coverage report:

You can choose between different reports to view assemblies, namespaces, methods and classes to find potential blind spots. I find this a valuable extension to your build server that it is hard to be without, once you have started using it.

Folder structure for a .NET application

I usually stick to the same base folder structure for my .NET applications. This one works well for both my open source projects and enterprise projects at work:

Branch name (usually ‘trunk’)	This root folder contains the solution file.
Application	All the projects of the application goes here. Each project in it’s own subfolder.
Build	I keep all build related stuff here, like: The signing key for the application A C# project holding the main MSBuild script file and any custom MSBuild tasks
Common	Usually not much here, except always a SolutionInfo.cs file.
Documentation	All developer-related documents of the project are kept here. This way a developer can easily get the latest version of a document without leaving Visual Studio. If the project contains auto-generated documentation (for instance using Sandcastle), these generator-projects are kept here as well.
Installers	All installer projects are kept here (Setup projects, WIX projects). Each project in it’s own subfolder.
Lib	This folder contains all the external assemblies used by the project. All Visual Studio projects reference the needed dll’s directly from this folder.
Tests	Contains all test projects, usually divided in unit tests, integration tests and test helpers. Each project in it’s own subfolder.

Note that the above is the physical folder structure. I always make sure that the version control has the excact same folder structure. Never try to build this folder structure from within Visual Studio. Instead build it by hand and add it to source control by hand.

In the main Visual Studio solution file however, I alter the structure slightly by putting the contents of the Application folder directly in the solution root, and mimic the rest of the folders using solution folders:

• Solution root
  • All projects of the Application folder
  • Build
  • Common
  • Documentation
  • Installers
  • Lib
  • Tests

Now, with everything being nicely structured, it’s time to do some coding.

Creating a .NET project’s development environment

This week a co-worker and I set up the envrionment for a new development project. It's starting to look like a pattern. Here is our list:

The development server

We use virtual servers running on some huge hardware farm somewhere. This is nice. We order a new Windows Server instance from our IT services, and within 24 hours it is up and running.

• IIS - used for staging our builds (if web)
• Visual Studio Database Edition GDR2
• SQL Server with:
  • CI database, used as the database for CI build integration tests
  • Staging Database (typically backing a nightly staging build, so the PM can track to progress)
  • Pre-Production Database – should always reflect the scheme of the current in-production build. We use this for schema comparisons.
• Instal Visual SVN (server)
• Install Team City 5.0  with:
  • CI Build
  • Staging Build (usually nightly)
  • Release Build (for the releases)

All the installations are basically "next, next, finish" installations, using default settings.

This setup means that every project has it's own Subversion server and build server running on the dev server. This works well for the small to medium sized projects we are having, and we have had no performance issues for our typical 1-10 team member projects.

The development environment

To keep the development environment consistent for all team members we prepare a complete development environment in a virtual machine, ready for distribution. The install list we're currently using is:

• Visual Studio 2008 (our company issue license is the Team Developer edition)
• Visual Studio Database Edition GDR2
• Visual SVN + Tortoise SVN
• ReSharper
• StyleCop
• StyleCop for ReSharper
• GhostDoc
• NUnit
• SQL Server Developer Edition
• Team City Visual Studio add-in
• Team City tray notifier
• WIX (Windows Installer Xml toolset)

After setting up the development environment, we run a sysprep and close it down, compress and distribute. The way sysprep works is that when the developer starts up this virtual machine for the first time, she is presented with a setup wizard resembling the last install steps in a clean Windows install. She can then give the virtual machine a name, set administrator password, etc. The virtual machine will now be unique.

I intentionally left our the version numbers for Windows Server, IIS and SQL Server. We used 2003/6.0/2005 for the latest projects, and are hoping that the next customer's hosting provider gives green light for 2008 versions.

Now we can start coding…