Moving from subversion to git can be a struggle, trying to understand what terms like checkout, commit, branch, remote, rebase all mean in the git world. I learned by experimenting in a demo repository, trying out various commands, and using gitk to visualize their impact. This post is broken up into two parts – after reading this, you may want to read the second part.

The gitk screen

I created a simple repository on github to walk through some scenarios. I'll start by creating a local copy of the repository:

d:\code>git clone git@github.com:joshuaflanagan/gitk-demo.git
Initialized empty Git repository in d:/code/gitk-demo/.git/
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 9 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.

d:\code>cd gitk-demo

d:\code\gitk-demo>gitk --all

There is a lot of information in this single screenshot:

1. The upper left pane shows the series of commits to this repository, with the most recent on top.
2. There have been three commits, all by Tony Stark.
3. The commit message for the most recent commit was "third commit"
4. There is a single local branch, named "master'", it points to the most recent commit
5. There is a single remote reference branch: the "master" branch from the remote repository named "origin", it also points to the most recent commit
6. The yellow dot next to the top commit indicates that is the snapshot currently in my working folder (referred to as HEAD)
7. I've highlighted the second commit, so that I can see its details in the lower pane
8. The commit SHA (unique identifier, similar to subversion revision number) of the second commit is 3d024dd9e4a83d8c6a9a143a68b75d4b872115a6
9. The lower right shows the list of files impacted by the second commit
10. The lower left shows the commit details, including the full diff
11. Clicking a file in the lower right pane scrolls the diff in the lower left pane to the corresponding section

A note about "master" and "origin"

When you first create a git repository, it starts with a single branch named "master". There is nothing special about this branch, other than it is the default. You are free to create a new one, and delete master (although, I don't see any reason to go against the default convention).

When you first clone a git repository, git will automatically create a remote for you named "origin". A remote is just a name used to manage references (URLs) to other repositories. There is nothing special about the "origin" remote, other than it is created for you. You are free to create a new one and delete origin. In fact, if you are working with multiple remotes, I recommend you delete the origin remote and create a new one for the same repository, but using a more descriptive name. For example, when I work with the FubuMVC source code, in my local repository I have a remote named "darth" which refers to the main repository owned by DarthFubuMVC, and a remote named "josh", which refers to my fork. If I had kept the name "origin", I would always have to remember which one I cloned from.

Branching

What happens when I create a branch?

d:\code\gitk-demo>git branch issue123

Press CTRL-F5 in the gitk window to refresh the repository view

We see the new branch marker for the issue123 branch points to the same commit as master and origina/master. It is important to note that the "master" is bold, indicating that is still the current branch. The bold branch label is equivalent to the asterisk in the command line output:

d:\code\gitk-demo>git branch
  issue123
* master

Now if I switch to the new branch and refresh gitk:

d:\code\gitk-demo>git checkout issue123
Switched to branch 'issue123'

(We're going to focus on information in the top pane from now on, so I'll hide the bottom part of gitk)

Note: For convenience, I could have created and switched to the new branch in a single command: git checkout –b issue123

Making changes

When I refer to the "current branch", I mean "the branch that will advance when I perform a commit". This is where the gitk visualization really starts to help. I'll make some changes to a file and then commit with the message "My first commit":

d:\code\gitk-demo>git commit -am "My first commit"
[issue123 f948bf8] My first commit
 1 files changed, 2 insertions(+), 1 deletions(-)

The issue123 branch now points to my new commit. Neither the master nor origin/master branch pointers have moved.

As I continue to commit, the current branch pointer moves with me:

d:\code\gitk-demo>git commit -am "Added another fruit"
[issue123 cac3c72] Added another fruit
 1 files changed, 1 insertions(+), 0 deletions(-)

But I thought it was a branch

Since I was working in a branch, I expected to see a tree stucture, with nodes turning off from the main "trunk". Something like this:

Instead, gitk shows all of the commits as a single straight line. When first using git, this was very confusing to me. My confusion stemmed from my misunderstanding of branches in git. Thinking about why gitk was showing all of the commits in a straight line finally brought the point home. In git, a branch is a label for a commit. The label moves to new commits as they are created. When you create a git branch, you are not changing anything in the structure of the repository or the source tree. You are just creating a new label.

Fast forward

After completing my work in the issue123 branch, I'll want to merge the changes back into master. Usually when I think of a merge, I think of comparing two trees and applying the differences from one onto the other. I imagine each commit being replayed on the other branch. Merging issue123 into master would require applying each of my two commits to the master branch. However, this work has already been done, when I first performed the commits. Because the master label hasn't moved since my work began on issue123, applying the diffs would end up with the same result. This is where the "single straight line" visualization really proves valuable – I can see that issue123 is directly ahead of master. Git is smart enough to recognize this situation and performs what it calls a fast-forward merge. A fast-forward merge isn't really a merge at all - since no file content comparisons are necessary – it simply moves the master branch label up to point to the commit pointed at by the issue123 label.

To merge the changes from issue123 into master, I first checkout (switch to) the master branch and then do the merge:

d:\code\gitk-demo>git checkout master
Switched to branch 'master'

d:\code\gitk-demo>git merge issue123
Updating bf37c64..cac3c72
Fast-forward
 fruits.txt     |    1 +
 vegetables.txt |    3 ++-
 2 files changed, 3 insertions(+), 1 deletions(-)

A few things to notice:

• The command-line output indicated the merge was a "Fast-forward".
• A new commit was not created. There is no new snapshot of the repository required, since there is no new version of any files/folders that didn't already exist in the repository.
• The remote origin/master branch has not moved. Everything we've done so far (except for the initial clone) has run completely locally, without contacting the github server.

Deleting a branch

The issue123 branch label is now redundant, since it points to the same commit as master. If there is no more work to do for issue123, we can safely get rid of the branch, without losing any historical information. If we later find out we need to make some changes to solve the issue, we can always create another branch (which is just a label). This is what it means when people say that branching in git is easy or lightweight.

d:\code\gitk-demo>git branch -d issue123
Deleted branch issue123 (was cac3c72).

Sharing with the world

As noted above, everything we've done so far has been in our local copy of the repository. The "master" branch at the "origin" remote has not moved. If I look at the github page for the repo, I can confirm that none of my commits exist.

To copy changes from my instance of the repository up to github's servers, I need to push from my master branch to the "master" branch of my remote named "origin".

d:\code\gitk-demo>git push origin master
Counting objects: 9, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 609 bytes, done.
Total 6 (delta 0), reused 0 (delta 0)
To git@github.com:joshuaflanagan/gitk-demo.git
   bf37c64..cac3c72  master –> master

Take note of the SHA1 ID of the latest commit, cac3c723.... Look back at the previous screenshots and notice that this identifier has not changed through all of the operations (merge, deleting the branch, etc). When we refresh the github page, we can see it has updated with all of the work I did locally. Notice the commit identifier on the web page matches up with the SHA1 ID we see locally. Also, there is no indication that any of the work was done on a separate branch – nobody ever needs to know. You are free to branch as much as you want locally without impacting a shared repository.

To be continued

Continue to part 2 to see the difference between merge and rebase.

In one picture:

In Hg, branches are metadata included in each changeset.  In the operation above, I created a branch, but that only marked the current directory with a branch name.  The branch won't actually show up unless I make a commit.  In fact, if I checkout to another revision, it's as if the "hg branch" command never happened.  When I list the branches, you won't see this new branch name.  The name in the PS window just comes from the PowerShell-Hg extension being smart.

One implication is that our team has to make phantom commits for the branch to officially "show up".  You start to see commits like "phantom commit" early on, then later they start saying "stupid commit".  It's the only way we could get a branch to show up locally and on the server.

In Git, a branch is nothing more than a pointer to a commit, and the commit itself carries no information about a branch name.  A much more flexible model in my experience.

This hit me today when I wanted to move a set of commits to a different branch, which turned out to be very difficult as the branch name was embedded in to the commit names.  Rebasing helped, except that the commits were already pushed to the server.  We wound up having to back out of the commits, even though these were on a different logical branch, simply because our build server got confused about these extra commits that were marked with the wrong branch name.

Perhaps Hg branches aren't broken and this design is intentional.  But it's annoying and less powerful and flexible than Git's more simple model, of just a pointer to a commit.

Side note for the Git folks – I've also wasted 2 hours of my life on Git when my .git folder suddenly went empty – and not from me accidentally deleting anything.  Lost all my dangling topic branches on that one.

On a recent large project, we had a goal early on that we didn't want to have a lot of QA folks manually testing our software.  Finding bugs through manual testing is incredibly time consuming and expensive, so we opted to try and build as much quality in to the product.  That's not to say that manual testing doesn't have its place, as humans are fantastic about using software in ways you didn't expect.

This was a long project, around 18 months, and will continue to have active development in the future.  Very early on we found that a good testing strategy was critical to the success of the project, especially for our team to be able to 1) continue to increase our velocity over time and 2) have the confidence to make both small and large changes in our application.

It took quite a while for us to settle on an effective strategy.  This was mostly because we had to learn how to design our application for testability, in all layers of the application.  Our team were all experienced in TDD before starting the project, but that wasn't the only skill we needed to create an effective testing strategy.

Levels of tests

Categorizing tests can get annoying.  You have functional tests, integration tests, unit tests, acceptance tests, slow tests, fast tests, UI tests, and on and on.  We found that our tests belonged in three main categories:

• Full-system tests
• Subcutaneous tests
• Unit tests

Each of these differs in the scope of what's being tested.  A full-system test exercises the application through the external interface, whether that's a browser, file drop, queue, WinForms app or whatnot.

Subcutaneous tests work at the layer directly below the external user interface.  In the context of a web application, a subcutaneous test in our case would be a form object sent through a command processor, with all the real classes and implementations in place.  We bypassed the Controller, which only contained UI layer logic, straight to the domain layer.  Send in form object, out pops success/failure.

Finally, we had unit tests.  Unit tests are designed to test one class, and can either be fast or slow tests.  Fast tests are the normal TDD tests, used to build out class design.  Test doubles are used as needed, but strictly interaction-based tests have less value unless we find the interactions very interesting.  We also have slow unit tests, which could also be classified as integration tests.  These would be things like repository tests, persistence tests, etc.

Our ratio of unit:subcutaneous:full-system tests hovered around something like 10:2:1.  We ended the project with something around 5000 unit tests, 1000 subcutaneous tests, and 500 full-system tests that used WatiN and Gallio to drive a browser.  The 6000 unit/subcutaneous tests executed in about 10 minutes, while the 500 UI tests completed in about 50 minutes.

Unit testing strategy

Unit tests were developed in a pretty strict TDD manner.  We write tests before any implementation is in place, and use the tests to drive the design of the code.  These tests help identify design issues, encapsulation problems, code smells and so on.

We strived to not write code that existed solely to enable testing.  That often meant that we had a design issue, and responsibilities were misplaced or encapsulation was violated.

As we got further down the pipeline in our project, we started to value interaction tests less and less.  Interaction testing through mocking is only really interesting if you're truly interested about interactions.  But more often, we were more interested in side-effects, and interactions were just an implementation detail.  What we often did instead is mock out slow or untestable pieces, like repositories, facades over external services, configuration classes, etc.  Otherwise, we limited mocking only to places where mocks were the only observation points for what we were interested in.

At some point in large projects, it can become obvious that your design needs a large-level refactoring, to extract out concepts to enable quicker delivery of features.  On our last project, some concepts unearthed included:

• AutoMapper
• Processing forms as individual command messages
• Input builders

With each of these, unit tests were actually a barrier to these refactorings.  But the barrier only existed because we had relied on these tests to capture all of the interesting behavior in our application.  To effectively allow large- and mid-size refactorings, we needed an additional level of testing.

Subcutaneous testing strategy

Subcutaneous tests, like their name implies, test everything just below the surface of the user interface.  In an MVC application, these would be tests for everything just below the controller.  For a web service, everything just below the endpoint.  The idea is that the topmost layer in an application does not perform any actual business logic, but just connect the external interfaces with the underlying services.

Subcutaneous tests are important because we want to be able to test business logic with the entirety of the system in play, with the exception of external connection points such as the user interface and external services.  While a unit test focuses on small-scale design, a subcutaneous test does not address design, but instead tests basic inputs and outputs of the system as a whole.

To build effective subcutaneous tests, we can try and build uniform pinch points through which common logic flows.  For example, we might build a command message handling system, or a common query interface.  In a recent project that processed batch files, each row in the file was transformed into a message.  We could then craft a message, send it through the system, and then verify all the side effects of processing that message.

Because subcutaneous tests address high-level behavior, rather than design, they are ideal for scenario-based testing strategies such as BDD or the Testcase Class per Fixture pattern.  If we want to be able to perform large refactorings, we need these high-level tests to create that wide-cast safety net for business behavior.  Subcutaneous tests are also great target points for calling features done, as they focus on more end-to-end logic.

While subcutaneous tests allow us to safely perform larger refactorings, they still do not provide a satisfactory level of confidence that our system will work in production.

Full system testing strategy

Our team originally called these tests "UI tests", until more and more of our projects entailed integration strategies where the inputs to our system weren't a browser, but instead messages, a REST endpoint, or FTP drops and batch file processing.  UI testing is a subset of full system testing.  The idea behind a full system test is that we want to test our software as it might be used in production.  For an MVC application, these would be browser-based tests.  For batch files, we would use actual files.  REST, actual HTTP requests.  Messages, real queues and messages.

If we want to know if our application works as expected, before it goes to production, one effective and efficient way to do so is to create an automated test that exercises the full system.  If my UI tests logs in to the application, places an order and I can verify that an order request was generated, I'm feeling pretty good about things.

One common misconception about full system tests is that they are black box tests.  While these have their merit, full system tests should have intimate knowledge about what's going on underneath the covers.  In fact, full system tests can even take advantage of the domain model to build up data, instead of a back-door system built solely for testing purposes.  One big mistake teams run in to is not following the same code paths in testing as they do in production, leading to wacky invalid, impossible states of the system.

In our projects, a full system test is the last code we write before we call a feature/story done, done, done.  Manual testing is just too expensive and unreliable for characterizing "done-ness" of a feature, but if I can do the exact same actions as would happen in production through the exact same external interfaces, that's success.

A holistic approach

In an application without tests, we've actually found it most valuable to start with full system testing, moving down towards unit tests as a means of a strategy for coverage.  We cast the widest net possible, but the simplest assertions first, then slowly move down towards unit-level logic.  In new applications, we tend not to focus on any one area, as all of these tests are critical to us for long-term maintainability of a system.

This testing strategy does require a level of investment.  We've found this holistic approach especially effective when we know that this application is critical to our client's business.  If an application is critical to business, it's going to require change.  If it's going to require change, we better be able to safely change it without affecting our client's business.