About Jamie Lenehan

Software Engineer working in the field of network optimisation products. Lots of C/C++/boost, embedded hardware, linux kernel, TCP, SSL, deep packet inspection, SNMP, SQL, IPv6, QoS, policy based routing, virtualisation and appliance management.

Moving from subversion to git with gitolite

The following notes provide details on the conversion of several repositories from subversion to git making use of two tools to help with the process:

  1. gitolite which provides a framework to help managing a central git repository. From https://github.com/sitaramc/gitolite
  2. svn2git to convert existing repositories from subversion to git. From: https://github.com/nirvdrum/svn2git

This is the second time I performed such conversions and it was a lot easier the second time around due to greater familiarity with git and the tools being used to perform the conversion. These notes should make it even easier the next time I need to do this.

Setting up gitolite

Gitolite is a management framework for git. This is installed onto the server and will be used to manage ACLs, available repositories, configuration and hooks etc.

1. Create the git group and user account

There is a single account that will be used for all access to the git repositories and this needs to be created on the server.

~#> adduser --system --shell /bin/bash --group --disabled-password --home /srv/git git
 Adding system user `git' (UID 125) ...
 Adding new group `git' (GID 125) ...
 Adding new user `git' (UID 125) with group `git' ...
 Creating home directory `/srv/git' ...

2. Change to the git user

We want to install gitolite into the new git account, so we need to su into it.

~#> su - git

3. Download the gitolite code

In this case I have downloaded the code into the src directory in the gitolite account.

$ pwd
 $ mkdir src
 $ cd src
 $ git clone git://github.com/sitaramc/gitolite
 Initialized empty Git repository in /srv/git/src/gitolite/.git/
 remote: Counting objects: 7695, done.
 remote: Compressing objects: 100% (2528/2528), done.
 remote: Total 7695 (delta 5273), reused 7410 (delta 5025)
 Receiving objects: 100% (7695/7695), 1.79 MiB | 426 KiB/s, done.
 Resolving deltas: 100% (5273/5273), done.

4. Install into ~/bin:

And now install gitolite. There are a few options provided in the gitolite documentation, in this case it is using the recommended method:

$ mkdir ~/bin
 $ ./gitolite/install -ln
 $ ls ~/bin

5. Acquire the initial ssh administrator key

The ssh public key of the initial administrator user is now needed. I copied this from my main workstation:

$ cd ~
$ scp johnsmith@nynaeve:.ssh/id_rsa.pub johnsmith.pub
johnsmith@nynaeve's password: 
id_rsa.pub                                                     100%  397     0.4KB/s   00:00    

6. Setup the initial administration repository

Now we are ready to create the gitolite-admin and test repositories.

$ export PATH=$PATH:/srv/git/bin
$  gitolite setup -pk johnsmith.pub
Initialized empty Git repository in /srv/git/repositories/gitolite-admin.git/
Initialized empty Git repository in /srv/git/repositories/testing.git/
WARNING: /srv/git/.ssh/authorized_keys missing; creating a new one

7. Edit .gitolite.rc

I made two changes here so that I could manage the repository configuration and hooks from the gitolite admin repository.

First to allow manage of configuration keys I modified GIT_CONFIG_KEYS. Note that this allows any config keys which may not be desirable in all situations. Read the gitolite documentation for more information

GIT_CONFIG_KEYS             =>  '.*',

The next change was to add LOCAL_CODE to set the local that gitolite would look for additional code, such as hooks. This was set to:
# Local code

LOCAL_CODE                  =>  "$ENV{HOME}/.gitolite/local",

This allows hooks to be committed to local/hooks/common directory in the gitolite-admin repository and have them actually take effect.

Managing gitolite

All management of gitolite is now performed via the gitolite-admin repository. This can be checked out using the account associated with the administrator ssh public key used during setup.

~/work%> git clone git@servername:gitolite-aCloning into gitolite-admin...
remote: Counting objects: 54, done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 54 (delta 10), reused 0 (delta 0)
Receiving objects: 100% (54/54), 10.16 KiB, done.
Resolving deltas: 100% (10/10), done.

To allow multiple public keys per user there are a few methods. I prefer the one file with different names per key.

~/work%> cd gitolite-admin/keydir
~/work/gitolite-admin/keydir%> git mv johnsmith.pub johnsmith@hosta.pub
~/work/gitolite-admin/keydir%> git add johnsmith@hostb.pub
~/work/gitolite-admin/keydir%> git commit -a

For the emailer I created local/hooks/common/post-receive and local/sender.cfg in the gitolite-admin repository and for configuration I placed the config keys in conf/gitolite.conf:

~/work/gitolite-admin%> ls
conf/  keydir/  local/
~/work/gitolite-admin%> ls local
hooks/  sender.cfg
~/work/gitolite-admin%> ls local/hooks/common
~/work/gitolite-admin%> cat conf/gitolite.conf
# Default hook settings
repo @all
    config hooks.mailinglist = johnsmith@example.org
    config hooks.emailprefix = "[%GL_REPO] "
    config hooks.maxdiffsize = 5000
    config hooks.project = "%GL_REPO"
    config hooks.link = "http://example.org/git/%GL_REPO/commit/?id=%s"
    config hooks.hostname = "example.org"

repo gitolite-admin
    RW+     =  johnsmith

repo testing
    RW+     =  johnsmith

These settings would depend on the emailer being used but give the basic idea on what is required.

Converting from subversion to git

The conversion process is done using svn2git and is reasonable simple. You should look at the svn2git documentation as it is reasonably straight foward and provides all the various options you are likely to need.

1. Install svn2git

sudo apt-get install git-core git-svn ruby rubygems
umask 022
sudo gem install svn2git

2. Create ~/.svn2git/authors

This will map subversion user names to git usernames. Alternatively you can pass the name of the authors files as a command line argument.

It just contains lines of the form:

johnsmith = John Smith <johnsmitch@example.com>

3. Convert your repository

Make sure you have either saved the subversion password, by directly accessing the repository with subversion, or that you use the password command line option.

There are different command line options to use depending on the layout of the subversion repository. Check the documentation for all the example.

To convert from a standard layout:

svn2git http://svn.example.com/path/to/repo

To convert from a repository that has no structure, just all the files in the root of the repository:

svn2git http://svn.example.com/path/to/repo --rootistrunk

To convert from a repository that has a project name at the top level, then the standard subversion structure underneath, but has no branches:

svn2git http://svn.example.com/path/to/repo --trunk projecta/trunk --tags projecta/tags --nobranches

Use the –verbose command line option if you need to see what is happening.

Use the –rebase if you have already converted the repository but new changes have been made and you want to get those additional changes. This is also useful if the subversion server times out and disconnects you during conversion (I had this happen when converting a very large repository hosted on a windows server when I needed to retry 30 to 40 times before managing to get the entire repository converted.)

Pushing a new repository to gitolite

1. Add the repository

In the gitolite-admin admin repository edit conf/gitolite.conf and define the new repository. This can be as simple as adding the following:

repo reponame
    RW+     =  johnsmith

Then commit the change and push this change to the server. This will create a new empty repository on the server.

2. Push up the existing repository

Change into your existing repository (such as the one just converted from subversion), then add your git server as a remote server and push up the new code:

git remote add origin git@server.example.com:reponame
git push origin master
git push --all
git push --tags

The push –all is only needed if additional branches (other than master) are present in the repository. The push –tags is only needed if there are tags in the repository.

3. Verify that it worked

Finally do a new clone of the repository and verify that the contents are correct:

mkdir ~/test
cd ~test
git clone git@server.example.com:reponame

Netgear WNDR3700

Two Netgear WNDR3700’s are now running here at twibble.org and both are loaded with locally built versions of openwrt.

The first is running as a router, switch, 2,4GHz access point, 5GHz wireless client. dhcp server and radvd server, while the second is running as a 5GHz access point, firewall (shorewall), vpn server (openswan), 6to4 gateway, pppoe client and router.

The 5GHz wireless bridge link carries multiple subnets and IPv6 and IPv4 traffic without any issues, unlike the bridge devices it replaced which could only handle single subnets and no IPv6. Previously a GRE tunnel was used over the link to allow them to be used here.

Beyonwiz DP-P1 hard disk upgrade

The trusty Beyonwiz DP-P1 had served me well for a few years but the 250G hard drive was starting to feel a bit small.

Research suggested that it was important that any new hard drive didn’t consume much more power than the original drive. If a SATA drive was to be used some a SATA to IDE adaptor would be needed and that would consume additional power which also needed to be taken into account.

Eventually I settled on the following components for the upgrade:

  1. Hitachi CinemaStar 7K1000.C 1TB 7200RPM SATA HDD
  2. Altronics D2335 SATA to IDE Converter Module

Attaching the converter to the SATA HDD effectively converted it into an IDE hard disk. It was then just a matter of swapping the existing hard drive and reconnecting the IDE cable and power cables.

The drive runs cooler and quieter than the original Samsumg 250G drive and disk related actions seem to be a lot more responsive. So deleting files, bringing up file player list etc are all feel quite a bit faster.

No problems have been observed in the year since the upgrade, including running the disk check after power outages. Most of the time the disk space used is sitting between 400G and 600G. Currently running firmware version 350.