Automated deployment using Subversion

It’s always nice to see something you have spent time developing being used in production.

A while ago I read an excellent blog post from Timothy Fitz called Continuous Deployment at IMVU: Doing the impossible fifty times a day, that got me started on trying to improve the way we deploy our websites at UKFast but just as important – being able to roll back to a previous version at the drop of a hat.

We had a few requirements such as, hook into current systems, deploy code with near zero downtime, run some post deploy commands, stop deploying on error and keep an archive of old deployments for simple roll back amongst other things.

As you’ve probably gathered by the title, we run Subversion as our version system, it works well as a standalone but that’s never enough for us at UKFast. The system that was developed so far has achieved everything that we need, new or existing sites are deployed usually within 30 seconds but that depends on the size of them, the freshly deployed code is activated online without any loss of service to visitors to the site which is probably one of the best features about this system. It also means that roll backs are done transparently and due to another feature, archives, depending on the (user configurable) number of archived versions we keep, we can roll back to a previous version almost instantaneously as the code is already deployed and ready to go – if we need to go back further past the archived version then a checkout is done on the old version.

Now once a site is deployed, that typically wouldn’t be the end of it. If someone had tried to perform the update manually then it most likely wouldn’t be case of just FTPing up the new website, for one, we usually have some heavy customisation on folder structure, permissions or just because we want something different. So the system needs to cope for this as well, it will allow users to issue post deploy commands from a predetermined list of allowed options. This has so far covered us from all the usual requests our web developers usually ask for and without have to give them a ssh shell!

In the event that any errors are detected at any point the deployment is halted and the user performing the update is notified, no need to roll code back in anyway as the new version hasn’t even gone live yet.

This system has meant we can stop using ssh / ftp accounts for website management, determine fine granular access to developers and mainly, allow us to update our websites at any stage of the day, knowing that it won’t impact site visitors and if it does, then we have a roll back process that takes us 10 seconds to perform and the site will be restored.

Unfortunately the way in a lot of our systems work means I can’t discuss it on an open blog, competitors might be watching. But if anyone wants to pick my brains then by all means, get in touch.

Share – Part Two

Well it seems that the most popular post on my blog is this one so I thought it deserved a bit of a follow up post.

I’ve just spent part of this evening re-organising my domain name and one of my old blogs to import all of the old posts into the current blog incarnation. Whilst doing that I noticed the original post about and how many comments it has received so far. It made me start thinking about what DCS say about their own customer service procedures, well according to the FAQs on the website, they should be quite a good company to deal with:

Q. When can I expect my delivery?
You can track the progress of your order using our on line tracking facility.

The so called tracking system is the basic of the basic, no based on my story which you can read about here, this is my tracking status to this day:

25/02/08: Received @ 16:01
26/02/08: Status = Part Dispatched
22/07/08: Status = Dispatched

Yeah – great!

We aim to deliver all orders as soon as possible. Orders received before 13.00 Monday to Friday for items that we hold in stock will generally be dispatched from our premises the same day. Orders received after 13.00 for items that we hold in stock will generally be dispatched the next working day (excluding weekends).

‘Generally’, what that seem to mean is, if we have it in stock (unlikely?) then they will ship it out asap, or if the supplier has it in then again, it will be shipped pretty damn quick, however, god forbid you order something from the website which they don’t stock and neither does the supplier – you either won’t see it, or it will take a looooong time. They don’t run a live stock system which makes me think that they don’t actually stock items themselves otherwise, how hard could a stock system be from them?

Should an item be out of stock or a special order we will obtain it as soon as possible. We will inform you via e mail if an item is out of stock and give you the opportunity to cancel your order.

No they won’t, I received nothing more than a small note on the bottom of the delivery item for the part delivery I received. Easily missed but I definitely didn’t receive an email. If my order hadn’t been part shipped and both items were out of stock then I wouldn’t have received any correspondence by the looks of things.

We normally dispatch items using a 48 hour parcel delivery service.

See above.

Despite our best efforts to deliver as soon as possible, unforseen circumstances could lead to a delay in delivery of goods. Any dates / times specified are approximate and should not be taken as guaranteed.

So, basically they don’t really have an idea of when you might get your items, they also won’t keep you upto date, leaving you the ‘customer’ to chase them.

One of the other things that is annoying with the way DCS does business, they take your money immediately whether they will be delivering your goods the next day or the next month – not a bad way to earn a little interest, no matter how small that might be.

One of the directors has recently setup another business which seems to as of yet, be currently in testing and not live however based on the past performance I would be wary about doing business with

Obviously all of this information is my opinion and is based on either my experience of DCS or from information obtained through a credit check.



Network Monitoring System – Demo

Well since the post I made about Alternative Nagios software generated a bit of interest (well, one comment is a bit for this blog :)),  it made me want to try out the software mentioned again to see how it fairs against Nagios which has been my monitoring software of choice for a considerable amount of time. I also thought, why not provide the software I install and test to anyone who might be interested in trying out some of the NMS systems out there.

So without further ado, below you will find the monitoring software I have installed and tested. These are fully functional in all aspects, the only thing limited is the ability to change passwords so that people can’t lock one another out and also, smtp is blocked on this server meaning that any alerts generated won’t get sent out.

username: admin
password: admin
Notes: You cannot add / edit / remove devices from this demo as Nagios relies on editing of configuration files directly which isn’t supported natively.

username: admin
password: admin

username: admin
password: admin

username: admin
password: admin

I’ve got to say, after going through and testing all of these, at the basic level of just adding a device to be monitored for things such as http, it’s an extremely difficult process on all of them (Nagios was the easiest but again, this is probably due to my time served with this software). From the perspective of simplicity, I can’t understand how such a task of adding a device whether by hostname or IP can be such a complicated task.

Anyway, feel free to make use of the online demo’s and if you would like me to add anymore NMS systems to this list then please feel free to get in touch and I will see what I can do.



Jane has kindly pointed out the login for Zabbix didn’t work – changed it to the actual correct details so now it should.


How far should companies go for end to end security?

For instance, your business is hosting your corporate website with a hosting company, the vetting process for this decision can be quite intense depending on the size of your company but obviously now concentrates on security amongst the service itself. However, security for your application doesn’t just stop at the hosting company, it goes much further than that, your DNS provider, SSL signing authority, etc.

At work, we are seeing more and more requests for information (RFI) in regards to tenders we are involved in. These processes are getting harder and more involved than years ago where the process was a bit more ‘laxed’, the sections on security from both a data storage and infrastructure point of view are becoming more and more detailed on the requirements but also more generic.  This is made all the more difficult at the RFI stage as 9 times out 10 you won’t know yet, what the potential client wants and security varies depending on requirements. For example, security from the point of view of a Colocation contract is vastley different than a Managed Hosting contract. They share similarities but the main service is very different meaning that our answers have to be more generic rather than tailored to the solution required.

This process however, seems to stop at the hosting company, even when they aren’t used for the whole solution. As I mentioned above, how much consideration is given to things like who the SSL signing authority is and how secure are there systems (MD5 considered harmful today), who is providing recursive DNS for the clients internal systems and have they patched for last years Source port vulnerability) all the way through to major vulnerabilities on operating systems typically found running within organisations (RPC Vulnerability for example or even the latest iWork 09 OSX vulnerability). The client side systems are usually covered with corporate security policies (you do have one of those don’t you?) but things like Zero Day exploits are a bit more difficult to defend against but not totally impossible. I mean just imagine if someone could hijack your domain name traffic and then spoof the SSL certificate used for your 10,000 transaction a day ecommerce site. The short and long term effects of something like that on a business could see the end of it before the charge backs on credit cards have hit shoppers bank accounts.

Based on the fact that companies seem to just stop at the stage of choosing a hosting company means that the solution as a whole is vulnerable to security holes from other suppliers or areas of the business. It wouldn’t matter how secure the hosting solution is if someone can redirect traffic to a fake website elsewhere. So the best answer to this question is, companies need to go as far as possible to gaurantee security for there solution. Extend the RFI out to more suppliers to ensure that all aspects are covered – or choose a one stop shop for the entire solution.


Nagios alternatives, are there any?

Nagios is one of the standard NMS (Network Monitoring Systems) available to businesses today, it’s wealth of features provide a very flexible system, it’s scalable and customisable so should fit into the most demanding environments – is this all that’s available to assist in keeping an eye on your IT infrastructure 24×7?

No, that’s the simple answer. The longer answer is more complicated and depends on what your System Administrators can support from an application point of view. A lot of the NMS systems available are your standard Perl / C implementation on a flat file or MySQL/PostgreSQL backend so should be supported on most systems. Others however require Java or Python support, whilst this is simple enough to install and run on most Linux distributions, what happens when things go wrong? How many companies have IT staff who have taken the time to learn Java/Python for the next big Web 2.0 site?

I’ve used Nagios for a while now (some 5 years) and have always gone back to the start and looked over the alternatives at various times to see what they offer in terms of features and integration. Integration, that’s the key for me when it comes to choosing an NMS system, if it’s not able to offer a level of integration with the systems we already have (and we’re willing to do some work to make that happen), then it’s a non-starter no matter how laden with features it might be. Over the past couple of years, a wider choice has become available which makes choosing to switch, that much harder! Having the luxury of time to test these newer systems isn’t something that’s available to everyone including me so my ‘experience’ of other systems is limited compared to my time with Nagios, however, I know what I want and need so it doesn’t always take long before commissioning an app to the dusty code graveyard.

Let’s get into some of the more popular NMS systems available at present and what my impression of them has been…..


OpenNMS is actually a really nice application to use for Network Monitoring, it’s discovery feature works really well and being configurable form XML files makes it extremely easy to setup and maintain. Installation is relatively straight forward if you are using the pre-packaged versions available or are a dab hand with Tomcat. The pre-configured range that I created for Network discovery worked fine and it detected all devices which responded to ICMP and monitoring of individual services/interfaces was simple if not time consuming if you have a large selection of devices to monitor.

The bad points for me are the fact that auto-discovery is the primary way of adding new devices to be monitored. You can add new devices in via the command line on the NMS server, this isn’t too much of an issue depending on how many new devices are added to your infrastructure on a day by day basis. If it’s a sizable amount then this isn’t going to be an option for long and with no way to add single devices in via a web interface your only options left is by some form of integration by way of a script. The next problem is managing the individual services/interfaces that are available for a particular device, this again appears to be a manual process with no easy way to integrate into your current NOC.

PostgreSQL is the supported DB of choice for this project, we haven’t at present migrated over to PostgreSQL which means that the maintenance of this solution would be higher than a MySQL based back end. That’s something to bear in mind in any solution you may migrate to or implement. PostgreSQL is gaining in popularity and features so this at some point will become a mute issue.


Wow! Zenoss appears to have improved quite a bit since the last version that I tested and looks to be highly recommended now. It’s features include a comprehensive API to allow integration with existing systems, this would enable the setup of monitoring new devices quite easy. Installation on platforms with supported binaries appears to be straight forward along with the configuration and setup of your first ‘Devices’ that need to be monitored. Auto-discovery is still an option and is more intelligent than OpenNMS, it provides a handy feature of ‘walking’ your network via routers to find all devices located on your network, this is quite a powerful feature on it’s own.

It supports the ability to expand your single monitoring server to a High Availability solution, whilst this isn’t quite out of the box, it really isn’t a complex setup for a Linux Sysadmin (Setup Guide). This enables you to grow your monitoring environment as your infrastructure grows or provide a level of redundancy to ensure that you know what is going on 24×7.

The changes and improvements that have been made since my last evaluation of Zenoss means that it’s about time that I tested it again – if it became a viable alternative then a lot of work would have to go into the migration from Nagios to Zenoss but it looks like it could be worthwhile.


I’m not a big a fan of the Documentation for Zabbix, everything is dumped into a single PDF which makes it difficult to filter out what is part of configuration and what is part of administration. For instance, to refresh my memory whether you could add a single host into the setup via the web administration, I checked the documentation. Now this was  a brief check, it was 00:15 but could I find anything other than auto-discovery? Nope, not a single thing, the system of course does allow this, you just have to struggle in the docs to find it. Not a great start but not a show stopper if an API was available – it doesn’t appear to be, I can see comments about this on the forum but so far nothing seems to have a materialised so far.

Repeat notifications has now been implemented since I last tested Zabbix, this is something that was extremely lacking in previous versions and is a must for any NOC, especially if your using email/sms/pager alerting. These methods are inherently unreliable when it comes to critical service so sending more than one alert is always a handy feature which meant that before, Zabbix would have sent a single alert when a device went down – and that was it, if you didn’t get that alert for whatever reason then you would be unaware of any issues until someone logged into the administartion system.

Distributed monitoring is included and seems to be extremely simple to setup, this is one of the better features of Zabbix and something worth considering if this is a requirement for your environment. In general Zabbix seems to have improved quite a lot since the last testing I did, the restrictive admin interface means it would be something that I wouldn’t really consider in a live environment.

Finally, a comprehensive list comparing the available NMS applications available is currently hosted on Wikipedia, go and check it out for a list of more NMS applications.

Share – DO NOT USE

Customer service skills in this country are pretty hard to find sometimes, especially for online companies where the face of that company could be an email address!

Well, for those who go off recomendations when deciding to use a new company, here’s one to avoid – This is my story of why.

On the 25th of Feb 08, I placed an order with for a Slik Pro 700 DX and a Manfrotto 676B Monopod. Total cost was £117.89. The online shop didn’t have a live stock system so I had no idea if either of the items were in stock, that wasn’t really and issue as I reaslise that small businesses can’t have the same or similar setups as some of the larger online shops + waiting for a week or two wasn’t really an issue either!

A few days later, a parcel arrived for me 🙂 Prompt delivery I thought. Upon opening the package, I realised that I’d only had sent to me, the Slik tripod. So, the next day I called up to enquire about the Monopod as I hadn’t had any notice that the Monopod was out of stock – or so I thought. The person who took my call was pretty rude and after I’d quoted my reference number ‘kindly’ pointed out to me that it was printed on the bottom of the delivery note that the Manfrotto would be delievered in a few days due to a lack of stock. FIne I thought, I got the main item I was after and as mentioned before, I could wait a few days.

A few days came and passed, no Manfrotto and no contact from, something I later find out not to expect from, no contact that is – oh and no Manfrotto either but more the lack of contact 🙂

Weeks later and no frotto still, no contact still so I thought, hey, I’ll check on the status of the order so again I gave a quick call only to find out that, surprise surprise, they didn’t have it and still didn’t know when it was going to turn up. At this point I just cancelled the remainder of the order and went into Jessops and bought one.

Some months later I was checking through my bank statements and thought I’d make sure the refund had gone through for the Manfrotto knowing what was like with their lack of contact. Fail, they hadn’t refunded me! So again, I called and this time got voicemail, time, after time during that day. No phone call back and to be honest, I very much doubt I would have got one so I emailed them to ask them what was going on. At this point they were quick to respond – with a basic apology and a promise to refund me the remaining balance of the order. Great, I also asked them to refund me the delivery cost for the order due to the 5 months they’ve now had my money for the order with no contact on what was happening.

22nd July 08, They responded, yipee! They agreed to refund me the cost of the Manfrotto – how could they not but they totally ignored my request for the delivery charge to be refunded so I emailed them again, asking once more, for the delivery cost to be refunded, to this day, they’ve ignored that email as well.

So, in short, avoid buying from, they DO NOT contact you when asked or to give you any updates on orders which don’t get processed immediately, they have no idea about customer service and don’t seem to care one bit.


Should you outsource your support abroad?

I’ve never been a big fan of companies outsourcing any part of their business to a foreign company to save money.  The bad press it has generated since it became popular to do has been quite substantial but it hasn’t put companies off doing it to save money – and lose business. I mean, is it really that cost affective to shift say your entire support department over to somewhere like India to save on wage and infrastructure costs but in turn, more than likely, lose at least a handle of your clients due to the lack of support that they may now receive?

Some companies are now making a big deal out of advertising that they don’t send your calls to anywhere other than say, your local branch like the Natwest do. Having today had to deal with, only briefly mind, but that was long enough, a company who have outsourced their support centre to India, it left me thinking that I could never even remotely consider outsourcing the support team I have to India. After the two calls I made with little joy in getting the information I needed, I didn’t know what else to do! The first phone call that I had the luck of having to make ended with me thinking that you wouldn’t want that type of company managing or having anything to do with your company. I called an 0871 number to speak to someone in support to try and get a number redirect working, when I was told that they couldn’t assist because basically they didn’t understand what it was I was referring to, they asked me to redial and press the same option I had for the current call!!!

I stated that surely that would just get me back through to them again and was told, “maybe”?!?! Unbelievable – but I tried anyway 🙂
So what happened, well I’m sure you can guess, I called, pressed 1 for support and got a lovely person on the end of the phone – from the same call centre. So went through the same thing as before to see if I got anyone else with at least some clue. Of course that didn’t work and I should have hung up a long time ago, or not even bothered to actually call them back in the first place.

To top all of this off as well, not only could I not speak to someone who understood our account and knew what I wanted or could point me in the right direction but the website for the company had zero contact details that were of any use or actually allowed me to speak to someone in the UK. Not what I would call customers service at all!