Everest Digital, LLC

The first thing that all good software projects have is a consistent documentation methodology. Unfortunately, from the smallest to the largest, almost all IT shops have inadequate documentation of their systems, policies, and procedures. Most professional software developers will tell you that even for a one-man software project, good documentation is critical to getting things done quickly and efficiently with as few errors as as possible. You may think that you can hold all that information in your head, but when you come back to a system six months after you last configured it, you are essentially a different person. You may not have any idea what you changed and why, which is just like a developer coming back to a section of code and having no idea what a particular function or subroutine does. A single comment around that subroutine, which would have taken five minutes when it was first written, would help. But six months later, the developer has to spend far more than five minutes figuring out what the code does so he can figure out how to change it safely.

For this reason, I recommend that you document virtually everything that can be documented. I'm not suggesting creating elaborate, 500-page policy manuals. However, when you order a new server, document its original hardware configuration, along with vendor information, hardware and software serial numbers, etc. Similarly, when one of your network engineers brings his kid to work, and the child decides to play "telephone operator" with your LAN's patch panels, it sure would be helpful to have documentation mapping out those interconnects.

Because of the nature of IT, I do not recommend that you have much in the way of printed documentation. In most fast-paced IT departments, printed documents get outdated quickly. Instead, I recommend that you implement a wiki on your corporate intranet. In this way, all of your staff members can update documentation as necessary, and you will even be able to roll back changes if people make mistakes. Plus, everyone will always be viewing the most current set of docs, so as long as you keep your wiki up to date, your IT people will always have the latest version of the information they need.

Probably the two most popular wiki programs are MediaWiki and TWiki. Both are free software licensed under the GPL. I strongly recommend TWiki for these documentation projects, as it was designed from the ground up to be an enterprise collaboration program.

The second component of any good software development team is source control, also known as software configuration management (SCM). For the unfamiliar, SCM programs track all changes to a given set of files, allowing you to roll back any mistakes you may make. They also let you compare various versions and revisions of your files.

While the need for SCM in a software development situation is obvious, the need in an IT department might be somewhat less so. However, if you are a sysadmin in any capacity, it is likely that you are a prime candidate for SCM. If you run Unix systems, just think about all the various shell scripts and other such files on your machines. These files are frequently modified and appended to, but what would happen if you accidentally broke your nightly backup script, and didn't realize it until the following morning? How quickly could you restore that functionality? Could you even remember exactly what changes you had made the previous day? An SCM system could address all of those issues for you quickly and easily.

Or what if you accidentally made a mistake in httpd.conf or an .htaccess file and took down part of your Web server? What if you didn't realize it was broken until several days later when an obscure corner of your Web site was no longer accessible? Chances are you wouldn't remember exactly what you had changed. You would have to pore over those files looking for a small (or not so small) bug in your changes. With SCM, you could just run a diff on your current file against the previous revision, and the system would show you exactly what lines you had changed, pointing out precisely the most probable location for the bug.

If you run Windows machines, consider for a moment what you would do if you accidentally deleted what you thought was an old domain logon script, only to realize that you deleted the new replacement logon script instead of the old one. Think of the headache of having to go to your backup tapes just to get that one file back. If you had all your logon scripts under source control, this would be a 30-second operation.

What would you do if you started getting calls saying that the HR department's intranet site was no longer working properly? You know your junior admin was making some changes to the source code yesterday, but you don't know exactly what he did. If you have these files under source control, you can see exactly what files he modified, what the changes were, and any comments he included when he checked the files back in to your SCM system.

These are just a few of the myriad cases where a good SCM system can save you time and prevent headaches. You can probably think of a million more uses for source control within your own enterprise. To get up and running quickly, I recommend that you check out the Subversion SCM system. It is free, open source software licensed under an Apache-style license, and it runs well on a large variety of systems. For Windows users, I also strongly endorse the TortoiseSVN client program. It provides a complete client for Subversion, implemented as a Windows shell extension. For more information about running and using Subversion, O'Reilly's book Version Control with Subversion is available online in its entirety, licensed under the Creative Commons Attribution license.

The third hallmark of a well-run software development team is the use of bug-tracking software. It is pretty much impossible to write software without some bugs into the code. In the same way, no IT department is without a list of open issues, else what would be the point of the IT department? However, the potential for these open issues to overwhelm even the most organized administrator makes the need for some kind of tracking software obvious. Besides, our mental energy is much better spent in ways other than trying to remember our to-do lists.

Issue tracking software is convenient in that not only can it be set up to keep track of your open issues, it can also handle incoming support requests via email, automatically creating a new case in the system when a new message comes in. The software can even assign a case number to the requester and send him a customizable autoreply containing important information such as your estimated response time and self-help sources. Some of these programs even provide end-user portals, which can include a way for users to track their open cases, as well as provide knowledgebases.

For most IT departments, I recommend Cerberus Helpdesk. Cerberus provides a free version that is identical to its full version in every way, except that it is limited to handling incoming emails for a single email address per domain. Its GUI is entirely Web-based (though a Java-based Windows workstation client is in beta). The interface allows you to view open issues from any Internet-accessible machine, running pretty much any OS. Cerberus provides a customer-facing portal as a free add-on, which allows customers to track open tickets, submit new tickets via the Web, and access the knowledgebase that is integrated into the standard Cerberus product. In addition, Cerberus provides a live over-the-Web chat interface as a paid add-on for providing live support via the Web.

IT departments have a lot they can learn from software development teams. If you're interested in more ideas, check out the archives of Joel Spolsky's blog, Joel on Software. There is a wealth of excellent material there, and for the most part, it is very accessible even for people who are not familiar with software development. Especially recommended are the articles "Painless Software Schedules," "The Joel Test: 12 Steps to Better Code," and "Painless Bug Tracking."

This article originally appeared February 27, 2006 in IT Manager's Journal. This version contains a few minor corrections from the original.