Rootkits were once billed as one of the biggest threats to antivirus vendors, but have they lived up to their name? We talk to some experts and find out.

When it comes to malicious software, rootkits have been one of the most hyped threats, in particular because they can do a lot of damage and can be hard to detect.

Take for instance the rootkit 'Rustock.C'. This is one of the most dangerous says security researcher Frank Boldewin. It installs itself on the Windows operating system and then uses a variety of sophisticated techniques that make it nearly impossible to detect or even analyse.

When he first started looking at the code earlier this year, Boldewin found it would simply cause his computer to crash. There was driver level encryption, which had to be decrypted, and it was written in assembly language, using 'spaghetti code structure' that made it extremely hard for Boldewin to figure out what the software was actually doing.

Analysing a rootkit is typically an evening's work for someone with Boldewin's technical skills. With Rustock.C, however, it took him days to figure out how the software worked. Because it is so hard to spot, Boldewin, a security researcher with German IT service provider GAD, believes that Rustock.C had been around for nearly a year before antivirus products began detecting it.

This is the story with rootkits. They're sneaky. But are they a major threat?

In late 2005, Mark Russinovich discovered the most famous rootkit. A windows security expert, Russinovich was baffled one day when he discovered a rootkit on his PC. After some sleuthing, he eventually discovered that copy protection software used by Sony BMG Music Entertainment actually used rootkit techniques to hide itself on computers.

Sony's software wasn't designed to do anything malicious, but it was virtually undetectable and extremely difficult to remove. Sony's rootkit became a major PR disaster for the company, which spent millions in legal settlements with users who were affected by the software.

Three years later, Russinovich, a technical fellow with Microsoft, still considers it the rootkit that caused the most trouble for computer users. But the Sony rootkit presaged problems for the antivirus vendors too. The fact that none of them had even noticed this software for about a year was a serious black eye for the security industry.

Though they got their start on Unix machines years earlier, at the time of the Sony fiasco, rootkits were considered the next big threat for antivirus vendors. Security researchers explored the use of virtualisation technology to hide rootkits and debated whether a completely undetectable rootkit could someday be created.

NEXT PAGE: Why rootkits don’t live up to the hype

  1. Just an annoyance or a serious worry?
  2. Why rootkits don’t live up to the hype
  3. Most malware doesn't even attempt to hide these days