The Ohio Dept. of Public Safety has successfully moved its mainframe applications to a Windows-based system, and all the work was done in-house.
The decision to keep the work in-house prompted much debate and skepticism, but there was no ducking the question that the agency had to retire the mainframe technology that had long supported its most important operations.
The project began in 2007 and the switchover was completed in March of this year. The mainframe is now unplugged.
The mainframe had supported legacy code in applications that dated back to the 1980s.
There were about 2,000 programs running on the mainframe, about 50 of which no one knew anything about. As engineers retired, knowledge about many of the applications running on the mainframe was lost.
At the beginning, all of the migration options were daunting.
Rewriting everything to Microsoft's .Net programming language would be too expensive and would take too long, but staying on a mainframe was too expensive as well.
Bringing in consultants to handle a transition to Windows might have cost as much as $10 million, said Keith Albert, the chief of IT governance and strategic direction for the department.
Therefore, the department trained its staff of long-time mainframe veterans who had worked mostly in IBM Pacbase code. The migration involved .Net training and a move from moving a hierarchical to a relational database.
The agency spent some $250,000 on training, and hired a few contractors to augment the staff.
The core of the development team included about 30 IT staff members who, by Albert's count, spent about 84,500 person-hours on the project.
There was some skepticism about the project at the start.
Albert said IT officials found little information about successful migrations and that was disconcerting. "We couldn't find anything out there that said we were going down the right path," he said.
The lack of information made an impression on Albert, who wrote a paper detailing the steps taken during the agency's migration and the issues the development team and management dealt with. The 18-page paper was completed last month and was posted Thursday on the department's Web site.
The document is a compelling read, with details about the problem, the options, the internal debate and the lessons learned along the way.
The title of the paper -- Exodus Project - Pigs Really Do Fly! -- is a remark made by a staff member who, early on, thought the project was impossible.
The Unisys Clearpath Dorado mainframe was long dependable for the agency, and was running more than 2,000 programs. However, the model in use at the agency had reached its end of life and it had to be replaced by this year.
Albert said he would have stayed on this mainframe for a number of years if Unisys extended its life as well as cut MIPS (Millions of Instructions per Second) charges, which were running about $1 million a year. The agency paid an additional $200,000 in maintenance fees.
The MIPS charges and the end-of-life was grating for Albert.
He compared the situation to "owning a car that runs fine, but there is only one place you can buy gas for the car and that gas station refuses to sell any more gas for the car after that x-date."
"There was no reason why we couldn't use the mainframe another five years," said Albert. The latest machine had been installed in 2001 but was actively maintained it by adding fiber cards, I/O processors, I/O channels, as needed.
But there were other motivations beyond MIPS for migrating off the mainframe, namely its Pacbase code. Working with it was becoming increasingly difficult to support as staff retired. "There are very few Pacbase programmers in the world anymore," said Albert.
There was skepticism internally about whether the legacy code could work on a Windows platform and as reliably as it did on the mainframe. But with about 30% of the IT staff eligible for retirement in three years and 50% eligible in five years, a migration was all but inevitable.
The agency will still need Pacbase skills for years to come, but the long-term plan is to rewrite the two million lines of legacy code for a .Net environment. "As long as my Pacbase programmers stick around long enough to keep things running, my .Net programmers can rewrite it," he said.
Albert's paper outlines some of the struggles faced by the development team.
There "was no expert judgment available or models to follow," wrote Albert. The initial work could have been completed more quickly "had the decision to re-platform not been repeatedly debated."
Albert also made note of the "valleys of despair" the team faced along the way as well as including a list of lessons learned.
Despite its challenges, the team had executive sponsorship throughout the project.
Users were also engaged. "The users were more confident with the change because the re-platforming approach meant the same code was executing, and that meant very few errors occurred," wrote Albert.
There were issues related to the code. There was no inventory of 5% of the code, and there were about 50 programs running that no one knew anything about. The people responsible for it had long left.
Some of the issues were overcome by using a Fujistu compiler that worked with Pacbase and allowed the legacy code to execute in a Windows environment.
The department sought bids for x86 systems. The agency selected Unisys ES7600 servers.
If Unisys had backed off on the need for MIPS charges, Albert said "we would have a much different conversation."
The MIPS charges have been a longstanding approach in the mainframe market said Charles King, an analyst at Pund-IT, says that as onerous as the charges may seem "the vast majority of mainframe owners don't seem to have a problem it."
Vendors argue that the mainframe constitutes is such a unique technology "that it deserves a unique kind of sales, service and maintenance model," said King.
Unisys, in a written response to a query about Ohio's migration, said the department "made its decision to transition off of its ClearPath system before Unisys introduced the new NextGen platforms and modernization technology."
"Evolving the ClearPath line to a common Intel-based architecture has enabled Unisys to improve price/performance options for our clients and reduce maintenance costs to levels comparable to those for other Intel-based platforms," Unisys said.
The mainframe and the Windows servers ran in parallel for three-and-half months before the switch became permanent.
In the end, Albert estimates that the project will save the agency between $7 million and $10 million over the next five years.
Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov, or subscribe to Patrick's RSS feed . His e-mail address is [email protected].
Read more about data center in Computerworld's Data Center Topic Center.