I wonder what went wrong?

  LastChip 17:19 14 Nov 2009

The Halifax Bank today, suffered a power outage that has caused widespread disruption to its services. click here

For a business that relies heavily on IT services, one has to ask why this happened.

My thinking is, in a major data-centre, if it suffers a power failure, the UPS's kick in and while they have a limited life, it should be sufficient time for the emergency power to come on-line (normally diesel generators) and restore normality, until such times as the main grid starts to function again.

If this works as planned, no disruption should occur, at least, not to front-line services.

At around 16:30, I tried to access one of my accounts and while I could log in, no data was available, so clearly, normal service had not yet resumed.

I wonder if this is yet another classic example of the bean counters saying, we don't need emergency generators and having to hire them in and get them on-line before any sort of remedial work could be done? Just a guess; but if it's more than that, I would think there are some very worried people in IT at the Halifax right now!

  Diemmess 17:56 14 Nov 2009

Well it IS Saturday!

Ten years ago or thereabouts, the BT main exchange for this part of the county lost power when an unrelated JCB cut the mains cable.

Standby generator/s wouldn't start and batteries began to run down. By the time a power source had been connected, peripheral exchanges had gone down and it was several more hours before everyone had a BT connection again.
999 was affected like everything else, but fortunately no major incidents occurred, and police cars had the job of touring the streets of a major city announcing that it was no use trying 999 for the present.

  spuds 18:56 14 Nov 2009

It wasn't only Halifax who had IT problems within the last few days. Barclaycard had some problems, and one bright spark in a call centre informed me that it would take two weeks to remedy the problem.The second person suggested that " Try tomorrow or call into your bank". Took four telephone calls over 12 hours before I was able to clear some statement accounts. Not a very happy bunny :O((

Had this very same problem about a year ago, and was informed that it was my problem if I was unable to make an on-line payment. That also took four telephone calls to resolve. I wonder if any of the management team have a 'The buck stops here' momento on their desks!.

  Forum Editor 23:44 14 Nov 2009

I was working in Richmond (Surrey, not Yorkshire), and quite suddenly one day the office lost its broadband connection. After a bit of faffing and phoning around we discovered that a digger driver had taken out a fibre-optic cable and removed internet access from hundreds of people.

All the emergency generators in the world wouldn't have helped in that situation. These things happen, and when they do we're all given a sharp reminder of how dependent we've become on the internet.

In this case the HBOS emergency generators brought the bank's ATM and card transaction services back up by 2:30 p.m. but it seems to have taken longer to restore the on-line banking system.

It shouldn't happen but it does, and it will no doubt do so in the future. We'll survive, but it is incredibly irritating when it occurs - there's never a good time.

  LastChip 01:02 15 Nov 2009

But in this instance, it was clearly reported as a power failure and I'm sure we both know, there are clear alternatives that one would expect to be in place, for a business the size of The Halifax Bank.

Even though I assume the data feed was not compromised, it would also be reasonable to expect two independent feeds to an operation of this size.

I'm also somewhat surprised, they (apparently) didn't have any capability to re-route through an alternative data centre.

I wouldn't have expected a single point of failure to bring a fairly major bank to its knees.

It is also interesting to speculate whether other banking operations route through this centre. For example, if their dealing desks rely on this centre to trade, what would it have cost them on a weekday when the markets are open?

  Forum Editor 09:13 15 Nov 2009

and when there's a problem it's not just a simple matter of re-routing traffic via another data centre.

I've done a fair bit of work with the banking industry, and I've seen at first hand how difficult it can be to keep an on-line service running securely when a system goes down.

I'm not defending HBOS here, I don't have all the facts, but I think it's a bit of an exaggeration to say that the bank was brought to its knees - that wasn't the case at all.

  LastChip 11:17 15 Nov 2009

But I don't think their customers (or even their staff) would see it as anything but bringing the service to its knees.

When customers can't access an ATM on a Saturday to get cash, or pay for goods via their credit card, or are limited to only £200 from a branch when they have a healthy account balance (incidentally, I think on balance, the last, was probably a good decision), how else would you describe it?

  spuds 13:41 15 Nov 2009

Over the last twelve months, I and a number of other people have had the 'Computers Gone Down' syndrome, so I am not sure if its an increasing event of perhaps daily life failures and tasks.

Here's three examples:
Local branch of a well known countrywide chain of newsagents. Their system went down, and the shop assistants initially suspended any transactions, until someone suggested that they used a calculator and notepad, and did the necessary inputs, when the system came back on-line. That worked very well.(Customer satisfaction!).

Another was with another well known store, who took the people on trust and completed the transactions.(Customer satisfaction!)

The third case was another fairly well known store whose manager decided that the store closed and people left their goods for possible collection later.(Quite a number of dissappointed people that day!).

Only hope that the National Grid doesn't have a routing problem, the we will be in a fine mess ;o)

This thread is now locked and can not be replied to.

Elsewhere on IDG sites

Alienware 17 R4 2017 review

These brilliant Lego posters show just what children's imaginations are capable of

Mac power user tips and hidden tricks

Comment réinitialiser votre PC, ordinateur portable ou tablette Windows ?