Indian Outsourcing Strikes Again!

[Ed P]

“British Airways brought to its knees world-wide by the catastrophic failure of their IT systems”.

Who could guess that an airline that is supposedly British would be so stupid as to outsource its critical IT to India. Unfortunately it is sadly the truth. Now it is remotely possible that computer operator error was not to blame, but sadly outsourcing of IT to India demonstrates that a company has not done a critical failure analysis and determined what cannot be allowed to fail and must be properly resourced with both competent staff and adequate equipment.

Brexit should take the opportunity to ensure that economic disincentives apply to all companies that outsource skilled jobs and systems overseas.

link

[dwynnehugh]

Might it not be the truth that for all national/multi-national companies the bottom line is £££££ and probably the accountants who ‘do the deal’ are more interested in company profits than total (if possible) data security and 24/7/365 reliability? These days how many IT decisions are made by accountants and not ‘people who know what they are doing?’

The more you meet people the more you understand why Noah took animals instead of humans

[Anonymous]

They officially blamed a Power Issue. I don’t understand how a power issue, which will be in one location, can take out the system for every plane around the globe.

What happened to having a redundancy?!

 

EDIT: I would take a quote from the Union with a pinch of salt of course though. They may not quite be impartial.

[Dave Rice]

Much less impartial than the management, believe me. I’ve been on the inside of outsourcing and it’s ALL about money.

The customer wants to save money and defers the pain / politics of redundancy and off shoring to someone else. The outsource company is after the assets, they don’t make money on the core contract. Both parties then only invest as little as possible, usually only in some shiny new things like apps that still run on the old core infrastructure.

It could well be a power supply issue that’s started a chain of events. But clearly it should never have happened. I have known a real case where a generator failover test failed because the diesel had been drained from it’s tank by the H&S brigade who didn’t tell IT. That was the London H&Q of a then Govt institution and thankfully would not have ground the business to a halt. All of my server rooms had a room UPS but they were never serviced or the batteries replaced (this was not in defence I hasten to add).

[Ed P]

I have worked with an Indian JV – I pity the poor BA expat who has to write the report explaining all this. CYA in the UK can be bad, but you ain’t seem nothing if you haven’t witnessed it in India.

The chances of a single point power failure cratering a world-wide network are slim to none. Accepting the official reason that it was a power failure, after much head wagging and nodding they will finally discover that it was all due to a poor untouchable with a mop who was immediately dismissed and has now disappeared.

[Dave Rice]

If you read the comments in the El Reg article there are some plausible scenarios where a power supply failure could trip a series of events. But of course the process design should never let that happen and that process should be properly tested. I have witnessed at first hand the fear of throwing the off switch and that has to be down to the culture.

I think the “brain split” scenario may well turn out to be the right answer, but we’ll never know.

[The Duke]

Not to worry, as the pound crashes, more call centres will return. Then we can take responsibility for our own breaches.

I don’t see what the surprise is that BA out source customer services. After all it isn’t a charity, and the air game is ruthless,run on fine margins.

[Ed P]

@Steve I could not find the ‘cynical’ emoji!

Of course I am not ‘surprised’ I am however surprised the the British Government fell into this trap. IT is a component of our strategic infrastructure and we should guard it jealously.

I have dealt with JVs, and Distributors over a greater part of the world (ex Eastern Europe or Russia) I would rank India as one of the most difficult places to find good reliable partners or agents. A major problem with Indian relationships is that many people still have a large anti-colonial chip on their shoulder. A requirement to immediately address a critical issue rapidly become a basis for discussion leading up to prolonged negotiation (very analogous to contract ‘Extras’) . A LOT of time can be wasted.  Dealing with any of India’s major surrounding countries is like a breath of fresh air in comparison.

[JayCeeDee]

Ed P wrote:

I could not find the ‘cynical’ emoji!

No – it’s not there, neither is the sarcasm emoji. Did we get the free version??? Damn, can’t find the p*sstaking or wind-up emoji!!

[The Duke]

I never know which emoji means what. Except for happy and sad. Also given each platform has different emojis, one thst you choose on your end that looks acceptable, can come across completely different on the recipients device.

Also to make my personal emoji life even more hell android in Android O has changed from their signature “squashed” emoji set, to a some horrid, probably more standard,  set of emoticons. They are ugly.

 

[Bob Williams]

The emoji to apply to any message to the BA management who allowed this to happen, should ber a very angry face on a very big hammer.

When the Thought Police arrive at your door, think -
I'm out.

[Alan Wood]

Seems it was jumping to conclusions to assume the BA problem was in India. The BA Chief Exec has just said it was a power surge in UK close to Heathrow that caused the problem.

[Ed P]

Even worse then, Tata totally failed to have a world-wide disaster recovery policy based on just  a one centre failure. Obviously there was no critical failure analysis and integration with the Tata infrastructure was naff.

We had a data centre right under a major flight path and one of our critical failure scenarios was a 747 crashing into the building and taking everything out, comms, systems, staff. The impacts of the failure of each major item  had aplan. All were then then merged into one overall horror scene. I will admit some scenarios needed 14 days to recover but all included sections on PR and close customer liaison. This was a key element in the Annual Audit (shudder) andfailure for the data centre management to have demonstrable rehearsed plans could be demotion or sackable. The disaster books were huge series of  tomes kept in two different locations plus the secure store.

Tata’s obvious failure to have such plans for a location that close to Heathrow is beyond belief. This is not an acceptable excuse – it is even worse!

[edit] I should have said our immediate backup was a ‘hot-site’ that would have recovered us to start of day. Still pretty horrible but it would buy time. I would have thought that today’s  tech, cloud and journalling only minutes would be lost and give a very viable situation. It is a shame the BBC obviously keep their own tech staff out of the loop or they could have primed the BBC reporter to respond with some horrible statements/questions.

[Dave Rice]

Alan, the location of the problem isn’t the issue. It’s the recovery plan.

I notice some very careful wording in use already.  “Alex Cruz has said the root cause of Saturday’s London flight-grounding IT systems ambi-cockup was “a power supply issue. They have all been local issues around a local data centre who has been managed and fixed by local resources,” he said.”

I’m sure that sparkies were not flown in from Mumbai and again that isn’t the issue. BA should be able to lose an entire data centre just like that. We know they have two in the UK, both near Heathrow Waterside, and I can’t believe both were taken out. I find it unbelievable that even 1 can be taken out by a power issue TBH.

I suspect that some old systems from various sources being glued together by people who have now left will be at the centre of it.

[Alan Wood]

I feel it is relevant considering the title of this thread:-

“Indian Outsourcing Strikes Again”.

together with several comments which refer to India.

I accept that wherever there is critical data such as this it should have a recovery plan based on several unconnected locations. I also find it strange that BA have two in the same location and both went down but I guess it would depend on what specific function each was providing and whether their data was interlinked. Again, if so, then to have one centre which is absolutely critical to the world wide operation of BA is bordering on criminal negligence.

[Ed P]

We obviously get into the realms of speculation as BA and Tata have been very tight-lipped about what actually happened. (Staff were warned not to talk) .So I’ll just lay-out observations:

a) Tata (an Indian conglomerate) have the responsibility for managing the IT operations of BA. i.e. outsourced to an Indian Company. Tata moved in their own people to take over UK jobs, in other words the location may have been the UK but Tata brought in cheaper labour who would have been unfamiliar with the UK hardware/comms – link

b) If nothing goes wrong, operating a data-centre is just a routine job. Long periods of boredom interspersed with some running around at month-ends or doing system restores etc. The real job of running a data centre kicks in when things break down or go wrong.

There should be a plan for every eventuality that will take down a data centre, with a hot-site being the primary route if and when the poop hits the fan. There should be regular rehearsals to ensure that everyone knows what to do – there is often a tome that handles the step by step procedures. A good tome will have PlanB sections for when these do not work. There was no Plan-B i.e. Indian outsourcing failed to deliver.

[edit]

Some limited data here.

and of course outsourcing brings its own problems in communication. link. Note comment about staff not being able to understand the fellow at the other end of the phone! It is bad enough ringing the BT help line but I’d hate having to do it for a fan full of poop!

 

c) Even if there was a total UK failure, for reasons unrelated to Tata (unlikely) why did this bring down the whole world network? The only reason I can think is that failure in the UK somehow brought down the Tata Corporate Cloud. implying very naff integration of the two. Total speculation based on the Tata glossy as there is no detail on the BA operation.

[Ed P]

The [edit] should go after c) – I have no idea what went wrong!

Incidentally the ‘cheaper’ Indian labour is all in getting few if any retirement/sick/holiday benefits, they actually probably get a fairly good contract salary.

[The Duke]

I think tata is one of the largest companies on the planet, iirc, so I’m sure they will sort it. Also they are meant to treat their staff very well indeed. One of the few multinationals that still hold staff happiness highly.

I seen a doc on them once, I’m sure the guy that started it all, was a cotton grower or fabric maker about 100+ years back

It was years ago I seen it, (probably 10+) I’d never heard of them before that. They now own a bit (alot) of everything worry wide.

[RSB]

I believe they do tetleys tea bags  :yes:

Americans: Over Sexed, Over Payed and Over here, Wat Wat!

[Ed P]

Tata’s IT salaries are here: (0.012 Rupees/£) . From the comments they do get sick pay

Approx/year:

Systems Engineer £6000

IT Analyst £9500

Associate Consultant £18700

Based on my (dated) India JV knowledge I’d say competitive with top quartile and 25% ish of UK

[Author]

Posts