Your Thoughts: TSB’s IT Blunder
The ongoing TSB IT meltdown has been strong evidence of the risks and challenges financial institutions face daily. It has caused mass uproar from customers and severely tarnished the bank’s overall reputation. TSB started a long-planned move of 1.3 billion customer records from its former parent company, Lloyds Banking Group, to Proteo4, a platform built […]
The ongoing TSB IT meltdown has been strong evidence of the risks and challenges financial institutions face daily. It has caused mass uproar from customers and severely tarnished the bank’s overall reputation.
TSB started a long-planned move of 1.3 billion customer records from its former parent company, Lloyds Banking Group, to Proteo4, a platform built by TSB’s Spanish owner, Banco Sabadell. The change-over, which started on Friday 20 April, was supposed to be completed over the weekend by 18:00 on Sunday. But on Monday morning millions of customers were unable to use online or mobile banking or had been given access to other people’s accounts.
Error messages and glitches meant paydays and company salaries were turned upside down across the UK. This has understandably caused a chain of problems across many sectors. TSB’s overall response has not been appreciated by the public and its customer service methods have been hugely questioned.
Below Finance Monthly lists some of Your Thoughts on TSB’s IT failure and its customer service approach.
Mark Hipperson, CTO, Centtrip:
Looking more closely at what happened and how the events evolved, it appears that some key IT best practices might have been omitted, such as:
- Production system access: it appears developers had access and were making live fixes to production. This is a big no-no in software development even in an ultra-agile DevOps environment.
- Rollback plan: when it all went wrong, it appeared there was no contingency plan or option to revert back.
- Incremental proving: it would have been more appropriate to first validate each change to ensure it was successful before moving to the next.
- Testing: It is pivotal to confirm all changes have been implemented successfully and work well. There are many different types of testing: user, operational, data migration, technical, unit and functional, which would have helped identify any issues before customers did.
- Early Live Support: it is crucial to make sure sufficient highly skilled staff are available immediately after the release in case things still go wrong.
And last but not least is proof of concepts (PoCs), which would have revealed any tech and planning errors. TSB should have run PoCs on test accounts, or even staff accounts, before the full release.
Alastair Graham, spokesperson, PIF:
Small business customers have reached a nadir in their relationship with traditional banking partners. Branch closures and the move of services online have meant that few now receive any active guidance or support from their bank in helping to grow their business.
At the same time, many feel that even basic banking services aren’t meeting their expectations. Even without issues such as the recent TSB banking crisis, businesses would like improvements to be made.Whether that is quicker account opening processes, simple lending or transparent and fair charges, the demand for alternatives is growing.
Tech innovations, combined with legislative changes such as Open Banking, mean that more products and services are being launched, designed specifically to meet the needs of small business customers. SMEs have already shown they will trust other providers when their banks fail to provide adequate services. This has been particularly evident where prepaid platforms offer more versatility, while still being a safe, secure and flexible method to transfer money.
Yaron Morgenstern, CEO, Glassbox Digital:
In today’s digital age, customer experience is more important than ever. This banking app drama has revealed how important it is to measure your consumer’s experience with complete visibility of any problems. This should really be an ongoing effort, and not just when you plan large scale back office migration. There are three fundamental tenets to an effective customer experience: observation of the customer journey via touchpoints, reshaping customer interactions, and rewiring the company’s services to align with customer expectations.
It is only through advanced digital analytics and AI technology that organisations can understand what is going through their customers’ minds. These are powerful tools for mapping out customers’ digital journeys from the moment they visit a website. This all goes to the heart of improving conversion in the digital customer journey.
Fabian Libeau, EMEA VP. RiskIQ:
The fact that TSB’s IT meltdown dragged on for such a long time, meant that customers were locked out of their accounts for extended periods. It also made them vulnerable to digital fraud in the form of phishing. TSB itself has warned more than five million customers that fraudsters have been attempting to take advantage of its IT breakdown to trick people into handing over information that could enable them to steal their money. Criminals exploiting brands to defraud stakeholders in this way is nothing new, and we know that financial institutions are a much-loved target for hackers, given the highly-sensitive and valuable information they’ve been entrusted with – it is therefore no wonder that cybercriminals are queuing up for an opportunity to impersonate the bank online.
Andy Barratt, UK Managing Director, Coalfire:
In the grand scheme of things, the TSB incident is perhaps not as significant an event as a nation-state hack like last year’s WannaCry. But it has still left many, including the ICO, concerned that a major ‘data breach’ occurred just weeks away from the implementation of the EU’s General Data Protection Regulation.
The power to hand out major fines that GDPR affords the regulator means that the price of poor data protection is about to become far easier to quantify. When the regulation comes into force at the end of the month, a breach like TSB’s would certainly require a Data Protection Impact Assessment and measures put in place to ensure a similar incident doesn’t happen in the future. At the very least, TSB will have put themselves on the ICO’s radar as ‘one to watch’ when GDPR comes into effect.
While the share price of Banco Sabadell, TSB’s Spanish parent, wasn’t overly affected by the incident, there could still be a significant financial consequence for the bank. We now know that a large number of customers are affected so the cost of rolling back any mistaken transactions as well as offering support, and potentially refunds, is likely to eat up a lot of operational resource. This event should be a reminder that data protection and the safeguarding of personal information has to be to priority for financial institutions.
Andy Barr, Founder, www.10Yetis.co.uk:
The best thing you can say about the TSB approach to public relations throughout its issues is that it is going to become the modern benchmark for university lecturers on how not to approach crisis communications.
From the very outset, TSB has failed in its approach to handling this ongoing crisis. Its messages have been wrong, even from its highest-level member of staff, the CEO. He has repeatedly issued statements that have been incorrect and that he has had to retract and apologise for.
TSB’s brand reputation is now circling the plughole and its Spanish owners could very well be forced down the route of a re-brand in the mid to longer term in order to try and recover their reputation. I fully expect a classic crisis communications recovery plan 101 to be rolled out, once this all dies down. Step one; apologise (usually full page ads), step two; announce an independent investigation, step three; a member of the C-Suite gets the Spanish Archer (El-bow), and then step four; another apology before trying to move on.
Whatever the final outcome, this has been a public relations disaster for TSB and they are very lucky that at the time that it happened there was so much other “hard news” going on such as Brexit, rail company re-nationalisation and, of course, Big Don, over the pond, constantly feeding the 24-hour news agenda.
Danny Bluestone, Founder & CEO, Cyber-Duck:
The TSB fiasco shows that many organisations vastly underestimate data migrations. Moving data on such a scale from an incumbent system to a different one is an inherently complex task. There are several steps to follow for a successful migration.
First and foremost, it begins with a considered strategy for structural changes that ensures no legacy data is made unusable and new functionality is accounted for. Banks like Monzo test new features within alpha and beta modes, so new pieces of functionality are tried and tested before a mass general public release. TSB would have been wise to utilise test scripts and automated testing to auto-test thousands of permutations from login to usage of the system. Relevant applications that monitor errors could have then detected issues early on.
TSB could have also used a run-book for deployment so all steps of deployment are documented. When an error was detected, TSB could have rolled back without data loss. Problems could also have arisen if TSB failed to use a testing environment that was identical to the production environment. As if there is even a slight difference, the user experience can break.
With regards to the application hosting, TSB should have an active engineering team monitoring performance 24/7. In our experience at Cyber-Duck – from working with numerous institutions including redesigning the Bank of England’s digital website – there really is no excuse for users to suffer. Complex data migrations can be dealt with in a secure and efficient manner if best practice methodology is followed.
Adam Alton, Senior Developer, Potato:
Software is difficult; Microsoft still hasn’t finished Windows. Trying to write a new piece of software or create a new system, and then migrate everything over to it in one go is likely to go badly. The chances of it working are incredibly slim. Instead, a migration in several parts would be better. Release small, release often. When Mark Zuckerberg said “move fast and break things”, you could interpret that as “you’re going to break things, so do frequent and small releases in order that you break as little as possible before you get a chance to fix it”. The problems with TSB’s migration appear to be multiple and disparate; error messages, slowness and capacity problems, users shown the wrong data. It seems unlikely that these stem from a single cause or single bug, so it would seem that they tried to do too much at once.
Coerced optimism: when under pressure to get something to work, it’s easy for a team of developers to wishfully believe that something is finished and working because they can’t see any problems, even though their experience tells them that the complexity of the system and the rushed job they’ve done means that it’s extremely unlikely to be free of issues. I wouldn’t be surprised if IT workers at TSB fell into this trap, leading to the premature announcements that the problems were resolved.
Denying that you have a problem is always a bad idea. Amazon Web Services (AWS) provide a detailed status dashboard giving a continuous and transparent view of any issues on their systems. They don’t deny that they occasionally hit problems but instead have a process in place for actively updating their customers with as much information as possible. This transparency and openness clearly win them a huge amount of customer trust.
Senthil Ravindran, EVP & Global Head, xLabs, Virtusa:
Fortunately for all involved, it seems as if the worst of TSB’s IT debacle is now behind it. But its botched migration led to more than 40,000 customer complaints in what was arguably the most high-profile banking error we’ve seen this year. Worse still, the technology itself isn’t to blame here – both previous owner Lloyd’s and the Proteo4UK system used by new owner Banco Sabadell have a good record in handling data. Instead, the responsibility here rests solely with TSB.
It mostly boils down to a lack of proper preparation on TSB’s part. Banks carry out small data migrations regularly, but a large-scale migration such as this typically calls for months of preparation. Actually moving the data isn’t the tricky bit; drawing the data from the siloes it’s stored in across the business and knowing how it’ll fit within the target system is the real challenge. This is why banks are increasingly looking to ‘sandbox’ the testing process; creating a synthetic environment with the data they hold to gauge how it’s likely to fit within a new system of record. Granted, this approach to testing doesn’t happen overnight, but when applied properly, it reassures banks that the actual migration will run smoothly.
This method would likely have spared TSB the disaster it has faced. Yet in reality, we’ll likely see similar high-profile stories appear over the coming months thanks to the combined pressures of GDPR and open banking. The former is forcing banks to bolster their data handling practices in order to avoid hefty financial penalties, while the latter is forcing banks to expose their data to all manner of third parties. Both initiatives are incredibly difficult for banks reliant on decades-old legacy IT systems to manage (indeed, it’s likely that the GDPR deadline this month may have added pressure on TSB to rush the migration through), and as the reality of this new banking environment begins to set in, expect to see other examples along the same lines as TSB’s.
We would also love to hear more of Your Thoughts on this, so feel free to comment below and tell us what you think!