SynopsisHDFC Bank is the largest credit card issuer with 1.49 crore cards is use according to RBI data. It is also one of the largest banks with regards to UPI transactions with 179.5 million transactions processed through its system in October.
MUMBAI: HDFC Bank‘s third technology outage in as many years is a consequence of its system not keeping pace with the bank’s strong growth in the last few years, experts said. The latest incident in November was caused by a power failure that was not rectified by back ups in time, bank executives said.
“We did not expect such a strong reaction from the central bank. We informed them about the problem in our data centre last month and were engaged with them to put in measures for the same. This hit us out of the blue because not being able to get new credit card customers would hit our profits for the next few months,” said a senior bank executive on the condition of anonymity.
HDFC Bank is the largest credit card issuer with 1.49 crore cards is use according to RBI data. It is also one of the largest banks with regards to UPI transactions with 179.5 million transactions processed through its system in October. All these businesses will be hit due to the RBI direction, the first such for the bank which is considered the best on the street.
In a message to the bank’s customers HDFC Bank CEO Shashidhar Jagdishan apologised for not being able to live up to “service quality and expectations.”
“We had two outages, one in Nov 2018 and second one in Dec 2019. We have taken help of external expertise, understood what needs to be done further and have substantially implemented the inputs to strengthen our IT infrastructure and systems. Unexpectedly another incident happened on Nov 21,2020 and the primary reason for the same is the power outage in our primary data centre. We are working on war footing to strengthen this area also now,” Jagdishan said.
The two outages Jagdishan was referring to was a failure of the bank’s new mobile app in December 2018 and a large server crash which led to a a suspension of digital transactions between December 2 and December 5 2019. Both these outages were blamed on higher than expected traffic by the bank.
The latest outage on November 21 was due to the shut down of the bank’s outsourced data centre in Navi Mumbai due to a power breakdown which could not be rectified in time as both the back up power supply, generators and standby batteries failed.
Former RBI deputy governor R Gandhi who also led the banking institute of technology of the RBI said HDFC Bank’s problems could be linked to a higher than anticipated increase in transactions.
“It looks like that they have not anticipated the increase in demand for digital transactions. All institutions including HDFC bank look at the trend growth in transactions and add additional capacity in terms of networks and storage. This time, we don’t know what happened whether they did not plan for the increase,” Gandhi said.
Sunil Gupta, cofounder of data centre infrastructure company Yotta Infrastructure said the recovery time for IT operarations could take 12 to 13 hours even if the power outage is only for 30 minutes.
“This is why the world’s most prominent certification bodies mandate that more than the main power source, data centers should focus on their own generating systems. Preventive maintenance of generator sets is also very important else they can fail when they are actually required at the time of an outage,” he said.
HDFC Bank CEO Jagdishan said the bank plans to fortify the identified areas for improvement. However, questions are being raised on the bank’s corporate goverance as it took a full three months for the bank to appoint a new chief information officer after Munish Mittal left earlier this year.
Experts say may be its time for the regulator to frame a clear policy of disaster recovery (DR) for banks.
“The framework should ensure that banks have a data centre and a near DR site within 50 kilometres distance and a far DR should be in another location. Drills should be mandated at these sites, and audit reports of these drills should be submitted to the RBI,” said Piyush Somani, CEO at ESDS Software Solution. A DR site is a facility an organization can use to recover and restore its technology infrastructure and operations when its primary data center becomes unavailable.