After it was revealed the recent network outage for 30 million customers who use the O2 network was caused by expired software certificates, we are taking a closer look at the importance of signing and maintaining software audits.
As a recap, on Thursday the 6th of December customers of the wider O2 network, including Tesco Mobile, Giffgaff, Lycamobile and Sky Mobile, started reporting their 3G and 4G internet was down. Business users were without the means to book jobs, use GPS or access their emails; consumers were without any internet connectivity, even the Transport for London’s live bus alerts suffered. As the duration of the 4G outage extended into Friday morning, the extent of the issue and its impact gained further momentum.
The cause of this upheaval of services? Initially the problem was put down to an undisclosed third-party supplier which later was confirmed to be Ericsson, who’s network handles over 40% of the world’s mobile traffic. A statement from their CEO stated that it was in fact ‘faulty software’ which caused the ‘network disturbances’.
‘‘An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers.’’ Ericsson CEO Börje Ekholm
So, what exactly is software code signing, why is it important and what is best practice?
Software Code Signing in Practice
Software code signing is the process of verifying software to ensure it has come from a legitimate source. As with your handwritten signature on important documentation, signed software has a digital signature which can be used to authenticate both the software and its origin. Essentially, when you receive a piece of software which has been digitally signed (from a reputable certificate authority), you have a level of reassurance that the software is safe to use.
In the past software was predominately sold and delivered as a physical product, boxed and ready to be used, usually from a well-known source. Today, as with many other interactions and transactions, we have shifted to online platforms to distribute software. While this digital model has many benefits, including faster, cheaper and continuous delivery, it also increases opportunities for code to be altered or corrupted.
Within the information security realm, you will often see something called the ‘CIA triad’, which despite appearances, is not anything to do with the Central Intelligence Agency. Instead, the acronym refers to ‘Confidentiality, Integrity and Availability’ and it is used to guide organisational information security policies and best practice.
In practice, if an organisation has a confidentiality breach, then data on their system has been viewed or accessed by an unauthorised individual or entity. A loss of integrity can vary in scale and impact, but essentially refers to data which has been modified, damaged or deleted by unauthorised means. Finally, we consider the availability of systems. Is the system or software available as and when needed by legitimate users?
The concept has been furthered with the addition of ‘non-repudiation’. Techopedia’s definition of non-repudiation is ‘method of guaranteeing message transmission between parties via digital signature and/or encryption’. Essentially, the digital signature itself is proof of origin and cannot be repudiated or denied at a later date.
These concepts are vital in preventing unauthorised modifications to applications and software, whilst allowing for legitimate updates from the manufacturer. Having a valid certificate on your distributed software helps prove its integrity and promotes user trust. The extra level of assurance can encourage use and enables the distributor to build a positive reputation for safe software.
A code signing certificate is only valid for a certain period of time, after which, the certificate can no longer be used. The length of the certificate depends entirely on the use case; some bespoke projects may have certificates lasting 15+ years while others may only last 12 months. Upon expiry, the validation of identity and trustworthiness of the entity noted in the certificate must be reaffirmed before the certificate can be re-issued.
In the case of Ericsson, we do not know the full story of the incident and practically, there could be a number of reasons for this snowball effect across multiple global telecommunications networks. According to the latest reports, O2 is expected to bill Ericsson ‘tens of millions of pounds’ and the estimates for the international bill have reached £100 million.
While the root cause has been established as an expired certificate, there are multiple protocols and procedures from both the software creator and distributor which should have safeguarded against this from happening. Examples of these protocols include appropriate lifecycle management for discovery, expiration and deployment of certificates as a standard, particularly so for critical services. Additionally, adopting stringent testing practices for any certificate renewals within staging environment should reveal any issues before being released into production.