14-10-2016, 09:07 #1
[EN] GlobalSign cert error sees browsers block top websites
Oct 14 2016
A revocation error at security certificate provider GlobalSign has sent parts of the internet into meltdown after web browsers refused to load websites incorrectly labelled unsafe.
Whilst attempting to clean up some of its root certificate links, GlobalSign revoked a cross certificate that had linked together two root certifications, which should not have been removed. GlobalSign manages a number of root secure sockets layer (SSL) certificates that authenticate the identity of internet hosts.
This revocation request caused browsers to infer that all certifications downstream of the cross-signed root had also been revoked.
It meant that some of the world's top websites - like Dropbox and The Guardian among many others, small and large - were labelled as 'insecure' by web browsers, preventing access for security reasons.
While the provider quickly removed the affected cross-certificate and cleared its caches, the onus is now on GlobalSign customers to replace their SSL certificates to restore access to their sites.
Additionally, the "global nature of CDN [content delivery networks] and the effectiveness of caching" meant that some of the corrupt certificates made their way to end user systems, GlobalSign said.
Affected sites could remain blocked by browsers for four days until the cached responses expire, given end users "cannot always eaily clear their caches, either through lack of knowledge or lack of permission", the certificate authority said.
The firm admitted the situation was "not ideal", and said in the meantime it would provide an alternative issuing certificate authority for customers that has been issued by a root not affected by the revoked cross.
"We are currently working on the detailed instructions to help you resolve the issue and will communicate those instructions to you shortly," GlobalSign chief product offier Lila Kee told customers.
14-10-2016, 09:15 #2
GlobalSign security certificate foul-up knocks out secure websites
A security certificate mix-up has frozen hundreds of thousands of websites.
Steven J. Vaughan-Nichols
October 14, 2016
If you can't get to some of your favorite websites today, it's may not have a thing to do with your browser or ISP. The blame likely goes to GlobalSign, a Belgium-based security certificate provider. The company fouled up a clean-up of some of their root certificates links. This resulted in many "secure" websites showing up as being insecure and, depending on your web browser, unavailable.
In a word: Yuck.
If you got an error message such as: "You cannot visit www.example.com right now because this certificate has been revoked," you ran into this problem. You can easily bypass this error in most web browsers and operating systems. So, for most non-technical users, any website using a GlobalSign certificate for security is essentially offline.
Here's how it happened. GlobalSign manages several root Secure Sockets Layer (SSL) certificates. A root SSL certificate is a certificate issued by a trusted certificate authority (CA). They are essential for the web's security.
For browser compatibility GlobalSign linked several cross-certificates between those roots to maximize. So far, so good. Then they decided to remove some of those links. In the process they revoked a cross-certificate linking two roots together that should not have been touched. It kept working... for a while. Then, their Online Certificate Status Protocol (OCSP) server started reporting that the cross-signed root had revoked all the downstream certificates.
The good news is that GlobalSign has removed the cross-certificate from the OCSP database and cleared all its caches.
The bad news is GlobalSign customers need to replace their SSL certificates. That's not too bad. It's what system and network administrators are paid to do.
The really bad news is those same corrupt certificates are now on end-user systems. There they will block the affected sites for as long as week. There are ways to fix this, but I only recommend them for powers users. Feeling up to the job? OK, here you go:
Windows users need to take the following steps:
Go to Start Menu > Run Type cmd and press Enter
Then enter the following command:
certutil -urlcache * delete
On a Mac, you need to open a terminal and enter the following command:
sudo rm /var/db/crls/*cache.db
This will delete the following files:
Finally, on a Linux desktop, you open a shell program and run this command:
If you then see a message such as "No such file or directory," then your desktop hasn't been set up to cache SSL certificates.
In all these cases, after taking these steps you should be able to reach the sites again once their system administrators have installed the new certificates.
14-10-2016, 09:23 #3
Dear Valued GlobalSign Customer,
As most of you are aware, we are experiencing an internal process issue (details below) that is impacting your business. While we have identified the root-cause, we deeply apologize for the problems this is causing you and wanted to ensure you that we are actively resolving the issue.
GlobalSign manages several root certificates and for compatibility and browser ubiquity reasons provides several cross-certificates between those roots to maximize the effectiveness across a variety of platforms. As part of a planned exercise to remove some of those links, a cross-certificate linking two roots together was revoked. CRL responses had been operational for 1 week, however an unexpected consequence of providing OCSP responses became apparent this morning, in that some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.
GlobalSign has since removed the cross-certificate from the OCSP database and cleared all caches. However, the global nature of CDNs and effectiveness of caching continued to push some of those responses out as far as end users. End users cannot always easily clear their caches, either through lack of knowledge or lack of permission. New users (visitors) are not affected as they will now receive good responses.
The problem will correct itself in 4 days as the cached responses expire, which we know is not ideal. However, in the meantime, GlobalSign will be providing an alternative issuing CA for customers to use instead, issued by a different root which was not affected by the cross that was revoked, but offering the same ubiquity and does not require to reissue the certificate itself.
We are currently working on the detailed instructions to help you resolve the issue and will communicate those instruction to you shortly.
Thank you for your patience.
Chief Product Officer
US +1 603-570-7060 | UK +44 1622 766 766 | EU +32 16 89 1900
14-10-2016, 09:37 #4
GlobalSign screw-up cancels websites' HTTPS certificates
Revoked certs may linger for days, locking people out of big and small sites
Alexander J Martin, Iain Thomson and Chris Williams
13 Oct 2016
Final update GlobalSign's efforts as a root certificate authority have gone TITSUP this afternoon – that's a total inability to support usual protocols.
The result is that many websites big and small have had their HTTPS certificates incorrectly scrapped, meaning that for some people their browsers no longer trust websites and refuse or are reluctant to access them.
Specifically, it appears GlobalSign inadvertently triggered the revocation of its intermediary certificates while updating a special cross-certificate. This smashed the chain of trust and ultimately nullified SSL/TLS certificates issued by GlobalSign to its customers. It could take days to fix, leaving folks unable to easily read their favorite webpages.
GlobalSign estimates it could take until the beginning of next week for websites' accidentally axed certs to be corrected. The organization has set up a support page for IT administrators and folks looking to fix broken HTTPS certificates.
GlobalSign claims the worldwide mass revocation is an "unexpected consequence" of internal changes it made, and that browsers and other software "incorrectly inferred" that certificates had been burned.
If you're not affected by today's outage – consider yourself lucky as the problem won't hit everyone due to the wide range of caching and revocation policies employed by different browsers, apps and other software. If your application hasn't picked up the revocations yet, it should be fine – if it has, you can try to delete your certificate revocation list cache (see the above link for instructions on Windows and macOS) to see if that helps.
"That's the unfortunate thing about PKI, different browsers have different update levels," GlobalSign's strategic projects director Steve Roylance told The Reg.
Wikipedia blocked in Google Chrome after its HTTPS cert was accidentally revoked
Just hours ago, it became clear that GlobalSign – a New Hampshire, US-based biz – was having troubles with its Online Certificate Status Protocol (OCSP), which is used for obtaining the revocation status of public key certificates which ensure that netizens are connecting to legit sites using SSL/TLS.
"We are currently experiencing a known issue which is causing certificate revocation/error messages to be displayed within some of our certificates," a rep for GlobalSign tweeted earlier.
We are currently experiencing issues with our OCSP which is causing certificate warning messages. We aim to fix this as soon as possible.
— GlobalSign (@globalsign) October 13, 2016
Responding to complaints on Twitter, GlobalSign said it had sorted out the issue on its end, but stressed it'll take time for the changes to work their way through the internet's maze of caches. The web company's status page states:
We are currently experiencing a known issue which is causing certificate revocation/error messages to be displayed within some of our certificates.
Unfortunately, the cache laundering is a tricky process that not everyone can follow, meaning less technology-literate peeps may struggle with certificate errors for some time.
If the OSCP/CRL cache clearing hasn't worked, we're still working on a resolution. We deeply apologize 4 the outage & will keep u updated.
— GlobalSign (@globalsign) October 13, 2016
As of publication, people are up in arms about how long it's going to take to correct the dodgy revocations.
@globalsign huge huge impact on our 5 webshops. do we have ETA on the fix?
— Rémi NGUYEN (@reminguyen) October 13, 2016
@globalsign We will start exploring other options very soon if there is no resolution or ETA in the next few hours.
— Hjalmar Theodorsson (@hjalmarth) October 13, 2016
Sites affected include the Financial Times, Guardian, Wikipedia, Logmein, and Dropbox.
This afternoon, a spokeswoman for GlobalSign shed some more light on the outage:
GlobalSign manages several root certificates and for compatibility and browser ubiquity reasons provides several cross-certificates between those roots to maximize the effectiveness across a variety of platforms.
As part of a planned exercise to remove some of those links, a cross-certificate linking two roots together was revoked. CRL responses had been operational for one week, however an unexpected consequence of providing OCSP responses became apparent this morning, in that some browsers incorrectly inferred that the cross-signed root had revoked intermediates, which was not the case.
GlobalSign has since removed the cross-certificate from the OCSP database and cleared all caches. However, the global nature of CDNs and effectiveness of caching continued to push some of those responses out as far as end users. End users cannot always easily clear their caches either through lack of knowledge or lack of permission. New users (visitors) are not affected as they will now receive good responses. The problem will correct itself in four days as the cached responses expire, which we know is not ideal. However, in the meantime, GlobalSign will be providing an alternative issuing CA for customers to use instead, issued by a different root which was not affected by the cross that was revoked but offering the same ubiquity.
Meanwhile, this is what GlobalSign's telling its customers...
@JZdziarski @TheRegister Here is what they sent out to us, the customers. pic.twitter.com/qADNP8x5ez
— Koen Rouwhorst (@koenrh) October 13, 2016