Last updated: 
4 months 1 week ago
Group Manager
Welcome to the Jisc Certificate Service group. For an update on the NEW Jisc certificate service please follow the below link. The New Jisc Certiface Service  The service offers a number of different X509 SSL certificates, including Extended Validation certificates that give users the highest possible assurance, as well as S/MIME email certificates for digitally signing emails. Jisc has an agreement with the Certificate Authority, QuoVadis who is the provider of the certificates. The service has been running since 2006 and has issued many thousands of certificates to organisations in UK research and education. This is a Community group where users can obtain relevant information, receive service updates and provide feedback.

QuoVadis Intermediate Revoke Update

18 January 2021 at 4:51pm

On Jan 14th, at 19:34:34 2021 GMT, Digicert revoked a version of the “QuoVadis Global SSL ICA G2” and “QuoVadis Global SSL ICA G3” intermediate certificates used to issue our OV certificates, without advance notification to Jisc. Many other users globally have been affected by this.                                                                                                                                                                                               

For anyone still having issues following our direct comms on friday, the guidance on fixing is pretty straightforward: change the intermediate you have configured in your webservers (e.g. apache, nginx, IIS) from the old version to the new version of the relevant intermediate – which in 99.9% of cases is the G3 intermediate. That needs replacing in the cert chain with the cert found here:

To aid identification, the fingerprints are:

  • The old, revoked version has a SHA1 fingerprint of E9:0B:CC:A3:D1:34:12:7E:F6:46:E8:54:72:3F:13:7D:79:71:DB:64
  • The new version has a SHA1 fingerprint of D4:66:18:CA:00:5D:4F:F3:7F:3B:14:00:93:D5:81:E0:63:CA:5A:E4.

So far, every instance we’ve had reported to us where that fix didn’t seem to work has been caused by caching issues (either in the browser, transparent proxies on network/VPNs, etc etc).

One behaviour we have seen is that whether users are affected is partly based on their browser and OS platform. Mac users see the issue – and the subsequent fix – faster than windows users. This is because Mac browsers seem to pick up on revocations of certs much faster, something to the Mac keychain vs Windows certificate store works, possibly. There seem to also be reports of some browsers still thinking the old chain is in place even though the new chain is being presented. A potential solution for this issue is to open a terminal and issue a “crlrefresh rpvv” command, which seems to fix the issue in some/many cases.

So to confirm, for anyone still having issues, the guidance is:

  1. Make sure the intermediate is updated on each server ASAP.
  2. (For IIS servers, you’ll have to import the new cert into the certificates snap-in, remove the old one, and restart IIS. Some are reporting having to reboot the server as well)
  3. Run that site through SSLLabs to confirm, in a way that will not be affected by caching, that everything is happy - https://www.ssllabs.com/ssltest/ - if your chain is misconfigured, or still using the old intermediate, you’ll get a “T” result. If configured correctly, will get e.g. an A+, A, B, etc (depending on the rest of your configuration).
  4. If you operate any transparent proxies on network, or on VPN appliances, etc, see if you can get the certs stored cleared.
  5. From then on, if users are still seeing issues, ask them to clear their browser cache.
  6. If that still doesn’t fix it, get in touch with us on certificates@jisc.ac.uk and we can try to diagnose further with you – but so far, doing steps 1-5 has usually gotten everyone working.

We have also received an official response from DigiCert + QuoVardis below.

-----------------------

As part of our efforts to remain current with browser root store compliance requirements and to advance industry best practices, QuoVadis has been rotating intermediate certificate authorities and providing new intermediates over the last several months. 

Accordingly, on January 14, 2021, QuoVadis revoked legacy certificates for the following CA versions:

Name

SHA256 Fingerprint of Revoked Version

QuoVadis Global SSL ICA G2

A4879EC0F36CF84B6F2ED87AE57EE3B94A0785C6862238CD45481084D152EB18

QuoVadis Global SSL ICA G3

CAB9C12DBDE3AD5D2BC0201B54B18BE209CD5E146AAA085ABBDF241B096DFF47

QuoVadis Grid ICA G2

74CE8C1631EF9F38E7A4197DA3F5474DBC34F001F2967C25B5999562BCC8C9D4

QuoVadis Enterprise Trust CA 2 G3

174E1DE77C8D93C68ECD2BD2EA6E191B584DB850277A834AAC898B7C80A91C70

End entity certificates issued after September 22, 2020 were issued with the new chain and not impacted. End entity certificates issued before that date may require the new intermediate CA installed in the chain. The current/updated CA certificates have been delivered via TrustLink Enterprise and the QuoVadis Repository since September 2020, when the intermediate CA rotations began.

The updated intermediate CA versions are: 

We understand the inconvenience this may cause some administrators, and our local support teams continue to assist any customer in need. We invite those requiring assistance to contact us at support.ch@quovadisglobal.com. For documentation on how to chain to the new intermediates, please see the knowledge base at: https://knowledge.digicert.com/quovadis.html.   

Comments

I'd like some clarification on the statement "End entity certificates issued after September 22, 2020 were issued with the new chain and not impacted. End entity certificates issued before that date may require the new intermediate CA installed in the chain."

The most recent certificate on my account is #313326, which was issued on 2021-01-13 (long after 2020-09-22). I've just double-checked - I re-downloaded the certificate zip from JCS, unpacked it and calculated the fingerprint:

$ openssl x509 -in 313326/RootCertificates/QuoVadisOVIntermediateCertificate.crt -noout -fingerprint -sha256
SHA256 Fingerprint=CA:B9:C1:2D:BD:E3:AD:5D:2B:C0:20:1B:54:B1:8B:E2:09:CD:5E:14:6A:AA:08:5A:BB:DF:24:1B:09:6D:FF:47

So that's the revoked ICA, in the pack for a certificate that was registered just one day before they messed it all up. 

I don't have any certificates issued since Digicert revoked the 'old' QV-SSL-G3, one would hope that they're not still issuing certificate packs with the revoked certificate, but it seems that up until the day before the ICA was revoked they were giving out the wrong intermediates.

As far as I can aee, we need to fix *all* OV certificates that have been issued by JCS, not just ones up to September.

Should we expect the same 'upgrade' on EV certificates?

Hi Steve - Digicert started issuing end entity certificates with the new intermediate as they say, however, only for those directly downloading using Trustlink (their backend). We weren't informed of the change so didn't know to change the intermediate distributed through our community site certificate service app, so I'm afraid all certs downloaded up until last friday at about 15:00 had the old intermediate present. Downloads after then have the new one.

EV certs are issued through a different intermediate that is unaffected by this issue.

Hi Rhys! I'm aware that the current issue doesn't affect EV certs, but the DigiCert/QuoVadis response makes reference to this being an "ongoing effort" rather than a one-off event.

It would be unfortunate if having had this incident we were to have a repeat with EV certificates. I'm asking in the hope of avoiding such embarrassment.

There seems to be a page that lists the current ICAs at https://knowledge.digicert.com/quovadis/ssl-certificates/ssl-general-top..., but since the links on the page do not seem to function I cannot confirm whether those published ICAs match the ones that we're currently using.

We use EV certs for our main institutional website, as well as SAML IdP and WebSSO. I'd really prefer to avoid a repeat of last weeks incident.

Can we get confirmation of if (or when) any other QV ICAs will be revoked?

Hi Steve,

To the best of our knowledge, there are no plans to be revoking the QV EV issuing intermediates.

Rhys.