Webhook SSL error with Let's Encrypt

I just created a ticket for this but I figured maybe someone else has some ideas. We configured a new webhook earlier but are getting the error response:

SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)

The development version we used over a week ago was working without issue.

We think the reason is likely due to a Let’s Encrypt root CA expiry that is out-of-date on the ShotGrid end. Or could this be cause by something on our end?

curling the endpoint works as expected, it is only ShotGrid that is unable to verify the SSL cert.

Hi, it looks like if we do a new build and deploy, it will discover Let’s Encrypt’s certificate changes automatically.
We will do an extra deploy today, to see if that helps you out! But the company does say there are other compatibility issues with some stacks. We have not heard from anybody about other certificate problems with webhook endpoints, so hopefully those issues will not affect you. A local test seems to confirm it will work with a new build.
I’ll ping here when that build is deployed.

I have confirmed that although @kevin-pxn is the only one who has reported the issue, 14 sites with webhooks have been affected by the “Let’s Encrypt” certificate expiry on Sept. 30. (I don’t yet know why it only affects those sites, or even why the new build fixes the issue. Certificate chain magic?)

Thank you for reporting this, @kevin-pxn , but with apologies, rather than do a Friday afternoon deploy, we would like to wait until Tuesday to update production with the new build.

This is because most SG webhook traffic has not been affected by this issue, and we are going into a long weekend for most of our staff, which would make it harder to mitigate if something new went wrong because of the deployment.

It was not an easy decision (we discussed pros and cons at length), but I hope you will understand! Even in public beta, we try to maintain a very high level of stability and reliability for the webhooks system.

Hi Neil, thanks for your reply. I completely understand not wanting to push updates to production on a Friday.

If you wouldn’t mind leaving an update on this thread when the update has been pushed, we can test if the issue has been resolved on our end.

In the meantime, enjoy the weekend :smiley: .

Okay, the deploy is now in production, completed at 13:53 UTC. I haven’t seen any recurrence of expired certificate errors since then, but they were sporadic, I will be looking forward to your confirmation, @kevin-pxn !