X-Sg-Signature is not found in headers any more?

Since the last 48 hours, we noticed that the newer webhooks post does not carry X-Sg-Signature header anymore.
Is there a change in the API? or we are doing something incorrect at our end?
FYI, Our integration was functional till March 17th, the last request that was processed correctly.
Regards,
@n00p

2 Likes

Unrelated, but out curiosity, are you using or have you tried to use Serverless Lambda to process your Webhooks? Having an issue with x-signature signature authenticating. Not sure if it’s Lambda or Python 3. But unable to match sha1 codes. However, we are able to properly Authenticator via flask.

1 Like

Hi @aka_anoop,

Just turns out I was doing signature validation tests yesterday. I just reran my test and I’m getting the signature header, no problem.

An excerpt of my Flask code that is getting the header looks like this and works fine:

    if 'X-SG-SIGNATURE' in request.headers:
        return request.headers['X-SG-SIGNATURE'][5:]

What behaviour are you seeing and if you drop the raw payload you get from Shotgun what does that look like? I’m game to try and help you figure this one out.

2 Likes

Yes we are using lambda on py 3.7. The setup was functional until last week. That’s the reason we didn’t check on that side.

Will verify the raw payload and get back to you @bouchep

2 Likes

Sorry for the late response. We did recently update the tech stack we use to deliver webhook notifications so that might be in play here. I just checked the code, we do send out the signature header (but as before, only if there is a “secret token” associated to the webhook).
I’ll run further tests with the old stack to see if I can pinpoint a difference in behavior…

2 Likes

I am also exploring lambda and webhooks. We are struggling with x-signatures but maybe having a different issue. I am getting the signature but unable to authenticate it. Maybe it’s related.

Also wondering if your doing anything special with your lambda. Are you using concurrent provisioning? Do you have a single lambda broker or are you sending every webhook to different lambda handlers? Are you using SQS to distribute your lambda load? Any insight would be great. Performance on lambda does not look that great when you expect a field on the page to update based on the lambda event. It’s generally fine if the work load happens in the background.

1 Like

I ran a few tests with the old and new stacks (no changes in the test client) and not having issues with the signatures.
In my tests I did use somewhat trivial secret tokens.
The signature does ingest the entire body of the request.
Can you guys further quantify the problem you are having?

  • Is this happening for all requests?
  • Is this happening for all webhooks?
  • If you use (different) simple secret keys, does that make a difference?
1 Like

Found the problem. Looks like the headers were changed to all lower case? The previous ones had X-Sg-Signature while the new one has x-sg-signature. Changing the key to that fixed the problem. Was this a recent change?

1 Like

We never ran in to problem with signature validation, in fact our integration has been active more than a month without any issues
Here is the snippet that we use, in case you find that useful.

body = event['body']
my_signature = 'sha1=' + hmac.new(WEBHOOK_SECRET.encode('utf-8'), body.encode('utf-8'), hashlib.sha1).hexdigest()
header_signature = event['headers']['x-sg-signature']

We have been using a single API gateway end point with multiple resources, each mapped to one lambda. The current use cases we have are not no frequent events, so we never ran in to scalability issues. Also we are not currently using the webhook events to trigger other entity updates yet as we are still migrating our old shotgun daemon based workloads to web hooks. So as of today, we haven’t run in to scalability issues yet. Will share more info once we run in to one.

This is helpful and as Patrick and I expected. I am a noob at AWS Lambda and our event data from the handler is yielding a dict. As a result we are unable to run an encode operation on it since its a dict. Furthermore, translating it to a string with something like json.dumps(event, sort_keys=True) appears to yield results that can’t be verified. How are you configuring your lambda handler to insure that event data is a string that you can run an encode on? Have you done something special with your serverless.yml file?

I am still unable to authenticate and not sure why. I think @aka_anoop has it working from AWS Lambda and I am trying to identify how he has gotten it to work.

@aka_anoop can you check your serverless.yml file to see if your setting the —raw argument? This might be why my authentication is not working.

  • --raw Pass data as a raw string even if it is JSON. If not set, JSON data are parsed and passed as an object.

Pretty sure we identified the issue. The issue is related to Serverless. I started by using Serverless to help simplify the packaging and deployment to AWS. More recently we started migrating to a direct deployment to AWS removing the need for moving parts. In doing so we noticed that the payload delivery between these two different delivery methods were different. Serverless payload is a json object while AWS payload is a json string that must be converted. Hence the differences in Sha signature validations. I do not see options to change Serverless at the moment. So, for the novice lambda users, stick to lambda native deploy if you need signature authentications.

2 Likes

Hey there Romey, I think I was in a similar situation as you. Using AWS lambda and serverless together. I was using whatever JSON I could dig up via requests in flask to get the data, but as you said something gets lost in the translation and the result signature is a mismatch.

Digging through the documentation for serverless, I found out you can grab the raw “context” and “event” data through an environment variable. (note these variables wont work locally with Postman, has to be run on the lambda)

print(request.environ[‘serverless.context’])
print(request.environ[‘serverless.event’])
[https://www.serverless.com/plugins/serverless-wsgi](https://Serverless Documentation)

When I fed the body.encode() into the signature I finally got a match.
Hope that helps, saved me from a lot of potential work.

Hi, PogOtter, would you be able to post a more detailed example?

I will expand the signature checking part of our documentation, since it sounds like this is a problem lots of people will run into, especially as it involves Flask and AWS serverless.

For anyone else who reads this, if you’ve solved any special signature challenges related to the quirks of a specific language/framework/environment, feel free to post them here and @ me. If I have a few, I will make a signature validation “recipes” section for the docs.

It is one thing that often gets harder when the framework gets more helpful!