Since the last 48 hours, we noticed that the newer webhooks post does not carry X-Sg-Signature header anymore.
Is there a change in the API? or we are doing something incorrect at our end?
FYI, Our integration was functional till March 17th, the last request that was processed correctly.
Regards,
@n00p
2 Likes
Unrelated, but out curiosity, are you using or have you tried to use Serverless Lambda to process your Webhooks? Having an issue with x-signature signature authenticating. Not sure if itâs Lambda or Python 3. But unable to match sha1 codes. However, we are able to properly Authenticator via flask.
1 Like
Hi @aka_anoop,
Just turns out I was doing signature validation tests yesterday. I just reran my test and Iâm getting the signature header, no problem.
An excerpt of my Flask code that is getting the header looks like this and works fine:
if 'X-SG-SIGNATURE' in request.headers:
return request.headers['X-SG-SIGNATURE'][5:]
What behaviour are you seeing and if you drop the raw payload you get from Shotgun what does that look like? Iâm game to try and help you figure this one out.
2 Likes
Yes we are using lambda on py 3.7. The setup was functional until last week. Thatâs the reason we didnât check on that side.
Will verify the raw payload and get back to you @bouchep
2 Likes
Sorry for the late response. We did recently update the tech stack we use to deliver webhook notifications so that might be in play here. I just checked the code, we do send out the signature header (but as before, only if there is a âsecret tokenâ associated to the webhook).
Iâll run further tests with the old stack to see if I can pinpoint a difference in behaviorâŚ
2 Likes
I am also exploring lambda and webhooks. We are struggling with x-signatures but maybe having a different issue. I am getting the signature but unable to authenticate it. Maybe itâs related.
Also wondering if your doing anything special with your lambda. Are you using concurrent provisioning? Do you have a single lambda broker or are you sending every webhook to different lambda handlers? Are you using SQS to distribute your lambda load? Any insight would be great. Performance on lambda does not look that great when you expect a field on the page to update based on the lambda event. Itâs generally fine if the work load happens in the background.
1 Like
I ran a few tests with the old and new stacks (no changes in the test client) and not having issues with the signatures.
In my tests I did use somewhat trivial secret tokens.
The signature does ingest the entire body of the request.
Can you guys further quantify the problem you are having?
- Is this happening for all requests?
- Is this happening for all webhooks?
- If you use (different) simple secret keys, does that make a difference?
1 Like
Found the problem. Looks like the headers were changed to all lower case? The previous ones had X-Sg-Signature while the new one has x-sg-signature. Changing the key to that fixed the problem. Was this a recent change?
1 Like
We never ran in to problem with signature validation, in fact our integration has been active more than a month without any issues
Here is the snippet that we use, in case you find that useful.
body = event['body']
my_signature = 'sha1=' + hmac.new(WEBHOOK_SECRET.encode('utf-8'), body.encode('utf-8'), hashlib.sha1).hexdigest()
header_signature = event['headers']['x-sg-signature']
We have been using a single API gateway end point with multiple resources, each mapped to one lambda. The current use cases we have are not no frequent events, so we never ran in to scalability issues. Also we are not currently using the webhook events to trigger other entity updates yet as we are still migrating our old shotgun daemon based workloads to web hooks. So as of today, we havenât run in to scalability issues yet. Will share more info once we run in to one.
This is helpful and as Patrick and I expected. I am a noob at AWS Lambda and our event data from the handler is yielding a dict. As a result we are unable to run an encode operation on it since its a dict. Furthermore, translating it to a string with something like json.dumps(event, sort_keys=True) appears to yield results that canât be verified. How are you configuring your lambda handler to insure that event data is a string that you can run an encode on? Have you done something special with your serverless.yml file?
I am still unable to authenticate and not sure why. I think @aka_anoop has it working from AWS Lambda and I am trying to identify how he has gotten it to work.
@aka_anoop can you check your serverless.yml file to see if your setting the âraw argument? This might be why my authentication is not working.
-
--rawPass data as a raw string even if it is JSON. If not set, JSON data are parsed and passed as an object.
Pretty sure we identified the issue. The issue is related to Serverless. I started by using Serverless to help simplify the packaging and deployment to AWS. More recently we started migrating to a direct deployment to AWS removing the need for moving parts. In doing so we noticed that the payload delivery between these two different delivery methods were different. Serverless payload is a json object while AWS payload is a json string that must be converted. Hence the differences in Sha signature validations. I do not see options to change Serverless at the moment. So, for the novice lambda users, stick to lambda native deploy if you need signature authentications.
2 Likes
Hey there Romey, I think I was in a similar situation as you. Using AWS lambda and serverless together. I was using whatever JSON I could dig up via requests in flask to get the data, but as you said something gets lost in the translation and the result signature is a mismatch.
Digging through the documentation for serverless, I found out you can grab the raw âcontextâ and âeventâ data through an environment variable. (note these variables wont work locally with Postman, has to be run on the lambda)
print(request.environ[âserverless.contextâ])
print(request.environ[âserverless.eventâ])
[Serverless Framework: Plugins](https://Serverless Documentation)
When I fed the body.encode() into the signature I finally got a match.
Hope that helps, saved me from a lot of potential work.
Hi, PogOtter, would you be able to post a more detailed example?
I will expand the signature checking part of our documentation, since it sounds like this is a problem lots of people will run into, especially as it involves Flask and AWS serverless.
For anyone else who reads this, if youâve solved any special signature challenges related to the quirks of a specific language/framework/environment, feel free to post them here and @ me. If I have a few, I will make a signature validation ârecipesâ section for the docs.
It is one thing that often gets harder when the framework gets more helpful!
Iâll see what else I can share, but the gist is that I know that to properly generate a signature you need the secret token to be encoded and combined the with the proper request body without any of the bits altered right?
Anyways only way at first I could find to fetch the body of the request was through the python requests module. But I think requests slightly alters it somehow as I kept getting mismatched results. Try things like requests.body() or requests.json().
However digging through serverless documentation I found out you can get the raw âbodyâ of the request through an environment variable.
From there I could get the signature and generate a signature, compare and get a match.
Probably canât share more than that, Im open to other methods to get the raw body data from the event, but thatâs the only way I could find that worked.