How to upload files using rest API (Command Line & Terminal)?

Been reviewing the upload steps in the API documentation but so far haven’t been able get it to work.

In this case I’m trying to automate Version uploads…

Request for ‘upload URL’ is

curl -X GET https//myshow.shotgrid.autodesk.com/api/v1/entity/Version/{VersionId}/_upload?\filename={FileName} \
-H 'Authorization: Bearer {BearerToken} \
-H 'Accept: application/json'

Result is

{
	"UrlRequest": {
		"data": {
			"timestamp": "2021-11-23T20:20:05Z",
			"upload_type": "Attachment",
			"upload_id": null,
			"storage_service": "s3",
			"original_filename": "[FileName]",
			"multipart_upload": false
		},
		"links": {
			"upload": "https://[s3domain].amazonaws.com/[longstring1]/[longstring2]/[FileName]
				?X-Amz-Algorithm=[Alg]
				&X-Amz-Credential=[Creds]
				&X-Amz-Date=[Date]
				&X-Amz-Expires=900
				&X-Amz-SignedHeaders=host
				&X-Amz-Security-Token=[Token]
				&X-Amz-Signature=[Signature]",
			"complete_upload": "/api/v1/entity/versions/{VersionId}/_upload"
		}
	}
}

then the Upload request is

curl -X PUT https//myshow.shotgrid.autodesk.com/api/v1/entity/Version/${VersionId}/_upload\
?filename=[FileName]\
&signature=[Signature]\
&user_id=15\
&user_type=HumanUser\
&expiration=[Expiration]` \
-H 'Content-Type: */* \
-H 'Accept: application/json' \
-H 'Authorization: Bearer ${BearerToken}'
-d '@[FilePath]'

And I get the following error

{
	"errors": [
		{
			"id": 1,
			"status": 400,
			"code": 103,
			"title": "Signature did not match",
			"source": {},
			"detail": null,
			"meta": null
		}
	]
}

Hi @Cmq

I believe that you need to upload your file to the upload location returned by your initial call.

Media is stored on Amazon’s S3, so this is where you need to upload the file, using all of the X-Amz-* headers in your call.

-Patrick

@patrick-hubert-adsk
Thank you for that. Is there specific documentation for this that you’re aware of?

I’m familiar with Amazon’s aws-cli library, but based your note, that’s not in play here.

Running with your suggestion I did the follwing…

		curl -X POST \
		-H 'X-Amz-Algorithm=XXX' \
		-H 'X-Amz-Credential=XXX' \
		-H 'X-Amz-Date=XXX' \
		-H 'X-Amz-Expires=900' \
		-H 'X-Amz-SignedHeaders=host' \
		-H 'X-Amz-Security-Token=XXX' \
		-H 'X-Amz-Signature=XXX' \
		-d '@/Volumes/Path/To/File' \
		'https://[subdomain].amazonaws.com/[longstring1]/[longstring]/[filename]'

The URL above was derived from the returned URL taking everything to the left of the query string. The headers were all derived by parsing the query string itself

Results so far are an error saying the data set included in the headers is too large (see below)

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<Error>
 <Code>RequestHeaderSectionTooLarge</Code>
 <Message>Your request header section exceeds the maximum allowed size.</Message
 <MaxSizeAllowed>8192</MaxSizeAllowed>
 <RequestId>XXX</RequestId>
 <HostId>XXX</HostId>
</Error>

Hello @Cmq

S3 uploads are done via the PUT http command.
You can use the upload link url like this:

curl -X PUT \
     -H 'Content-Type=%content_type%' \
     -H 'x-amz-meta-original-filename=%filename%' \
     -d '@/Volumes/Path/To/Upload/Image' \
     '%receivedUploadLink%'

Where
%content_type% is the content type of the file you want to upload ( ex: image/jpg )
%filename% is the filename of the file you want to upload ( ex: image.jpg )
%receivedUploadLink% is the “UrlRequest” > “links” > “upload” field of the Upload URL request result

1 Like

I also ran into this problem in my own troubleshooting, and I was able to perform a file upload using the REST API. Unfortunately, the docs are a little confusing, so it was a bit tricky to see what was needed to be done. However, with the help of @daniel.sauve comment, this is what is needed to be done to upload a file to ShotGrid.

  1. Create the Upload URL
some_file="my_zip.zip"
curl -s -X GET \
   -H "Accept: application/json" \
   -H "Authorization: Bearer ${access_token}" \
   "https://${YOUR_SITE}/api/v1/entity/${ENTITY_TYPE/${ENTITY_ID}/_upload?filename=${some_file}"
  1. Using the upload link found in the response above, pass that into this CURL response
curl -s -X PUT \
   -H "Content-Type: application/zip" \
   -H "x-amz-meta-original-filename=${some_file}" \
   -T "${some_file}" \
   "${UPLOAD_LINK}"

Note that in my use case, I was uploading a ZIP, hence using -T as well as specifying that Content-Type . Also, this CURL does NOT return anything. If you look at the curl status code, a 200 means it worked. You also don’t need to pass in an auth token in the header; the upload URL being a S3 endpoint will have credentials in its URL query.

  1. Use the information from the first CURL to create a JSON body to pass into the final CURL to complete the upload.
curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -H "Authorization: Bearer ${access_token}" \
  -d "{\"upload_info\":{\"timestamp\":\"${timestamp}\",\"upload_type\":\"${upload_type}\",\"upload_id\":${upload_id},\"storage_service\":\"${storage_service}\",\"original_filename\":\"${original_filename}\",\"multipart_upload\":${multipart_upload}},\"links\":{\"upload\":\"${upload_url}\",\"complete_upload\":\"${complete_url}\"},\"upload_data\":{\"display_name\":\"${some_file}\"}}" \
"https://${YOUR_SITE}${complete_upload_url}"

This is needed in order to actually SEE the uploaded file in ShotGrid. Like the second CURL call, this does NOT return anything, only a status code. A status code of 201 means that this call worked.

As for some of the parameters, a S3 URL for the upload_link doesn’t need a upload_id to be specified in the final CURL call. If you need to get that upload_id, query the entity that is associated with this upload and examine its fields. For instance,

real_upload_id=$(curl -s -X GET \
   -H "Accept: application/json" \
   -H "Authorization: Bearer ${access_token}" \
   "https://${YOUR_SITE}/api/v1/entity/CustomNonProjectEntity01/${ENTITY_ID}" | jq -r '.data["attributes"]["sg_content"]["id"]'
)

This example allowed me to get the upload ID of the entity that I was uploading towards.

I figured I’ll leave this here for anyone else who runs into this issue in their searching online. Hope this helps!