Support exporting source media to google drive

[philmcmahon]

What does this change?

This PR adds a new option to the export page to export the original source media to google drive. As part of this I've redesigned the export page - it now looks like this:

The export has several stages - see below for screenshots of that

On the assumption that, following this change, users will frequently be exporting more than one file, all exported files (even if there's just one) are now stored in a subfolder of the 'Guardian Transcribe Tool' folder (suffixed with date/time in case they have multiple files with the same name).

I decided to use a lambda to perform the export to google drive. This has the advantages of being easier to setup than an ECS task, and faster to start. It has the disadvantage that we can only export files up to 10GB (the maximum ephemeral storage), and we only have 15 minutes to do the upload. In my (limited) testing, I found that the lambda was able to export a 1.2GB file in 70 seconds, so I suspect we'll be limited more by the max file size than the timeout - but only just.

I had to use a separate lambda function for this rather than the API itself because API gateway has a 30s timeout, and once the lambda returns a http response it gets terminated. There are workarounds to this but I couldn't find anything that works nicely with serverless-express so I decided to create a separate function (this has the advantage that we don't need our API lambda to have loads of memory/disk space).

Some error reporting exists for if the file is too large. I still need to add an error for if the lambda times out whilst performing the export - might leave for a future PR though.

The feature relies on the file extension to tell google drive what type the file is - this seems to work reasonably well. A future feature could run apache tika or something similar on the file to determine the file type.

In theory the uploadFileToGoogleDrive function should be streaming the file 128MB at a time, in practice I found that the function ran out of memory when uploading a 1.2Gb file when the lambda only had 512MB. This needs more investigation - for now I have set the memory to 2GB. I think it's worth getting in as is because my 1.2GB test file was off a 1h30 youtube video, and I suspect many videos will be under this length. Might be a bit of fun though to try and work out how memory management in node works.

How to test

This is currently live on CODE, you can try it out here https://transcribe.code.dev-gutools.co.uk/

Screenshots

[github-actions]

Deploy build 791 of investigations::transcription-service to CODE

All deployment options

• Deploy build 791 of investigations::transcription-service to CODE
• Deploy parts of build 791 to CODE by previewing it first
• What's on CODE right now?

---

From guardian/actions-riff-raff.

[github-actions]

Deploy build 671 of investigations::transcription-service-repository to CODE

All deployment options

• Deploy build 671 of investigations::transcription-service-repository to CODE
• Deploy parts of build 671 to CODE by previewing it first
• What's on CODE right now?

---

From guardian/actions-riff-raff.