|
| 1 | +# ImageAPI |
| 2 | +This project is experimenting with different providers of cloud based web hooks and storage. |
| 3 | +[Note: This is in the very early stages of the experimentation and is not currently functional. The specs and approach may change without any notice.] |
| 4 | + |
| 5 | +The base use case is the ability to upload remote images into a cloud storage bucket as well as to retrieve and resize them. |
| 6 | +(Note: some providers have resizing capabilities inbuilt [eg. GCP], this project will examine other methods to implement the same) |
| 7 | + |
| 8 | +## API Requirements |
| 9 | +- Upload Images via URL |
| 10 | +- Download Images / provide Signed URL |
| 11 | +- Resize Images |
| 12 | +- 80 day expiry |
| 13 | + |
| 14 | +### Additional Considerations |
| 15 | +- Real time updates for large uploads |
| 16 | +- Performance [Optimization for write heavy load] |
| 17 | +- Caching |
| 18 | +- Security [Who can access which images] |
| 19 | +- Resilience [] |
| 20 | + |
| 21 | +## Setup |
| 22 | +1. Create Project. |
| 23 | +2. Add Project_ID to config.json |
| 24 | +TODO: add script to deploy.sh to add Project_ID to bucket name |
| 25 | + |
| 26 | +### Service Accounts |
| 27 | + |
| 28 | +## Leads |
| 29 | +Consider spinning up app in Firebase |
| 30 | +Resumable file uploads: https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload |
| 31 | +https://github.com/mkahn5/gcloud-resumable-uploads/blob/master/views/index.ejs |
| 32 | +Google Websocket demo: https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/master/appengine/websockets |
| 33 | +Firebase approach to uploads: https://firebase.google.com/docs/storage/web/upload-files#monitor_upload_progress |
| 34 | +Pub/Sub: https://github.com/googleapis/nodejs-pubsub/ |
| 35 | + |
| 36 | + |
| 37 | +## TODO |
| 38 | +- Filetype verification |
| 39 | +- Partitioned file uploads |
| 40 | +- Caching |
| 41 | + - Match Catch expiry to Bucket item expiration |
| 42 | + |
| 43 | +## Architecture |
| 44 | +The API endpoint footprint is small and doesn't need to maintain any state. This makes it a prime candidate for Lamba-style webhooks. (Benefits: Auto-scaling, minimal upkeep, smaller code footprint etc.) |
| 45 | +User will interact with a single REST endpoint (/images) through GET and PUT requests. |
| 46 | + |
| 47 | + |
| 48 | + |
| 49 | + |
| 50 | + |
| 51 | + |
| 52 | +### Provider |
| 53 | +Note: GCP's Node 8 driver is still Beta. |
| 54 | + |
| 55 | +### Implementation |
| 56 | +#### Item Expiry |
| 57 | +The item expiry can be handled directly in the data bucket policy. |
| 58 | +`gsutil mb --retention 80d gs://ImageAPI` |
| 59 | +(For a more nuanced/extensible approach we could specify a more detailed lifecycle policy or have a lambda function launched via scheduled CRON job - eg. via cloud scheduler) |
| 60 | + |
| 61 | + |
| 62 | +## Implementation Notes |
| 63 | +### Code/Folder Structure - gcloud deploy limitations |
| 64 | +The `gcloud deploy` command, unfortunately, appears to only work on a singular local index.js file (or remote repositories). |
| 65 | + |
| 66 | +### (Prepackaged solutions) |
| 67 | +Google's AppEngine provides ready made image servicing, including resizing and cropping. |
| 68 | +https://cloud.google.com/appengine/docs/standard/python/refdocs/google.appengine.api.images#google.appengine.api.images.get_serving_url |
| 69 | +https://medium.com/google-cloud/uploading-resizing-and-serving-images-with-google-cloud-platform-ca9631a2c556 |
| 70 | + |
| 71 | +### Performance considerations |
| 72 | +#### HTTP Requests to Google Cloud |
| 73 | +We're currently utilizing Google's prebuilt node module for access. For a smaller footprint, to aid with speed of load for the cloud functions, we could use a streamlined request library to interact directly with their REST API. |
0 commit comments