Skip to content

Commit 31d7d2c

Browse files
CaffeineFusionCaffeineFusion
committed
initial brainstorming and GCP experimentation
0 parents  commit 31d7d2c

11 files changed

Lines changed: 1364 additions & 0 deletions

File tree

‎.eslintrc.json‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{
2+
"env": {
3+
"node": 1,
4+
"es6": true
5+
},
6+
"extends": "eslint:recommended",
7+
"parserOptions": {
8+
"ecmaVersion": 2018,
9+
"sourceType": "module"
10+
},
11+
"rules": {
12+
"indent": [
13+
"error",
14+
"tab",
15+
{
16+
"SwitchCase": 1
17+
}
18+
],
19+
"quotes": [
20+
"error",
21+
"single"
22+
],
23+
"semi": [
24+
"error",
25+
"always"
26+
],
27+
"multi-string": "on",
28+
"no-console":"off",
29+
"allowTemplateLiterals":true
30+
31+
}
32+
}

‎.gcloudignore‎

Whitespace-only changes.

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
samples
2+
node_modules
3+
keys

‎README.MD‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# ImageAPI
2+
This project is experimenting with different providers of cloud based web hooks and storage.
3+
[Note: This is in the very early stages of the experimentation and is not currently functional. The specs and approach may change without any notice.]
4+
5+
The base use case is the ability to upload remote images into a cloud storage bucket as well as to retrieve and resize them.
6+
(Note: some providers have resizing capabilities inbuilt [eg. GCP], this project will examine other methods to implement the same)
7+
8+
## API Requirements
9+
- Upload Images via URI
10+
- Download Images
11+
- Resize Images
12+
- 80 day expiry
13+
14+
### Additional Considerations
15+
- Real time updates for large uploads
16+
- Performance [Optimization for write heavy load]
17+
- Caching
18+
- Security [Who can access which images]
19+
- Resilience []
20+
21+
## Setup
22+
1. Create Project.
23+
2. Add Project_ID to config.json
24+
TODO: add script to deploy.sh to add Project_ID to bucket name
25+
26+
## Leads
27+
Consider spinning up app in Firebase
28+
Resumable file uploads: https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload
29+
https://github.com/mkahn5/gcloud-resumable-uploads/blob/master/views/index.ejs
30+
Google Websocket demo: https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/master/appengine/websockets
31+
Firebase approach to uploads: https://firebase.google.com/docs/storage/web/upload-files#monitor_upload_progress
32+
Pub/Sub: https://github.com/googleapis/nodejs-pubsub/
33+
34+
35+
## Questions
36+
- How will client be informed of workerised file upload failure? Web Socket?
37+
- How do we provide information on file before it has been fully uploaded to the server?
38+
- How to verify correct filetype?
39+
40+
41+
## Architecture
42+
The API endpoint footprint is small and doesn't need to maintain any state. This makes it a prime candidate for Lamba-style webhooks. (Benefits: Auto-scaling, minimal upkeep, smaller code footprint etc.)
43+
User will interact with a single REST endpoint (/images) through GET and POST requests.
44+
45+
46+
For Uploading the images, we need the POST endpoint to initiate one or more upload operations. In this case, we'll trigger one or more instances of a Lambda function. (Phase 2 could utilize a Pub/Sub service with workers consuming the upload tasks - Beyond the scope of this demo)
47+
48+
49+
For real-time feedback on the upload process, we could have the client long-poll the GET status. However, because we want to have multiple image uploads of undefined size, a better approach would be to use a Web Socket where the client can be notified of status updates of each upload. This would also provide a means for real-time upload states - a chunked upload can provide %age upload completions to the client side in real-time.
50+
Two approaches could be used on POST - return a 101 status and direct the client to upgrade to a Websocket. Return a 102 status
51+
52+
### Provider
53+
Note: GCP's Node 8 driver is still Beta.
54+
55+
### Components
56+
2 External End Points - GET and PUT corresponding to:
57+
3 Lambdas - GET, PUT, and uploadImage.
58+
1 Data Store.
59+
60+
### Implementation
61+
#### Item Expiry
62+
The item expiry can be handled directly in the data bucket policy.
63+
`gsutil mb --retention 80d gs://ImageAPI`
64+
(For a more nuanced/extensible approach we could specify a more detailed lifecycle policy or have a lambda function launched via scheduled CRON job - eg. via cloud scheduler)
65+
66+
#### Resizing
67+
68+
Prepackaged Google solution:
69+
https://medium.com/google-cloud/uploading-resizing-and-serving-images-with-google-cloud-platform-ca9631a2c556
70+
71+
#### Image Upload
72+
73+
### Known limitations
74+
- File upload size limits ()
75+
76+
## Implementation Notes
77+
### Code/Folder Structure - gcloud deploy limitations
78+
The `gcloud deploy` command, unfortunately, appears to only work on a singular local index.js file (or remote repositories).
79+
Rather than coalescing these into a single index file, the deployment script points to the remote locations in the Github repository. (Noted the issue with the documentation on this point.)
80+
81+
### (Prepackaged solutions)
82+
Google's AppEngine provides ready made image servicing, including resizing and cropping.
83+
https://cloud.google.com/appengine/docs/standard/python/refdocs/google.appengine.api.images#google.appengine.api.images.get_serving_url
84+
85+
### Performance considerations
86+
#### HTTP Requests to Google Cloud
87+
We're currently utilizing Google's prebuilt node module for access. For a smaller footprint, to aid with speed of load for the cloud functions, we could use a streamlined request library to interact directly with their REST API.

‎config.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"project_id":"307972859230",
3+
"bucket_name":"image_api_307972859230",
4+
"keys": {
5+
"cloud_storage": {
6+
"path":"./keys/cloud_storage.json"
7+
}
8+
}
9+
}

‎deploy.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
gsutil mb --retention 80d gs://image_api # need to provide a unique identifier
2+
3+
gcloud functions deploy getImage --source='src' --runtime=nodejs8 --trigger-http
4+
gcloud functions deploy getImage --source='src/getImage.js' --runtime=nodejs8 --trigger-http
5+
gcloud functions deploy getImage --source='src/postImage.js' --runtime=nodejs8 --trigger-http
6+
# gcloud functions deploy image --trigger-http
7+
# --trigger-bucket=ImageAPI
8+
9+
# Samples - https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/master/functions/imagemagick
10+
# Samples - https://cloud.google.com/functions/docs/quickstart

‎experiments/gcp/example_template.js‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/** GCP example for reference **/
2+
3+
// Imports the Google Cloud client library
4+
const {Storage} = require('@google-cloud/storage');
5+
6+
// Creates a client
7+
const storage = new Storage();
8+
9+
/**
10+
* TODO(developer): Uncomment the following lines before running the sample.
11+
*/
12+
// const bucketName = 'Name of a bucket, e.g. my-bucket';
13+
// const filename = 'Local file to upload, e.g. ./local/path/to/file.txt';
14+
15+
// Uploads a local file to the bucket
16+
await storage.bucket(bucketName).upload(filename, {
17+
// Support for HTTP requests made with `Accept-Encoding: gzip`
18+
gzip: true,
19+
metadata: {
20+
// Enable long-lived HTTP caching headers
21+
// Use only if the contents of the file will never change
22+
// (If the contents will change, use cacheControl: 'no-cache')
23+
cacheControl: 'public, max-age=31536000',
24+
},
25+
});
26+
27+
console.log(`${filename} uploaded to ${bucketName}.`);

‎experiments/gcp/upload.js‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
'use strict';
2+
3+
const request = require('request');
4+
5+
const config = require('./../config.json');
6+
7+
const {Storage} = require('@google-cloud/storage');
8+
const storage = new Storage({
9+
projectId: config.project_id,
10+
keyFilename: './../keys/cloud_storage.json'
11+
});
12+
13+
// Returns a Promise object which resolves to a stream.
14+
function uploadByURL(url, bucket, fileName) {
15+
let file = bucket.file(fileName);
16+
return new Promise((resolve, reject) => {
17+
request(url)
18+
.on('response', (response) => { response.pause(); resolve(response); });
19+
}).then((response) => { return response.pipe(file.createWriteStream({ gzip: true })); });
20+
}
21+
22+
exports.upload = function (url, fileName) {
23+
return uploadByURL(url, storage.bucket, fileName);
24+
};

‎index.js‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
'use strict';
2+
const url = require('url');
3+
const uuid = require('uuidv4');
4+
const request = require('request');
5+
const rp = require('request-promise');
6+
7+
const config = require('./config.json');
8+
9+
const Storage = require('@google-cloud/storage');
10+
const storage = new Storage({
11+
projectId: config.project_id,
12+
keyFilename: config.keys.cloud_storage.path
13+
});
14+
15+
16+
function parseURL(urlString) {
17+
try {
18+
return url.parse(urlString);
19+
}
20+
catch(e) {
21+
//
22+
}
23+
console.log('index.js, parseURL not written');
24+
//return(url);
25+
}
26+
27+
function checkURLStatus(url) {
28+
return new Promise((resolve, reject) => {
29+
const request = require('request');
30+
31+
request({method: 'HEAD', url}, function (error, response) {
32+
if(error || response.statusCode !== 200) {
33+
reject({url, statusCode:response.statusCode, error:error?error:'inaccessibleURL', message:'Remote URL could not be reached.'});
34+
}
35+
resolve(url);
36+
});
37+
});
38+
}
39+
40+
/**
41+
* verifyURL - pre-upload checks to ensure we notify client of failures prior to attempted upload.
42+
**/
43+
function verifyURL(url) {
44+
return new Promise((resolve, reject) => {
45+
if(!parseURL(url))
46+
reject({url, error:'invalidURL', message:'URL could not be parsed. Please check that URL is valid'});
47+
else {
48+
checkURLStatus(url)
49+
.then(resolve)
50+
.catch(reject);
51+
}
52+
});
53+
}
54+
55+
// For testing
56+
async function uploadLocalFile(fileName, bucketName) {
57+
await storage.bucket(bucketName).upload(fileName, {
58+
gzip: true,
59+
metadata: {
60+
cacheControl: 'public, max-age=31536000',
61+
},
62+
});
63+
}
64+
65+
/*
66+
const bucket = gcs.bucket('bucket_name');
67+
const gcsname = 'test.pdf';
68+
const file = bucket.file(gcsname);
69+
var pdfdata = "binary_pdf_file_string";
70+
var buff = Buffer.from(pdfdata, 'binary').toString('utf-8');
71+
72+
const stream = file.createWriteStream({
73+
metadata: {
74+
contentType: 'application/pdf'
75+
}
76+
});
77+
stream.on('error', (err) => {
78+
console.log(err);
79+
});
80+
stream.on('finish', () => {
81+
console.log(gcsname);
82+
});
83+
stream.end(new Buffer(buff, 'base64'));
84+
*/
85+
86+
function uploadByURL(url, endpoint, location) {
87+
//let message = { uploadedFiles: `${imageURL}` };
88+
let file = myBucket.file('my-file');
89+
request(url)
90+
.pipe(endpoint.upload());
91+
92+
//res.status(200).json(message);x
93+
}
94+
95+
function getImage(req, res) {
96+
let message = req.query.message || req.body.message || 'Hello World!';
97+
switch (req.get('content-type')) {
98+
case 'application/JSON':
99+
100+
break;
101+
case 'image/jpeg':
102+
103+
break;
104+
case 'image/png':
105+
106+
break;
107+
}
108+
res.status(200).send(message);
109+
}
110+
111+
112+
// Lambdas
113+
exports.image = (req, res) => {
114+
if(!res.body.urls) res.status(400).json({error:'noURLs', message:'No URLs were provided.'});
115+
let urls = res.body.urls;
116+
switch(req.method) {
117+
case 'POST':
118+
Promise.all(
119+
urls.map((url) => {
120+
return verifyURL(url)
121+
.then(uploadByURL);
122+
})
123+
)
124+
// TODO: Change status IDs depending on error/s from Promises.
125+
.then(res.status(202).json)
126+
.catch(res.status(400).json);
127+
break;
128+
case 'GET':
129+
getImage(req, res);
130+
break;
131+
}
132+
};
133+
134+
exports.upload = (req, res) => {
135+
136+
res.status(200).send({});
137+
};

0 commit comments

Comments
 (0)