Skip to content

Commit c085c36

Browse files
eruniondimitropoulosfilfreireDavis Martinirajtaghlidi
authored
feat(rebase): updating our fork with the latest upstream (readmeio#180)
* Total Overhaul (but with all the same fixtures!) (readmeio#248) see Kong#248 and the commits therein for more context. Essentially: - the original client fixtures were unchanged, which hopefully means the - all source code is now in strict mode TypeScript - tests are now all in jest - the file structure was reorganized so that everything for a particular client is in one place - the CLI is updated and now using yargs - all dependencies were updated and some (i.e. `format.utils`) were able to be removed entirely - more work left to do (including CI with GitHub Actions, for example), but this is a start * fix: case where if `postData.params` is missing some targets crash (readmeio#258) Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: compatibility issues on node 14 with `Object.hasOwn()` (readmeio#252) * fix: typo in the httpie `style` option not being correctly applied (readmeio#254) Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: axios targets not sending `x-www-form-urlencoded` properly (readmeio#255) Co-authored-by: Dimitri Mitropoulos <[email protected]> * feat: addition of a PHP target for Guzzle (readmeio#253) Co-authored-by: Dimitri Mitropoulos <[email protected]> * Add Github Build Workflow (readmeio#250) (readmeio#251) * Add Github Build Workflow (readmeio#250) * Edit job name * Replace install with ci on GH workflow * Add matrix with major node versions (14, 16, 18) * Disable fail-fast * Remove node v14 from build GH action * feat: native upload support in python `requests` snippets (readmeio#259) Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: `multipart/form-data` header issues with node/js fetch targets (readmeio#257) Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: headers not being properly applied to R httr snippets (readmeio#263) Co-authored-by: Dimitri Mitropoulos <[email protected]> * Fix build workflow dispatch rules (readmeio#265) * Chore: Remove travis links (readmeio#266) * Remove travis links * Update README.md Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: issue where query strings in R wouldn't be properly concatenated (readmeio#269) * fix: issue where query strings in R wouldn't be properly concatenated * adds (and respects) indent options to httr, plus double looping fix now, indent is respected, and also avoiding running Object.keys twice per run since we can just run it once with .entries Co-authored-by: Dimitri Mitropoulos <[email protected]> * add header namesspace to prevent header errors (readmeio#247) * add header namesspace to prevent header errors * update fixtures Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: stop implicitly coercing warning in Swift snippet generator (readmeio#195) * swift/nsurlsession adds `as Any` to print for error * adds OVERWRITE_EVERYTHING to ease fixture snapshot resetting * updates fixtures Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: clj-http handling of literal null JSON bodies (readmeio#283) Co-authored-by: Sergey Zakharchenko <[email protected]> * fix: prevent crash in Swift/Objc with checking length of input body post params (readmeio#192) Co-authored-by: Dimitri Mitropoulos <[email protected]> * fix: cUrl target should encode x-www-form-urlencoded post data params (readmeio#198) Co-authored-by: Dimitri Mitropoulos <[email protected]> * feat: Add support for insecureSkipVerify (readmeio#285) * go/native: adds insecureSkipVerify * node/native: adds insecureSkipVerify * python/python3: adds insecureSkipVerify * ruby/native: adds insecureSkipVerify * shell/curl: adds insecureSkipVerify Co-authored-by: Tim Perry <[email protected]> * feat: implementing cleaner handling of JSON in cURL snippets (readmeio#256) Co-authored-by: Dimitri Mitropoulos <[email protected]> * chore: minor cleanup (readmeio#286) * feat: use curl's --compressed option for requests that accept it (readmeio#287) * feat: make Python snippets simpler, clearer & more consistent (readmeio#288) Co-authored-by: Tim Perry <[email protected]> * feat: change the default response code for Python Requests (readmeio#181) Co-authored-by: Dimitri Mitropoulos <[email protected]> * feat: PHP JSON body encoding (readmeio#291) * php/curl: use json_encode for CURLOPT_POSTFIELDS * php/http1: use json_encode when body is JSON * php/http2: use json_encode when body is JSON Co-authored-by: Andrii Kostenko <[email protected]> * Async/Await (top level) support in JavaScript snippets (readmeio#292) * Exclude package.json from build to fix output paths (readmeio#294) * Exclude package.json from build to fix output paths * keeps bin pointing at cli output * makes rootDir explicit * removes unused cli build scripts Co-authored-by: Dimitri Mitropoulos <[email protected]> * Fix crash when building nsurlsession snippets for empty params (readmeio#295) * removes `require 'openssl'` from ruby target (no longer needed) (readmeio#296) * Escape quotes in headers correctly in all languages (readmeio#289) * updates README (readmeio#299) Co-authored-by: Filipe Freire <[email protected]> * ioutil -> io (deprecated) (readmeio#305) * chore: undoing unwanted changes * chore: revert more unwanted changes * chore: reverting more unwanted changes * fix: fixing broken test snapshots and libcurl not escaping * fix: a bunch of broken tests * fix: removing dead code * fix: remove support for `insecureSkipVerify` as we dont need or want it * fix: removing top-level await changes for axios * fix: revert top-level await changes for js:fetch * fix: remove some problematic changes to `node:request` * fix: retaining line trimming in powershell snippets * fix: bug in php snippets where booleans were casted to null * fix: revert problematic changes to python:requests * fix: revert more unwanted changes * chore: temporarily skipping the integration suite * fix: broken snapshot * fix: disabling the integration suite from being run without inside docker * fix: integration suite * fix: integration suite --------- Co-authored-by: Dimitri Mitropoulos <[email protected]> Co-authored-by: Filipe Freire <[email protected]> Co-authored-by: Davis Martin <[email protected]> Co-authored-by: iraj taghlidi <[email protected]> Co-authored-by: Sergey Zakharchenko <[email protected]> Co-authored-by: Julien Giovaresco <[email protected]> Co-authored-by: Tim Perry <[email protected]> Co-authored-by: Andrii Kostenko <[email protected]> Co-authored-by: Tim Perry <[email protected]> Co-authored-by: Alexander Weber <[email protected]>
1 parent 8276996 commit c085c36

170 files changed

Lines changed: 677 additions & 354 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -160,13 +160,15 @@ There are some major differences between this library and the [httpsnippet](http
160160
* Does not do any HAR schema validation. It's just assumed that the HAR you're supplying to the library is already valid.
161161
* The main `HTTPSnippet` export contains an `options` argument for an `harIsAlreadyEncoded` option for disabling [escaping](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) of cookies and query strings in URLs.
162162
* We added this because all HARs that we interact with already have this data escaped and this option prevents them from being double encoded, thus corrupting the data.
163+
* Does not support the `insecureSkipVerify` option on `go:native`, `node:native`, `ruby:native`, and `shell:curl` as we don't want snippets generated for our users to bypass SSL certificate verification.
163164
* Node
164165
* `fetch`
165-
* Body payloads are treated as an object literal and wrapped within `JSON.stringify()`. We do this to keep those targets looking nicer with those kinds of payloads. This also applies to the JS `fetch` target as wel.
166+
* Body payloads are treated as an object literal and wrapped within `JSON.stringify()`. We do this to keep those targets looking nicer with those kinds of payloads. This also applies to the JS `fetch` target as well.
166167
* `request`
167168
* Does not provide query string parameters in a `params` argument due to complexities with query encoding.
168-
* PHP → `guzzle`
169-
* Snippets have `require_once('vendor/autoload.php');` prefixed at the top.
169+
* PHP
170+
* `guzzle`
171+
* Snippets have `require_once('vendor/autoload.php');` prefixed at the top.
170172
* Python
171173
* `python3`
172174
* Does not ship this client due to its incompatibility with being able to support file uploads.

src/fixtures/requests/application-json.js

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ module.exports = {
1818
],
1919
postData: {
2020
mimeType: 'application/json',
21-
text: '{"number":1,"string":"f\\"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":{}}],"boolean":false}',
21+
text: '{"number":1,"string":"f\\"oo","arr":[1,2,3],"nested":{"a":"b"},"arr_mix":[1,"a",{"arr_mix_nested":[]}],"boolean":false}',
2222
},
2323
},
2424
response: {
@@ -36,7 +36,7 @@ module.exports = {
3636
mimeType: 'application/json',
3737
text: JSON.stringify({
3838
args: {},
39-
data: '{\n "number": 1,\n "string": "f\\"oo",\n "arr": [\n 1,\n 2,\n 3\n ],\n "nested": {\n "a": "b"\n },\n "arr_mix": [\n 1,\n "a",\n {\n "arr_mix_nested": {}\n }\n ],\n "boolean": false\n}',
39+
data: '{\n "number": 1,\n "string": "f\\"oo",\n "arr": [\n 1,\n 2,\n 3\n ],\n "nested": {\n "a": "b"\n },\n "arr_mix": [\n 1,\n "a",\n {\n "arr_mix_nested": []\n }\n ],\n "boolean": false\n}',
4040
files: {},
4141
form: {},
4242
headers: {
@@ -48,7 +48,7 @@ module.exports = {
4848
1,
4949
'a',
5050
{
51-
arr_mix_nested: {},
51+
arr_mix_nested: [],
5252
},
5353
],
5454
boolean: false,

src/fixtures/requests/headers.js

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ module.exports = {
2323
name: 'x-bar',
2424
value: 'Foo',
2525
},
26+
{
27+
name: 'quoted-value',
28+
value: '"quoted" \'string\'',
29+
},
2630
],
2731
},
2832
response: {

src/helpers/escape.test.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
import { escapeString } from './escape';
2+
3+
describe('Escape methods', () => {
4+
describe('escapeString', () => {
5+
it('does nothing to a safe string', () => {
6+
expect(escapeString('hello world')).toBe('hello world');
7+
});
8+
9+
it('escapes double quotes by default', () => {
10+
expect(escapeString('"hello world"')).toBe('\\"hello world\\"');
11+
});
12+
13+
it('escapes newlines by default', () => {
14+
expect(escapeString('hello\r\nworld')).toBe('hello\\r\\nworld');
15+
});
16+
17+
it('escapes backslashes', () => {
18+
expect(escapeString('hello\\world')).toBe('hello\\\\world');
19+
});
20+
21+
it('escapes unrepresentable characters', () => {
22+
expect(
23+
escapeString('hello \u0000') // 0 = ASCII 'null' character
24+
).toBe('hello \\u0000');
25+
});
26+
});
27+
});

src/helpers/escape.ts

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
export interface EscapeOptions {
2+
/**
3+
* The delimiter that will be used to wrap the string (and so must be escaped
4+
* when used within the string).
5+
* Defaults to "
6+
*/
7+
delimiter?: string;
8+
9+
/**
10+
* The char to use to escape the delimiter and other special characters.
11+
* Defaults to \
12+
*/
13+
escapeChar?: string;
14+
15+
/**
16+
* Whether newlines (\n and \r) should be escaped within the string.
17+
* Defaults to true.
18+
*/
19+
escapeNewlines?: boolean;
20+
}
21+
22+
/**
23+
* Escape characters within a value to make it safe to insert directly into a
24+
* snippet. Takes options which define the escape requirements.
25+
*
26+
* This is closely based on the JSON-stringify string serialization algorithm,
27+
* but generalized for other string delimiters (e.g. " or ') and different escape
28+
* characters (e.g. Powershell uses `)
29+
*
30+
* See https://tc39.es/ecma262/multipage/structured-data.html#sec-quotejsonstring
31+
* for the complete original algorithm.
32+
*/
33+
export function escapeString(rawValue: any, options: EscapeOptions = {}) {
34+
const { delimiter = '"', escapeChar = '\\', escapeNewlines = true } = options;
35+
36+
const stringValue = rawValue.toString();
37+
38+
return [...stringValue]
39+
.map(c => {
40+
if (c === '\b') {
41+
return `${escapeChar}b`;
42+
} else if (c === '\t') {
43+
return `${escapeChar}t`;
44+
} else if (c === '\n') {
45+
if (escapeNewlines) {
46+
return `${escapeChar}n`;
47+
}
48+
49+
return c; // Don't just continue, or this is caught by < \u0020
50+
} else if (c === '\f') {
51+
return `${escapeChar}f`;
52+
} else if (c === '\r') {
53+
if (escapeNewlines) {
54+
return `${escapeChar}r`;
55+
}
56+
57+
return c; // Don't just continue, or this is caught by < \u0020
58+
} else if (c === escapeChar) {
59+
return escapeChar + escapeChar;
60+
} else if (c === delimiter) {
61+
return escapeChar + delimiter;
62+
} else if (c < '\u0020' || c > '\u007E') {
63+
// Delegate the trickier non-ASCII cases to the normal algorithm. Some of these
64+
// are escaped as \uXXXX, whilst others are represented literally. Since we're
65+
// using this primarily for header values that are generally (though not 100%
66+
// strictly?) ASCII-only, this should almost never happen.
67+
return JSON.stringify(c).slice(1, -1);
68+
}
69+
70+
return c;
71+
})
72+
.join('');
73+
}
74+
75+
/**
76+
* Make a string value safe to insert literally into a snippet within single quotes,
77+
* by escaping problematic characters, including single quotes inside the string,
78+
* backslashes, newlines, and other special characters.
79+
*
80+
* If value is not a string, it will be stringified with .toString() first.
81+
*/
82+
export const escapeForSingleQuotes = (value: any) => escapeString(value, { delimiter: "'" });
83+
84+
/**
85+
* Make a string value safe to insert literally into a snippet within double quotes,
86+
* by escaping problematic characters, including double quotes inside the string,
87+
* backslashes, newlines, and other special characters.
88+
*
89+
* If value is not a string, it will be stringified with .toString() first.
90+
*/
91+
export const escapeForDoubleQuotes = (value: any) => escapeString(value, { delimiter: '"' });

src/integration.test.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ availableTargets()
9797
.filter(target => target.cli)
9898
.filter(testFilter('key', environmentFilter()))
9999
.forEach(({ key: targetId, cli: targetCLI, title, extname: fixtureExtension, clients }) => {
100-
describe(`${title} integration tests`, () => {
100+
(process.env.NODE_ENV === 'test' ? describe.skip : describe)(`${title} integration tests`, () => {
101101
clients.filter(testFilter('key', clientFilter(targetId))).forEach(({ key: clientId }) => {
102102
// If we're in an HTTPBin-powered Docker environment we only want to run tests for the
103103
// client that our Docker has been configured for.

src/targets/c/libcurl/client.ts

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import type { Client } from '../../targets';
22

33
import { CodeBuilder } from '../../../helpers/code-builder';
4+
import { escapeForDoubleQuotes } from '../../../helpers/escape';
45

56
export const libcurl: Client = {
67
info: {
@@ -26,7 +27,7 @@ export const libcurl: Client = {
2627
push('struct curl_slist *headers = NULL;');
2728

2829
headers.forEach(header => {
29-
push(`headers = curl_slist_append(headers, "${header}: ${headersObj[header]}");`);
30+
push(`headers = curl_slist_append(headers, "${header}: ${escapeForDoubleQuotes(headersObj[header])}");`);
3031
});
3132

3233
push('curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, headers);');

src/targets/c/libcurl/fixtures/application-json.c

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,6 @@ struct curl_slist *headers = NULL;
77
headers = curl_slist_append(headers, "content-type: application/json");
88
curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, headers);
99

10-
curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, "{\"number\":1,\"string\":\"f\\\"oo\",\"arr\":[1,2,3],\"nested\":{\"a\":\"b\"},\"arr_mix\":[1,\"a\",{\"arr_mix_nested\":{}}],\"boolean\":false}");
10+
curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, "{\"number\":1,\"string\":\"f\\\"oo\",\"arr\":[1,2,3],\"nested\":{\"a\":\"b\"},\"arr_mix\":[1,\"a\",{\"arr_mix_nested\":[]}],\"boolean\":false}");
1111

1212
CURLcode ret = curl_easy_perform(hnd);

src/targets/c/libcurl/fixtures/headers.c

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ struct curl_slist *headers = NULL;
77
headers = curl_slist_append(headers, "accept: application/json");
88
headers = curl_slist_append(headers, "x-foo: Bar");
99
headers = curl_slist_append(headers, "x-bar: Foo");
10+
headers = curl_slist_append(headers, "quoted-value: \"quoted\" 'string'");
1011
curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, headers);
1112

1213
CURLcode ret = curl_easy_perform(hnd);

src/targets/clojure/clj_http/fixtures/application-json.clj

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,5 @@
55
:string "f\"oo"
66
:arr [1 2 3]
77
:nested {:a "b"}
8-
:arr_mix [1 "a" {:arr_mix_nested {}}]
8+
:arr_mix [1 "a" {:arr_mix_nested []}]
99
:boolean false}})

0 commit comments

Comments
 (0)