Initializing without a token
Alternatively, you can create a client without a token, and use it with multiple workspaces as long as you supply a `token` when you call a method. ```javascript const { WebClient } = require('@slack/web-api'); // Initialize a single instance for the whole app const web = new WebClient(); // Find a token in storage (database) before making an API method call (async () => { // Some fictitious database const token = await db.findTokenByTeam(teamId, enterpriseId) // Call the method const result = web.auth.test({ token }); })(); ```Using a dynamic method name
If you want to provide the method name as a string so that you can decide which method to call dynamically or to call a method that might not be available in your version of the client, use the `WebClient.apiCall(methodName, [options])` method. The API method call above can also be written as follows: ```javascript const conversationId = '...'; (async () => { // Using apiCall() allows the app to call any method and to do it programmatically const response = await web.apiCall('chat.postMessage', { text: 'Hello world!', channel: conversationId, }); })(); ```More error types
There are a few more types of errors that you might encounter, each with one of these `code`s: * `ErrorCode.RequestError`: A request could not be sent. A common reason for this is that your network connection is not available, or `api.slack.com` could not be reached. This error has an `original` property with more details. * `ErrorCode.RateLimitedError`: The Web API cannot fulfill the API method call because your app has made too many requests too quickly. This error has a `retryAfter` property with the number of seconds you should wait before trying again. See [the documentation on rate limit handling](https://slack.dev/node-slack-sdk/web-api/#rate-limits) to understand how the client will automatically deal with these problems for you. * `ErrorCode.HTTPError`: The HTTP response contained an unfamiliar status code. The Web API only responds with `200` (yes, even for errors) or `429` (rate limiting). If you receive this error, it's likely due to a problem with a proxy, a custom TLS configuration, or a custom API URL. This error has the `statusCode`, `statusMessage`, `headers`, and `body` properties containing more details.Using functional iteration
The `.paginate()` method can accept up to two additional parameters. The third parameter, `stopFn`, is a function that is called once for each page of the result, and should return `true` when the app no longer needs to get another page. The fourth parameter is `reducerFn`, which is a function that gets called once for each page of the result, but can be used to aggregate a result. The value it returns is used to call it the next time as the `accumulator`. The first time it gets called, the `accumulator` is undefined. ```javascript (async () => { // The first two parameters are the method name and the options object. const done = await web.paginate('something.list', { name: 'value' }, // The third is a function that receives each page and should return true when the next page isn't needed. (page) => { /* ... */ }, // The fourth is a reducer function, similar to the callback parameter of Array.prototype.reduce(). // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce // The accumulator is initialized to undefined. (accumulator, page, index) => { /* ... */ }, ); }); ``` The returned value is a `Promise`, but what it resolves to depends on whether or not you include the fourth (optional) parameter. If you don't include it, the resolved value is always `undefined`. In this case, its used for control flow purposes (resuming the rest of your program), and the function in the third parameter is used to capture a result. If you do include the fourth parameter, then the resolved value is the value of the `accumulator`. This is a familiar pattern for people that use _functional programming_.Sending log output somewhere besides the console
You can also choose to have logs sent to a custom logger using the `logger` option. A custom logger needs to implement specific methods (known as the `Logger` interface): | Method | Parameters | Return type | |--------------|-------------------|-------------| | `setLevel()` | `level: LogLevel` | `void` | | `setName()` | `name: string` | `void` | | `debug()` | `...msgs: any[]` | `void` | | `info()` | `...msgs: any[]` | `void` | | `warn()` | `...msgs: any[]` | `void` | | `error()` | `...msgs: any[]` | `void` | A very simple custom logger might ignore the name and level, and write all messages to a file. ```javascript const { createWriteStream } = require('fs'); const logWritable = createWriteStream('/var/my_log_file'); // Not shown: close this stream const web = new WebClient(token, { // Creating a logger as a literal object. It's more likely that you'd create a class. logger: { debug(...msgs): { logWritable.write('debug: ' + JSON.stringify(msgs)); }, info(...msgs): { logWritable.write('info: ' + JSON.stringify(msgs)); }, warn(...msgs): { logWritable.write('warn: ' + JSON.stringify(msgs)); }, error(...msgs): { logWritable.write('error: ' + JSON.stringify(msgs)); }, setLevel(): { }, setName(): { }, }, }); ```cheerio
Fast, flexible & lean implementation of core jQuery designed specifically for the server.
```js let cheerio = require('cheerio') let $ = cheerio.load('
Hello world
') $('h2.title').text('Hello there!') $('h2').addClass('welcome') $.html() //=>Hello there!
``` ## Installation `npm install cheerio` ## Features __❤ Familiar syntax:__ Cheerio implements a subset of core jQuery. Cheerio removes all the DOM inconsistencies and browser cruft from the jQuery library, revealing its truly gorgeous API. __ϟ Blazingly fast:__ Cheerio works with a very simple, consistent DOM model. As a result parsing, manipulating, and rendering are incredibly efficient. Preliminary end-to-end benchmarks suggest that cheerio is about __8x__ faster than JSDOM. __❁ Incredibly flexible:__ Cheerio wraps around @FB55's forgiving [htmlparser2](https://github.com/fb55/htmlparser2/). Cheerio can parse nearly any HTML or XML document. ## Sponsors Does your company use Cheerio in production? Please consider [sponsoring this project](https://opencollective.com/cheerio#sponsor). Your help will allow maintainers to dedicate more time and resources to its development and support.
## Backers [Become a backer](https://opencollective.com/cheerio#backer) to show your support for Cheerio and help us maintain and improve this open source project. 
## API ### Markup example we'll be using: ```html - Apple
- Orange
- Pear
- ...
- ...
- ...
- ...
- ...
.filter( selection )
.filter( element )
.filter( function(index) ) Iterates over a cheerio object, reducing the set of selector elements to those that match the selector or pass the function's test. When a Cheerio selection is specified, return only the elements contained in that selection. When an element is specified, return only that element (if it is contained in the original selection). If using the function method, the function is executed in the context of the selected element, so `this` refers to the current element. Selector: ```js $('li').filter('.orange').attr('class'); //=> orange ``` Function: ```js $('li').filter(function(i, el) { // this === el return $(this).attr('class') === 'orange'; }).attr('class') //=> orange ``` #### .not( selector )
.not( selection )
.not( element )
.not( function(index, elem) ) Remove elements from the set of matched elements. Given a jQuery object that represents a set of DOM elements, the `.not()` method constructs a new jQuery object from a subset of the matching elements. The supplied selector is tested against each element; the elements that don't match the selector will be included in the result. The `.not()` method can take a function as its argument in the same way that `.filter()` does. Elements for which the function returns true are excluded from the filtered set; all other elements are included. Selector: ```js $('li').not('.apple').length; //=> 2 ``` Function: ```js $('li').not(function(i, el) { // this === el return $(this).attr('class') === 'orange'; }).length; //=> 2 ``` #### .has( selector )
.has( element ) Filters the set of matched elements to only those which have the given DOM element as a descendant or which have a descendant that matches the given selector. Equivalent to `.filter(':has(selector)')`. Selector: ```js $('ul').has('.pear').attr('id'); //=> fruits ``` Element: ```js $('ul').has($('.pear')[0]).attr('id'); //=> fruits ``` #### .first() Will select the first element of a cheerio object ```js $('#fruits').children().first().text() //=> Apple ``` #### .last() Will select the last element of a cheerio object ```js $('#fruits').children().last().text() //=> Pear ``` #### .eq( i ) Reduce the set of matched elements to the one at the specified index. Use `.eq(-i)` to count backwards from the last selected element. ```js $('li').eq(0).text() //=> Apple $('li').eq(-1).text() //=> Pear ``` #### .get( [i] ) Retrieve the DOM elements matched by the Cheerio object. If an index is specified, retrieve one of the elements matched by the Cheerio object: ```js $('li').get(0).tagName //=> li ``` If no index is specified, retrieve all elements matched by the Cheerio object: ```js $('li').get().length //=> 3 ``` #### .index() #### .index( selector ) #### .index( nodeOrSelection ) Search for a given element from among the matched elements. ```js $('.pear').index() //=> 2 $('.orange').index('li') //=> 1 $('.apple').index($('#fruit, li')) //=> 1 ``` #### .end() End the most recent filtering operation in the current chain and return the set of matched elements to its previous state. ```js $('li').eq(0).end().length //=> 3 ``` #### .add( selector [, context] ) #### .add( element ) #### .add( elements ) #### .add( html ) #### .add( selection ) Add elements to the set of matched elements. ```js $('.apple').add('.orange').length //=> 2 ``` #### .addBack( [filter] ) Add the previous set of elements on the stack to the current set, optionally filtered by a selector. ```js $('li').eq(0).addBack('.orange').length //=> 2 ``` ### Manipulation Methods for modifying the DOM structure. #### .append( content, [content, ...] ) Inserts content as the *last* child of each of the selected elements. ```js $('ul').append('
- //
- Apple //
- Orange //
- Pear //
- Plum //
- //
- Apple //
- Orange //
- Pear //
- Plum //
- //
- Plum //
- Apple //
- Orange //
- Pear //
- //
- Plum //
- Apple //
- Orange //
- Pear //
- //
- Apple //
- Plum //
- Orange //
- Pear //
- //
- Apple //
- Plum //
- Orange //
- Pear //
- //
- Plum //
- Apple //
- Orange //
- Pear //
- //
- Plum //
- Apple //
- Orange //
- Pear //
- //
- Apple //
- Orange //
- //
- Apple //
- Orange //
- Plum //
- //
- Apple //
- Orange //
- Plum //
//
// - //
- Apple //
- Orange //
- Plum //
//
// //
// //
// .css( [ propertyNames] )
.css( [propertyName], [value] )
.css( [propertName], [function] )
.css( [properties] ) Get the value of a style property for the first element in the set of matched elements or set one or more CSS properties for every matched element. ### Rendering When you're ready to render the document, you can use the `html` utility function: ```js $.html() //=>
- //
- Apple //
- Orange //
- Pear //
This is content.
') cheerio.text($('div')) //=> This is content. ``` ### Miscellaneous DOM element methods that don't fit anywhere else #### .toArray() Retrieve all the DOM elements contained in the jQuery set as an array. ```js $('li').toArray() //=> [ {...}, {...}, {...} ] ``` #### .clone() #### Clone the cheerio object. ```js var moreFruit = $('#fruits').clone() ``` ### Utilities #### $.root Sometimes you need to work with the top-level root element. To query it, you can use `$.root()`. ```js $.root().append('- ...
- ,
- ) → true slick.contains(, ) → false --- ## The Parser Parse a CSS selector string into a JavaScript object ### `parse` selector into object Parse a CSS Selector String into a Selector Object. Expects: String Returns: SelectorObject slick.parse("#foo > bar.baz") → SelectorObject ### format ### `#foo > bar.baz` [[ { "combinator":" ", "tag":"*", "id":"foo" }, { "combinator":">", "tag":"bar", "classList": ["baz"], "classes": [{"value":"baz", "match": RegExp }]} ]] ### `h1, h2, ul > li, .things` [ [{ "combinator":" ", "tag": "h1" }], [{ "combinator":" ", "tag": "h2" }], [{ "combinator":" ", "tag": "ul" }, { "combinator": ">", "tag": "li" }], [{ "combinator":" ", "tag": "*", "classList": ["things"], "classes": [{"value": "things", "match": RegExp }] }] ]
=========================================
END OF slick@1.12.2 NOTICES AND INFORMATION
%% smart-buffer@4.1.0 NOTICES AND INFORMATION BEGIN HERE:
=========================================
The MIT License (MIT) Copyright (c) 2013-2017 Josh Glazebrook Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
=========================================
END OF smart-buffer@4.1.0 NOTICES AND INFORMATION
%% socketcluster-client@16.0.1 NOTICES AND INFORMATION BEGIN HERE:
=========================================
(The MIT License) Copyright (c) 2013-2018 SocketCluster.io Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
=========================================
END OF socketcluster-client@16.0.1 NOTICES AND INFORMATION
%% socketcluster-server@16.0.0 NOTICES AND INFORMATION BEGIN HERE:
=========================================
(The MIT License) Copyright (c) 2013-2018 SocketCluster.io Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
=========================================
END OF socketcluster-server@16.0.0 NOTICES AND INFORMATION
%% socks-proxy-agent@4.0.2 NOTICES AND INFORMATION BEGIN HERE:
=========================================
socks-proxy-agent ================ ### A SOCKS proxy `http.Agent` implementation for HTTP and HTTPS [](https://travis-ci.org/TooTallNate/node-socks-proxy-agent) This module provides an `http.Agent` implementation that connects to a specified SOCKS proxy server, and can be used with the built-in `http` or `https` modules. It can also be used in conjunction with the `ws` module to establish a WebSocket connection over a SOCKS proxy. See the "Examples" section below. Installation ------------ Install with `npm`: ``` bash $ npm install socks-proxy-agent ``` Examples -------- #### `http` module example ``` js var url = require('url'); var http = require('http'); var SocksProxyAgent = require('socks-proxy-agent'); // SOCKS proxy to connect to var proxy = process.env.socks_proxy || 'socks://127.0.0.1:9050'; console.log('using proxy server %j', proxy); // HTTP endpoint for the proxy to connect to var endpoint = process.argv[2] || 'http://nodejs.org/api/'; console.log('attempting to GET %j', endpoint); var opts = url.parse(endpoint); // create an instance of the `SocksProxyAgent` class with the proxy server information var agent = new SocksProxyAgent(proxy); opts.agent = agent; http.get(opts, function (res) { console.log('"response" event!', res.headers); res.pipe(process.stdout); }); ``` #### `https` module example ``` js var url = require('url'); var https = require('https'); var SocksProxyAgent = require('socks-proxy-agent'); // SOCKS proxy to connect to var proxy = process.env.socks_proxy || 'socks://127.0.0.1:9050'; console.log('using proxy server %j', proxy); // HTTP endpoint for the proxy to connect to var endpoint = process.argv[2] || 'https://encrypted.google.com/'; console.log('attempting to GET %j', endpoint); var opts = url.parse(endpoint); // create an instance of the `SocksProxyAgent` class with the proxy server information var agent = new SocksProxyAgent(proxy); opts.agent = agent; https.get(opts, function (res) { console.log('"response" event!', res.headers); res.pipe(process.stdout); }); ``` #### `ws` WebSocket connection example ``` js var WebSocket = require('ws'); var SocksProxyAgent = require('socks-proxy-agent'); // SOCKS proxy to connect to var proxy = process.env.socks_proxy || 'socks://127.0.0.1:9050'; console.log('using proxy server %j', proxy); // WebSocket endpoint for the proxy to connect to var endpoint = process.argv[2] || 'ws://echo.websocket.org'; console.log('attempting to connect to WebSocket %j', endpoint); // create an instance of the `SocksProxyAgent` class with the proxy server information var agent = new SocksProxyAgent(proxy); // initiate the WebSocket connection var socket = new WebSocket(endpoint, { agent: agent }); socket.on('open', function () { console.log('"open" event!'); socket.send('hello world'); }); socket.on('message', function (data, flags) { console.log('"message" event! %j %j', data, flags); socket.close(); }); ``` License ------- (The MIT License) Copyright (c) 2013 Nathan Rajlich <nathan@tootallnate.net> Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
=========================================
END OF socks-proxy-agent@4.0.2 NOTICES AND INFORMATION
%% socks@2.3.3 NOTICES AND INFORMATION BEGIN HERE:
=========================================
The MIT License (MIT) Copyright (c) 2013 Josh Glazebrook Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
=========================================
END OF socks@2.3.3 NOTICES AND INFORMATION
%% sort-keys@1.1.2 NOTICES AND INFORMATION BEGIN HERE:
=========================================
The MIT License (MIT) Copyright (c) Sindre Sorhus
All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of this software nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ========================================= END OF sprintf-js@1.0.3 NOTICES AND INFORMATION %% sshpk@1.16.1 NOTICES AND INFORMATION BEGIN HERE: ========================================= Copyright Joyent, Inc. All rights reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ========================================= END OF sshpk@1.16.1 NOTICES AND INFORMATION %% statuses@1.5.0 NOTICES AND INFORMATION BEGIN HERE: ========================================= The MIT License (MIT) Copyright (c) 2014 Jonathan Ong Harry Potter Harry Potter ## Get text values directly `````javascript var xml = "Harry Potter Harry Potter Harry Potter [0-9]{4}) -? # year \n\ (? [0-9]{2}) -? # month \n\ (? [0-9]{2}) # day ', 'x'); // XRegExp.exec gives you named backreferences on the match result var match = XRegExp.exec('2012-02-22', date); match.day; // -> '22' // It also includes optional pos and sticky arguments var pos = 3, result = []; while (match = XRegExp.exec('<1><2><3><4>5<6>', /<(\d+)>/, pos, 'sticky')) { result.push(match[1]); pos = match.index + match[0].length; } // result -> ['2', '3', '4'] // XRegExp.replace allows named backreferences in replacements XRegExp.replace('2012-02-22', date, '${month}/${day}/${year}'); // -> '02/22/2012' XRegExp.replace('2012-02-22', date, function (match) { return match.month + '/' + match.day + '/' + match.year; }); // -> '02/22/2012' // In fact, all XRegExps are RegExps and work perfectly with native methods date.test('2012-02-22'); // -> true // The *only* caveat is that named captures must be referred to using numbered backreferences '2012-02-22'.replace(date, '$2/$3/$1'); // -> '02/22/2012' // If you want, you can extend native methods so you don't have to worry about this // Doing so also fixes numerous browser bugs in the native methods XRegExp.install('natives'); '2012-02-22'.replace(date, '${month}/${day}/${year}'); // -> '02/22/2012' '2012-02-22'.replace(date, function (match) { return match.month + '/' + match.day + '/' + match.year; }); // -> '02/22/2012' date.exec('2012-02-22').day; // -> '22' // Extract every other digit from a string using XRegExp.forEach XRegExp.forEach('1a2345', /\d/, function (match, i) { if (i % 2) this.push(+match[0]); }, []); // -> [2, 4] // Get numbers within tags using XRegExp.matchChain XRegExp.matchChain('1 2 3 4 a 56', [ XRegExp('(?is).*?'), /\d+/ ]); // -> ['2', '4', '56'] // You can also pass forward and return specific backreferences var html = 'XRegExp\ Google'; XRegExp.matchChain(html, [ {regex: //i, backref: 1}, {regex: XRegExp('(?i)^https?://(? [^/?#]+)'), backref: 'domain'} ]); // -> ['xregexp.com', 'www.google.com'] // XRegExp.union safely merges strings and regexes into a single pattern XRegExp.union(['a+b*c', /(dogs)\1/, /(cats)\1/], 'i'); // -> /a\+b\*c|(dogs)\1|(cats)\2/i ~~~ These examples should give you the flavor of what's possible, but XRegExp has more syntax, flags, utils, options, and browser fixes that aren't shown here. You can even augment XRegExp's regular expression syntax with addons (see below) or write your own. See [xregexp.com](http://xregexp.com/) for more details. ## Addons In browsers, you can either load addons individually, or bundle all addons together with XRegExp by loading `xregexp-all.js`. XRegExp's [npm](http://npmjs.org/) package uses `xregexp-all.js`, which means that the addons are always available when XRegExp is installed on the server using npm. ### XRegExp Unicode Base In browsers, first include the Unicode Base script: ~~~ html ~~~ Then you can do this: ~~~ js var unicodeWord = XRegExp('^\\p{L}+$'); unicodeWord.test('Русский'); // -> true unicodeWord.test('日本語'); // -> true unicodeWord.test('العربية'); // -> true ~~~ The base script adds `\p{Letter}` and its alias `\p{L}`, but other Unicode categories, scripts, blocks, and properties require addon packages. Try these next examples after additionally including `unicode-scripts.js`: ~~~ js XRegExp('^\\p{Hiragana}+$').test('ひらがな'); // -> true XRegExp('^[\\p{Latin}\\p{Common}]+$').test('Über Café.'); // -> true ~~~ XRegExp uses the Unicode 6.1 Basic Multilingual Plane. ### XRegExp.build In browsers, first include the script: ~~~ html ~~~ You can then build regular expressions using named subpatterns, for readability and pattern reuse: ~~~ js var time = XRegExp.build('(?x)^ {{hours}} ({{minutes}}) $', { hours: XRegExp.build('{{h12}} : | {{h24}}', { h12: /1[0-2]|0?[1-9]/, h24: /2[0-3]|[01][0-9]/ }, 'x'), minutes: /^[0-5][0-9]$/ }); time.test('10:59'); // -> true XRegExp.exec('10:59', time).minutes; // -> '59' ~~~ Named subpatterns can be provided as strings or regex objects. A leading `^` and trailing unescaped `$` are stripped from subpatterns if both are present, which allows embedding independently useful anchored patterns. `{{…}}` tokens can be quantified as a single unit. Backreferences in the outer pattern and provided subpatterns are automatically renumbered to work correctly within the larger combined pattern. The syntax `({{name}})` works as shorthand for named capture via `(? {{name}})`. Named subpatterns cannot be embedded within character classes. See also: *[Creating Grammatical Regexes Using XRegExp.build](http://blog.stevenlevithan.com/archives/grammatical-patterns-xregexp-build)*. ### XRegExp.matchRecursive In browsers, first include the script: ~~~ html ~~~ You can then match recursive constructs using XRegExp pattern strings as left and right delimiters: ~~~ js var str = '(t((e))s)t()(ing)'; XRegExp.matchRecursive(str, '\\(', '\\)', 'g'); // -> ['t((e))s', '', 'ing'] // Extended information mode with valueNames str = 'Here is example'; XRegExp.matchRecursive(str, 'an
- ]) → [, , ] slick.search("#foo > bar.baz", document, []) → [
, "div.rocks") → true slick.matches(
, "div.rocks") → false slick.matches(
, ', 'Apply only to these teams')
.parse(process.argv);
let options = {};
if (Commander.teams) {
options.teamIds = Commander.teams.split(',');
}
(async function() {
try {
await new ContentCombiner().go(options);
process.exit();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-cloud-services 0000775 0000000 0000000 00000001556 14607772627 0025053 0 ustar 00root root 0000000 0000000 #!/bin/bash
#desc# stop/start/status all cs services in in the playground
function usage {
echo "usage: `basename $0` {action}"
exit 1
}
function signal_services {
local action=$1
cs_api-service $action
cs_broadcaster-service $action
cs_mailin-service $action
cs_outbound_email-service $action
[ "$action" == status ] && mdb-service status || echo "skippign mongo (mdb-service $action)"
}
mongoRunning=`mdb-service status|wc -l`
case "$1" in
start)
[ $mongoRunning -eq 0 ] && mdb-service start && sleep 1
cs_api-service start -- --one_worker
cs_broadcaster-service start -- --one_worker
cs_mailin-service start -- --one_worker
cs_outbound_email-service start -- --one_worker
sleep 1
echo
echo
signal_services status
;;
stop)
signal_services stop
sleep 1
signal_services status
;;
status)
signal_services status
;;
*) usage;;
esac
codestream-server-api-15.4.0/api_server/bin/cs_api-copy-team-data 0000775 0000000 0000000 00000020244 14607772627 0024724 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python3
#desc# Copy codestream team data from one mongo db to another
import os
import sys
import argparse
import pickle
import glob
import re
from pymongo import MongoClient
from bson.objectid import ObjectId
sys.path.append(os.environ['DT_TOP'] + "/lib")
from sysUtils import dumpDict, readJsonFileIgnoreComments, someValueOrNone
def getTeamIdFromTeamName(cfgData):
teamsCol = MongoClient(cfgData['sourceUrl'])[cfgData['db']]['teams']
c = teamsCol.count_documents({'name': cfgData['teamName']})
if c != 1:
print("there are {} teams named {}. Use teamId".format(c, cfgData['teamName']))
exit(1)
team = teamsCol.find_one({'name': cfgData['teamName']})
if cfgData['debug']: print("{} has teamId {}".format(cfgData['teamName'], team['_id']))
return str(team['_id'])
def dumpCollection(cfgData, whichUrl, collection, queryFilter, dumpDir):
if not os.path.isdir(dumpDir): os.mkdir(dumpDir)
docList = list(MongoClient(cfgData[whichUrl])[cfgData['db']][collection].find(queryFilter))
for doc in docList:
doc['_id'] = str(doc['_id'])
with open(dumpDir + "/" + collection + ".data", 'wb') as binDumpFile:
pickle.dump(docList, binDumpFile)
if cfgData['verbose']:
print("dumped {} document(s) from the {} collection".format(len(docList), collection))
def loadCollection(loadFileName):
collectionData = None
with open(loadFileName, 'rb') as binLoadFile:
collectionData = pickle.load(binLoadFile)
for doc in collectionData:
if len(doc['_id']) == 24:
doc['_id'] = ObjectId(doc['_id'])
return collectionData
def mongoDeleteManyWithFilter(cfgData, whichUrl, collection, collectionQueryMap):
try:
result = MongoClient(cfgData[whichUrl])[cfgData['db']][collection].delete_many(collectionQueryMap[collection])
if result.acknowledged != True:
print("delete_many was not acknowledged")
exit(1)
delCount = result.deleted_count
if cfgData['verbose']:
print("{} documents deleted from {}".format(delCount, collection))
return delCount
except:
exc_type, exc_obj, exc_traceback = sys.exc_info()
print("Error during delete_many for collection", collection)
print("Type: {}, Obj: {}, Tb: {}".format(exc_type, exc_obj, exc_traceback))
exit(1)
def mongoInsertMany(cfgData, whichUrl, dumpDir = None):
if not dumpDir: dumpDir = cfgData['dumpDir'] + "/" + cfgData['teamId']
fileSpec = dumpDir + "/*.data"
collectionFiles = glob.glob(fileSpec)
if not collectionFiles:
print("no files to load found ({})".format(fileSpec))
exit(1)
for loadFileName in sorted(collectionFiles):
collection = re.sub('\.data$', '', os.path.basename(loadFileName))
collectionData = loadCollection(loadFileName)
if collectionData:
try:
result = MongoClient(cfgData[whichUrl])[cfgData['db']][collection].insert_many(collectionData)
if result.acknowledged != True:
print("insert was not acknowledged")
return False
idList = result.inserted_ids
if len(idList) != len(collectionData):
print("number of records inserted ({}) does not match number loaded ({})".format(len(idList), len(collectionData)))
return False
if cfgData['verbose']:
print("{} documents inserted into {}".format(len(idList), collection))
except:
exc_type, exc_obj, exc_traceback = sys.exc_info()
print("Error during insert_many for collection", collection)
print("Type: {}, Obj: {}, Tb: {}".format(exc_type, exc_obj, exc_traceback))
return False
else:
print("no documents in {} to load".format(collection))
return True
parser = argparse.ArgumentParser(description='copy selected codestream data')
parser.add_argument('--verbose', action='store_true', default=False, help="verbose mode")
parser.add_argument('--debug', action='store_true', default=False, help="debug mode")
# config
parser.add_argument('--cfg', type=str, help="config file")
parser.add_argument('--sourceUrl', type=str, help="source mongo connect Url")
parser.add_argument('--destUrl', type=str, help="dest mongo connect Url")
parser.add_argument('--teamId', type=str, help="teamId")
parser.add_argument('--teamName', type=str, help="team name (case sensitive) - only useful if it matches on team")
parser.add_argument('--db', type=str, help="Codestream Database")
parser.add_argument('--dumpDir', type=str, help="Dump directory")
# behavior
parser.add_argument('--load', action='store_true', default=False, help="load data from dump (default is false)")
parser.add_argument('--no-dump', action='store_true', default=False, help="Use pre-existing dump (default is false)")
parser.add_argument('--pre-delete', action='store_true', default=False, help="delete pre-existing documents on the target before loading")
parser.add_argument('--restore-on-fail', action='store_true', default=True, help="try to restore original data if copy fails")
args = parser.parse_args()
if args.debug: args.verbose = True
cfgData = readJsonFileIgnoreComments(args.cfg) if args.cfg else {}
if args.sourceUrl: cfgData['sourceUrl'] = args.sourceUrl
if args.destUrl: cfgData['destUrl'] = args.destUrl
if args.teamId: cfgData['teamId'] = args.teamId
if args.teamName: cfgData['teamName'] = args.teamName
if args.dumpDir: cfgData['dumpDir'] = args.dumpDir
if args.db: cfgData['db'] = args.db
if args.debug: cfgData['debug'] = True
if args.verbose: cfgData['verbose'] = True
if not someValueOrNone(cfgData, 'dumpDir'):
cfgData['dumpDir'] = os.environ['CS_API_TMP'] + "/csdump"
if not someValueOrNone(cfgData, 'db'):
cfgData['db'] = "codestream"
if not someValueOrNone(cfgData, 'debug'):
cfgData['debug'] = False
if not someValueOrNone(cfgData, 'verbose'):
cfgData['verbose'] = False
# find the teamId if a teamName was provided
if someValueOrNone(cfgData, 'teamName') and 'teamId' not in cfgData:
cfgData['teamId'] = getTeamIdFromTeamName(cfgData)
if args.debug: dumpDict(cfgData, None, "cfgData:")
queryTeamId = { "teamId": cfgData['teamId'] }
queryTeamIds = { "teamIds": cfgData['teamId'] }
queryTeamObjectId = { "_id": ObjectId(cfgData['teamId']) }
collectionQueryMap = {
'users': queryTeamIds,
'companies': queryTeamIds,
'repos': queryTeamId,
'streams': queryTeamId,
'posts': queryTeamId,
'markers': queryTeamId,
'markerLocations': queryTeamId,
'teams': queryTeamObjectId
}
restoreDir = cfgData['dumpDir'] + "/dest-backup/" + cfgData['teamId']
if not args.no_dump:
if os.path.isdir(cfgData['dumpDir']):
print("dump directory {} already exists".format(cfgData['dumpDir']))
exit(1)
os.mkdir(cfgData['dumpDir'])
os.mkdir(cfgData['dumpDir'] + "/dest-backup")
if cfgData['verbose']: print("==== saving documents from sourceUrl")
for collection in sorted(collectionQueryMap.keys()):
dumpCollection(cfgData, 'sourceUrl', collection, collectionQueryMap[collection], cfgData['dumpDir'] + "/" + cfgData['teamId'])
if args.load:
if args.restore_on_fail:
if cfgData['verbose']: print("==== saving documents from destUrl for restore if needed (--restore-on-fail)")
for collection in sorted(collectionQueryMap.keys()):
if not os.path.isdir(restoreDir): os.mkdir(restoreDir)
dumpCollection(cfgData, 'destUrl', collection, collectionQueryMap[collection], restoreDir)
if args.pre_delete:
if cfgData['verbose']: print("==== pre-deleting documents from destUrl")
for collection in sorted(collectionQueryMap.keys()):
numDeleted = mongoDeleteManyWithFilter(cfgData, 'destUrl', collection, collectionQueryMap)
if cfgData['verbose']: print("==== loading documents destUrl")
success = mongoInsertMany(cfgData, 'destUrl')
if not success and not args.restore_on_fail:
exit(1)
elif not success:
cfgData['verbose'] = True
print("=======================================================================")
print("****** Data load failed - attempting restore of original content ******")
print("=======================================================================")
print("=== Step 1. Delete partially copied docs...")
for collection in sorted(collectionQueryMap.keys()):
numDeleted = mongoDeleteManyWithFilter(cfgData, 'destUrl', collection, collectionQueryMap)
print("=== Step 2. Loading original data...")
success = mongoInsertMany(cfgData, 'destUrl', restoreDir)
if not success:
print("That didn't work either. You're on your own. Terribly sorry about that.")
exit(1)
print("=== Restore succeesful")
print("DON'T FORGET TO RE-CREATE THE INDEXES IDIOT")
exit(0)
codestream-server-api-15.4.0/api_server/bin/cs_api-deactivate_inactive_teams.js 0000775 0000000 0000000 00000013663 14607772627 0027725 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// deactivate all teams whose last activity preceded a certain time
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Deactivator = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/lib/util/deactivator');
const FS = require('fs');
Commander
.option('-t, --time ', 'If last activity was before this timestamp, deactivate the team')
.option('--created, --created', 'Treat the timestamp as a createdAt date, ignore activity considerations')
.option('--dryrun, --dryrun', 'Do a dry run, don\'t actually deactivate any teams')
.option('--numposts, --numposts ', 'Don\'t deactivate teams whose post count exceeds this value')
.option('--emails, --emails ', 'Output list of emails of deactivated users to this file')
.option('--ignore, --ignore ', 'Ignore the org IDs given in this file')
.option('--throttle, --throttle ', 'Throttle deactivations by this time interval')
.parse(process.argv);
if (!Commander.time) {
console.warn('timestamp required');
process.exit();
}
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class InactiveTeamDeactivator {
constructor (options) {
Object.assign(this, options);
this.throttle = this.throttle || 1000;
}
async go () {
await this.openMongoClient();
await this.openIgnoreFile();
await this.openEmailFile();
await this.doDeactivate();
await this.finish();
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// read in file of company IDs to ignore
async openIgnoreFile () {
if (!this.ignoreFile) { return; }
const contents = FS.readFileSync(this.ignoreFile, 'utf8').replace(/\r\n/g, '\n');
this.ignoreCompanyIds = contents.split('\n');
console.log(`Will ignore ${this.ignoreCompanyIds.length} company IDs...`);
}
// open an output stream to write emails to, representing all users deactivated
async openEmailFile () {
if (!this.emailsOutputFile) { return; }
this.emailsOutputStream = FS.createWriteStream(this.emailsOutputFile);
}
// step through all the teams, determine which have no active users, and deactivate them
async doDeactivate () {
const result = await this.data.teams.getByQuery(
{
deactivated: false,
createdAt: { $lt: this.time }
},
{
stream: true,
overrideHintRequired: true,
fields: ['id']
}
);
let team;
this.numTeamsDeactivated = 0;
do {
team = await result.next();
if (team) {
await this.checkAndDeactivateTeam(team);
}
} while (team);
result.done();
if (this.dryrun) {
console.log(`${this.numTeamsDeactivated} teams would have been deactivated`);
} else {
console.log(`${this.numTeamsDeactivated} teams were deactivated`);
}
}
// check for activity for a single team, and deactivate as needed
async checkAndDeactivateTeam (team) {
console.log(`Checking team ${team.id}...`);
if (this.ignoreCompanyIds && this.ignoreCompanyIds.includes(team.companyId)) {
console.log(`Ignoring team ${team.id} which is in the ignore list`);
await Wait(this.throttle / 10);
return;
}
if (this.created) {
// if the created flag is set, just deactivate any team
// (the createdAt timestamp has already been applied)
return this.deactivateTeam(team);
}
const users = await this.data.users.getByQuery(
{
deactivated: false,
teamIds: team.id,
isRegistered: true
},
{
overrideHintRequired: true,
fields: ['lastLogin']
}
);
let lastLogin = 0;
users.forEach(u => {
if (u.lastLogin > lastLogin) {
lastLogin = u.lastLogin;
}
});
if (lastLogin < this.time) {
if (this.numPosts) {
const numPosts = this.data.posts.countByQuery(
{
teamId: team.id
},
{
overrideHintRequired: true
}
);
if (numPosts >= this.numPosts) {
console.log(`Team ${team.id} will not be deactivated because they have ${numPosts} posts`);
await Wait(this.throttle / 10);
return;
}
}
return this.deactivateTeam(team);
} else {
await Wait(this.throttle / 10);
}
}
// deactivate this team
async deactivateTeam (team) {
if (this.dryrun) {
console.log(`Would have deactivated team ${team.id}`);
} else {
await new Deactivator().go({
mongoClient: this.mongoClient,
teamIdOrName: team.id,
deactivateTeamlessUsers: true,
emailsOutputStream: this.emailsOutputStream
});
console.log(`Team ${team.id} deactivated`);
}
this.numTeamsDeactivated++;
await Wait(this.throttle);
}
async finish () {
if (this.emailsOutputStream) {
this.emailsOutputStream.close();
}
}
}
(async function() {
try {
const time = Commander.time ? parseInt(Commander.time, 10) : NaN;
if (isNaN(time)) {
throw 'invalid or missing timestamp';
}
const options = {
time,
emailsOutputFile: Commander.emails,
ignoreFile: Commander.ignore,
dryrun: Commander.dryrun,
created: Commander.created
}
if (Commander.throttle) {
options.throttle = parseInt(Commander.throttle, 10);
if (isNaN(options.throttle)) {
throw 'invalid throttle value';
}
}
if (Commander.numposts) {
options.numPosts = parseInt(Commander.numposts, 10);
if (isNaN(options.numPosts)) {
throw 'invalid post count';
}
}
await ApiConfig.loadPreferredConfig();
await new InactiveTeamDeactivator(options).go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-deidentify.js 0000775 0000000 0000000 00000016266 14607772627 0024667 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/*************************************************************************************************************************************************
*************************************************************************************************************************************************
*************************************************************************************************************************************************
* NOTE - THIS SCRIPT IS STILL IN PROGRESS ... DO NOT USE UNTIL MORE FULLY TESTED
*************************************************************************************************************************************************
*************************************************************************************************************************************************
*************************************************************************************************************************************************
*/
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
Commander
.option('-t, --team ', 'Deidentify team')
.option('-u, --user ', 'Deidentify user')
.option('--strings, --deidentify-strings', 'Quote-enclosed, comma-separated list of strings to redact')
.parse(process.argv);
if (!Commander.team && !Commander.user) {
console.warn('teamId or userId required');
process.exit();
}
let Options = {
teamId: Commander.team,
userId: Commander.user,
deidentifyStrings: Commander.deidentifyStrings
};
const COLLECTIONS = ['companies', 'teams', 'users', 'posts', 'codemarks', 'reviews'];
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, 100);
});
}
class Deidentifier {
async go (options) {
Object.assign(this, options);
this.logger = this.logger || console;
await ApiConfig.loadPreferredConfig();
await this.openMongoClient();
await this.getTargetObject();
await this.getUsers();
await this.deidentifyUsers();
await this.deactivateTeam();
}
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async getTargetObject () {
if (this.teamId) {
this.team = await this.data.teams.getById(this.teamId);
if (!this.team) {
throw `could not fetch team ${this.teamId}`;
}
} else if (this.userId) {
this.user = await this.data.users.getById(this.userId);
if (!this.user) {
throw `could not fetch user ${this.userId}`;
}
}
else {
throw 'must provide team ID or user ID';
}
}
async getUsers () {
if (this.team) {
this.users = await this.data.users.getByIds(this.team.memberIds);
} else {
this.users = [this.user];
}
}
async deidentifyUsers () {
for (const user of this.users) {
console.log(`Deidentifying user ${user.id}...`);
await this.deidentifyUser(user);
await Wait(100);
}
}
async deidentifyUser (user) {
await this.deidentifyUserContent(user);
if (this.team && (user.teamIds || []).find(tid => tid !== this.team.id)) {
console.log(`******** Not deidentifying user ${user.id} who is on multiple teams`);
return;
}
const now = Date.now();
const { email, fullName } = user;
const emailParts = (email || '').split('@');
const domainParts = emailParts[1] && emailParts[1].split('.');
const fullNameParts = (fullName || '').split(' ');
const redactedEmail = `${this.redactString(emailParts[0])}-deactivated${now}@${domainParts.map(part => this.redactString(part)).join('.')}`;
const redactedFullName = fullNameParts.map(part => this.redactString(part)).join(' ');
const op = {
$set: {
email: redactedEmail,
searchableEmail: redactedEmail,
fullName: redactedFullName,
deactivated: true,
},
$unset: {
firstName: true,
lastName: true,
providerInfo: true,
accessTokens: true,
timeZone: true,
sessions: true,
preferences: true,
pubNubToken: true,
providerIdentities: true,
broadcasterToken: true,
modifiedReposModifiedAt: true,
status: true,
modifiedRepos: true,
countryCode: true,
compactModifiedRepos: true,
secondaryEmails: true
}
};
if (this.team) {
op.$pull = {
teamIds: this.team.id
};
if (this.team.companyId) {
op.$pull.companyIds = this.team.companyId;
}
}
await this.data.users.updateDirect(
{ _id: this.data.users.objectIdSafe(user.id) },
op
);
}
redactString (str) {
return '*'.repeat(str.length);
}
async deidentifyUserContent (user) {
const teamIds = user.teamIds || [];
for (let teamId of teamIds) {
await this.deidentifyUserContentFromTeam(user, teamId);
}
}
async deidentifyUserContentFromTeam (user, teamId) {
for (let collection of ['posts', 'codemarks', 'reviews']) {
await this.deidentifyUserContentInCollection(collection, user, teamId);
}
}
async deidentifyUserContentInCollection (collection, user, teamId) {
const result = await this.data[collection].getByQuery({ teamId }, { stream: true });
let item;
do {
item = await result.next();
if (item) {
if (await this.deidentifyUserContentInItem(collection, item, user)) {
await Wait(100);
}
}
} while (item);
result.done();
}
async deidentifyUserContentInItem(collection, item, user) {
const set = { };
['text', 'title'].forEach(contentAttr => {
const strings = (this.deidentifyStrings || '').split(',');
const stringsOrAttrs = [...strings, '__email', '__fullName'];
stringsOrAttrs.forEach(stringOrAttr => {
let index = -1;
do {
let value;
if (stringOrAttr.startsWith('__')) {
value = user[stringOrAttr.substring(2)] || '';
} else {
const match = stringOrAttr.match(/"(.*)"/);
if (match) {
value = match[1];
} else {
value = stringOrAttr;
}
}
if (value.length > 0) {
index = (item[contentAttr] || '').toLowerCase().indexOf(value.toLowerCase());
if (index !== -1) {
const before = item[contentAttr].substring(0, index - 1);
const redacted = this.redactString(value);
const after = item[contentAttr].substring(before.length + value.length);
item[contentAttr] = set[contentAttr] = `${before}${redacted}${after}`;
}
}
} while (index !== -1);
});
});
if (Object.keys(set).length > 0) {
console.log(`Deidentifying user ${user.id} from ${collection} item ${item.id}...`);
await this.data[collection].updateDirect(
{ _id: this.data[collection].objectIdSafe(item.id) },
{ $set: set }
);
return true;
}
}
async deactivateTeam () {
if (this.team) {
await this.data.teams.updateDirect(
{ _id: this.data.teams.objectIdSafe(this.team.id) },
{ $set: { deactivated: true } }
);
}
}
}
(async function() {
try {
await new Deidentifier().go(Options);
process.exit();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-delete_code_errors.js 0000775 0000000 0000000 00000005650 14607772627 0026366 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// delete all code errors and associated posts from a given company
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
Commander
.option('-c, --companyId ', 'ID of the company')
.parse(process.argv);
if (!Commander.companyId) {
console.warn('companyId required');
process.exit();
}
class CodeErrorDeleter {
constructor (options) {
Object.assign(this, options);
}
async go () {
await this.openMongoClient();
await this.doDelete();
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// do the deletion
async doDelete () {
const company = await this.data.companies.getById(this.companyId);
if (!company) {
console.warn('COMPANY NOT FOUND: ' + this.companyId);
}
const teamId = company.everyoneTeamId;
const codeErrors = await this.data.codeErrors.getByQuery({ teamId }, { overrideHintRequired: true });
console.log(`Found ${codeErrors.length} code errors for company ${this.companyId}`);
const codeErrorIds = codeErrors.map(ce => ce._id);
const postIds = codeErrors.map(ce => ce.postId);
const childPosts = await this.data.posts.getByQuery(
{
teamId,
parentPostId: { $in: postIds }
},
{ overrideHintRequired: true }
);
console.log(`Found ${childPosts.length} child posts`);
const childPostIds = childPosts.map(p => p._id.str);
const grandchildPosts = await this.data.posts.getByQuery(
{
teamId,
parentPostId: { $in: childPostIds }
},
{ overrideHintRequired: true }
);
console.log(`Found ${grandchildPosts.length} grandchild posts`);
const grandchildPostIds = grandchildPosts.map(p => p._id.str);
const allChildPostIds = [...childPostIds, ...grandchildPostIds];
console.log(`Deleting markers...`);
await this.data.markers.deleteByQuery({ teamId, postId: { $in: allChildPostIds } });
console.log(`Deleting codemarks...`);
await this.data.codemarks.deleteByQuery({ teamId, postId: { $in: allChildPostIds } });
console.log(`Deleting posts...`);
const allPostIds = [...postIds, ...allChildPostIds];
await this.data.posts.deleteByIds(allPostIds);
console.log(`Deleting code errors...`);
await this.data.codeErrors.deleteByIds(codeErrorIds);
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new CodeErrorDeleter({companyId: Commander.companyId}).go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-help 0000775 0000000 0000000 00000000152 14607772627 0023043 0 ustar 00root root 0000000 0000000 #!/bin/bash
#desc# brief command summary of the cs_api (CodeStream API) sandbox
dt-help $CS_API_TOP/bin
codestream-server-api-15.4.0/api_server/bin/cs_api-intake_new_relic_orgs.js 0000775 0000000 0000000 00000006044 14607772627 0027070 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// desc// dump records mapping New Relic account IDs to org IDs to mongo
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
const ReadLine = require('readline');
const FS = require('fs');
Commander
.option('-f, --file ', 'CSV file containing data to pull in')
.option('-s, --start ', 'Start at this number')
.option('-l, --limit ', 'Limit to this number of records')
.parse(process.argv);
const COLLECTIONS = ['newRelicOrgs'];
const THROTTLE_TIME = 1000;
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class IntakeEngine {
constructor (options) {
Object.assign(this, options);
}
async go () {
await this.openMongoClient();
await this.openFileStream();
await this.writeCollection();
await this.finish();
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// open a file stream to read from
async openFileStream () {
this.fileStream = await FS.createReadStream(this.file);
this.readLine = ReadLine.createInterface({
input: this.fileStream
});
}
// write out the account to org mapping collection, with throttling
async writeCollection () {
let nRead = 0;
let nWritten = 0;
if (!this.start) {
await this.data.newRelicOrgs.deleteByQuery({});
}
for await (const line of this.readLine) {
nRead++;
const [accountId, orgId] = line.split(',');
const accountIdNum = parseInt(accountId, 10);
if (!accountIdNum) continue;
if (this.start && nRead < this.start) {
continue;
}
await this.data.newRelicOrgs.create({
accountId: accountIdNum,
orgId
}, { noVersion: true });
nWritten++;
if (this.limit && nWritten === this.limit) {
console.log('Reached limit');
break;
}
if (nRead % 1000 == 0) {
console.log(`Read ${nRead}, wrote ${nWritten}`);
await Wait(THROTTLE_TIME);
}
}
}
async finish () {
this.fileStream.close();
}
}
(async function() {
try {
const file = Commander.file;
if (!file) {
throw 'must provide teamId or all';
}
let limit, start;
if (Commander.limit) {
limit = parseInt(Commander.limit);
if (isNaN(limit)) {
throw 'limit is not a number'
}
}
if (Commander.start) {
start = parseInt(Commander.start);
if (isNaN(start)) {
throw 'start is not a number'
}
}
await ApiConfig.loadPreferredConfig();
await new IntakeEngine({ file, limit, start }).go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-load-test 0000775 0000000 0000000 00000002525 14607772627 0024015 0 ustar 00root root 0000000 0000000 #!/bin/bash
function usage {
echo "usage: $0 -N threads [-M max_iterations_per_thread]"
echo " $0 -S"
exit
}
function run_test {
cd $CS_API_TOP
rightnow=`date +%s`
log="$logdir/test-$rightnow"
touch $log.running
echo "Launching $log"
(./test.sh; mv $log.running $log.done; echo "test $rightnow done") >$log.log 2>&1 &
sleep 1
}
function stop_all {
pids=`ps auxww|grep mocha|grep -v grep |grep $CS_API_NAME|awk '{print $2}'`
[ -z "$pids" ] && echo "nothing running" || kill $pids
exit
}
function exec_test {
run_test
running=1
cycle=0
while [ $running -gt 0 ]; do
[ $running -lt $n_threads ] && run_test
sleep 5
cycle=`expr $cycle + 1`
num_complete=`ls $logdir/*.done 2>/dev/null|wc -l`
if [ $max_times -gt 0 ]; then
[ $num_complete -ge $max_times ] && echo "max count reached" && stop_all
fi
running=`ls $logdir/*.running 2>/dev/null|wc -l`
if [ $(( $cycle % 5 )) -eq 0 ]; then
printf "`date` Running: %d Complete: %d\n" $running $num_complete
fi
done
}
n_threads=0
max_times=0
while getopts "N:M:S" arg
do
case $arg in
S) stop_all;;
N) n_threads=$OPTARG;;
M) max_times=$OPTARG;;
*) usage;;
esac
done
shift `expr $OPTIND - 1`
[ $n_threads -le 0 ] && usage
start_time=`date +%Y%m%d-%H%M%S`
logdir=$CS_API_LOGS/load-test-$start_time
mkdir $logdir
echo "Test logs will reside in $logdir"
exec_test
codestream-server-api-15.4.0/api_server/bin/cs_api-merge_companies.js 0000775 0000000 0000000 00000004540 14607772627 0025670 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// manually merge one company into another, deleting the merged company after we are done
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/node_modules/commander');
const TeamMerger = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/teams/team_merger');
Commander
.option('-f, --fromCompanyIds ', 'Comma-separated IDs of the companies to merge and delete')
.option('-t, --toCompanyId ', 'ID of the company to merge to')
.option('--dryrun', 'Do a dry run with informational messages, but don\'t actually DO anything')
.option('--merge-teams', 'Also merge the team for the from company, before merging to the to company')
.parse(process.argv);
if (!Commander.fromCompanyIds || !Commander.toCompanyId) {
Commander.help();
}
class CompanyMerger {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
await this.process();
this.logger.log('DONE');
}
catch (error) {
this.logger.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async process () {
for (let fromCompanyId of this.fromCompanyIds) {
await new TeamMerger({
logger: this.logger,
data: this.data,
dryRun: this.dryRun
}).mergeCompanies(fromCompanyId, this.toCompanyId);
}
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
const fromCompanyIds = Commander.fromCompanyIds.split(',');
await new CompanyMerger().go({
fromCompanyIds: fromCompanyIds.map(id => id.toLowerCase()),
toCompanyId: Commander.toCompanyId.toLowerCase(),
dryRun: Commander.dryrun,
mergeTeams: Commander.mergeTeams
});
}
catch (error) {
this.logger.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-merge_flagged_repos.js 0000775 0000000 0000000 00000003635 14607772627 0026517 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// after teams have been merged into a single company, merge repos that have been flagged for merge
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/node_modules/commander');
const RepoMerger = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/repos/repo_merger');
Commander
.option('-t, --teamId ', 'ID of the team to scan for repos in need of merging')
.option('--dryrun', 'Do a dry run with informational messages, but don\'t actually DO anything')
.parse(process.argv);
if (!Commander.teamId) {
Commander.help();
}
class RepoMergerForTeam {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
await this.process();
this.logger.log('DONE');
}
catch (error) {
this.logger.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async process () {
await new RepoMerger({
logger: this.logger,
data: this.data,
dryRun: this.dryRun
}).mergeFlaggedRepos(this.teamId);
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new RepoMergerForTeam().go({
teamId: Commander.teamId.toLowerCase(),
dryRun: Commander.dryrun,
logger: console
});
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-merge_repos.js 0000775 0000000 0000000 00000010616 14607772627 0025043 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// manually merge a repo's remotes to another, delete the merged repo, and adjust all codemarks, file streams, and markers accordingly
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/node_modules/commander');
// need these collections from mongo
const COLLECTIONS = ['repos', 'markers', 'streams', 'users'];
Commander
.option('-f, --fromRepo ', 'ID of the repo to merge and delete')
.option('-t, --toRepo ', 'ID of the repo to merge to')
.option('--invalidate-user-sessions', 'Invalidate the sessions of all users on the team, to wipe any cached data')
.parse(process.argv);
if (!Commander.fromRepo || !Commander.toRepo) {
Commander.help();
}
class RepoMerger {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
await this.process();
console.log('DONE');
}
catch (error) {
this.logger.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async process () {
await this.getRepos();
await this.mergeRepos();
await this.fixMarkers();
await this.fixFileStreams();
await this.invalidateSessions();
}
async getRepos () {
this.fromRepo = await this.data.repos.getById(this.fromRepoId);
if (!this.fromRepo) {
this.logger.error(`Repo ${Commander.fromRepo} not found`);
process.exit();
}
this.toRepo = await this.data.repos.getById(this.toRepoId);
if (!this.toRepo) {
this.logger.error(`Repo ${Commander.toRepo} not found`);
process.exit();
}
if (this.fromRepo.teamId !== this.toRepo.teamId) {
this.logger.error('Repos must be from the same team');
process.exit();
}
}
async mergeRepos () {
let fromRemotes = this.fromRepo.remotes || [];
if (this.fromRepo.url && this.fromRepo.normalizedUrl && this.fromRepo.companyIdentifier) {
fromRemotes.push({
url: this.fromRepo.url,
normalizedUrl: this.fromRepo.normalizedUrl,
companyIdentifier: this.fromRepo.companyIdentifier
});
}
let toRemotes = this.toRepo.remotes || [];
if (this.toRepo.url && this.toRepo.normalizedUrl && this.toRepo.companyIdentifier) {
toRemotes.push({
url: this.toRepo.url,
normalizedUrl: this.toRepo.normalizedUrl,
companyIdentifier: this.toRepo.companyIdentifier
});
}
fromRemotes = fromRemotes.filter(fromRemote => {
return !toRemotes.find(toRemote => toRemote.normalizedUrl === fromRemote.normalizedUrl);
});
toRemotes = [...toRemotes, ...fromRemotes];
this.logger.log(`Updating repo ${this.toRepo.id}...`);
await this.data.repos.updateById(this.toRepoId, { remotes: toRemotes });
this.logger.log(`Disassociating repo ${this.fromRepo.id}...`);
await this.data.repos.updateById(this.fromRepoId, { deactivated: true, teamId: `WAS ${this.fromRepo.teamId}` });
}
async fixMarkers () {
this.logger.log('Adjusting markers...');
await this.data.markers.updateDirect(
{ teamId: this.fromRepo.teamId, repoId: this.fromRepo.id },
{ $set: { repoId: this.toRepo.id } }
);
}
async fixFileStreams () {
this.logger.log('Adjusting file streams...');
await this.data.streams.updateDirect(
{ teamId: this.fromRepo.teamId, repoId: this.fromRepo.id },
{ $set: { repoId: this.toRepo.id } }
);
}
async invalidateSessions () {
if (!this.invalidateUserSessions) {
return;
}
this.logger.log('Invalidating user access tokens...');
await this.data.users.updateDirect(
{ teamIds: this.fromRepo.teamId },
{ $set: { 'accessTokens.web.invalidated': true } }
);
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new RepoMerger().go({
fromRepoId: Commander.fromRepo,
toRepoId: Commander.toRepo,
invalidateUserSessions: Commander.invalidateUserSessions
});
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-migrate_to_company_centric.js 0000775 0000000 0000000 00000005156 14607772627 0030126 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// migrate all companies that have not yet been migrated to company-centric paradigm
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
const MigrationHandler = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/company_centric_migration/migration_handler');
Commander
.option('--dryrun', 'Do a dry run, meaning don\'t actually write anything to our database, but report on numbers')
.parse(process.argv);
const THROTTLE_TIME = 1000;
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class Migrator {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
await this.openMongoClient();
await this.doMigrations();
}
catch (error) {
console.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// step through the companies that have not yet been migrated, and migrate them
async doMigrations () {
const result = await this.data.companies.getByQuery(
{
hasBeenMigratedToCompanyCentric: { $ne: true },
deactivated: false
},
{
stream: true,
overrideHintRequired: true,
sort: { _id: -1 }
}
);
let company;
let n = 0;
do {
company = await result.next();
if (company) {
await this.migrateCompany(company);
n++;
}
} while (company);
result.done();
console.log(`${n} companies were migrated`);
}
// migrate a single company
async migrateCompany (company) {
console.log(`Migrating company ${company.id}:${company.name}...`);
const migrationHandler = new MigrationHandler({
data: this.data,
logger: console,
company,
dryRun: this.dryrun
});
await migrationHandler.handleMigration();
if (migrationHandler.didMigrateMultiTeamCompany) {
await Wait(1000);
} else {
await Wait(100);
}
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new Migrator().go({ dryrun: Commander.dryrun });
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-migrate_to_nr_azure.js 0000775 0000000 0000000 00000013456 14607772627 0026600 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// migrate all companies that have not yet been migrated to New Relic / Azure as identity provider
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
const NewRelicIDPMigrationHandler = require('../lib/util/newrelic_idp_migration_handler');
const NewRelicIDP = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/newrelic_idp/newrelic_idp');
const FS = require('fs');
Commander
.option('--dryrun', 'Do a dry run, meaning don\'t actually write anything to our database, but report on numbers')
.option('--throttle ', 'Throttle processing each user by this amount of time')
.option('--company ', 'Migrate only this company')
.option('--companyfile ', 'Migrate only those companies listed (by ID) in this file')
.option('--verbose', 'Verbose logging output')
.option('--nr', 'Only migrate NR-connected orgs')
.option('--incremental', 'Incremental migration, include companies that have already been migrated, just looking for un-migrated users')
//.option('--setidppwd', 'Set the flag that says verify password before first login after migration')
.parse(process.argv);
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class Migrator {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
await this.openMongoClient();
await this.readCompanyFile();
await this.doMigrations();
}
catch (error) {
console.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
const logQueries = !!Commander.verbose;
this.mongoClient = new MongoClient({
collections: ['__all'],
dryRunMode: !!Commander.dryrun,
queryLogging: logQueries,
logger: logQueries ? console : undefined
});
try {
this.config = ApiConfig.getPreferredConfig();
await this.mongoClient.openMongoClient({ ...this.config.storage.mongo });
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
throw `unable to open mongo client: ${message}`;
}
}
// read company IDs from file, as needed
async readCompanyFile () {
if (this.companyfile) {
this.companyIds = FS.readFileSync(this.companyfile).toString().split("\n").filter(c => c).map(c => c.trim());
} else if (this.company) {
this.companyIds = [this.company];
}
}
// step through the companies that have not yet been migrated, and migrate them
async doMigrations () {
this.idp = new NewRelicIDP();
await this.idp.initialize(this.config);
/*
// do not allow us to run without setting IDP password when in production
if (this.config.sharedGeneral.isProductionCloud && !this.setIDPPassword) {
console.warn('Cannot run IDP migration in production without setidppwd flag set');
process.exit(1);
}
*/
this.migrationHandler = new NewRelicIDPMigrationHandler({
data: this.data,
logger: console,
dryRun: this.dryrun,
verbose: this.verbose,
throttle: this.throttle,
incremental: this.incremental,
//setIDPPassword: this.setIDPPassword,
//passwordPlaceholder: this.config.integrations.newRelicIdentity.passwordKey,
idp: this.idp
});
const query = this.companyIds ?
{
_id: this.data.companies.inQuerySafe(this.companyIds)
} : {
deactivated: false
};
if (!this.companyIds && this.nrConnectedOnly) {
query.nrOrgIds = { $exists: true };
}
if (!this.incremental) {
query.linkedNROrgId = { $exists: false };
}
const result = await this.data.companies.getByQuery(query, {
stream: true,
overrideHintRequired: true,
sort: { _id: 1 }
});
let company;
let totalUsersMigrated = 0;
let totalUsersExisting = 0;
let totalCompaniesMigrated = 0;
let totalCompaniesPartiallyMigrated = 0;
let totalErrors = 0;
do {
company = await result.next();
if (company) {
const info = await this.migrateCompany(company);
if (info.error) {
console.error(`****** COMPANY ${company.id} COULD NOT BE MIGRATED: ${info.error}`);
totalErrors++;
} else if (info.numUserErrors) {
console.error(`****** COMPANY ${company.id} HAD ${info.numUserErrors} THAT COULD NOT BE MIGRATED`);
totalCompaniesPartiallyMigrated++;
} else {
totalUsersMigrated += info.numUsersMigrated;
totalUsersExisting += info.numUsersExisting;
totalCompaniesMigrated++;
}
}
} while (company);
result.done();
const which = Commander.dryrun ? 'would have been' : 'were'
console.log(`${totalCompaniesMigrated} companies and ${totalUsersMigrated} users ${which} migrated, ${totalUsersExisting} users already existed`);
if (totalCompaniesPartiallyMigrated) {
console.log(`${totalCompaniesPartiallyMigrated} ${which} only partially migrated`);
}
}
// migrate a single company
async migrateCompany (company) {
return this.migrationHandler.migrateCompany(company);
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
let throttle = 100;
if (Commander.throttle) {
throttle = parseInt(Commander.throttle);
if (isNaN(throttle)) throw 'throttle not a number';
}
await new Migrator().go({
dryrun: !!Commander.dryrun,
throttle,
nrConnectedOnly: Commander.nr,
incremental: !!Commander.incremental,
//setIDPPassword: !!Commander.setidppwd,
verbose: !!Commander.verbose,
company: Commander.company,
companyfile: Commander.companyfile
});
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
console.error(message);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-migrate_to_one_user_per_org.js 0000775 0000000 0000000 00000007123 14607772627 0030301 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// migrate all companies that have not yet been migrated to one-user-per-org paradigm
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
const OneUserPerOrgMigrationHandler = require('../lib/util/one_user_per_org_migration_handler');
Commander
.option('--dryrun', 'Do a dry run, meaning don\'t actually write anything to our database, but report on numbers')
.option('--throttle ', 'Throttle processing each user by this amount of time')
.option('--verbose', 'Verbose logging output')
.parse(process.argv);
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class Migrator {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
await this.openMongoClient();
await this.doMigrations();
}
catch (error) {
console.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
const logQueries = !!Commander.verbose;
this.mongoClient = new MongoClient({
collections: ['__all'],
dryRunMode: !!Commander.dryrun,
queryLogging: logQueries,
logger: logQueries ? console : undefined
});
try {
await this.mongoClient.openMongoClient({ ...ApiConfig.getPreferredConfig().storage.mongo });
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
throw `unable to open mongo client: ${message}`;
}
}
// step through the companies that have not yet been migrated, and migrate them
async doMigrations () {
this.migrationHandler = new OneUserPerOrgMigrationHandler({
data: this.data,
logger: console,
dryRun: this.dryrun,
verbose: this.verbose,
throttle: this.throttle,
tokenSecret: this.tokenSecret
});
const result = await this.data.companies.getByQuery(
{
hasBeenMigratedToOneUserPerOrg: { $ne: true },
deactivated: false
},
{
stream: true,
overrideHintRequired: true,
sort: { _id: 1 }
}
);
let company;
let totalUsersMigrated = 0;
let totalCompaniesMigrated = 0;
let totalUserRecordsCreated = 0;
do {
company = await result.next();
if (company) {
const info = await this.migrateCompany(company);
totalUsersMigrated += info.numUsersMigrated;
totalUserRecordsCreated += info.numUserRecordsCreated;
totalCompaniesMigrated++;
}
} while (company);
result.done();
const which = Commander.dryrun ? 'would have been' : 'were'
console.log(`\n\n${totalCompaniesMigrated} companies and ${totalUsersMigrated} users ${which} migrated, ${totalUserRecordsCreated} new user records ${which} created`);
}
// migrate a single company
async migrateCompany (company) {
return this.migrationHandler.migrateCompany(company);
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
let throttle = 100;
if (Commander.throttle) {
throttle = parseInt(Commander.throttle);
if (isNaN(throttle)) throw 'throttle not a number';
}
await new Migrator().go({
dryrun: !!Commander.dryrun,
throttle,
verbose: !!Commander.verbose,
tokenSecret: ApiConfig.getPreferredConfig().sharedSecrets.auth
});
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
console.error(message);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-modules-report 0000775 0000000 0000000 00000001067 14607772627 0025102 0 ustar 00root root 0000000 0000000 #!/bin/bash
tmpfile=/tmp/module-check-$$
modhashfile=$CS_API_TOP/sandbox/node-modules-check.txt
cd $CS_API_TOP
time find -s node_modules -type f -exec shasum {} \; | tee -a $tmpfile
if [ -f $modhashfile ]; then
x=`diff $modhashfile $tmpfile|wc -l`
if [ $x -eq 0 ]; then
echo "node_modules integrety confirmed against $modhashfile"
/bin/rm $tmpfile
else
echo "node_modules integrity against $modhashfile failed. There are $x differences"
echo
echo "run: dif $tmpfile $modhashfile"
fi
else
mv $tmpfile $modhashfile
echo
echo "Saved $modhashfile"
fi
codestream-server-api-15.4.0/api_server/bin/cs_api-mongo-config-tool 0000775 0000000 0000000 00000000423 14607772627 0025451 0 ustar 00root root 0000000 0000000 #!/usr/bin/env bash
# desc# command line utility for working with the mongo config
[ -z "$CSSVC_BACKEND_ROOT" ] && echo "CSSVC_BACKEND_ROOT not defined. Your environment is not setup" && exit 1
$CSSVC_BACKEND_ROOT/shared/codestream_configs/bin/mongo-config-loader.js "$@"
codestream-server-api-15.4.0/api_server/bin/cs_api-move_to_mono_repo.js 0000775 0000000 0000000 00000030206 14607772627 0026256 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// merge a repo's remotes to another, delete the merged repo, and adjust all codemarks, file streams, and markers accordingly
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CS_API_TOP + '/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CS_API_TOP + '/config/config');
const Commander = require('commander');
const FS = require('fs');
// need these collections from mongo
const COLLECTIONS = ['repos', 'markers', 'streams', 'users', 'reviews'];
Commander
.option('-f, --fromRepo ', 'ID of the repo to merge and delete')
.option('-t, --toRepo ', 'ID of the repo to merge to, if not provided a new repo will be created')
.option('-m, --mappings ', 'Mappings file which maps commit SHAs from the repo being merged')
.option('-p, --pathToPrepend ', 'Append this path part to the path for files from the repo being merged')
.option('--dryrun', 'Do a dry run with informational messages, but don\'t actually DO anything')
.option('--verbose', 'Output verbose logging messages (mainly having to do with reviews')
.option('--throttle ', 'Time to wait for each operation, to avoid mongo overload')
.parse(process.argv);
if (!Commander.fromRepo || !Commander.toRepo) {
Commander.help();
}
class RepoMerger {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.config = ApiConfig.getPreferredConfig();
await this.openMongoClient();
await this.process();
this.log('DONE');
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
console.error(`Merge failed: ${message}`);
if (error instanceof Error) {
console.error(error.stack);
}
}
process.exit();
}
wait (n) {
return new Promise(resolve => {
setTimeout(resolve, n);
});
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(this.config.storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async process () {
await this.readMappings();
await this.getRepos();
await this.copyCommitHashes();
await this.convertFileStreams();
await this.convertMarkers();
await this.convertReviews();
}
async readMappings () {
if (!this.mappingsFile) {
this.mappings = {};
return;
}
this.log('Reading mappings file...');
return new Promise((resolve, reject) => {
let data;
try {
data = FS.readFileSync(this.mappingsFile).toString();
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
return reject(`Unable to read mappings file: ${message}`);
}
try {
data = data.split('\n');
this.mappings = {};
data.forEach(line => {
line = line.trim();
if (!line || line.startsWith('old')) return;
const map = line.split(' ');
this.mappings[map[0]] = map[1];
});
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
throw 'Unable to parse mappings json data: ' + message;
}
resolve();
});
}
async getRepos () {
this.log('Finding repos...');
this.fromRepo = await this.data.repos.getById(this.fromRepoId);
if (!this.fromRepo) {
throw `Repo ${Commander.fromRepo} not found`;
}
this.toRepo = await this.data.repos.getById(this.toRepoId);
if (!this.toRepo) {
throw `Repo ${Commander.toRepo} not found`;
}
if (this.fromRepo.teamId !== this.toRepo.teamId) {
throw 'Repos must be from the same team';
}
}
async copyCommitHashes () {
this.log('Copying commit hashes...');
const toRepoHashes = this.toRepo.knownCommitHashes || [];
(this.fromRepo.knownCommitHashes || []).forEach(fromHash => {
const toHash = this.mappings[fromHash] || fromHash;
toRepoHashes.push(toHash);
});
try {
if (this.dryrun) {
const numHashes = (this.fromRepo.knownCommitHashes || []).length;
return this.log(` (Would have copied ${numHashes} known commit hashes)`);
}
await this.data.repos.updateDirect(
{ id: this.data.repos.objectIdSafe(this.toRepo.id) },
{ $set: { knownCommitHashes: toRepoHashes } }
);
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
throw `Unable to copy commit hashes: ${message}`;
}
}
async convertFileStreams () {
this.log('Converting file streams...');
const fileStreams = await this.data.streams.getByQuery(
{
teamId: this.fromRepo.teamId,
repoId: this.fromRepo.id
},
{
overrideHintRequired: true
}
);
for (const stream of fileStreams) {
await this.convertFileStream(stream);
}
if (this.dryrun) {
this.log(` (Would have converted ${fileStreams.length} file streams)`);
}
else {
this.log(`Converted ${fileStreams.length} file streams`);
}
}
async convertFileStream (stream) {
this.log(`\tStream ${stream.id}...`);
const set = {
repoId: this.toRepo.id,
file: `${this.pathToPrepend}/${stream.file}`
};
if (!this.dryrun) {
await this.data.streams.updateDirect(
{ id: this.data.streams.objectIdSafe(stream.id) },
{ $set: set }
);
await this.wait(this.throttle);
}
}
async convertMarkers () {
this.log('Converting markers...');
const markers = await this.data.markers.getByQuery(
{
teamId: this.fromRepo.teamId,
repoId: this.fromRepo.id
},
{
overrideHintRequired: true
}
);
for (const marker of markers) {
await this.convertMarker(marker);
}
if (this.dryrun) {
this.log(` (Would have converted ${markers.length} markers)`);
}
else {
this.log(`Converted ${markers.length} markers`);
}
}
async convertMarker (marker) {
this.log(`\tMarker ${marker.id}`);
const set = {
repoId: this.toRepo.id
};
if (marker.file) {
set.file = `${this.pathToPrepend}/${marker.file}`;
}
if (!marker.referenceLocations && marker.commitHashWhenCreated && marker.locationWhenCreated) {
set.referenceLocations = [{
commitHash: this.mappings[marker.commitHashWhenCreated] || marker.commitHashWhenCreated,
location: marker.locationWhenCreated,
flags: {
convertedForMonoRepo: true
}
}];
}
else {
set.referenceLocations = [];
marker.referenceLocations.forEach(rl => {
rl.commitHash = this.mappings[rl.commitHash] || rl.commitHash;
set.referenceLocations.push(rl);
});
}
if (!this.dryrun) {
await this.data.markers.updateDirect(
{ id: this.data.markers.objectIdSafe(marker.id) },
{ $set: set }
);
await this.wait(this.throttle);
}
}
async convertReviews () {
this.log('Converting reviews...');
let reviews = await this.data.reviews.getByQuery(
{
teamId: this.fromRepo.teamId
},
{
overrideHintRequired: true
}
);
reviews = reviews.filter(review => {
return (review.reviewChangesets || []).find(changeset => changeset.repoId === this.fromRepo.id);
});
for (const review of reviews) {
await this.convertReview(review);
}
if (this.dryrun) {
this.log(` (Would have converted ${reviews.length} reviews)`);
}
else {
this.log(`Converted ${reviews.length} reviews`);
}
}
async convertReview (review) {
this.log(`\tReview ${review.id}...`);
this.convertReviewChangesets(review);
this.convertReviewDiffs(review.reviewDiffs);
this.convertCheckpointReviewDiffs(review.checkpointReviewDiffs);
const set = {
reviewChangesets: review.reviewChangesets,
reviewDiffs: review.reviewDiffs,
checkpointReviewDiffs: review.checkpointReviewDiffs
};
if (!this.dryrun) {
await this.data.reviews.updateDirect(
{ id: this.data.reviews.objectIdSafe(review.id) },
{ $set: set }
);
await this.wait(this.throttle);
}
}
convertReviewChangesets (review) {
this.debug(`\tConverting ${review.reviewChangesets.length} change sets...`);
let n = 0;
review.reviewChangesets.forEach(changeset => {
n++;
this.debug(`\t\tChangeset ${n}`);
this.debug(`\t\t\tSetting the repo ID to ${this.toRepo.id}`);
changeset.repoId = this.toRepo.id;
this.convertChangesetCommits(changeset);
if (changeset.modifiedFiles) {
this.convertChangesetModifiedFiles(changeset.modifiedFiles);
}
if (changeset.modifiedFilesInCheckpoint) {
this.convertChangesetModifiedFiles(changeset.modifiedFilesInCheckpoint);
}
});
}
convertChangesetCommits (changeset) {
this.debug(`\t\t\tConverting ${(changeset.commits || []).length} commits...`);
(changeset.commits || []).forEach(commit => {
if (commit.info && commit.info.ref) {
const oldRef = commit.info.ref;
commit.info.ref = this.mappings[commit.info.ref] || commit.info.ref;
this.debug(`\t\t\t\tref ${oldRef} to ${commit.info.ref}`);
}
if (commit.sha) {
const oldSha = commit.sha;
commit.sha = this.mappings[commit.sha] || commit.sha;
this.debug(`\t\t\t\tsha ${oldSha} to ${commit.sha}`);
}
});
}
convertChangesetModifiedFiles (modifiedFiles) {
this.debug(`\t\t\tConverting ${(modifiedFiles || []).length} modified files...`);
(modifiedFiles || []).forEach(file => {
if (file.file) {
const oldFile = file.file;
file.file = `${this.pathToPrepend}/${file.file}`;
this.debug(`\t\t\t\tfile ${oldFile} to ${file.file}`);
}
if (file.oldFile) {
const oldOldFile = file.oldFile;
file.oldFile = `${this.pathToPrepend}/${file.oldFile}`;
this.debug(`\t\t\t\toldFile ${oldOldFile} to ${file.oldFile}`);
}
});
}
convertReviewDiffs (reviewDiffs) {
reviewDiffs[this.toRepo.id] = reviewDiffs[this.fromRepo.id];
delete reviewDiffs[this.fromRepo.id];
this.debug(`\treivewDiffs repo ID ${this.fromRepo.id} became ${this.toRepo.id}}`);
const reviewDiff = reviewDiffs[this.toRepo.id];
this.debug('\tConverting review diff...');
this.convertReviewDiff(reviewDiff);
}
convertCheckpointReviewDiffs (checkpointReviewDiffs) {
this.debug(`\tConverting ${(checkpointReviewDiffs || []).length} checkpoint review diffs...`);
(checkpointReviewDiffs || []).forEach(reviewDiff => {
if (reviewDiff.repoId === this.fromRepo.id) {
reviewDiff.repoId = this.toRepo.id;
this.debug(`\t\tCheckpoint reviewDiff repo ID became ${reviewDiff.repoId}`);
}
this.convertReviewDiff(reviewDiff.diffs);
});
}
convertReviewDiff (reviewDiff) {
[
'leftBaseSha',
'rightBaseSha',
'latestCommitSha'
].forEach(shaField => {
if (reviewDiff[shaField]) {
const oldField = reviewDiff[shaField];
reviewDiff[shaField] = this.mappings[reviewDiff[shaField]] || reviewDiff[shaField];
this.debug(`\t\t${shaField} from ${oldField} to ${reviewDiff[shaField]}`);
}
});
[
'leftDiffs',
'rightDiffs',
'rightReverseDiffs',
'rightToLatestCommitDiffs',
'latestCommitToRightDiffs'
].forEach(diffField => {
if (reviewDiff[diffField]) {
this.debug(`\t\tConverting diffs for ${diffField}`);
this.convertDiffs(reviewDiff[diffField]);
}
});
delete reviewDiff[this.fromRepo.id];
}
convertDiffs (diffs) {
this.debug(`\t\t\tConverting ${diffs.length} diffs...`);
diffs.forEach(diff => this.convertDiff(diff));
}
convertDiff (diff) {
if (diff.oldFileName) {
const oldFileName = diff.oldFileName;
diff.oldFileName = `${this.pathToPrepend}/${diff.oldFileName}`;
this.debug(`\t\t\t\toldFileName from ${oldFileName} to ${diff.oldFileName}`);
}
if (diff.newFileName) {
const newFileName = diff.newFileName;
diff.newFileName = `${this.pathToPrepend}/${diff.newFileName}`;
this.debug(`\t\t\t\tnewFileName from ${newFileName} to ${diff.newFileName}`);
}
}
log (msg) {
console.log(msg);
}
debug (msg) {
if (this.verbose) {
console.log(msg);
}
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new RepoMerger().go({
fromRepoId: Commander.fromRepo,
toRepoId: Commander.toRepo,
mappingsFile: Commander.mappings,
remote: Commander.remote,
pathToPrepend: Commander.pathToPrepend,
dryrun: Commander.dryrun,
verbose: Commander.verbose,
throttle: Commander.throttle ? parseInt(Commander.throttle, 10) : 0
});
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-msteams_bot_app_pkg_creator 0000775 0000000 0000000 00000013004 14607772627 0027650 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python3
# desc# create the app packages for integrating codestream to msteams
"""Create MS Team connector app packages for CS <-> MS Teams integration
SYNOPSIS
Builds a zip file from a manifest.
"""
optionsDoc = """ACTIONS
OPTIONS
"""
epilogDoc = """EXAMPLES
Examples are needed.
"""
import os
from posixpath import dirname
import shutil
import glob
import re
import json
import devtools.core as dtCore
import devtools.resources.resources as dtResources
import devtools.templates.template as dtTemplate
# sys.path.append(os.environ['DT_TOP'] + "/lib")
# import sysUtils as su
# import awsCS
# create the app package (zip file) for the specified env & botId
# and optionally distribute it
def makeAppPackage(globalData, args, manifestTemplate):
outDir = f"{globalData['distRoot']}/{args.env}"
if not os.path.exists(outDir):
os.mkdir(outDir)
# create the manifest
envString = "" if args.env == "prod" else f"-{args.env}"
# manifest = su.expandTemplate(manifestTemplate, {'env': envString, 'botId': args.botId})
tmpl = dtTemplate.template(template=manifestTemplate)
manifest = tmpl.expand({'env': envString, 'botId': args.botId})
manifestData = json.loads(manifest)
outFile = open(f"{outDir}/manifest.json", "w")
outFile.write(manifest)
outFile.close()
# copy additional resource files
for f in ['color.png', 'outline.png']:
shutil.copyfile(f"{globalData['templateDir']}/{f}", f"{outDir}/{f}")
# create the zip file named based on env and version
zipName = f"codestream-msteamsbot{envString}-{manifestData['version']}.zip"
dirStack = dtCore.util.pushDir(outDir)
dtCore.exec.execCommand(f"zip -q {outDir}/../{zipName} *", abortOnFail=True, verbose=args.verbose)
dtCore.util.popDir(dirStack)
dtCore.exec.execCommand(f"find {outDir} -type f")
dtCore.exec.execCommand(f"ls -l {dirname(outDir)}/{zipName}")
# distribute the zip file via S3
if args.distribute:
cmd = f"aws s3 cp {globalData['distRoot']}/{zipName} {globalData['distributionKeyPrefix']}/{zipName} --sse"
dtCore.exec.execCommand(cmd, abortOnFail=True, verbose=args.verbose)
# create the package contents from a key management database config file
def makeAppPackageFromEnvFile(globalData, manifestTemplate, envFile):
rc = 0
match = re.search(r'.*\/teams-(.+).cfg$', envFile)
if match:
env = "pd" if match.group(1) == "dev" else match.group(1)
# envData = su.readKeyValueFile(envFile)
envData = dtCore.io.readKVFileToDict(envFile)
# su.dumpDict(envData)
if 'botAppId' in envData:
makeAppPackage(globalData, env, envData['botAppId'], manifestTemplate)
else:
print(f"botAppId not defined for {env} environment")
else:
dtCore.io.printErr(f"could not determine env from {envFile}")
rc = 1
return rc
globalData = {
# 'distributionKeyPrefix': f"{awsCS.awsCS().getCloudFrontProperty('assets', 's3BucketPrefix')}/mstbot",
# 'distributionKeyPrefix': f"{rm.cloudFrontDistribution('assets', 's3BucketPrefix')}/mstbot",
'botDir': f"{os.environ['CS_API_TOP']}/etc/msteamsbot",
'templateDir': f"{os.environ['CS_API_TOP']}/etc/msteamsbot/template",
'distRoot': f"{os.environ['CS_API_TOP']}/etc/msteamsbot/dist",
}
defaultManifestTemplate = f"{globalData['templateDir']}/manifest.json"
arguments = dtCore.options.arguments(__doc__, epilogDoc, optionsDoc, modules=[
dtCore.logger.moduleOptions,
dtCore.io.moduleOptions,
dtResources.moduleOptions,
])
# arguments.parser.add_argument('--debug', action='store_true', default=False, help='show debugging output')
# arguments.parser.add_argument('--verbose', action='store_true', default=False, help='show verbose output')
arguments.parser.add_argument('-e', '--env', type=str, help="environment")
arguments.parser.add_argument('-b', '--botId', type=str, help="bot registration app ID")
arguments.parser.add_argument('--use-keydb', action='store_true', default=False, help="get app IDs from key management DB")
arguments.parser.add_argument('--distribute', action='store_true', default=False, help="copy zips to S3 using the AWS CLI")
arguments.parser.add_argument('--manifest-template', type=str, default=defaultManifestTemplate, help=f"for manifest creation (def = {defaultManifestTemplate})")
args = arguments.parseArgs()
options = arguments.options
# logger = dtCore.logger.logger(options=options)
# options.pack('logger', logger)
rm = dtResources.manager(options=options)
options.pack('rm', rm)
if args.distribute:
globalData['distributionKeyPrefix'] = f"{rm.cloudFrontDistribution('assets', 's3BucketPrefix')}/mstbot"
if args.env == "pd" and args.use_keydb:
args.env = "dev"
print("key-db secrets for 'pd' are stored as 'dev'. Resetting environment to 'dev'")
ec = 0
if args.debug:
dtCore.io.dumpData(globalData, msg="globalData:")
if args.verbose:
print(f"making app package using {args.manifest_template}")
manifestTemplate = dtCore.io.readFileIgnoreComments(args.manifest_template)
if not os.path.exists(globalData['distRoot']):
print(f"creating {globalData['distRoot']}")
os.mkdir(globalData['distRoot'])
if args.env and args.botId:
makeAppPackage(globalData, args, manifestTemplate)
elif args.use_keydb:
if not os.environ['KM_PKI']:
dtCore.io.printErr("where is the key management database (KM_PKI is not defined)?")
exit(1)
# read environments & botIds from the secrets database
globString = f"teams-{args.env}.cfg" if args.env else "teams-*.cfg"
if args.debug:
print(f"globString = {globString}")
for envFile in glob.glob(f"{os.environ['KM_PKI']}/secrets/microsoft/{globString}"):
ec += makeAppPackageFromEnvFile(globalData, manifestTemplate, envFile)
else:
arguments.parser.print_help()
exit(1)
exit(ec)
codestream-server-api-15.4.0/api_server/bin/cs_api-msteams_to_codestream.js 0000775 0000000 0000000 00000031020 14607772627 0027105 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
const CodemarkIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/codemarks/indexes');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
const StreamIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/streams/indexes');
const PubNub = require('pubnub');
const PubNubClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/pubnub/pubnub_client_async');
const UUID = require('uuid').v4;
const OS = require('os');
Commander
.option('-t, --teamId ', 'Team ID to convert from MSTeams, specify "all" for all MSTeams teams (be careful!)')
.option('--dryrun', 'Do a dry run, meaning don\'t actually write anything to our database, but report on numbers')
.option('--throttle ', 'Pause this number of milliseconds between teams')
.option('--verbose', 'Verbose output')
.option('--debug', 'Debug output')
.option('--dieonwarn ', 'Die on a warning', parseInt)
.parse(process.argv);
const COLLECTIONS = ['teams', 'posts', 'codemarks', 'users', 'streams', 'teams', 'markers'];
const DEFAULT_THROTTLE_TIME = 1000;
const ThrottleTime = Commander.throttle ? parseInt(Commander.throttle) : DEFAULT_THROTTLE_TIME;
var Logger = console;
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class TeamCodemarkConverter {
constructor (options) {
Object.assign(this, options);
this.streamSeqNums = {};
this.streams = {};
this.warnCount = 0;
this.totalCodemarks = 0;
this.numSkipped = 0;
this.codemarksConverted = 0;
}
// fetch all the replies to codemarks for a team and turn them into posts
async convertForTeam (team) {
this.team = team;
await this.getTeamUsers();
await this.setUserFlags();
await this.ensureTeamStream();
await this.processCodemarks();
await this.updateTeam();
await this.clearUserFlags();
this.log(`\tCODEMARKS: ${this.totalCodemarks}`);
this.log(`\tSKIPPED: ${this.numSkipped}`);
this.log(`\tCONVERTED: ${this.codemarksConverted}`);
this.log(`\tWARNINGS: ${this.warnCount}`);
this.log('==============================================================');
}
// get all the users on the given team
async getTeamUsers () {
this.debug('\tGetting users on team...');
const users = await this.data.users.getByQuery(
{
teamIds: this.team.id
},
{
hint: UserIndexes.byTeamIds
}
);
this.users = users.reduce((usersHash, user) => {
usersHash[user.id] = user;
return usersHash;
}, {});
this.debug(`\t${Object.keys(this.users).length} users found on team`);
}
// set maintenance mode, must set password, and clear provider info flags for all users on the team
async setUserFlags () {
if (Commander.dryrun) {
this.log('\tWould have set user flags');
return;
}
else {
this.log('\tSetting user flags...');
}
const op = {
$set: {
inMaintenanceMode: true,
mustSetPassword: true,
clearProviderInfo: true
}
};
await this.data.users.updateDirect({ teamIds: this.team.id }, op);
await this.sendOpsToUsers(op);
}
// send pubnub message for all users on team
async sendOpsToUsers (op) {
const requestId = UUID();
this.log(`\tSending Pubnub op to ${Object.keys(this.users).length} users, reqid=${requestId}...`);
await Promise.all(Object.keys(this.users).map(async userId => {
await this.sendUserOp(userId, op, requestId);
}));
}
// send pubnub message for user
async sendUserOp (userId, op, requestId) {
// send pubnub update on user's me-channel
const message = {
user: Object.assign(op, { id: userId }),
requestId
};
const channel = `user-${userId}`;
try {
await this.pubnub.publish(
message,
channel
);
}
catch (error) {
// this doesn't break the chain, but it is unfortunate
this.warn(`WARNING: Unable to publish user op to channel ${channel}: ${JSON.stringify(error)}`);
}
}
// make sure we have a team stream for the given team
async ensureTeamStream () {
this.debug('\tLooking for team stream...');
const teamStreams = await this.data.streams.getByQuery(
{
teamId: this.team.id,
isTeamStream: true
},
{
hint: StreamIndexes.byIsTeamStream
}
);
if (teamStreams.length > 0) {
this.teamStream = teamStreams[0];
this.debug(`\tFound team stream ${this.teamStream.id}`);
return;
}
this.debug('\tCreating a team stream...');
const id = this.data.streams.createId().toString();
const stream = {
id,
version: 1,
deactivated: false,
createdAt: this.team.createdAt,
modifiedAt: this.team.createdAt,
teamId: this.team.id,
type: 'channel',
name: 'general',
isTeamStream: true,
privacy: 'public',
creatorId: this.team.creatorId,
sortId: id,
nextSeqNum: 1
};
if (Commander.dryrun) {
this.log(`\tWould have created team stream for team ${this.team.id}`);
this.teamStream = stream;
}
else {
this.log(`\tCreating team stream for team ${this.team.id}...`);
this.teamStream = await this.data.streams.create(stream);
}
this.verbose(stream);
}
// process all the codemarks for the team
async processCodemarks () {
this.debug('\tProcessing codemarks...');
let codemark;
do {
let n = 0;
const query = {
teamId: this.team.id,
deactivated: false
};
if (codemark) {
query.createdAt = { $gt: codemark.createdAt };
}
this.debug(`\tInitiating new query: ${JSON.stringify(query)}...`);
const result = await this.data.codemarks.getByQuery(
query,
{
stream: true,
hint: CodemarkIndexes.byTeamId,
sort: { createdAt: 1 }
}
);
do {
codemark = await result.next();
if (codemark) {
try {
await this.processCodemark(codemark);
}
catch (error) {
this.warn(`WARNING: Error thrown processing codemark ${codemark.id}, ignoring: ${error}`);
}
await Wait(ThrottleTime);
n++;
}
} while (codemark && n < 500);
result.done();
} while (codemark);
}
// process a single codemark for a team
async processCodemark (codemark) {
this.totalCodemarks++;
this.log(`\tProcessing codemark #${this.totalCodemarks}: ${codemark.id}...`);
// link-type codemarks are not posted on MSTeams
if (codemark.type === 'link') {
this.log(`\t\tSkipping link-type codemark ${codemark.id}`);
this.numSkipped++;
return;
}
// create a post pointing to this codemark on CodeStream
const postId = this.data.posts.createId().toString();
const post = await this.createCodemarkPost(codemark, postId);
// update the codemark and markers to actually point to the post
await this.updateCodemark(codemark, post);
await this.updateMarkers(codemark, post);
this.codemarksConverted++;
}
// create a post pointing to a given codemark, the post will go into the team stream
async createCodemarkPost (codemark, postId) {
const seqNum = this.bumpSeqNum(this.teamStream);
const postData = {
id: postId,
version: 1,
deactivated: false,
createdAt: codemark.createdAt,
modifiedAt: codemark.createdAt,
creatorId: codemark.creatorId,
codemarkId: codemark.id,
teamId: codemark.teamId,
streamId: this.teamStream.id,
seqNum
};
let post;
if (Commander.dryrun) {
this.log('\t\tWould have created post for codemark');
post = postData;
}
else {
this.log('\t\tCreating post for codemark...');
post = await this.data.posts.create(postData);
}
this.verbose(postData);
return post;
}
// each post we put in a stream has to have an ever-increasing sequence number, we'll keep
// track of that here
bumpSeqNum (stream) {
this.streamSeqNums[stream.id] = this.streamSeqNums[stream.id] || 0;
return ++this.streamSeqNums[stream.id];
}
// update the codemark to point to the created post
async updateCodemark (codemark, post) {
const op = {
$unset: {
providerType: true
},
$set: {
postId: post.id,
streamId: post.streamId
}
};
if (Commander.dryrun) {
this.log('\t\tWould have updated codemark');
}
else {
await this.data.codemarks.applyOpById(codemark.id, op);
this.log('\t\tUpdating codemark...');
}
this.verbose(op);
Object.assign(codemark, op.$set);
delete codemark.providerType;
}
// update markers associated with the codemark to point to the codemark's post and stream
async updateMarkers (codemark, post) {
const query = {
teamId: this.team.id,
codemarkId: codemark.id
};
const op = {
$set: {
postStreamId: post.streamId,
postId: post.id
}
};
if (Commander.dryrun) {
this.log('\t\tWould have updated markers');
}
else {
this.log('\t\tUpdating markers...');
await this.data.markers.updateDirect(query, op);
}
this.verbose(query);
this.verbose(op);
}
// update the team to no longer be a MSTeams team
async updateTeam () {
const providerIdentities = this.team.providerIdentities.filter(pi => !pi.startsWith('msteams'));
const op = {
$unset: {
'providerInfo.msteams': true
},
$set: {
providerIdentities
}
};
if (Commander.dryrun) {
this.log('\tWould have updated team');
}
else {
await this.data.teams.applyOpById(this.team.id, op);
this.log(`\tUpdating team ${this.team.id}...`);
}
this.verbose(op);
Object.assign(this.team, op.$set);
delete this.team.providerIdentities;
}
// clear maintenance mode flags for all users on the team
async clearUserFlags () {
if (Commander.dryrun) {
this.log('\tWould have cleared user maintenance mode flags');
return;
}
else {
this.log('\tClearing user maintenance mode flags...');
}
const op = {
$unset: {
inMaintenanceMode: true
}
};
await this.data.users.updateDirect({ teamIds: this.team.id }, op);
await this.sendOpsToUsers(op);
}
log (msg) {
Logger.log(msg);
}
warn (msg) {
Logger.warn(msg);
this.warnCount++;
if (Commander.dieonwarn && this.warnCount === Commander.dieonwarn) {
process.exit();
}
}
debug (msg) {
Commander.debug && Logger.log(msg);
}
verbose (obj) {
Commander.verbose && Logger.log(JSON.stringify(obj, undefined, 10));
}
}
class MSTeamsConverter {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
await this.openMongoClient();
await this.openPubnubClient();
if (this.teamId === 'all') {
await this.processAllTeams();
}
else {
await this.processSingleTeam();
}
}
catch (error) {
Logger.error(error);
process.exit();
}
process.exit();
}
// process all MSTeams teams in our database! (this could take some time...)
async processAllTeams () {
const result = await this.data.teams.getByQuery(
{
'providerIdentities': /msteams::/,
deactivated: false
},
{
stream: true,
overrideHintRequired: true,
sort: { _id: -1 }
}
);
let team;
do {
team = await result.next();
if (team) {
await this.processTeam(team);
await Wait(ThrottleTime);
}
} while (team);
result.done();
}
// process a single MSTeams team, as given by team ID
async processSingleTeam () {
const team = await this.data.teams.getById(this.teamId);
if (!team) {
throw 'team not found: ' + this.teamId;
}
return this.processTeam(team);
}
// process a single MSTeams team
async processTeam (team) {
Logger.log(`Processing team ${team.id}...`);
await new TeamCodemarkConverter({ data: this.data, pubnub: this.pubnubClient }).convertForTeam(team);
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// open a Pubnub client for broadcasting the changes
async openPubnubClient () {
this.pubnub = new PubNub(Object.assign({}, ApiConfig.getPreferredConfig().broadcastEngine.pubnub, { uuid: 'API-' + OS.hostname() }));
this.pubnubClient = new PubNubClient({ pubnub: this.pubnub });
await this.pubnubClient.init();
}
}
(async function() {
try {
const teamId = Commander.teamId;
if (!teamId) {
throw 'must provide teamId or all';
}
await ApiConfig.loadPreferredConfig();
await new MSTeamsConverter().go({ teamId });
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-nullify_slack_tokens.js 0000775 0000000 0000000 00000010243 14607772627 0026752 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
const PubNub = require('pubnub');
const PubNubClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/pubnub/pubnub_client_async');
const UUID = require('uuid').v4;
const OS = require('os');
Commander
.option('-u, --userId ', 'Team ID to fetch Slack replies for, specify "all" for all Slack teams (be careful!)')
.option('--throttle ', 'Pause this number of milliseconds between users')
.option('--dryrun', 'Do a dry run, meaning don\'t actually write the replies to our database, but report on numbers')
.parse(process.argv);
const COLLECTIONS = ['users'];
const DEFAULT_THROTTLE_TIME = 1000;
const Logger = console;
const ThrottleTime = Commander.throttle ? parseInt(Commander.throttle) : DEFAULT_THROTTLE_TIME;
// wait this number of milliseconds
const Wait = function (time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class Nullifier {
// main entry point
async go(options = {}) {
try {
Object.assign(this, options);
await this.openMongoClient();
await this.openPubnubClient();
if (this.userId) {
await this.processSingleUser();
}
else {
await this.processAllUsers();
}
}
catch (error) {
Logger.error(error);
process.exit();
}
process.exit();
}
async processSingleUser() {
const user = await this.data.users.getById(this.userId);
if (!user) {
throw 'user not found: ' + this.userId;
}
return this.processUser(user);
}
async processAllUsers () {
const result = await this.data.users.getByQuery(
{
'providerInfo': { $exists: true },
deactivated: false
},
{
stream: true,
overrideHintRequired: true,
sort: { _id: -1 }
}
);
let user;
do {
user = await result.next();
if (user) {
const processed = await this.processUser(user);
await Wait(processed ? ThrottleTime : 0);
}
} while (user);
result.done();
}
async processUser (user) {
let op = {
$unset: {
},
$set: {
modifiedAt: Date.now(),
version: user.version + 1
}
};
Object.keys(user.providerInfo).forEach(teamId => {
if (teamId === 'slack') {
op.$unset['providerInfo.slack'] = true;
}
if (user.providerInfo[teamId].slack) {
op.$unset[`providerInfo.${teamId}.slack`] = true;
}
});
if (Object.keys(op.$unset).length === 0) {
console.log(`User ${user.id} has provider info but no slack token`);
return false;
}
if (Commander.dryrun) {
console.log(`Would have updated user ${user.id}: ${JSON.stringify(op, undefined, 5)}`);
return;
}
else {
console.log(`Updating user ${user.id}...`);
}
await this.updateUser(user, op);
await this.sendOpToUser(user, op);
}
async updateUser (user, op) {
await this.data.users.updateDirect({ _id: this.data.users.objectIdSafe(user.id) }, op);
}
async sendOpToUser (user, op) {
const requestId = UUID();
const message = {
user: Object.assign(op, {
id: user.id,
$version: {
before: user.version,
after: user.version + 1
}
}),
requestId
};
const channel = `user-${user.id}`;
await this.pubnubClient.publish(
message,
channel
);
}
async openMongoClient() {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async openPubnubClient() {
this.pubnub = new PubNub(Object.assign({}, ApiConfig.getPreferredConfig().broadcastEngine.pubnub, { uuid: 'API-' + OS.hostname() }));
this.pubnubClient = new PubNubClient({ pubnub: this.pubnub });
await this.pubnubClient.init();
}
}
(async function () {
try {
await ApiConfig.loadPreferredConfig();
await new Nullifier().go({ userId: Commander.userId });
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-one-to-one_nr_orgs.js 0000775 0000000 0000000 00000017351 14607772627 0026250 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// ensure that each NR org is associated with at most one CS org
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
const CompanyIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/companies/indexes');
const PostIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/posts/indexes');
Commander
.option('--dryrun, --dryrun', 'Do a dry run, don\'t actually change any data')
.option('--verbose, --verbose', 'Print extra log messages')
.option('--throttle, --throttle ', 'Throttle write operations by this time interval')
.parse(process.argv);
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class UniquifyNRCSOrgAssociations {
constructor (options) {
Object.assign(this, options);
this.throttle = this.throttle || 1000;
}
async go () {
await this.openMongoClient();
await this.uniquifyNROrgIds();
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async uniquifyNROrgIds () {
const result = await this.data.companies.getByQuery(
{
deactivated: false,
isNRConnected: true
},
{
stream: true,
overrideHintRequired: true
}
);
let company;
const orgIds = new Set();
do {
company = await result.next();
if (company && company.nrOrgIds && company.nrOrgIds.length > 0) {
const orgId = company.nrOrgIds[0];
if (orgIds.has(orgId)) continue;
orgIds.add(orgId);
const result = await this.data.companies.getByQuery(
{
nrOrgIds: [orgId]
},
{
hint: CompanyIndexes.byNROrgId
}
);
const companies = result.filter(_ => !_.deactivated);
if (companies.length > 1) {
if (this.verbose) {
console.log(`Found ${companies.length} companies with nrOrgId ${orgId}`);
}
await this.processCompanies(companies);
}
}
} while (company);
}
async processCompanies (companies) {
const companiesToClear = [];
const [popularCompaniesByUser, companiesToClearByUser] = await this.getCompaniesByUser(companies);
if (this.verbose) {
console.log(`${popularCompaniesByUser.length} popular companies by user`);
console.log(`${companiesToClearByUser.length} unpopular companies by user`);
}
for (const c of companiesToClearByUser) {
console.log(`- ${c.name}`);
}
companiesToClear.push(...companiesToClearByUser);
if (popularCompaniesByUser.length > 1) {
const [popularCompaniesByPost, companiesToClearByPost] = await this.getCompaniesByPost(popularCompaniesByUser);
if (this.verbose) {
console.log(`${popularCompaniesByPost.length} popular companies by post`);
console.log(`${companiesToClearByPost.length} unpopular companies by post`);
}
for (const c of companiesToClearByPost) {
console.log(`- ${c.name}`);
}
companiesToClear.push(...companiesToClearByPost);
if (popularCompaniesByPost.length > 1) {
const [popularCompaniesByDate, companiesToClearByDate] = await this.getCompaniesByRecentLogin(popularCompaniesByPost);
if (this.verbose) {
console.log(`${popularCompaniesByDate.length} companies with recent logins`);
console.log(`${companiesToClearByDate.length} companies with older logins`);
}
for (const c of companiesToClearByDate) {
console.log(`- ${c.name}`);
}
companiesToClear.push(...companiesToClearByDate);
if (popularCompaniesByDate.length > 1) {
popularCompaniesByDate.sort((a, b) => a.createdAt - b.createdAt);
if (this.verbose) {
console.log(`${popularCompaniesByDate.slice(1).length} leftover companies`);
}
for (const c of popularCompaniesByDate.slice(1)) {
console.log(`- ${c.name}`);
}
companiesToClear.push(...popularCompaniesByDate.slice(1));
}
}
}
await this.clearCompanies(companiesToClear);
}
async getCompaniesByUser (companies) {
const companyUserCount = [];
for (const c of companies) {
companyUserCount.push({
company: c,
count: await this.getUserCount(c)
});
}
return this.filterCompaniesByCount(companyUserCount);
}
async getCompaniesByPost (companies) {
const companyPostCount = [];
for (const c of companies) {
companyPostCount.push({
company: c,
count: await this.getPostCount(c)
});
}
return this.filterCompaniesByCount(companyPostCount);
}
async getCompaniesByRecentLogin (companies) {
const companyLoginDates = [];
for (const c of companies) {
companyLoginDates.push({
company: c,
count: await this.getMostRecentLogin(c)
});
}
return this.filterCompaniesByCount(companyLoginDates);
}
filterCompaniesByCount (companyData) {
const sorted = companyData.sort((a, b) => b.count - a.count);
const popularCompanies = [];
const companiesToClear = [];
if (sorted[0].count === sorted[1].count) {
for (const c of sorted) {
if (c.count === sorted[0].count) {
popularCompanies.push(c.company);
} else {
companiesToClear.push(c.company);
}
}
} else {
popularCompanies.push(sorted[0].company);
companiesToClear.push(...(sorted.slice(1).map(_ => _.company)));
}
return [
popularCompanies,
companiesToClear
];
}
async getUserCount (company) {
const result = await this.data.users.getByQuery(
{
teamIds: company.everyoneTeamId
},
{
hint: UserIndexes.byTeamId,
fields: ['deactivated', 'isRegistered']
}
);
return result.filter(_ => !_.deactivated && _.isRegistered).length;
}
async getPostCount (company) {
return await this.data.posts.countByQuery(
{
teamId: company.everyoneTeamId
},
{
hint: PostIndexes.byTeamId
}
);
}
async getMostRecentLogin (company) {
const result = await this.data.users.getByQuery(
{
teamIds: company.everyoneTeamId
},
{
hint: UserIndexes.byTeamId,
fields: ['deactivated', 'isRegistered', 'lastLogin']
}
);
const filtered = result.filter(_ => !_.deactivated && _.isRegistered && _.lastLogin);
filtered.sort((a, b) => b.lastLogin - a.lastLogin);
return filtered[0].lastLogin;
}
async clearCompanies (companies) {
if (this.dryrun) {
console.log(`Would have cleared data for ${companies.length} companies`);
return;
}
const companyIds = companies.map(_ => this.mongoClient.mongoCollections.companies.objectIdSafe(_.id));
await this.mongoClient.mongoCollections.companies.updateDirect(
{ id: { $in: companyIds } },
{ $unset: { nrOrgIds: 1, isNRConnected: 1 } }
);
for (const c of companies) {
await this.mongoClient.mongoCollections.users.updateDirect(
{ teamIds: c.everyoneTeamId },
{
$unset: {
[`providerInfo.${c.everyoneTeamId}.newrelic`]: 1
}
},
{ hint: UserIndexes.byTeamId }
);
}
await Wait(this.throttle);
}
}
(async function () {
try {
const options = {
dryrun: Commander.dryrun,
verbose: Commander.verbose
};
if (Commander.throttle) {
options.throttle = parseInt(Commander.throttle, 10);
if (isNaN(options.throttle)) {
throw 'invalid throttle value';
}
}
await ApiConfig.loadPreferredConfig();
await new UniquifyNRCSOrgAssociations(options).go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-populate_nr_org_ids.js 0000775 0000000 0000000 00000013020 14607772627 0026562 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// clear out the values for nrOrgIds and nrAccountIds on all companies
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
const { request, gql } = require('graphql-request');
Commander
.option('--dryrun, --dryrun', 'Do a dry run, don\'t actually change any data')
.option('--throttle, --throttle ', 'Throttle write operations by this time interval')
.parse(process.argv);
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class PopulateNewRelicOrgIds {
constructor (options) {
Object.assign(this, options);
this.throttle = this.throttle || 1000;
}
async go () {
this.newRelicApiUrl = ApiConfig.getPreferredConfig().sharedGeneral.newRelicApiUrl || 'https://api.newrelic.com';
await this.openMongoClient();
await this.processCompanies();
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async processCompanies () {
const result = await this.data.companies.getByQuery(
{
deactivated: false,
// isNRConnected: true
},
{
stream: true,
overrideHintRequired: true
}
);
let company, users;
this.numCompaniesPopulated = 0;
this.numCompaniesFailed = 0;
do {
company = await result.next();
if (company) {
users = await this.getUsersWithApiKeys(company);
let orgId = null;
for (let user of users) {
try {
const accessToken = (
user.providerInfo[company.everyoneTeamId] &&
user.providerInfo[company.everyoneTeamId].newrelic &&
user.providerInfo[company.everyoneTeamId].newrelic.accessToken
) || (
user.providerInfo.newrelic &&
user.providerInfo.newrelic.accessToken
);
orgId = await this.getOrganizationId(accessToken);
} catch (error) {
console.warn(error.message);
}
if (orgId) {
console.log(`Setting orgId for company ${company.id} to ${orgId}`);
await this.setOrgId(company, orgId);
break;
}
}
if (!orgId) {
console.log(`Unable to find an authorized user for company ${company.id}`);
this.numCompaniesFailed++;
}
}
} while (company);
result.done();
if (this.dryrun) {
console.log(`Would have populated nrOrgIds on ${this.numCompaniesPopulated} companies`);
console.log(`Would have failed to populate nrOrgIds for ${this.numCompaniesFailed} companies`);
} else {
console.log(`Populated nrOrgIds on ${this.numCompaniesPopulated} companies`);
console.log(`Failed to populate nrOrgIds for ${this.numCompaniesFailed} companies`);
}
}
async getUsersWithApiKeys (company) {
let result = await this.data.users.getByQuery(
{
teamIds: company.everyoneTeamId
},
{
hint: UserIndexes.byTeamId
}
);
result = result.filter(_ =>
!_.deactivated &&
_.isRegistered &&
_.lastLogin &&
_.providerInfo && ((
_.providerInfo[company.everyoneTeamId] &&
_.providerInfo[company.everyoneTeamId].newrelic
) || (
_.providerInfo.newrelic
))
);
result.sort((a, b) => a.lastLogin - b.lastLogin);
return result;
}
// fetch the user's organization ID from NR
async getOrganizationId (apiKey) {
const baseUrl = this.newRelicApiUrl;
const url = baseUrl + '/graphql';
const query = gql`{
actor {
organization {
id
}
}
}`;
const headers = {
'Api-Key': apiKey,
'Content-Type': 'application/json',
'NewRelic-Requesting-Services': 'CodeStream',
"X-Query-Source-Capability-Id": "CODESTREAM",
"X-Query-Source-Component-Id": "codestream.api"
};
const response = await request(url, query, {}, headers);
if (response && response.actor && response.actor.organization && response.actor.organization.id) {
return response.actor.organization.id;
}
return undefined;
}
async setOrgId (company, orgId) {
let set;
if (company.nrOrgIds && !company.isNRConnected) {
set = { isNRConnected: true };
} else if (!company.nrOrgIds) {
set = { nrOrgIds: [orgId], isNRConnected: true };
}
if (set) {
this.numCompaniesPopulated++;
if (this.dryrun) {
console.log(`Would have set nrOrgIds for company ${company.id} to ${orgId}...`);
} else {
console.log(`Setting nrOrgIds to ${orgId} for company ${company.id}...`);
await this.mongoClient.mongoCollections.companies.updateDirect(
{ id: this.mongoClient.mongoCollections.companies.objectIdSafe(company.id) },
{ $set: set }
);
}
} else {
console.log(`NR props already set for company ${company.id}`);
}
await Wait(this.throttle);
}
}
(async function() {
try {
const options = {
dryrun: Commander.dryrun
};
if (Commander.throttle) {
options.throttle = parseInt(Commander.throttle, 10);
if (isNaN(options.throttle)) {
throw 'invalid throttle value';
}
}
await ApiConfig.loadPreferredConfig();
await new PopulateNewRelicOrgIds(options).go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-reinvite_users.js 0000775 0000000 0000000 00000020354 14607772627 0025602 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// look for users due for a reinvite, and queue message to send them an email as needed
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const PubNub = require('pubnub');
const PubNubClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/pubnub/pubnub_client_async');
const OS = require('os');
const SQSClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/aws/sqs_client');
const AWS = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/aws/aws');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
const SignupTokenIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/signup_token_indexes');
// need these collections from mongo
const COLLECTIONS = ['users', 'reinviteUsersLastRunAt', 'signupTokens'];
const THROTTLE_TIME = 100;
const REINVITE_INTERVAL = 24 * 60 * 60 * 1000;
const RUN_INTERVAL = 50 * 60 * 1000;
// wait this number of milliseconds
const Wait = function (time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class Reinviter {
// main entry point
async go (options = {}) {
this.numProcessed = 0;
this.numInvited = 0;
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
if (await this.abortIfDoneAlready()) {
this.log('Not running, another process has already run');
process.exit();
}
await this.openPubnubClient();
await this.openSQSClient();
await this.updateLastRunAt();
await this.process();
this.log(`${this.numProcessed} users processed`);
this.log(`${this.numInvited} users invited`);
process.exit();
}
// if another instance of this script already ran, just abort
async abortIfDoneAlready () {
const lastRunAt = await this.data.reinviteUsersLastRunAt.getByQuery({}, { overrideHintRequired: true });
if (lastRunAt && lastRunAt[0] && lastRunAt[0].lastRunAt > Date.now() - RUN_INTERVAL) {
return true;
}
}
// open a mongo client to read from
async openMongoClient () {
this.log('Connecting to mongo...');
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(this.config.storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// open a Pubnub client for broadcasting the changes
async openPubnubClient () {
this.log('Connecting to PubNub...');
this.pubnub = new PubNub(Object.assign({}, this.config.broadcastEngine.pubnub, { uuid: 'API-' + OS.hostname() }));
this.pubnubClient = new PubNubClient({ pubnub: this.pubnub });
await this.pubnubClient.init();
}
// open an SQS client for queueing emails
async openSQSClient () {
this.log('Initializing SQS...');
this.aws = new AWS(this.config.queuingEngine.awsSQS);
this.sqsClient = new SQSClient({ aws: this.aws, logger: this.logger });
const queueName = this.config.queuingEngine[this.config.queuingEngine.selected].outboundEmailQueueName;
await this.sqsClient.createQueue({
name: queueName,
logger: this.logger
});
}
// set flag for the given user or all the users in the given team
async process () {
if (this.isWeekend()) {
this.log('It is local weekend, not running');
return;
}
const result = await this.data.users.getByQuery(
{
needsAutoReinvites: { $gt: 0 }
},
{
stream: true,
hint: UserIndexes.byNeedsAutoReinvites,
sort: { _id: 1 }
}
);
let user;
do {
user = await result.next();
if (user) {
await this.reinviteUserAsNeeded(user);
await Wait(THROTTLE_TIME);
}
} while (user);
result.done();
}
// set flag for the given user
async reinviteUserAsNeeded (user) {
let clearUser = false;
if (user.isRegistered) {
this.log(`User ${user.email} is now registered, not re-inviting`);
clearUser = true;
} else if (user.deactivated) {
this.log(`User ${user.email} has been deactivated, not re-inviting`);
clearUser = true;
} else if (!user.needsAutoReinvites) {
this.log(`User ${user.email} has no more auto-reinvites, not re-inviting`);
clearUser = true;
} else if (user.lastInviteSentAt > Date.now() - REINVITE_INTERVAL) {
this.log(`User ${user.email} has had an invite within ${REINVITE_INTERVAL} ms, not re-inviting (yet)`);
this.numProcessed++;
return;
}
if (clearUser) {
return this.clearUser(user);
} else {
return this.reinviteUser(user);
}
}
// clear auto re-invites for this user, no longer needed
async clearUser (user) {
const op = {
$unset: {
needsAutoReinvites: true,
autoReinviteInfo: true
}
};
await this.data.users.updateDirect({ _id: this.data.users.objectIdSafe(user.id) }, op);
this.numProcessed++;
}
// re-invite this user by sending an invite email, updating the user, and publishing user changes
async reinviteUser (user) {
let teamId, forceCompanyCentricInviteCopy;
if (user.autoReinviteInfo && user.autoReinviteInfo.teamId) {
teamId = user.autoReinviteInfo.teamId;
} else if (user.teamIds && user.teamIds.length === 1) {
teamId = user.teamIds[0];
} else if (user.inviteCode) {
const token = await this.data.signupTokens.getOneByQuery(
{ token: user.inviteCode },
{ hint: SignupTokenIndexes.byToken }
);
if (token) {
teamId = token.teamId;
} else {
forceCompanyCentricInviteCopy = true;
}
}
// queue invite email for send by outbound email service
this.log(`Triggering auto invite email to ${user.email}...`);
const message = {
type: 'invite',
userId: user.id,
teamId,
forceCompanyCentricInviteCopy,
...user.autoReinviteInfo
};
const queueName = this.config.queuingEngine[this.config.queuingEngine.selected].outboundEmailQueueName;
await this.sqsClient.sendMessage(queueName, message);
// update the user indicating invite has been sent
const op = {
$set: {
lastInviteType: 'autoReinvitation',
modifiedAt: Date.now(),
lastInviteSentAt: Date.now(),
version: user.version + 1
},
$inc: {
numInvites: 1
}
};
if (user.needsAutoReinvites === 1) {
op.$unset = {
needsAutoReinvites: true,
autoReinviteInfo: true
};
} else {
op.$inc.needsAutoReinvites = -1;
}
await this.data.users.updateDirect({ _id: this.data.users.objectIdSafe(user.id) }, op);
// publish the change to all teams
op.$version = {
before: user.version,
after: user.version + 1
};
const teamIds = user.teamIds || [];
await Promise.all(teamIds.map(async teamId => {
const message = {
requestId: `autoReinvite-${3-user.needsAutoReinvites}-${teamId}-${user.id}`,
users: [op]
};
try {
await this.pubnubClient.publish(
message,
'team-' + teamId
);
}
catch (error) {
// this doesn't break the chain, but it is unfortunate...
this.log(`Could not publish user update message to team ${teamId}: ${JSON.stringify(error)}`);
}
}));
this.numProcessed++;
this.numInvited++;
}
// is it the weekend now? we don't work weekends
// weekend defined as starting at 6 PM Friday, ending at 8 AM Monday
isWeekend () {
const now = new Date();
const day = now.getDay();
const hour = now.getHours();
switch (day) {
case 0: // sunday
case 7: // saturday
return true;
case 1: // monday
return hour < 8;
case 6: // friday
return hour >= 18;
default:
return false;
}
}
// update when this script was last run so other running instances don't try
async updateLastRunAt () {
await this.data.reinviteUsersLastRunAt.updateDirect(
{ },
{ $set: { lastRunAt: Date.now() } },
{ upsert: true}
);
}
log (msg) {
const now = new Date().toISOString();
this.logger.log(`${now} - ${msg}`);
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new Reinviter().go({ config: ApiConfig.getPreferredConfig() });
}
catch (error) {
const msg = error instanceof Error ? error.message : JSON.stringify(error);
const stack = error instanceof Error ? error.stack : '';
const now = new Date().toISOString();
console.log(`${now} - ERROR: ${msg}\n${stack}`);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-rename_grok_user.js 0000775 0000000 0000000 00000005321 14607772627 0026060 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// rename the Grok user to AI
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const TeamIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/teams/indexes');
Commander
.option('--dryrun, --dryrun', 'Do a dry run, don\'t actually change any data')
.option('--throttle, --throttle ', 'Throttle write operations by this time interval')
.parse(process.argv);
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class RenameGrokUser {
constructor (options) {
Object.assign(this, options);
this.throttle = this.throttle || 1000;
}
async go () {
await this.openMongoClient();
await this.processTeams();
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
async processTeams () {
const result = await this.data.teams.getByQuery(
{},
{
stream: true,
hint: TeamIndexes.byProviderIdentities
}
);
let team;
let count = 0;
do {
team = await result.next();
if (team && team.grokUserId) {
try {
if (this.dryrun) {
console.log(`Would have renamed user ${team.grokUserId}...`);
count++;
} else {
console.log(`Renaming user ${team.grokUserId}...`);
await this.mongoClient.mongoCollections.users.updateDirect(
{ id: this.mongoClient.mongoCollections.users.objectIdSafe(team.grokUserId) },
{ $set: { username: 'AI' } }
);
await Wait(this.throttle);
count++;
}
} catch (ex) {
console.log(`Failed to rename user ${team.grokUserId}: ${ex}`);
}
}
} while (team);
if (this.dryrun) {
console.log(`Would have renamed up to ${count} users`);
} else {
console.log(`Ensured ${count} users have the correct name`);
}
}
}
(async function() {
try {
const options = {
dryrun: Commander.dryrun
};
if (Commander.throttle) {
options.throttle = parseInt(Commander.throttle, 10);
if (isNaN(options.throttle)) {
throw 'invalid throttle value';
}
}
await ApiConfig.loadPreferredConfig();
await new RenameGrokUser(options).go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-restore-local-db 0000775 0000000 0000000 00000001064 14607772627 0025254 0 ustar 00root root 0000000 0000000 #!/bin/bash
function usage {
echo "usage: $0 [-D] -d designator"
exit 1
}
dropCodestreamDb=1
while getopts "Dd:" arg
do
case $arg in
D) dropCodestreamDb=0;;
d) dumpTag=$OPTARG;;
esac
done
[ -z "$dumpTag" ] && echo "-d is required" && usage
dumpDir=$CS_API_SANDBOX/data/dump.$dumpTag
[ ! -d $dumpDir ] && echo "$dumpDir not found" && exit 1
if [ $dropCodestreamDb -eq 1 ]; then
echo "mongo --host localhost codestream --eval \"db.dropDatabase()\""
mongo --host localhost codestream --eval "db.dropDatabase()"
fi
mongorestore --host localhost $dumpDir
codestream-server-api-15.4.0/api_server/bin/cs_api-run_migrations.js 0000775 0000000 0000000 00000002760 14607772627 0025575 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// run needed database migrations against current mongo database
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const MigrationsHelper = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/migrations/migrations_helper');
class MigrationRunner {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
await this.process();
}
catch (error) {
this.logger.error(error);
throw error;
}
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all', 'migrationVersion'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
throw `unable to open mongo client: ${message}`;
}
}
async process () {
await new MigrationsHelper({
data: this.data,
logger: this.logger
}).runMigrations();
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new MigrationRunner().go();
}
catch (error) {
console.error(error);
process.exit(1);
}
process.exit(0);
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-sb-tool 0000775 0000000 0000000 00000001405 14607772627 0023474 0 ustar 00root root 0000000 0000000 #!/bin/bash
#desc# playground utility tool
. $DT_TOP/lib/sandbox_release_tool.sh || exit 1
function usage {
local c=`basename $0`
echo "
usage:
Backend services playground utility. Commands are executed across all 4 projects.
$(sandtool_help_common_args)
"
exit 1
}
[ -z "$1" ] && usage
ec=0
[ -z "$CS_API_TOP" ] && echo "api sandbox not loaded" && ec=1
[ -z "$CS_BROADCASTER_TOP" ] && echo "broadcaster sandbox not loaded" && ec=1
[ -z "$CS_MAILIN_TOP" ] && echo "inbound email sandbox not loaded" && ec=1
[ -z "$CS_OUTBOUND_EMAIL_TOP" ] && echo "outbound email sandbox not loaded" && ec=1
[ $ec -eq 1 ] && exit 1
SANDTOOL_REPO_LIST="$CS_API_TOP $CS_BROADCASTER_TOP $CS_MAILIN_TOP $CS_OUTBOUND_EMAIL_TOP"
sandtool_process_common_args "$@" && exit
usage
codestream-server-api-15.4.0/api_server/bin/cs_api-sendgrid_to_segment.js 0000775 0000000 0000000 00000021022 14607772627 0026550 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// pull latest messages from SendGrid and ping Segment for each delivered email
/* eslint no-console: 0 */
'use strict';
const Fetch = require('node-fetch');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const AnalyticsNode = require('analytics-node');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT +
'/shared/server_utils/mongo/mongo_client');
const Commander = require('commander');
Commander.option('-f, --force', 'Force execution').parse(process.argv);
if (!Commander.force) {
Commander.help();
}
const ONE_DAY = 24 * 60 * 60 * 1000;
const MESSAGE_LIMIT = 1000;
const THROTTLE_TIME = 200;
const DEBUG = false;
const RUN_INTERVAL = 22 * 55 * 60 * 1000; // hours x min x sec x 1000 milisec. 22h 55m allows for random sleep when run from cron
const CATEGORIES = [
'email_confirmation',
'invitation',
'notification',
'notification_invite',
'password_reset',
'reinvitation',
'weekly',
'weekly_invite',
];
class TransferSendGridData {
constructor(options) {
Object.assign(this, options);
this.messages = [];
this.numRateLimitHits = 0;
}
async go() {
await this.openMongoClient();
if (await this.abortIfDoneAlready()) {
console.log('*** This script was already run within 24 hours, aborting... ***');
return;
}
await this.updateLastRunAt();
await this.connectToSegment();
await this.getSendGridMessages();
await this.writeToSegment();
console.log('Waiting 5 seconds for tracking to complete...');
await this.wait(5000);
}
// open a mongo client to read from
async openMongoClient() {
this.mongoClient = new MongoClient({ collections: ['users', 'sendGridToSegmentLastRunAt'] });
try {
console.log('Connecting to Mongo...');
await this.mongoClient.openMongoClient(this.config.storage.mongo);
this.data = this.mongoClient.mongoCollections;
} catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
throw `unable to open mongo client: ${message}`;
}
}
// if another instance of this script already ran, just abort
async abortIfDoneAlready() {
const lastRunAt = await this.data.sendGridToSegmentLastRunAt.getByQuery(
{},
{ overrideHintRequired: true }
);
if (lastRunAt && lastRunAt[0] && lastRunAt[0].lastRunAt > Date.now() - RUN_INTERVAL) {
return true;
}
}
// update when this script was last run so other running instances don't try
async updateLastRunAt() {
await this.data.sendGridToSegmentLastRunAt.updateDirect(
{},
{ $set: { lastRunAt: Date.now() } },
{ upsert: true }
);
}
async connectToSegment() {
console.log('Connecting to Segment...');
this.segment = new AnalyticsNode(this.config.telemetry.segment.token);
}
async getSendGridMessages() {
console.log('Fetching SendGrid messages...');
for (let category of CATEGORIES) {
await this.getSendGridMessagesByCategory(category);
}
this.messages.sort((b, a) => {
return a.last_event_time.localeCompare(b.last_event_time);
});
}
async getSendGridMessagesByCategory(category) {
let numMessages = 0;
const now = Date.now();
const lastMidnight = this.midnight(now);
const previousMidnight = lastMidnight - ONE_DAY;
let lastTime = lastMidnight;
let nPages = 0;
do {
nPages++;
console.log(`\tFetching page ${nPages} of SendGrid messages, category="${category}"...`);
let messages = await this.getSendGridMessagePage(category, lastTime, previousMidnight);
messages = messages.filter((message) => {
return !this.messages.find((haveMessage) => haveMessage.msg_id === message.msg_id);
});
console.log(`\t\t(fetched ${messages.length} "${category}" messages)`);
if (DEBUG) {
console.log(
'####################################################################################'
);
console.log(JSON.stringify(messages, undefined, 5));
console.log(
'####################################################################################'
);
}
messages = messages.filter((message) => {
return message.opens_count > 0;
});
numMessages = messages.length;
this.messages = [...this.messages, ...messages];
if (numMessages > 0) {
console.log(`\t\t(${messages.length} "${category}" emails have been opened)`);
const lastMessage = messages[messages.length - 1];
lastTime = new Date(lastMessage.last_event_time);
}
} while (numMessages > 0 && this.messages.length < MESSAGE_LIMIT);
}
async getSendGridMessagePage(category, latest, earliest) {
const token = this.config.emailDeliveryService.sendgrid.apiKey;
const apiKey = token.split('.')[1];
const latestISO = new Date(latest).toISOString();
const earliestISO = new Date(earliest).toISOString();
const query = `
api_key_id="${apiKey}" AND
last_event_time BETWEEN TIMESTAMP "${earliestISO}" AND TIMESTAMP "${latestISO}" AND
status="delivered" AND
(Contains(categories,"${category}"))
`;
const limit = Math.min(1000, MESSAGE_LIMIT);
const params = `limit=${limit}&query=${encodeURIComponent(query)}`;
const url = `https://api.sendgrid.com/v3/messages?${params}`;
const response = await Fetch(url, {
headers: {
Authorization: `Bearer ${token}`,
'Content-Type': 'application/json',
},
});
const text = await response.text();
let json;
if (!response.ok) {
if (response.statusText === 'Too Many Requests') {
console.warn('*** SendGrid said Too Many Requests, waiting 5 minutes... ***');
await this.wait(300000);
return this.getSendGridMessagePage(category, latest, earliest);
}
console.warn('status=' + response.statusText);
try {
json = JSON.parse(text);
} catch (error) {
console.error(
`JSON parsing of error response failed with "${error.message}", text was:\n${text}`
);
process.exit(1);
}
if (json.errors && json.errors[0] && json.errors[0].message === 'too many requests') {
this.numRateLimitHits++;
if (this.numRateLimitHits === 20) {
console.error('Too many rate limit hits, aborting...');
process.exit(1);
}
console.warn('*** Hit SendGrid rate limit, waiting 5 minutes... ***');
await this.wait(300000);
return this.getSendGridMessagePage(category, latest, earliest);
}
console.error(`Unable to fetch from SendGrid: ${response.statusText}`);
console.error(`returned: ${JSON.stringify(json, 0, 5)}`);
console.error(`response headers: ${JSON.stringify(response.headers.raw(), 0, 5)}`);
process.exit(1);
}
try {
json = JSON.parse(text);
} catch (error) {
console.error(`JSON parsing failed with "${error.message}", text was:\n${text}`);
process.exit(1);
}
json.messages.forEach((message) => {
message.category = category;
});
return json.messages;
}
async writeToSegment() {
console.log(`Writing ${this.messages.length} events to Segment...`);
let n = 0;
for (let message of this.messages) {
n++;
await this.writeMessageToSegment(message, n, this.messages.length);
await this.wait(THROTTLE_TIME);
}
}
async writeMessageToSegment(message, n, total) {
const user = await this.data.users.getOneByQuery({
searchableEmail: message.to_email.toLowerCase(),
});
if (!user) {
console.warn(`*** Could not find matching user for ${message.to_email} ***`);
return;
}
try {
console.log(`\t${n} of ${total}: ${message.to_email}...`);
const trackObject = {
event: 'Sendgrid Opens',
userId: user.id,
timestamp: new Date(message.last_event_time),
integrations: { All: false, Postgres: true }, // only send to PostGres
properties: {
distinct_id: user.id,
$email: user.email,
name: user.fullName || '',
msg_id: message.msg_id,
category: message.category,
},
};
if (user.registeredAt) {
trackObject.properties['$created'] = new Date(user.registeredAt).toISOString();
trackObject.integrations.Intercom = true; // also send to Intercom, if user is registered
}
this.segment.track(trackObject);
} catch (error) {
const msg = error instanceof Error ? error.message : JSON.stringify(error);
console.warn(`*** Exception writing to Segment: ${msg} ***`);
}
}
midnight(timestamp) {
const msSinceMidnightGmt = timestamp % ONE_DAY;
return timestamp - msSinceMidnightGmt;
}
async wait(ms) {
await new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
}
(async function () {
try {
const Config = await ApiConfig.loadPreferredConfig();
await new TransferSendGridData({ config: Config }).go();
} catch (error) {
const msg = error instanceof Error ? error.message : JSON.stringify(error);
console.error(msg);
process.exit(1);
}
process.exit(0);
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-service 0000775 0000000 0000000 00000005252 14607772627 0023561 0 ustar 00root root 0000000 0000000 #!/bin/bash
#desc# CodeStream API service init script
function usage {
echo "usage: $0 { start | stop | status | restart } [-if1] [-- server-args]" >&2
echo >&2
echo " -f run in foreground"
echo " -1 add the server-arg --one_worker"
echo " -i force index recreation"
exit 1
}
function startup {
if [ "$CS_API_SETUP_MONGO" == true ]; then
if [ -n "$CS_API_MONGO_USER" -a ! -f $CS_API_SANDBOX/mongo-user.configured ]; then
echo "Creating mongo user $CS_API_MONGO_USER as dbOwner of codestream db"
mdb-add-user -u $CS_API_MONGO_USER -p $CS_API_MONGO_PASS -r dbOwner -D $CS_API_MONGO_DATABASE
touch $CS_API_SANDBOX/mongo-user.configured
fi
if [ ! -f $CS_API_SANDBOX/mongo-indexes.configured -o $reindex -eq 1 ]; then
echo "Running ensure-indexes.js..."
echo "touch $CS_API_SANDBOX/mongo-indexes.configured to prevent this on future starts."
(cd $CS_API_TOP && bin/ensure-indexes.js build)
else
echo "******"
echo "$CS_API_SANDBOX/mongo-indexes.configured was found."
echo "If you want to force index creation use the -i flag with the init script"
echo "or remove the abovementioned file."
echo "usage: $0 {start | restart} [-i]"
echo "******"
fi
fi
echo "Starting $CS_API_TOP/bin/api_server.js. Logging to $CS_API_LOGS/api.log"
if [ $run_fg -eq 1 ]; then
$CS_API_TOP/bin/api_server.js $server_args
else
echo "`date`: cs_api-service is starting the API" | tee -a $CS_API_LOGS/init.err.log >>$CS_API_LOGS/init.out.log
nohup $CS_API_TOP/bin/api_server.js $server_args >>$CS_API_LOGS/init.out.log 2>>$CS_API_LOGS/init.err.log &2 && exit 1
if [ -n "$CS_API_TEST_ONLY" ]; then
echo "This sandbox is configured as a test-only client (CS_API_TEST_ONLY)"
exit 1
fi
[[ ! ( "$1" == start || "$1" == stop || "$1" == status || "$1" == restart ) ]] && usage
. $DT_TOP/lib/init_funcs.sh
action=$1
run_fg=0
server_args=""
reindex=0
OPTIND=2
while getopts "f1i" arg
do
case $arg in
i) reindex=1;;
1) server_args="--one_worker";;
f) run_fg=1;;
*) usage;;
esac
done
shift `expr $OPTIND - 1`
ec=0
server_args="$server_args $*"
case "$action" in
start)
startup
;;
stop)
service_stop $CS_API_TOP/bin/api_server.js "useForce,$CS_API_INIT_STOP_OPTS" "$CS_API_INIT_FAIL_MSG"
ec=$?
sleep 5
service_status $CS_API_TOP/bin/api_server.js
;;
status)
service_status $CS_API_TOP/bin/api_server.js
;;
restart)
service_stop $CS_API_TOP/bin/api_server.js "$CS_API_INIT_STOP_OPTS" "$CS_API_INIT_FAIL_MSG"
if [ $ec -ne 0 ]; then
echo "error stopping service"
else
startup
fi
;;
*)
usage
;;
esac
exit $ec
codestream-server-api-15.4.0/api_server/bin/cs_api-set_isnrconnected.js 0000775 0000000 0000000 00000006726 14607772627 0026254 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// set isNRConnected for all companies with NR-connected users
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
Commander
.option('--dryrun, --dryrun', 'Do a dry run, don\'t actually change any data')
.option('--throttle, --throttle ', 'Throttle write operations by this time interval')
.parse(process.argv);
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class SetIsNRConnected {
constructor (options) {
Object.assign(this, options);
this.throttle = this.throttle || 1000;
}
async go () {
await this.openMongoClient();
await this.processCompanies();
process.exit();
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: ['__all'] });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// iterate over companies without isNRConnected and check for connected users
async processCompanies () {
const result = await this.data.companies.getByQuery(
{
deactivated: false,
isNRConnected: { $ne: true }
},
{
stream: true,
overrideHintRequired: true
}
);
let company;
let count = 0;
do {
company = await result.next();
if (company) {
if (await this.checkForNRConnectedUsers(company)) {
try {
if (this.dryrun) {
console.log(`Would have set isNRConnected for company ${company.id}...`);
count++;
} else {
console.log(`Setting isNRConnected for company ${company.id}...`);
await this.mongoClient.mongoCollections.companies.updateDirect(
{ id: this.mongoClient.mongoCollections.companies.objectIdSafe(company.id) },
{ $set: { isNRConnected: true } }
);
await Wait(this.throttle);
count++;
}
} catch (ex) {
console.log(`Failed to set isNRConnected for company ${company.id}: ${ex}`);
}
}
}
} while (company);
if (this.dryrun) {
console.log(`Would have set isNRConnected on ${count} companies`);
} else {
console.log(`Set isNRConnected on ${count} companies`);
}
}
// check if any users are NR-connected
async checkForNRConnectedUsers (company) {
let result = await this.data.users.getByQuery(
{
teamIds: company.everyoneTeamId
},
{
hint: UserIndexes.byTeamId
}
);
return result.some(_ =>
!_.deactivated &&
_.isRegistered &&
_.providerInfo && ((
_.providerInfo[company.everyoneTeamId] &&
_.providerInfo[company.everyoneTeamId].newrelic
) || (
_.providerInfo.newrelic
))
);
}
}
(async function() {
try {
const options = {
dryrun: Commander.dryrun
};
if (Commander.throttle) {
options.throttle = parseInt(Commander.throttle, 10);
if (isNaN(options.throttle)) {
throw 'invalid throttle value';
}
}
await ApiConfig.loadPreferredConfig();
await new SetIsNRConnected(options).go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-set_user_flag.js 0000775 0000000 0000000 00000007057 14607772627 0025363 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
//desc// set (or clear) a flag for a user or all members of a team
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const PubNub = require('pubnub');
const PubNubClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/pubnub/pubnub_client_async');
const Commander = require('commander');
const OS = require('os');
const UUID = require('uuid').v4;
// need these collections from mongo
const COLLECTIONS = ['teams', 'users'];
Commander
.option('-t, --teamId ', 'Set flag for all members of this team')
.option('-u, --userId ', 'Set flag for this user')
.option('-f, --flag ', 'The flag to set or clear')
.option('-c, --clear', 'Instead of setting the flag, clear it')
.parse(process.argv);
if ((!Commander.teamId && !Commander.userId) || !Commander.flag) {
Commander.help();
}
class SetFlag {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
await this.openPubnubClient();
await this.process();
console.log('DONE');
}
catch (error) {
this.logger.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// open a Pubnub client for broadcasting the changes
async openPubnubClient () {
this.pubnub = new PubNub(Object.assign({}, ApiConfig.getPreferredConfig().broadcastEngine.pubnub, { uuid: 'API-' + OS.hostname() }));
this.pubnubClient = new PubNubClient({ pubnub: this.pubnub });
await this.pubnubClient.init();
}
// set flag for the given user or all the users in the given team
async process () {
let userIds = [];
if (this.teamId) {
const team = await this.data.teams.getById(this.teamId);
if (!team) {
throw 'team not found';
}
userIds = team.memberIds;
}
else {
userIds = [this.userId];
}
await Promise.all(userIds.map(async userId => {
await this.setFlagForUser(userId);
}));
this.logger.log(`Updated ${userIds.length} users and sent broadcaster messages`);
}
// set flag for the given user
async setFlagForUser (userId) {
let op;
// set or clear flag
if (this.clear) {
op = { $unset: { [this.flag]: true } };
}
else {
op = { $set: { [this.flag]: true } };
}
await this.data.users.updateDirect({ _id: this.data.users.objectIdSafe(userId) }, op);
// send pubnub update on user's me-channel
const requestId = UUID();
const message = {
user: Object.assign(op, { id: userId }),
requestId
};
const channel = `user-${userId}`;
try {
await this.pubnubClient.publish(
message,
channel
);
}
catch (error) {
// this doesn't break the chain, but it is unfortunate
this.logger.warn(`Unable to publish user op to channel ${channel}: ${JSON.stringify(error)}`);
}
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new SetFlag().go({
teamId: Commander.teamId,
userId: Commander.userId,
flag: Commander.flag,
clear: Commander.clear
});
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-set_user_password.js 0000775 0000000 0000000 00000003741 14607772627 0026310 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// utility script to set a user's password
/* eslint no-console: 0 */
'use strict';
const PasswordHasher = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/password_hasher');
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
const Commander = require('commander');
Commander
.requiredOption('-e, --email ', 'Email of user whose password is to change')
.requiredOption('-p, --password ', 'Password to change to')
.option('--force-set', 'Force the user to change their password next time they log in')
.parse(process.argv);
(async function() {
await ApiConfig.loadPreferredConfig();
const mongoClient = new MongoClient({ collections: ['users'] });
try {
await mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
}
catch (error) {
console.error(`unable to open mongo client: ${JSON.stringify(error)}`);
process.exit();
}
const userCollection = mongoClient.mongoCollections.users;
const hash = await new PasswordHasher({ password: Commander.password }).hashPassword();
try {
const users = await userCollection.getByQuery(
{ searchableEmail: Commander.email.toLowerCase() },
{ hint: UserIndexes.bySearchableEmail }
);
if (users.length === 0) {
console.error('User not found: ' + Commander.email);
process.exit();
}
const set = {
passwordHash: hash
};
if (Commander.forceSet) {
set.mustSetPassword = true;
}
await mongoClient.mongoCollections['users'].updateDirect(
{ searchableEmail: Commander.email.toLowerCase() },
{ $set: set }
);
}
catch (error) {
console.error(`unable to set password hash for user: ${JSON.stringify(error)}`);
process.exit();
}
console.log('Password hash changed');
process.exit();
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-slack_replies_to_codestream.js 0000775 0000000 0000000 00000077416 14607772627 0030277 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
const CodemarkIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/codemarks/indexes');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
const StreamIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/streams/indexes');
const PasswordHasher = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/password_hasher');
const PubNub = require('pubnub');
const PubNubClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/pubnub/pubnub_client_async');
const UUID = require('uuid').v4;
const OS = require('os');
const { WebClient } = require('@slack/web-api');
Commander
.option('-t, --teamId ', 'Team ID to fetch Slack replies for, specify "all" for all Slack teams (be careful!)')
.option('--dryrun', 'Do a dry run, meaning don\'t actually write the replies to our database, but report on numbers')
.option('--throttle ', 'Pause this number of milliseconds between teams')
.option('--verbose', 'Verbose output')
.option('--debug', 'Debug output')
.option('--dieonwarn ', 'Die on a warning', parseInt)
.option('--ignorestreams ', 'Ignore the given comma-separated list of stream IDs')
.option('--ignoreusers ', 'Ignore codemarks authored by the given comma-separated list of user IDs')
.option('--setpasswords', 'Set temporary passwords for users who don\t have one after migration is complete')
.option('--clearslackcreds', 'Clear Slack credentials for all users on each team after migration is complete')
.parse(process.argv);
const COLLECTIONS = ['teams', 'posts', 'codemarks', 'users', 'streams', 'teams', 'markers'];
const DEFAULT_THROTTLE_TIME = 1000;
const Logger = console;
const ThrottleTime = Commander.throttle ? parseInt(Commander.throttle) : DEFAULT_THROTTLE_TIME;
const IgnoreStreams = Commander.ignorestreams ? Commander.ignorestreams.split(',') : [];
const IgnoreUsers = Commander.ignoreusers ? Commander.ignoreusers.split(',') : [];
// for paginated requests, we'll throttle repeated requested according to the tier
// the Slack API request is on for rate limiting
const SLACK_TIER_THROTTLE_TIMES = {
1: 60000,
2: 3000,
3: 1200,
4: 600
};
// wait this number of milliseconds
const Wait = function(time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
};
class TeamReplyFetcher {
constructor (options) {
Object.assign(this, options);
this.slackClients = {};
this.streamSeqNums = {};
this.slackStreams = {};
this.streams = {};
this.warnCount = 0;
this.totalCodemarks = 0;
this.totalReplies = 0;
this.numSkipped = 0;
this.codemarksConverted = 0;
}
// fetch all the replies to codemarks for a team and turn them into posts
async fetchRepliesForTeam (team) {
this.team = team;
await this.getTeamUsers();
await this.setUserFlags();
await this.ensureTeamStream();
await this.processCodemarks();
await this.updateTeam();
if (Commander.setpasswords) {
await this.setUserPasswords();
}
if (Commander.clearslackcreds) {
await this.clearSlackCredentials();
}
await this.clearUserFlags();
this.log(`\tCODEMARKS: ${this.totalCodemarks}`);
this.log(`\tSKIPPED: ${this.numSkipped}`);
this.log(`\tCONVERTED: ${this.codemarksConverted}`);
this.log(`\tREPLIES: ${this.totalReplies}`);
this.log(`\tWARNINGS: ${this.warnCount}`);
this.log('==============================================================');
}
// get all the users on the given team
async getTeamUsers () {
this.debug('\tGetting users on team...');
const users = await this.data.users.getByQuery(
{
teamIds: this.team.id
},
{
hint: UserIndexes.byTeamIds
}
);
this.users = users.reduce((usersHash, user) => {
usersHash[user.id] = user;
return usersHash;
}, {});
this.debug(`\t${Object.keys(this.users).length} users found on team`);
}
// set maintenance mode, must set password, and clear provider info flags for all users on the team
async setUserFlags () {
if (Commander.dryrun) {
this.log('\tWould have set user flags');
return;
}
else {
this.log('\tSetting user flags...');
}
const op = {
$set: {
inMaintenanceMode: true,
mustSetPassword: true,
clearProviderInfo: true
}
};
await this.data.users.updateDirect({ teamIds: this.team.id }, op);
await this.sendOpsToUsers(op);
}
// send pubnub message for all users on team
async sendOpsToUsers (op) {
const requestId = UUID();
this.log(`\tSending Pubnub op to ${Object.keys(this.users).length} users, reqid=${requestId}...`);
await Promise.all(Object.keys(this.users).map(async userId => {
await this.sendUserOp(userId, op, requestId);
}));
}
// send pubnub message for user
async sendUserOp (userId, op, requestId) {
// send pubnub update on user's me-channel
const message = {
user: Object.assign(op, { id: userId }),
requestId
};
const channel = `user-${userId}`;
try {
await this.pubnub.publish(
message,
channel
);
}
catch (error) {
// this doesn't break the chain, but it is unfortunate
this.warn(`WARNING: Unable to publish user op to channel ${channel}: ${JSON.stringify(error)}`);
}
}
// make sure we have a team stream for the given team
async ensureTeamStream () {
this.debug('\tLooking for team stream...');
const teamStreams = await this.data.streams.getByQuery(
{
teamId: this.team.id,
isTeamStream: true
},
{
hint: StreamIndexes.byIsTeamStream
}
);
if (teamStreams.length > 0) {
this.teamStream = teamStreams[0];
this.debug(`\tFound team stream ${this.teamStream.id}`);
return;
}
this.debug('\tCreating a team stream...');
const id = this.data.streams.createId().toString();
const stream = {
id,
version: 1,
deactivated: false,
createdAt: this.team.createdAt,
modifiedAt: this.team.createdAt,
teamId: this.team.id,
type: 'channel',
name: 'general',
isTeamStream: true,
privacy: 'public',
creatorId: this.team.creatorId,
sortId: id,
nextSeqNum: 1
};
if (Commander.dryrun) {
this.log(`\tWould have created team stream for team ${this.team.id}`);
this.teamStream = stream;
}
else {
this.log(`\tCreating team stream for team ${this.team.id}...`);
this.teamStream = await this.data.streams.create(stream);
}
this.verbose(stream);
}
// process all the codemarks for the team, looking for replies to convert into CodeStream posts
async processCodemarks () {
this.debug('\tProcessing codemarks...');
let codemark;
do {
let n = 0;
const query = {
teamId: this.team.id,
deactivated: false
};
if (codemark) {
query.createdAt = { $gt: codemark.createdAt };
}
this.debug(`\tInitiating new query: ${JSON.stringify(query)}...`);
const result = await this.data.codemarks.getByQuery(
query,
{
stream: true,
hint: CodemarkIndexes.byTeamId,
sort: { createdAt: 1 }
}
);
do {
codemark = await result.next();
if (codemark) {
try {
await this.processCodemark(codemark);
}
catch (error) {
this.warn(`WARNING: Error thrown processing codemark ${codemark.id}, ignoring: ${error}`);
}
await Wait(ThrottleTime);
n++;
}
} while (codemark && n < 500);
result.done();
} while (codemark);
}
// process a single codemark for a team
async processCodemark (codemark) {
this.totalCodemarks++;
this.log(`\tProcessing codemark #${this.totalCodemarks}: ${codemark.id}...`);
// link-type codemarks are not posted on slack
if (codemark.type === 'link') {
this.log(`\t\tSkipping link-type codemark ${codemark.id}`);
this.numSkipped++;
return;
}
// for each codemark, we'll use the slack access token of the codemark creator to make our slack API calls
this.debug(`\t\tGetting user slack client for creator of codemark, ${codemark.creatorId}...`);
const slackClient = await this.getUserSlackClient(this.team, codemark);
if (!slackClient) {
this.numSkipped++;
return;
}
if (typeof slackClient === 'string') {
this.warn(`WARNING: Could not connect to slack on behalf of codemark ${codemark.id} creator ${codemark.creatorId}, skipping this codemark: ${slackClient}`);
this.numSkipped++;
return;
}
// get the Slack stream for this codemark's post on Slack
this.debug('\t\tGetting the Slack stream...');
const slackStream = await this.getSlackStream(codemark, slackClient);
if (IgnoreStreams.includes(slackStream.id)) {
this.debug(`\t\tIgnoring stream ${slackStream.id} per instruction`);
this.numSkipped++;
return;
}
const slackStreamInfo = {
id: slackStream.id,
name: slackStream.name,
latestText: slackStream.latest && slackStream.latest.text,
creator: slackStream.creator,
archived: slackStream.is_archived,
private: slackStream.is_private,
channel: slackStream.is_channel,
im: slackStream.is_im,
mpim: slackStream.is_mpim
};
this.debug(`\t\tSlack stream: ${JSON.stringify(slackStreamInfo, undefined, 5)}`);
// don't bother for slack streams that are archived
if (slackStream.is_archived) {
this.log('\t\tStream is archived, ignoring codemark');
this.numSkipped++;
return;
}
// ensure we have the appropriate stream for this codemark
this.debug(`\t\tEnsuring stream for this codemark, slack stream ${slackStream.id}...`);
const stream = await this.ensureStream(slackStream, slackClient, codemark);
// create a post pointing to this codemark on CodeStream, and then handle the actual replies
const postId = this.data.posts.createId().toString();
const replies = await this.processCodemarkReplies(codemark, postId, stream, slackClient, slackStream) || {};
const numReplies = Object.keys(replies).length;
const post = await this.createCodemarkPost(codemark, postId, stream, numReplies);
// update the codemark and markers to actually point to the post
await this.updateCodemark(codemark, post, replies);
await this.updateMarkers(codemark, post);
this.codemarksConverted++;
this.totalReplies += (numReplies || 0);
}
// create a post pointing to a given codemark, possibly also creating a stream for the post to go in
async createCodemarkPost (codemark, postId, stream, numReplies) {
const seqNum = this.bumpSeqNum(stream);
const postData = {
id: postId,
version: 1,
deactivated: false,
createdAt: codemark.createdAt,
modifiedAt: codemark.createdAt,
creatorId: codemark.creatorId,
codemarkId: codemark.id,
teamId: codemark.teamId,
streamId: stream.id,
seqNum,
numReplies
};
let post;
if (Commander.dryrun) {
this.log('\t\tWould have created post for codemark');
post = postData;
}
else {
this.log('\t\tCreating post for codemark...');
post = await this.data.posts.create(postData);
}
this.verbose(postData);
return post;
}
// ensure we have an appropriate CodeStream stream for a codemark
async ensureStream (slackStream, slackClient, codemark) {
let stream;
// cache the codemark stream for each Slack stream
if (this.streams[slackStream.id]) {
stream = this.streams[slackStream.id];
this.debug(`\t\tFound stream ${stream.id} for Slack stream ${slackStream.id}`);
return stream;
}
// for public channels on Slack, we'll just use the team stream on CodeStream
if (!slackStream.is_im && !slackStream.is_mpim && !slackStream.is_private) {
this.debug(`\t\tWill use team stream ${this.teamStream.id} for Slack stream ${slackStream.id}`);
this.streams[slackStream.id] = this.teamStream;
return this.teamStream;
}
// for private channels, groups, and DMs, make sure we have all the members,
// create "faux users" as needed
this.debug('\t\tEnsuring stream members...');
const members = await this.ensureStreamMembers(slackClient, slackStream);
// create the stream we need ... note that if it's a CodeStream DM, there might already be a
// stream with the necessary membership
const memberIds = members.map(m => m.id);
const type = (slackStream.is_im || slackStream.is_mpim) ? 'direct' : 'channel';
this.debug(`\t\tCreating ${type} stream with ${memberIds.length} members...`);
stream = await this.createStream(type, memberIds, slackStream.name, slackStream, codemark);
this.streams[slackStream.id] = stream;
return stream;
}
// ensure we have CodeStream users for all the necessary members of a stream
async ensureStreamMembers (slackClient, slackStream) {
// first make sure we have all the Slack users for the workspace we are attached to
await this.ensureSlackUsers(slackClient);
// get the membership of the Slack stream
this.debug('\t\tFetching Slack stream members...');
const memberIds = await this.slackApiPaginated(
slackClient,
'conversations.members',
{
channel: slackStream.id
},
'members',
4
);
this.debug(`\t\tMembers are: ${memberIds}`);
// for each member of the Slack stream, ensure we have a corresponding CodeStream user
this.debug(`\t\tEnsuring ${memberIds.length} users...`);
const members = [];
for (let memberId of memberIds) {
const slackUser = this.slackUsers[memberId];
if (!slackUser) {
this.warn(`WARNING: Slack user ${memberId} not found among slack users for workspace, will not be a member of stream created for ${slackStream.id}`);
}
else {
const user = await this.ensureUser(slackUser);
members.push(user);
}
}
return members;
}
// ensure we have a CodeStream user corresponding to the given Slack user
async ensureUser (slackUser) {
// first look through our existing users, looking for a match on the Slack ID, this could
// be an actual registered user, a "faux user", or a match by email
this.debug(`\t\t\tFinding match for Slack user ${slackUser.id}...`);
let userId = Object.keys(this.users).find(userId => {
const ourUser = this.users[userId];
return (
(ourUser.providerIdentities && ourUser.providerIdentities[`slack::${slackUser.id}`]) ||
(ourUser.externalUserId === `slack::${this.team.id}::${slackUser.team_id}::${slackUser.id}`) ||
(slackUser.profile && slackUser.profile.email && slackUser.profile.email === ourUser.email)
);
});
const user = userId && this.users[userId];
if (user) {
this.debug(`\t\t\tFound matching user ${user.id} for Slack user ${slackUser.id}`);
return user;
}
// no match found, so create a "faux user"
this.debug(`\t\t\tCreating faux user for Slack user ${slackUser.id}...`);
return await this.createFauxUser(slackUser);
}
// create a "faux user" - an unregistered user to stand in for the given Slack user
// this user doesn't appear as an actual member of the team in the CodeStream UI, but they
// have a CodeStream user ID for purposes of reply authorship
async createFauxUser (slackUser) {
const id = this.data.users.createId().toString();
const now = Date.now();
const userData = {
id,
version: 1,
deactivated: false,
createdAt: now,
modifiedAt: now,
username: slackUser.name,
externalUserId: `slack::${this.team.id}::${slackUser.team_id}::${slackUser.id}`,
creatorId: id,
teamIds: [this.team.id],
companyIds: [this.team.companyId]
};
if (slackUser && slackUser.profile && slackUser.profile.email) {
userData.email = slackUser.profile.email;
}
if (slackUser && slackUser.profile && slackUser.profile.real_name) {
userData.fullName = slackUser.profile.real_name;
}
if (slackUser && slackUser.tz) {
userData.timeZone = slackUser.tz;
}
let user;
if (Commander.dryrun) {
this.log(`\t\t\tWould have created faux user for slack user ${slackUser.id}`);
user = userData;
}
else {
this.log(`\t\t\tCreating faux user for slack user ${slackUser.id}...`);
user = await this.data.users.create(userData);
}
this.verbose(userData);
this.users[user.id] = user;
return user;
}
// create a stream of the given type with the given members
async createStream (type, memberIds, name, slackStream, codemark) {
let stream;
if (type === 'direct') {
// for direct streams, we might already have a stream with the same members
this.debug('\t\tLooking for match to direct stream...');
const streams = this.data.streams.getByQuery(
{
teamId: this.team.id,
memberIds: memberIds,
type: 'direct'
},
{
hint: StreamIndexes.byMembers
}
);
if (streams[0]) {
this.debug(`\t\tFound matching direct stream ${streams[0].id}`);
this.streams[slackStream.id] = streams[0];
return streams[0];
}
}
const id = this.data.streams.createId().toString();
const now = Date.now();
const streamData = {
id,
version: 1,
deactivated: false,
createdAt: now,
modifiedAt: now,
teamId: this.team.id,
type,
memberIds,
privacy: 'private',
creatorId: codemark.creatorId,
nextSeqNum: 1,
sortId: id
};
if (type === 'channel') {
streamData.name = name;
}
if (Commander.dryrun) {
this.log(`\t\tWould have created ${type} stream for codemark ${codemark.id}`);
stream = streamData;
}
else {
this.log(`\t\tCreating ${type} stream for codemark ${codemark.id}...`);
stream = await this.data.streams.create(streamData);
}
this.verbose(streamData);
this.streams[slackStream.id] = stream; // cache the stream as being the right one for the given Slack stream
return stream;
}
// ensure we have all the Slack users for the workspace attached to the gien team
async ensureSlackUsers (slackClient) {
if (this.slackUsers) {
this.debug('\t\tAlready have Slack users for this workspace');
return;
}
this.debug('\t\tFetching Slack users for this workspace...');
const users = await this.slackApiPaginated(
slackClient,
'users.list',
{ },
'members',
2
);
this.slackUsers = users.reduce((usersHash, user) => {
usersHash[user.id] = user;
return usersHash;
}, {});
this.debug(`\t\tFound ${Object.keys(this.slackUsers).length} users in this workspace`);
}
// each post we put in a stream has to have an ever-increasing sequence number, we'll keep
// track of that here
bumpSeqNum (stream) {
this.streamSeqNums[stream.id] = this.streamSeqNums[stream.id] || 0;
return ++this.streamSeqNums[stream.id];
}
// get the Slack stream this particular codemark was posted in
async getSlackStream (codemark, slackClient) {
// first check our cache
if (this.slackStreams[codemark.streamId]) {
this.debug(`\t\tFound Slack stream ${codemark.streamId}`);
return this.slackStreams[codemark.streamId];
}
// now call out to the Slack API
this.debug(`\t\tGetting Slack stream ${codemark.streamId}...`);
const response = await this.slackApi(
slackClient,
'conversations.info',
{
channel: codemark.streamId
}
);
this.slackStreams[codemark.streamId] = response.channel;
return response.channel;
}
// process all the replies to a given codemark, creating CodeStream posts as needed
async processCodemarkReplies (codemark, postId, stream, slackClient, slackStream) {
this.debug('\t\tProcessing codemark replies...');
try {
const replies = {};
const threadTs = codemark.postId.split('|')[1];
const messages = await this.slackApiPaginated(
slackClient,
'conversations.replies',
{
channel: codemark.streamId,
ts: threadTs
},
'messages',
3
);
for (let message of messages) {
if (message.ts !== threadTs) {
const reply = await this.processCodemarkReply(codemark, message, postId, stream, slackClient);
replies[`${codemark.streamId}|${message.ts}`] = reply.id;
}
}
return replies;
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
if (error.endsWith('thread_not_found') && slackStream.latest && slackStream.latest.text === 'This content can\'t be displayed.') {
this.log(`\t\tSkipping codemark ${codemark.id}, appears to have been deleted`);
return;
}
const slackStreamInfo = {
id: slackStream.id,
name: slackStream.name,
latestText: slackStream.latest && slackStream.latest.text,
creator: slackStream.creator,
archived: slackStream.is_archived
};
this.warn(`WARNING: Error getting replies to codemark ${codemark.id}, slack post ID ${codemark.postId}, slack stream ID ${slackStream.id}:${slackStream.name}, skipping this codemark: ${message}\nSLACK STREAM:\n${JSON.stringify(slackStreamInfo, undefined, 5)}`);
//this.warn(JSON.stringify(slackStream, undefined, 5));
}
}
// process a single codemark reply, creating a CodeStream post for it
async processCodemarkReply (codemark, message, postId, stream, slackClient) {
// first make sure we have all the Slack users for the workspace we are attached to
await this.ensureSlackUsers(slackClient);
// make sure we have an author for the reply
const slackUserId = message.user || (message.root && message.root.user);
if (!slackUserId) {
this.warn(`WARNING: Message ${message.ts} has no slack ser ID, cannot create reply`);
return;
}
const slackUser = this.slackUsers[slackUserId];
if (!slackUser) {
this.warn(`WARNING: Slack user ${slackUserId} not found among slack users for workspace, cannot create reply for message ${message.ts}`);
return;
}
const user = await this.ensureUser(slackUser);
// create a post for the reply
this.log(`\t\t\tProcessing reply ${message.ts} to codemark ${codemark.id}...`);
const id = this.data.posts.createId().toString();
const createdAt = parseInt(message.ts.split('.')[0], 10) * 1000;
const seqNum = this.bumpSeqNum(stream);
const postData = {
id,
version: 1,
deactivated: false,
numReplies: 0,
createdAt,
modifiedAt: createdAt,
streamId: stream.id,
parentPostId: postId,
teamId: this.team.id,
creatorId: user.id,
text: message.text,
seqNum
};
let reply;
if (Commander.dryrun) {
this.log('\t\t\tWould have created reply post to codemark');
reply = postData;
}
else {
this.log('\t\t\tCreating reply post to codemark...');
reply = await this.data.posts.create(postData);
}
this.verbose(postData);
return reply;
}
// update the codemark to point to the created post
async updateCodemark (codemark, post, replies) {
const numReplies = Object.keys(replies).length;
const op = {
$unset: {
providerType: true
},
$set: {
postId: post.id,
streamId: post.streamId,
numReplies
}
};
if (codemark.pinnedReplies) {
op.$set.pinnedReplies = [];
for (let i = 0; i < (codemark.pinnedReplies || []).length; i++) {
const replyId = replies[codemark.pinnedReplies[i]];
if (replyId) {
op.$set.pinnedReplies.push(replyId);
}
}
}
if (Commander.dryrun) {
this.log('\t\tWould have updated codemark');
}
else {
await this.data.codemarks.applyOpById(codemark.id, op);
this.log('\t\tUpdating codemark...');
}
this.verbose(op);
Object.assign(codemark, op.$set);
delete codemark.providerType;
}
// update markers associated with the codemark to point to the codemark's post and stream
async updateMarkers (codemark, post) {
const query = {
teamId: this.team.id,
codemarkId: codemark.id
};
const op = {
$set: {
postStreamId: post.streamId,
postId: post.id
}
};
if (Commander.dryrun) {
this.log('\t\tWould have updated markers');
}
else {
this.log('\t\tUpdating markers...');
await this.data.markers.updateDirect(query, op);
}
this.verbose(query);
this.verbose(op);
}
// update the team to no longer be a slack team
async updateTeam () {
const providerIdentities = this.team.providerIdentities.filter(pi => !pi.startsWith('slack'));
const op = {
$unset: {
'providerInfo.slack': true
},
$set: {
providerIdentities
}
};
if (Commander.dryrun) {
this.log('\tWould have updated team');
}
else {
await this.data.teams.applyOpById(this.team.id, op);
this.log(`\tUpdating team ${this.team.id}...`);
}
this.verbose(op);
Object.assign(this.team, op.$set);
delete this.team.providerIdentities;
}
// update the users on the given team by giving them a temporary password so they can sign in
async setUserPasswords () {
this.log('\t\tSetting temporary passwords...');
for (let userId in this.users) {
const user = this.users[userId];
if (user.isRegistered && !user.passwordHash) {
await this.setUserPassword(user);
}
else if (user.externalUserId) {
this.debug(`\t\t\tNOTE: user ${user.id} is "faux" user ${user.externalUserId}`);
}
else if (!user.isRegistered) {
this.debug(`\t\t\tNOTE: user ${user.id}:${user.email} is not registered`);
}
else if (user.passwordHash) {
this.debug(`\t\t\tNOTE: user ${user.id}:${user.email} has a password`);
}
}
}
// update the given user by giving them a temporary password so they can sign in
async setUserPassword (user) {
const passwordHash = await new PasswordHasher({
password: 'temp123'
}).hashPassword();
const updateData = { passwordHash };
if (Commander.dryrun) {
this.log(`\t\tWould have set temporary password for user ${user.id}:${user.email}`);
}
else {
await this.data.users.updateById(user.id, updateData);
this.log(`\t\tSetting temporary password for user ${user.id}:${user.email}...`);
}
this.verbose(updateData);
Object.assign(user, updateData);
}
// clear the slack credentials for all users on the team
async clearSlackCredentials () {
const query = {
teamIds: this.team.id
};
const op = {
$unset: {
[`providerInfo.${this.team.id}.slack`]: true
}
};
if (Commander.dryrun) {
this.log('\t\tWould have cleared all Slack credentials');
}
else {
this.log('\t\tClearing all Slack credentials');
await this.data.users.updateDirect(query, op);
}
this.verbose(query);
this.verbose(op);
}
// clear maintenance mode flags for all users on the team
async clearUserFlags () {
if (Commander.dryrun) {
this.log('\tWould have cleared user maintenance mode flags');
return;
}
else {
this.log('\tClearing user maintenance mode flags...');
}
const op = {
$unset: {
inMaintenanceMode: true
}
};
await this.data.users.updateDirect({ teamIds: this.team.id }, op);
await this.sendOpsToUsers(op);
}
// get a slack client object used to call out to the Slack API, we use the creator of the codemark
// for the access token for the slack client
async getUserSlackClient (team, codemark) {
if (IgnoreUsers.includes(codemark.creatorId)) {
this.log(`\t\tIgnoring user ${codemark.creatorId} by instruction`);
return;
}
// do we already have a slack client for this codemark's creator?
if (this.slackClients[codemark.creatorId]) {
this.debug(`\t\tAlready have Slack client for ${codemark.creatorId}`);
return this.slackClients[codemark.creatorId];
}
// do we already have the author of this codemark? (we should)
let user;
if (this.users[codemark.creatorId]) {
this.debug(`\t\tAlready have user ${codemark.creatorId}`);
user = this.users[codemark.creatorId];
if (user.deactivated) {
this.debug('\t\tNOTE: this user is deactivated');
}
}
// but just in case, find it in the database
if (!user) {
this.debug(`\t\tFetching user ${codemark.creatorId}`);
user = await this.data.users.getById(codemark.creatorId);
if (!user) {
return 'user not found';
}
}
// now make sure that user has a Slack access token for this team
this.users[codemark.creatorId] = user;
const userId = user.id;
const teamId = team.id;
const token = (
user.providerInfo &&
user.providerInfo[teamId] &&
user.providerInfo[teamId].slack &&
user.providerInfo[teamId].slack.accessToken
) || (
user.providerInfo &&
user.providerInfo.slack &&
user.providerInfo.slack.accessToken
);
if (!token) {
return 'user has no access token';
}
try {
// form the slack client object
this.debug(`\t\tMaking Slack client for ${codemark.creatorId}...`);
const client = new WebClient(token);
const authTestResponse = await client.auth.test();
if (authTestResponse && authTestResponse.ok) {
this.slackClients[userId] = client;
return client;
}
else {
return 'access token test failed';
}
}
catch (error) {
return `error testing access token to user ${user.email}: ${error}`;
}
}
// make a slack call
async slackApi (slackClient, moduleMethod, data) {
try {
this.debug(`\t\t\t\t\t\t\t(SLACK API: ${moduleMethod}...)`);
const [module, method] = moduleMethod.split('.');
this.verbose(data);
return await slackClient[module][method](data);
}
catch (error) {
throw (`slack call failed: ${error}`);
}
}
// make a paginated slack call, accumulating the results, with possible throttling
// based on the rate limiting tier of the slack API request
async slackApiPaginated (slackClient, moduleMethod, data, collection, tier) {
let responses = [];
do {
this.debug(`\t\t\t\t\t\t(SLACK API PAGINATED: ${moduleMethod}...)`);
const response = await this.slackApi(slackClient, moduleMethod, data);
if (collection) {
responses = [...responses, ...response[collection]];
}
else {
responses.push(response);
}
data.cursor = (response.response_metdata && response.response_metadata.next_cursor) || undefined;
const throttle = SLACK_TIER_THROTTLE_TIMES[tier] || 0;
if (data.cursor) {
await Wait(throttle);
}
} while (data.cursor);
return responses;
}
log (msg) {
Logger.log(msg);
}
warn (msg) {
Logger.warn(msg);
this.warnCount++;
if (Commander.dieonwarn && this.warnCount === Commander.dieonwarn) {
process.exit();
}
}
debug (msg) {
Commander.debug && Logger.log(msg);
}
verbose (obj) {
Commander.verbose && Logger.log(JSON.stringify(obj, undefined, 10));
}
}
class SlackReplyFetcher {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
await this.openMongoClient();
await this.openPubnubClient();
if (this.teamId === 'all') {
await this.processAllTeams();
}
else {
await this.processSingleTeam();
}
}
catch (error) {
Logger.error(error);
process.exit();
}
process.exit();
}
// process all Slack teams in our database! (this could take some time...)
async processAllTeams () {
const result = await this.data.teams.getByQuery(
{
'providerIdentities': /slack::/,
deactivated: false
},
{
stream: true,
overrideHintRequired: true,
sort: { _id: -1 }
}
);
let team;
do {
team = await result.next();
if (team) {
await this.processTeam(team);
await Wait(ThrottleTime);
}
} while (team);
result.done();
}
// process a single Slack team, as given by team ID
async processSingleTeam () {
const team = await this.data.teams.getById(this.teamId);
if (!team) {
throw 'team not found: ' + this.teamId;
}
return this.processTeam(team);
}
// process a single Slack team
async processTeam (team) {
Logger.log(`Processing team ${team.id}...`);
await new TeamReplyFetcher({ data: this.data, pubnub: this.pubnubClient }).fetchRepliesForTeam(team);
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// open a Pubnub client for broadcasting the changes
async openPubnubClient () {
this.pubnub = new PubNub(Object.assign({}, ApiConfig.getPreferredConfig().broadcastEngine.pubnub, { uuid: 'API-' + OS.hostname() }));
this.pubnubClient = new PubNubClient({ pubnub: this.pubnub });
await this.pubnubClient.init();
}
}
(async function() {
try {
const teamId = Commander.teamId;
if (!teamId) {
throw 'must provide teamId or all';
}
await ApiConfig.loadPreferredConfig();
await new SlackReplyFetcher().go({ teamId });
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-start-docker 0000775 0000000 0000000 00000000307 14607772627 0024517 0 ustar 00root root 0000000 0000000 #!/bin/bash
#desc# Start script for inside a docker container
cd $CS_API_TOP
echo "API docker start script is running"
bin/set-globals.js
bin/ensure-indexes.js build
bin/api_server.js --one_worker
codestream-server-api-15.4.0/api_server/bin/cs_api-tc-buildstep 0000775 0000000 0000000 00000005011 14607772627 0024511 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python3
# desc# Standard interface to CodeStream's TeamCity build process for the VS Code Extension
# Relevant documentation
# https://github.com/TeamCodeStream/teamcity_tools/blob/master/README.md#Structured_Builds
import os
import sys
sys.path.append(os.environ['DT_TOP'] + "/lib")
import sysUtils as su
import buildUtils
# su.printErr("this script needs to be customized")
# exit(1)
# 1. add filename extensions for all assets created (tgz, vsix, zip, ...)
assetExtensionList = ['tgz']
# 2. add any additional repos to be included (do not include primary repo)
additionalRepos = []
# 3. choose the build type (npm, intellijPlugin, custom)
buildType = buildUtils.buildType.npm
args = buildUtils.parseStandardBuildArgs()
buildOptions = {
'verbose': args.verbose,
'assetExtensionList': assetExtensionList,
'additionalRepos': additionalRepos
}
# 4. Set this if you want your applied git tags to be something other than 'v' (v1.2.3)
shortName = os.environ['CS_API_SHORT_NAME'] if 'CS_API_SHORT_NAME' in os.environ else "api"
buildOptions['gitTagPrefix'] = f"{shortName}-"
if args.verbose:
print("buildstep running with", args.action)
build = buildUtils.build(buildType, **buildOptions)
if not build:
su.printErr("could not get a build object")
exit(1)
exitCode = 0
# 5. Make sure the default actions apply, if not make changes as necessary.
# Use 'build.execCmd("command string")' to execute custom actions.
# https://github.com/TeamCodeStream/teamcity_tools/blob/master/README.project-build-types.md
if args.action == 'prep':
build.prep() # set teamcity variables / parameters
elif args.action == 'build':
# build.build() # complete build so services can be started
print("there's nothing to build")
elif args.action == 'citest':
# build.citest() # run the continuous integration test suite
exitCode = build.execCmd("cs_api-service start && sleep 2 && npm run test:ci", returnOnError=True)
build.execCmd("cs_api-service stop")
elif args.action == 'pack':
build.pack(csServerMonoRepoDir='api_server') # create asset info file and package assets
elif args.action == 'publish':
build.publish() # publish assets for distribution
elif args.action == 'tag':
build.tag() # update version info and apply tags
elif args.action == 'release':
build.release() # release assets to the general public
else:
su.printErr("invalid build args")
exit(1)
exit(exitCode)
codestream-server-api-15.4.0/api_server/bin/cs_api-update-plan.js 0000775 0000000 0000000 00000013336 14607772627 0024750 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// NOTE: THIS SCRIPT IS OBSOLETE
//desc// update payment plan and trial info in mongo and on Intercom
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Intercom = require('intercom-client');
const Indexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/companies/indexes');
const Strftime = require('strftime');
const UserIndexes = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes');
// need these collections from mongo
const COLLECTIONS = ['companies', 'teams', 'users', 'updatePlanLastRunAt'];
// throttle the updates so we don't stress mongo or intercom
const NO_UPDATE_THROTTLE_TIME = 10;
const UPDATE_THROTTLE_TIME = 1000;
const RUN_INTERVAL = 23 * 60 * 60 * 1000;
class PlanUpdater {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
if (await this.abortIfDoneAlready()) {
return;
}
await this.openIntercomClient();
await this.updateLastRunAt();
await this.process();
}
catch (error) {
this.logger.error(error);
throw error;
}
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// if another instance of this script already ran, just abort
async abortIfDoneAlready () {
const lastRunAt = await this.data.updatePlanLastRunAt.getByQuery({}, { overrideHintRequired: true });
if (lastRunAt && lastRunAt[0] && lastRunAt[0].lastRunAt > Date.now() - RUN_INTERVAL) {
return true;
}
}
// open an Intercom client to write to
async openIntercomClient () {
// NOTE: THE LINE BELOW MAKES THIS SCRIPT OBSOLETE, WE NO LONGER HAVE AN INTERCOM KEY
//this.intercomClient = new Intercom.Client({ token: ApiConfig.getPreferredConfig().telemetry.intercom.token });
}
// look for all companies that are in trial, and for each one, change its plan as needed
async process () {
const result = await this.data.companies.getByQuery(
{
plan: /^[0-9]+DAYTRIAL$/,
deactivated: false
},
{
stream: true,
hint: Indexes.byPlan
}
);
let company;
do {
company = await result.cursor.next();
if (company) {
await this.processCompany(company);
}
} while (company);
result.done();
}
// for this company, see if its trial is expired, if so, put it on either:
// TRIALEXPIRED or FREEPLAN ... companies that were on 14DAYTRIAL go directly to TRIALEXPIRED,
// in other words, there is no FREEPLAN for those companies ... companies on the old 30DAYTRIAL
// can go on FREEPLAN if number of members if <= 5
async processCompany (company) {
const now = Date.now();
const date = Strftime('%Y-%m-%d %H:%M:%S.%L');
if (company.trialEndDate > now) {
this.logger.log(`${date}: Company ${company._id} ("${company.name}") is still in trial`);
return await this.wait(NO_UPDATE_THROTTLE_TIME);
}
// Unused - if reserrected, we should go through the LicenseManager
const users = await this.getRegisteredUsers(company);
let newPlan;
if (users.length > 5 || company.plan !== '30DAYTRIAL') {
newPlan = 'TRIALEXPIRED';
}
else {
newPlan = 'FREEPLAN';
}
this.logger.log(`${date}: Updating company ${company._id} ("${company.name}") to ${newPlan}`);
await this.updateIntercom(company, newPlan);
await this.updateMongo(company, newPlan);
await this.wait(UPDATE_THROTTLE_TIME);
}
// get the registered users in a company
async getRegisteredUsers (company) {
const teams = await this.data.teams.getByIds(company.teamIds);
const teamIds = teams.map(team => team._id.toString());
return await this.data.users.getByQuery(
{
teamIds: {$in: teamIds},
isRegistered: true,
deactivated: false
},
{
hint: UserIndexes.byTeamIds,
fields: ['_id']
}
);
}
// update the plan in mongo
async updateMongo (company, newPlan) {
try {
await this.data.companies.updateDirect(
{ _id: this.data.companies.objectIdSafe(company._id) },
{ $set: { plan: newPlan } }
);
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
this.logger.warn(`Update to company ${company._id} failed: ${message}`);
}
}
// update the plan in intercom, we'll make sure the trial dates are updated as well
async updateIntercom (company, plan) {
const update = {
company_id: company._id,
plan,
custom_attributes: {
trialStart_at: Math.floor(company.trialStartDate/1000),
trialEnd_at: Math.floor(company.trialEndDate/1000)
}
};
try {
await this.intercomClient.companies.update(update);
}
catch (error) {
const message = error instanceof Error ? error.message: JSON.stringify(error);
this.logger.error(`Unable to update company ${company._id} on Intercom: ${message}`);
}
}
// wait this number of milliseconds
wait (time) {
return new Promise(resolve => {
setTimeout(resolve, time);
});
}
// update when this script was last run so other running instances don't try
async updateLastRunAt () {
await this.data.updatePlanLastRunAt.updateDirect(
{ },
{ $set: { lastRunAt: Date.now() } },
{ upsert: true}
);
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new PlanUpdater().go();
}
catch (error) {
this.logger.error(error);
process.exit(1);
}
process.exit(0);
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-update_company_plan.js 0000775 0000000 0000000 00000007400 14607772627 0026553 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// NOTE: THIS SCRIPT IS OBSOLETE
//desc// manually set a payment plan for a company in mongo and on Intercom
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Intercom = require('intercom-client');
const Commander = require('commander');
// need these collections from mongo
const COLLECTIONS = ['companies'];
const parseDate = function(date) {
const timestamp = Date.parse(date);
if (isNaN(timestamp)) {
console.error('Invalid date: ' + date);
process.exit(1);
}
return timestamp;
};
Commander
.option('-c, --companyId ', 'CodeStream ID of the company whose plan to change')
.option('-p, --plan ', 'Name of plan to change to')
.option('-s, --start ', 'Set planStartDate to this date (best to put the date in quotes)', parseDate)
.option('-t, --trial-start-date ', 'Set trialStartDate to this date (best to put the date in quotes)', parseDate)
.option('-e, --trial-end-date ', 'Set trialEndDate to this date (best to put the date in quotes)', parseDate)
.parse(process.argv);
if (!Commander.companyId) {
Commander.help();
}
class PlanUpdater {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
await this.openIntercomClient();
await this.process();
console.log('DONE');
}
catch (error) {
this.logger.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// open an Intercom client to write to
async openIntercomClient () {
// NOTE: THE LINE BELOW MAKES THIS SCRIPT OBSOLETE, WE NO LONGER HAVE AN INTERCOM KEY
//this.intercomClient = new Intercom.Client({ token: ApiConfig.getPreferredConfig().telemetry.intercom.token });
}
// change the company's plan in both mongo and on Intercom
async process () {
await this.updateMongo();
await this.updateIntercom();
}
// update the plan in mongo
async updateMongo () {
const set = {};
if (Commander.plan) {
set.plan = Commander.plan;
}
if (Commander.start) {
set.planStartDate = Commander.start;
}
if (Commander.trialStartDate) {
set.trialStartDate = Commander.trialStartDate;
}
if (Commander.trialEndDate) {
set.trialEndDate = Commander.trialEndDate;
}
try {
await this.data.companies.updateDirect(
{ _id: this.data.companies.objectIdSafe(this.companyId) },
{ $set: set }
);
}
catch (error) {
const message = error instanceof Error ? error.message : JSON.stringify(error);
console.warn(`Update to company ${this.companyId} failed: ${message}`);
}
}
// update the plan in intercom, we'll make sure the trial dates are updated as well
async updateIntercom () {
const update = {
company_id: this.companyId,
plan: this.plan
};
try {
await this.intercomClient.companies.update(update);
}
catch (error) {
const message = error instanceof Error ? error.message: JSON.stringify(error);
console.error(`Unable to update company ${this.companyId} on Intercom: ${message}`);
}
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new PlanUpdater().go({
companyId: Commander.companyId,
plan: Commander.plan
});
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_api-vars 0000775 0000000 0000000 00000000122 14607772627 0023063 0 ustar 00root root 0000000 0000000 #!/bin/bash
#desc# Display sandbox environment variables
dt-vars -p CS_API "$@"
codestream-server-api-15.4.0/api_server/bin/cs_convert_to_codemarks.js 0000775 0000000 0000000 00000010545 14607772627 0026200 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Commander = require('commander');
Commander
.option('-d, --destroy', 'Destroy the old properties while creating the new ones')
.parse(process.argv);
const COLLECTIONS = ['posts', 'markers', 'codemarks'];
class Converter {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
const markers = await this.data.markers.getByQuery(
{ },
{ overrideHintRequired: true }
);
await Promise.all(markers.map(async marker => {
await this.convertMarker(marker);
}));
}
catch (error) {
this.logger.error(error);
process.exit();
}
process.exit();
}
async convertMarker (marker) {
let post;
if (!marker.providerType && marker.postId) {
post = await this.data.posts.getById(marker.postId);
}
let codemark;
if (!marker.codemarkId) {
codemark = await this.makeCodemark(marker, post);
await this.updateMarker(marker, post, codemark);
}
if (post && codemark && !post.codemarkId) {
await this.updatePost(post, codemark);
}
if (Commander.destroy) {
if (post) {
await this.removeFromPost(post);
}
await this.removeFromMarker(marker);
}
}
async makeCodemark (marker, post) {
const codemark = {
teamId: marker.teamId,
streamId: marker.postStreamId || (post && post.streamId),
postId: marker.postId,
markerIds: [marker.id.toString()],
fileStreamIds: [marker.fileStreamId || (marker.codeBlock && marker.codeBlock.streamId) || marker.streamId],
deactivated: marker.deactivated,
createdAt: marker.createdAt,
modifiedAt: marker.modifiedAt,
creatorId: marker.creatorId
};
if (marker.providerType) {
codemark.providerType = marker.providerType;
}
else if (marker.postId.match(/\|/)) {
codemark.providerType = 'slack';
}
if (post) {
['type', 'color', 'status', 'title', 'assignees', 'text'].forEach(prop => {
if (post[prop]) {
codemark[prop] = post[prop];
}
});
if (!post.type) {
codemark.type = 'comment';
}
}
if (marker.numComments && marker.numComments > 1) {
codemark.numReplies = marker.numComments - 1;
}
return this.data.codemarks.create(codemark);
}
async updateMarker (marker, post, codemark) {
const codeBlock = Object.assign({}, (post && post.codeBlocks && post.codeBlocks[0]) || {}, marker.codeBlock || {});
if (codeBlock) {
Object.assign(marker, codeBlock);
if (marker.location) {
marker.locationWhenCreated = marker.location;
}
delete marker.markerId;
delete marker.preContext;
delete marker.postContext;
delete marker.remotes;
}
marker.fileStreamId = codemark.fileStreamIds[0];
marker.postStreamId = marker.postStreamId || (post && post.streamId);
marker.codemarkId = codemark.id.toString();
await this.data.markers.update(marker);
}
async updatePost (post, codemark) {
post.codemarkId = codemark.id.toString();
await this.data.posts.update(post);
}
async removeFromPost (post) {
const unset = {
$unset: {
codeBlocks: true,
commitHashWhenPosted: true
}
};
if (post.type !== 'comment') {
unset.$unset.text = true;
}
await this.data.posts.updateDirect(
{ id: this.data.posts.objectIdSafe(post.id) },
unset
);
}
async removeFromMarker (marker) {
const unset = {
$unset: {
codeBlock: true,
location: true,
streamId: true,
numComments: true,
commitHash: true,
preContext: true,
postContext: true,
itemId: true
}
};
await this.data.markers.updateDirect(
{ id: this.data.markers.objectIdSafe(marker.id) },
unset
);
}
// open a mongo client to do the dirty work
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new Converter().go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_deactivate.js 0000775 0000000 0000000 00000001654 14607772627 0024100 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const Deactivator = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/lib/util/deactivator');
Commander
.option('-t, --team ', 'Deactivate team')
.option('-r, --repo ', 'Deactivate repo')
.option('-u, --user ', 'Deactivate user')
.option('--dtu, --deactivate-teamless-users', 'Deactivate teamless users when done')
.parse(process.argv);
if (!Commander.team && !Commander.repo && !Commander.user) {
console.warn('team, repo, or user required');
process.exit();
}
let options = {
teamIdOrName: Commander.team,
repoId: Commander.repo,
userIdOrEmail: Commander.user,
deactivateTeamlessUsers: Commander.deactivateTeamlessUsers
};
(async function() {
try {
await new Deactivator().go(options);
process.exit();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_delete 0000775 0000000 0000000 00000001146 14607772627 0022612 0 ustar 00root root 0000000 0000000 #!/bin/sh
ARGS=$@
doit () {
$(dirname "$0")/cs_delete.js $ARGS
}
if [[ $1 = "-h" || $1 = "--help" ]]; then
doit;
exit;
fi
makeSure () {
while true; do
read -p "Really? You're sure? Type 'YES' or 'NO' " yn
case $yn in
YES ) echo "OK. Here goes..."; doit; break;;
NO ) exit;;
* ) echo "Please answer YES or NO.";;
esac
done
}
while true; do
read -p "Do you really want to do this deletion? This action can not be undone! " yn
case $yn in
[Yy]* ) makeSure; break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
codestream-server-api-15.4.0/api_server/bin/cs_delete.js 0000775 0000000 0000000 00000001751 14607772627 0023227 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const Deleter = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/lib/util/deleter');
Commander
.option('-t, --team ', 'Delete team')
.option('-r, --repo ', 'Delete repo')
.option('-u, --user ', 'Delete user')
.option('--dtu, --delete-teamless-users', 'Delete teamless users when done')
.parse(process.argv);
if (!Commander.team && !Commander.repo && !Commander.user) {
console.warn('team, repo, or user required');
process.exit();
}
if (Commander.repo) {
console.warn('Deleting by repo is no longer supported');
process.exit();
}
let options = {
teamIdOrName: Commander.team,
//repoId: Commander.repo,
userIdOrEmail: Commander.user,
deleteTeamlessUsers: Commander.deleteTeamlessUsers
};
(async function() {
try {
await new Deleter().go(options);
process.exit();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_delete_test_data 0000775 0000000 0000000 00000000334 14607772627 0024640 0 ustar 00root root 0000000 0000000 #!/bin/bash
[ -z "$CS_API_MONGO_URL" ] && echo "CS_API_MONGO_URL not defined" && exit 1
echo mongo $CS_API_MONGO_URL $CS_API_TOP/bin/cs_delete_test_data.js
mongo $CS_API_MONGO_URL $CS_API_TOP/bin/cs_delete_test_data.js
codestream-server-api-15.4.0/api_server/bin/cs_delete_test_data.js 0000664 0000000 0000000 00000000602 14607772627 0025246 0 ustar 00root root 0000000 0000000 // mongo script to remove all data created by API server unit tests,
// which put a special flag into all data created called _forTesting
// make eslint happy
/* globals db */
const Collections = [
'companies',
'teams',
'repos',
'streams',
'posts',
'markers',
'markerLocations',
'users'
];
for (let collection of Collections) {
db[collection].remove({_forTesting: true});
}
codestream-server-api-15.4.0/api_server/bin/cs_dev_secrets.js 0000775 0000000 0000000 00000000636 14607772627 0024274 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
const shellescape = require('shell-escape');
const { readVaultDevSecrets } = require('../../shared/server_utils/dev_secrets');
async function main() {
const env = await readVaultDevSecrets();
for (const key in env) {
console.log(`export ${key}=${shellescape([env[key]])}`);
}
}
(async () => {
try {
await main();
} catch (e) {
console.error('Error', e);
}
process.exit();
})();
codestream-server-api-15.4.0/api_server/bin/cs_merge_teams.js 0000775 0000000 0000000 00000001664 14607772627 0024260 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const TeamMerger = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/teams/team_merger');
Commander
.option('-f, --from ', 'Teams to merge')
.option('-t, --to ', 'Team to merge to')
.parse(process.argv);
if (!Commander.from) {
console.warn('At least one "from" team is required');
process.exit();
}
let fromTeamIds = Commander.from.split(',').map(id => id.toLowerCase());
let toTeamId;
if (Commander.to) {
toTeamId = Commander.to.toLowerCase();
}
else {
toTeamId = fromTeamIds[0];
fromTeamIds.splice(0, 1);
}
if (fromTeamIds.length === 0) {
console.warn('Provide at least two teams');
process.exit();
}
const options = { fromTeamIds, toTeamId };
(async function() {
try {
await new TeamMerger().go(options);
process.exit();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_msteams_bot_manifest_creator.js 0000664 0000000 0000000 00000002305 14607772627 0027700 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// utility script to create a manifest file for an MS Teams bot
/* eslint no-console: 0 */
'use strict';
const Commander = require('commander');
const fs = require('fs');
Commander
.option('-b, --botId ', 'Bot id')
.option('-e, --env ', 'environment (pd, prod, brian)')
.parse(process.argv);
if (!Commander.botId && !Commander.env) {
Commander.help();
}
(async function () {
try {
var text = fs.readFileSync(process.env.CSSVC_BACKEND_ROOT + '/api_server/etc/msteamsbot/template/manifest.json', 'utf8');
text = text.replace(/{{botId}}/g, Commander.botId);
if (Commander.env === 'prod') {
// don't put any env for prod
text = text.replace(/{{env}}/g, '');
}
else {
text = text.replace(/{{env}}/g, `-${Commander.env}`);
}
let json = JSON.parse(text);
let outputDir = `${process.env.CSSVC_BACKEND_ROOT}/api_server/etc/msteamsbot/dist`;
if (!fs.existsSync(outputDir)) fs.mkdirSync(outputDir);
outputDir += '/' + Commander.env;
if (!fs.existsSync(outputDir)) fs.mkdirSync(outputDir);
fs.writeFileSync(`${outputDir}/manifest.json`, JSON.stringify(json, null, 4));
process.exit();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/cs_set_reporting_group.js 0000775 0000000 0000000 00000007351 14607772627 0026067 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
'use strict';
// set, update or delete the reportingGroup attribute of
// a company
/* eslint no-console: 0 */
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Mongodb = require('mongodb');
const MongoClient = Mongodb.MongoClient;
const Commander = require('commander');
Commander
.option('-n, --dryrun', 'dry run only')
.option('-c, --company ', 'Company Name')
.option('-i, --company-id ', 'Company ID')
.option('-r, --report', 'Report company, don\'t change DB')
.option('-g, --group ', 'string used to group companies for reporting or "null" to remove the attribute')
.parse(process.argv);
if ((!Commander.company && !Commander.companyId) ||
(Commander.company && Commander.companyId)) {
console.log('Only one value, Company or Company ID is required');
Commander.outputHelp();
process.exit(1);
}
if ((!Commander.group && !Commander.report) ||
(Commander.group && Commander.report)) {
console.log('Only one value, Group or Report is required');
Commander.outputHelp();
process.exit(1);
}
if (Commander.dryrun) {
console.log('dry run mode - I won\'t actually do anything to the db');
}
const queryCollection = async function(csDb, collection, query) {
// console.log('finding', collection, query);
try {
return await csDb.collection(collection).find(query).toArray();
}
catch(error) {
console.log('error', error);
process.exit(1);
}
};
(async function() {
let db;
// connect to mongo
// console.log('connecting...');
try {
await ApiConfig.loadPreferredConfig();
db = await MongoClient.connect(ApiConfig.getPreferredConfig().storage.mongo.url, ApiConfig.getPreferredConfig().storage.mongo.tlsOptions);
}
catch (error) {
console.log('mongo connect error', error);
process.exit(1);
}
let csDb = db.db(ApiConfig.getPreferredConfig().storage.mongo.database);
// find matching companies
let query = Commander.companyId ?
{ _id : new Mongodb.ObjectId(Commander.companyId) } :
{ name : Commander.company };
let companies = await queryCollection(csDb, 'companies', query);
// no companies found
if(companies.length === 0) {
console.log('no matching companies found');
process.exit(1);
}
// multiple companies found - give useful feedback
if(companies.length > 1 || Commander.report) {
if(companies.length > 1)
console.log('multiple companies found');
for (let companyIdx in companies) {
let thisCompany = companies[companyIdx];
let memberOIDs = thisCompany.memberIds.map(function(id) { return Mongodb.ObjectId(id);});
let members = await queryCollection(csDb, 'users', {_id: {$in: memberOIDs}});
let membersToDisplay = [];
for (let membersIdx in members) {
let thisMember = members[membersIdx];
let displayString = thisMember.fullName + ' (' + thisMember.searchableEmail + ')';
membersToDisplay.push(displayString);
}
let companyDisplayObject = {
Name: thisCompany.name,
CompanyId: thisCompany._id,
reportingGroup: thisCompany.reportingGroup,
Members: membersToDisplay
};
console.log('Company', companyIdx, ':', companyDisplayObject, '\n');
}
console.log('select a unique CompanyId and rerun the script with -i');
process.exit(1);
}
// one company found
let operation = Commander.group === 'null' ?
{ $unset: { reportingGroup: '' } }:
{ $set: { reportingGroup: Commander.group.toLowerCase() } };
console.log('updating company', companies[0].name, 'with ID', companies[0]._id, 'using', operation);
if (Commander.dryrun) {
process.exit(0);
}
try {
await csDb.collection('companies').updateOne(
{_id: Mongodb.ObjectId(companies[0]._id)},
operation);
}
catch(error) {
console.log('update error', error);
process.exit(1);
}
process.exit(0);
})();
codestream-server-api-15.4.0/api_server/bin/cs_update_intercom.js 0000775 0000000 0000000 00000005025 14607772627 0025145 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// NOTE: THIS SCRIPT IS OBSOLETE
// one-time update of intercom with all the trial data in the mongo database
/* eslint no-console: 0 */
'use strict';
const MongoClient = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/mongo/mongo_client');
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const Intercom = require('intercom-client');
const COLLECTIONS = ['teams'];
class IntercomUpdater {
// main entry point
async go (options = {}) {
try {
Object.assign(this, options);
this.logger = this.logger || console;
await this.openMongoClient();
await this.openIntercomClient();
await this.process();
console.log('DONE');
}
catch (error) {
this.logger.error(error);
process.exit();
}
process.exit();
}
// open a mongo client to read from
async openMongoClient () {
this.mongoClient = new MongoClient({ collections: COLLECTIONS });
try {
await this.mongoClient.openMongoClient(ApiConfig.getPreferredConfig().storage.mongo);
this.data = this.mongoClient.mongoCollections;
}
catch (error) {
throw `unable to open mongo client: ${JSON.stringify(error)}`;
}
}
// open an Intercom client to write to
async openIntercomClient () {
// NOTE: THE LINE BELOW MAKES THIS SCRIPT OBSOLETE, WE NO LONGER HAVE AN INTERCOM KEY
//this.intercomClient = new Intercom.Client({ token: ApiConfig.getPreferredConfig().telemetry.intercom.token });
}
async process () {
const result = await this.data.teams.getByQuery({deactivated: false}, { stream: true, overrideHintRequired: true, sort: { _id: -1 } });
let team;
do {
team = await result.cursor.next();
if (team && team.plan) {
await this.processTeam(team);
}
} while (team);
result.done();
}
async processTeam (team) {
const update = {
company_id: team._id,
plan: team.plan,
custom_attributes: {
trialStart_at: Math.floor(team.trialStartDate/1000),
trialEnd_at: Math.floor(team.trialEndDate/1000)
}
};
console.log(`Updating team ${team._id}:"${team.name}" with`, update);
try {
await this.intercomClient.companies.update(update);
await new Promise(resolve => {
setTimeout(resolve, 300);
});
}
catch (error) {
console.warn('****** UNABLE TO UPDATE TEAM ' + team._id, error);
await new Promise(resolve => {
setTimeout(resolve, 100);
});
}
}
}
(async function() {
try {
await ApiConfig.loadPreferredConfig();
await new IntercomUpdater().go();
}
catch (error) {
console.error(error);
process.exit();
}
})();
codestream-server-api-15.4.0/api_server/bin/ensure-indexes.js 0000775 0000000 0000000 00000011341 14607772627 0024232 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// run this script to ensure the appropriate indexes are present in your mongo database,
// the api server will not run properly without these indexes in place
// you can also run this script to drop all the indexes
//
// to drop, supply "drop" as a command line argument
// to build, supply "build" as a command line argument
//
// you can supply both, that will drop and build (rebuild from scratch)
'use strict';
/* eslint no-console: 0 */
const AllModuleIndexes = {
companies: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/companies/indexes'),
teams: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/teams/indexes'),
repos: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/repos/indexes'),
streams: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/streams/indexes'),
posts: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/posts/indexes'),
markers: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/markers/indexes'),
codemarks: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/codemarks/indexes'),
codemarkLinks: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/codemarks/codemark_link_indexes'),
reviews: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/reviews/indexes'),
codeErrors: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/code_errors/indexes'),
users: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/indexes'),
signupTokens: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/signup_token_indexes'),
messages: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/broadcaster/indexes'),
msteams_conversations: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/msteams_conversations/indexes'),
msteams_states: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/msteams_states/indexes'),
msteams_teams: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/msteams_teams/indexes'),
reposByCommitHash: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/repos/repo_by_commit_hash_indexes'),
gitLensUsers: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/gitlens_user_indexes'),
newRelicOrgs: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/newrelic_comments/new_relic_org_indexes'),
entities: require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/entities/indexes')
};
const AllFinished = {
indexes: 0,
drops: 0,
indexed: 0,
dropped: 0
};
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const MongoClient = require('mongodb').MongoClient;
const Drop = process.argv.find(arg => arg === 'drop');
let Build = process.argv.find(arg => arg === 'build');
if(!Drop && !Build) {
Build = 'build';
}
const DropIndexes = async function(db, collection) {
if (!Drop) { return; }
AllFinished.drops++;
const collectionObj = db.collection(collection);
try {
await collectionObj.dropIndexes();
}
catch (error) {
return console.log('error dropping indexes on collection', collection, error);
}
AllFinished.dropped++;
};
const BuildIndexes = async function(db, collection) {
if (!Build) { return; }
const moduleIndexes = AllModuleIndexes[collection];
const collectionObj = db.collection(collection);
for (let indexName in moduleIndexes) {
const index = moduleIndexes[indexName];
AllFinished.indexes++;
console.log('ensuring index on collection', collection, index);
try {
await collectionObj.createIndex(index);
}
catch (error) {
return console.log('error', error);
}
AllFinished.indexed++;
console.log('indexed collection, index', collection, index);
}
};
const DoCollection = async function(db, collection) {
await DropIndexes(db, collection);
await BuildIndexes(db, collection);
};
function WaitUntilFinished() {
console.log('waiting to finish', AllFinished);
if( (AllFinished.drops === AllFinished.dropped) && (AllFinished.indexes === AllFinished.indexed)) {
console.log('we\'re done');
process.exit(0);
}
setTimeout(WaitUntilFinished, 2000);
}
(async function() {
let mongoClient, db;
try {
const mongoUrl = ApiConfig.configIsMongo()
? ApiConfig.options.mongoUrl
: (await ApiConfig.loadPreferredConfig()).storage.mongo.url;
const mongoTlsOpts = ApiConfig.configIsMongo()
? ApiConfig.options.mongoTlsOpts
: (await ApiConfig.loadPreferredConfig()).storage.mongo.tlsOptions;
mongoClient = await MongoClient.connect(mongoUrl, Object.assign({ useNewUrlParser: true }, mongoTlsOpts));
db = mongoClient.db();
}
catch (error) {
console.log('mongo connect error', error);
process.exit(1);
}
for (let collection in AllModuleIndexes) {
await DoCollection(db, collection);
}
WaitUntilFinished();
})();
codestream-server-api-15.4.0/api_server/bin/hash_password.js 0000775 0000000 0000000 00000000665 14607772627 0024150 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
// run this script to generate a password hash based on an input password
/* eslint no-console: 0 */
const PasswordHasher = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/modules/users/password_hasher');
const Password = process.argv[2];
new PasswordHasher({ password: Password }).hashPassword((error, hash) => {
if (error) {
console.error(JSON.stringify(error));
}
else {
console.log(hash);
}
});
codestream-server-api-15.4.0/api_server/bin/set-globals.js 0000775 0000000 0000000 00000002346 14607772627 0023515 0 ustar 00root root 0000000 0000000 #!/usr/bin/env node
/* eslint no-console: 0 */
const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
const MongoClient = require('mongodb').MongoClient;
const globalsToSet = [ "serviceGatewayAuth" ];
(async function () {
let mongoClient, db;
try {
console.log("set-globals...");
const mongoUrl = ApiConfig.configIsMongo()
? ApiConfig.options.mongoUrl
: (await ApiConfig.loadPreferredConfig()).storage.mongo.url;
const mongoTlsOpts = ApiConfig.configIsMongo()
? ApiConfig.options.mongoTlsOpts
: (await ApiConfig.loadPreferredConfig()).storage.mongo.tlsOptions;
mongoClient = await MongoClient.connect(mongoUrl, Object.assign({ useNewUrlParser: true }, mongoTlsOpts));
db = mongoClient.db();
for (const global of globalsToSet) {
const item = await db.collection('globals').findOne({ tag: global });
if (!item) {
await db.collection('globals').insertOne({ enabled: true, tag: global });
} else {
if (!item.enabled) {
await db.collection('globals').updateOne({ tag: global }, { $set: { enabled: true } });
}
}
console.log(`${global} set to true`);
}
process.exit(0);
} catch (error) {
console.log('mongo connect error', error);
process.exit(1);
}
})();
codestream-server-api-15.4.0/api_server/config/ 0000775 0000000 0000000 00000000000 14607772627 0021430 5 ustar 00root root 0000000 0000000 codestream-server-api-15.4.0/api_server/config/config.js 0000664 0000000 0000000 00000003426 14607772627 0023240 0 ustar 00root root 0000000 0000000 // api configuration
'use strict';
const Fs = require("fs");
const StructuredConfigFactory = require(process.env.CSSVC_BACKEND_ROOT + '/shared/codestream_configs/lib/structured_config');
const customConfigFunc = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/custom_config');
// The restartRequired() method is meant to compare two configurations (prior &
// current) in order to determine if a 'restart' (defined by the application) is
// required.
//
// These two configuration params refer to the customzed configs if a
// customConfig option is used, otherwise they refer to the native configs.
//
// The return value can be any type and will be passed back to the caller of the
// restartRequired() method (usually it's boolean, but it doesn't have to be).
//
// function customRestartFunc(priorConfig, currentConfig) {
// }
var Config;
// Pre-set config variables in the process environment
// This block should not fire when running in a legacy sandbox
if (!Config && !process.env.CS_API_SANDBOX) {
const EnvDataFile = `${process.env.CSSVC_BACKEND_ROOT}/api_server/config/${process.env.CSSVC_ENV}.json`;
if (Fs.existsSync(EnvDataFile)) {
for (const [key, value] of Object.entries(JSON.parse(Fs.readFileSync(EnvDataFile, 'UTF-8')))) {
if (!process.env[key]) {
process.env[key] = value;
}
}
process.env['CS_API_SANDBOX'] = process.env.CSSVC_BACKEND_ROOT;
process.env['CS_API_LOGS'] = `${process.env.CSSVC_BACKEND_ROOT}/log`;
}
}
Config = Config || StructuredConfigFactory.create({
configFile: process.env.CS_API_CFG_FILE || process.env.CSSVC_CFG_FILE,
mongoUrl: process.env.CSSVC_CFG_URL,
mongoTlsCAFile: process.env.CSSVC_MONGO_CLIENT_CERT_FILE,
showConfigProperty: 'apiServer.showConfig',
// customRestartFunc,
customConfigFunc
});
module.exports = Config;
codestream-server-api-15.4.0/api_server/config/dev-secrets.json 0000664 0000000 0000000 00000004666 14607772627 0024563 0 ustar 00root root 0000000 0000000 [
"AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY",
"BROADCAST_ENGINE_PUBNUB_BLUE_KEY_PUBLISH_KEY",
"BROADCAST_ENGINE_PUBNUB_BLUE_KEY_SECRET_KEY",
"BROADCAST_ENGINE_PUBNUB_BLUE_KEY_SUBSCRIBE_KEY",
"BROADCAST_ENGINE_PUBNUB_GREEN_KEY_PUBLISH_KEY",
"BROADCAST_ENGINE_PUBNUB_GREEN_KEY_SECRET_KEY",
"BROADCAST_ENGINE_PUBNUB_GREEN_KEY_SUBSCRIBE_KEY",
"CSSVC_AUTH_SECRET",
"EMAIL_DELIVERY_SENDGRID_APIKEY",
"ENVIRONMENT_GROUP_SECRETS_REQUEST_AUTH",
"INTEGRATIONS_ASANA_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_ASANA_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_BITBUCKET_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_BITBUCKET_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_DEVOPS_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_DEVOPS_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_GITHUB_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_GITHUB_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_GITLAB_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_GITLAB_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_JIRA_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_JIRA_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_LINEAR_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_LINEAR_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_MSTEAMS_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_MSTEAMS_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_MSTEAMS_CLOUD_BOT_APP_ID",
"INTEGRATIONS_MSTEAMS_CLOUD_BOT_APP_PASSWORD",
"INTEGRATIONS_NEWRELICGROK_CLOUD_API_KEY",
"INTEGRATIONS_NEWRELIC_CLOUD_ACCOUNT_NUMBER",
"INTEGRATIONS_NEWRELIC_CLOUD_BROWSER_INGEST_KEY",
"INTEGRATIONS_NEWRELIC_CLOUD_LICENSE_INGEST_KEY",
"INTEGRATIONS_NEWRELIC_CLOUD_TELEMETRY_ENDPOINT",
"INTEGRATIONS_NEWRELIC_CLOUD_WEBVIEW_AGENT_ID",
"INTEGRATIONS_NEWRELIC_CLOUD_WEBVIEW_APP_ID",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_NEW_RELIC_CLIENT_ID",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_NEW_RELIC_CLIENT_SECRET",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_PASSWORD_KEY",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_USER_SERVICE_SECRET",
"INTEGRATIONS_SLACK_CLOUD_APP_CLIENT_ID",
"INTEGRATIONS_SLACK_CLOUD_APP_CLIENT_SECRET",
"INTEGRATIONS_SLACK_CLOUD_APP_ID",
"INTEGRATIONS_SLACK_CLOUD_APP_SIGNING_SECRET",
"INTEGRATIONS_TRELLO_CLOUD_APP_CLIENT_ID",
"STORAGE_MONGO_URL",
"TELEMETRY_SEGMENT_TOKEN",
"TELEMETRY_SEGMENT_WEB_TOKEN",
"TELEMETRY_ENDPOINT",
"UNIVERSAL_SECRETS_TELEMETRY",
"UNIVERSAL_SECRETS_TELEMETRY",
"SHARED_SECRETS_COMMENT_ENGINE",
"SHARED_SECRETS_COMMENT_ENGINE_ALT",
"SHARED_SECRETS_CONFIRMATION_CHEAT",
"SHARED_SECRETS_COOKIE",
"SHARED_SECRETS_MAIL",
"SHARED_SECRETS_SIGNUP_FLOW_JWT",
"SHARED_SECRETS_SUBSCRIPTION_CHEAT"
]
codestream-server-api-15.4.0/api_server/config/local.json 0000664 0000000 0000000 00000003604 14607772627 0023420 0 ustar 00root root 0000000 0000000 {
"QUEUING_ENGINE_AWS_SQS_REGION": "us-east-1",
"BROADCAST_ENGINE_PUBNUB_PREFERRED_KEY_COLOR": "blue",
"CS_API_AUTH_ORIGIN": "https://auth.cdstrm.dev/no-auth",
"CS_API_CALLBACK_ENV": "local",
"CS_API_DONT_WANT_AWS": "true",
"CS_API_MARKETING_SITE_URL": "https://newrelic.com/codestream",
"CS_API_PORT": "12078",
"CS_API_PUBLIC_API_URL": "https://localhost.newrelic.com:12079",
"CSSVC_EMAIL_SUPPRESS_EMAILS": "true",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_CREDENTIALS_SERVICE_HOST": "https://staging-credential-service.nr-ops.net",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_GRAPHQL_HOST": "https://nerd-graph.staging-service.nr-ops.net",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_IDP_SERVICE_HOST": "https://idp-service.staging-service.nr-ops.net",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_LOGIN_SERVICE_HOST": "https://staging-login.newrelic.com",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_NEW_RELIC_REGION": "us01",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_ORG_SERVICE_HOST": "https://staging-organization-service.nr-ops.net",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_SIGNUP_SERVICE_HOST": "https://signup-processor.staging-service.newrelic.com",
"INTEGRATIONS_NEW_RELIC_IDENTITY_CLOUD_USER_SERVICE_HOST": "https://staging-user-service.nr-ops.net",
"INTEGRATIONS_NEWRELICGROK_CLOUD_API_URL": "https://nerd-completion.staging-service.nr-ops.net/v1/chat/completions",
"SERVICE_ENV": "staging",
"SHARED_GENERAL_NEW_RELIC_API_URL": "https://staging-api.newrelic.com",
"SHARED_GENERAL_NEW_RELIC_LANDING_SERVICE_URL": "https://landing.staging-service.newrelic.com",
"SHARED_GENERAL_NEW_RELIC_SEC_API_URL": "https://nrsec-workflow-api.staging-service.newrelic.com",
"SHARED_GENERAL_RUN_TIME_ENVIRONMENT": "local",
"US_VAULT_HOST": "https://vault-prd1a.r10.us.nr-ops.net:8200",
"BASE_VAULT_PATH": "containers/teams/codestream/services/codestream-server/base/",
"DEV_SECRETS_FILE": "api_server/config/dev-secrets.json"
}
codestream-server-api-15.4.0/api_server/docs/ 0000775 0000000 0000000 00000000000 14607772627 0021113 5 ustar 00root root 0000000 0000000 codestream-server-api-15.4.0/api_server/docs/README.md 0000664 0000000 0000000 00000000666 14607772627 0022402 0 ustar 00root root 0000000 0000000 # API Documentation Library
- [Integrations](integrations/README.md)
- [RabbitMQ Notes](rabbitmq.md)
- [Version Compatibility Matrix](version-matrix.md)
- [Unified Config File](unified-cfg-file.md) - Details on the unified structured config file used for backend services.
- [TeamCity build information](build.md)
- [On-Prem Development](onprem-development.md) - Setting up a sandbox for onprem development.
- [Pubnub Access](pubnub.md)
codestream-server-api-15.4.0/api_server/docs/build.md 0000664 0000000 0000000 00000002277 14607772627 0022544 0 ustar 00root root 0000000 0000000 # API Server Build Information
TeamCity Project: [API](http://redirector.codestream.us/builds/api)
## GitFlow and Brief Overview
Read the [Build Overview](https://redirector.codestream.us/ops/workflows) page on the Ops Wiki site.
## Assets
| Type | Desc |
| --- | --- |
| info | asset info file |
| tgz | tarball of repo following an **npm install** for x86_64 |
| docker image | docker images are only stored in dockerhub repos |
| Asset Env | Asset | Branch | Location |
| --- | --- | --- | --- |
| dev | tgz | develop | [Asset server (dev/api-server)](http://assets.codestream.us/artifacts/dev/api-server/) |
| prod | tgz | master | [Asset server (prod/api-server)](http://assets.codestream.us/artifacts/prod/api-server/) |
| dev | docker image | develop | Published to dockerhub as [teamcodestream/api-onprem-dev](https://cloud.docker.com/u/teamcodestream/repository/docker/teamcodestream/api-onprem-dev) |
| prod | docker image | master | Published to dockerhub as [teamcodestream/api-onprem](https://cloud.docker.com/u/teamcodestream/repository/docker/teamcodestream/api-onprem) |
## Build Documentation
[TeamCity build documentation](https://github.com/TeamCodeStream/teamcity_tools/tree/master/docs)
codestream-server-api-15.4.0/api_server/docs/integrations/ 0000775 0000000 0000000 00000000000 14607772627 0023621 5 ustar 00root root 0000000 0000000 codestream-server-api-15.4.0/api_server/docs/integrations/README.md 0000664 0000000 0000000 00000010765 14607772627 0025111 0 ustar 00root root 0000000 0000000 # Integrations
## 3rd Party Services
- [Asana](asana.md)
- [Azure DevOps](azure-devops.md)
- [Bitbucket](bitbucket.md)
- [Github](github.md)
- [Gitlab](gitlab.md)
- [Intercom](intercom.md)
- [Jira](jira.md)
- [Linear](linear.md)
- [MS Teams](msteams.md)
- [New Relic](newrelic.md)
- [Slack](slack.md)
- [Trello](trello.md)
## OAuth & SSO Handshake
### OAuth Without SSO
1. User presses the **Connect to X** (3rd party) button in the webview.
1. Webview notifies the **LSP agent** which uses the IDE’s internal apis to
hand-off a CS API request (URL) to a browser. This request initiates the
handshake (`/web/provider-auth/`).
1. Browser makes CS API request which returns a 302 redirect link to the 3rd
party site (referred to as the _**auth-me**_ link for purpose of this
write-up). The redirect link contains:
- CodeStream state object
- 3rd party App ID
- Callback URL (which we will call the provider-token URL) which also
includes a state object containing the codestream runTime environment
- App-specific scopes being requested
1. Browser makes the _**auth-me**_ request to the 3rd party site which ensures
the request callback is valid & prompts the user to approve access to its
service.
1. The 3rd party responding to the _**auth-me**_ request sends the callback URL
(`/no-auth/provider-token` URL) as a 302 redirect to the browser along with:
- A short-lived authorization code
- The state object passed in from CodeStream
1. The browser, responding to the callback redirect, makes a
`/no-auth/provider-token` request of the CS API which then:
1. Validates the CodeStream state object
1. Makes a server-to-server request from the CS API to the 3rd party service
using the short-lived authorization code in order to get an **access token**.
1. Stores the **access token** in the CodeStream database
1. Responds with a 302 redirect for the browser to request a **Successful
Auth** page on the marketing site.
1. Publishes a broadcast message containing the **access token** on the
user’s **me-channel** which the client's LSP agent receives. The client is
now authorized and has the necessary token to place API requests directly
to the 3rd party service.
### OAuth With SSO
Similar to non-SSO OAuth with a bit more apparatus.
1. User presses Sign-Up with X button in webview
1. CodeStream client generates a sign-up token (a guid)
1. This sign-up token is included in the initial CS API request URL (step 2
above)
1. This sign-up token is also embedded within the state object within the
auth-me link (step 3 above)
1. Once the IDE api’s pass the request off to a browser, the lsp agent enters a
loop, polling the CS API waiting for the server to say that access is granted
(the /no-auth/check-signup request).
1. The provider-token request (step 6 above) decodes and parses the embedded
state object, including the sign-up token in the server-to-server call.
1. Once the CS API server completes the provider-token request, the polling
check-signup request will respond successfully (access allowed) with a data
package containing the me-object which includes the access token.
### Shared Development Apps
As CodeStream supports any number of development environments, including
ephemeral ones, we have a mechanism to accommodate using one OAuth callback for
multiple environments. In these cases, the state object that’s included in the
callback URL (step 3 above) will contain the name of the CodeStream environment
(sharedGeneral.runTimeEnvironment). Our nginx configuration will parse this
state out of the URL’s query string and redirect it to the correct environment.
This does not work in all cases. Some 3rd party apps are restrictive in their
operation.
In these cases, the apiServer.authOrigin URL will be used to construct the
callback URL in lieu of the apiServer.publicApiUrl value.
### Internal Environments
For environments that are only accessible behind the VPN, some integrations,
notably **Slack** & **MS Teams**, which provide for server-to-server calls
initiated by the 3rd party (such as **Interactive Components** in slack), will
need to go through a proxy.
In these cases a predetermined URL will be configured in the CodeStream
integration connector app on the 3rd party’s system, to send the requests to the
CodeStream proxy which, in turn, will proxy them to the correct environment.
Today this is the same hostname as is used for the Shared Development Apps above
(this should change).
codestream-server-api-15.4.0/api_server/docs/integrations/asana.md 0000664 0000000 0000000 00000002404 14607772627 0025226 0 ustar 00root root 0000000 0000000 # Asana Integration
## Setting up an OAuth App
* You will need an Asana account (obviously) and be logged into it.
* Go to the [Manage Developer Apps](https://app.asana.com/0/my-apps) page. If
that link doesn't work directly, the [asana OAuth
documentation](https://developers.asana.com/docs/oauth) includes this link to
[register an application](https://app.asana.com/-/account_api). On that page,
you should find a link to **Manage Developer Apps**.
* Create a new app and fill in the basic information
* On the left nav bar, select Configure > OAuth
* Add a redirect URL for your API installation. Similar to this:
```
https://api.codestream.com/no-auth/provider-token/asana
```
* Copy the **Client ID** and **Client Secret** from that page and add them as a
sub-section of the **integrations** section in your CodeStream configuration
file, like this:
```
{
"integrations": {
"asana": {
"cloud": {
"appClientId": "******************",
"appClientSecret": "********************************",
"disabled": false
}
}
}
}
```
## References
* https://developers.asana.com/docs/authentication-basics
* https://app.asana.com/0/developer-console
codestream-server-api-15.4.0/api_server/docs/integrations/azure-devops.md 0000664 0000000 0000000 00000002067 14607772627 0026574 0 ustar 00root root 0000000 0000000 # Azure DevOps Integration
## Creating an OAuth App
The DevOps integration requires an app associated with an Azure DevOps
organization. The OAuth app will have an owner.
* Login to Azure DevOps as a member of the organization and go to
https://app.vsaex.visualstudio.com/me
* `Applications & Services > Create New Application`
* Complete all fields. Make sure the **Authorization Callback URL** is set to
something similar to this:
```
https://api.codestream.com/no-auth/provider-token/azuredevops
```
* Select the following scopes:
- Identity (read)
- Work Items (read and write)
* Copy the App ID and Client Secret (_not_ the App Secret) and add them to the
CodeStream server configuration file in the **integrations** section like so:
```
{
"integrations": {
"devops": {
"cloud": {
"appClientId": "******************",
"appClientSecret": "******************************** VERY LONG *******************",
"disabled": false
}
}
}
}
```
codestream-server-api-15.4.0/api_server/docs/integrations/bitbucket.md 0000664 0000000 0000000 00000002726 14607772627 0026126 0 ustar 00root root 0000000 0000000 # Bitbucket Ingtegration
## Setting up an OAuth Consumer
* You will need a Bitbucket (Atlassian) account (obviously) and be logged into it.
* On the Profile drop down, select the workspace you want to own the OAuth
consumer app or select "All Workspaces" if you want to create a new workspace.
On the "All Workspaces" page you can create a new one if you want.
* Once you've selected your workspace, select `Settings > OAuth Consumers > Add Consumer`
* Complete the fields, of note is the callback URL which should look something like this:
```
https://api.codestream.com/no-auth/provider-token/bitbucket
```
* For scopes, select the following:
- Account: email, read
- Issues: read, write
- Workspace Membership: read
- Projects: read
- Repositories: read, write
- Pull Requests: read, write
- Snippets: read
* A key and secret will be generated for the consumer app. These values should
be added to your CodeStream server configuration file in the **integrations**
section with `appClientId` holding the **Key** and the `appClientSecret`
holding the **Secret**.
```
{
"integrations": {
"bitbucket": {
"cloud": {
"appClientId": "******************",
"appClientSecret": "********************************",
"disabled": false
}
}
}
}
```
## Reference Links
* https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/
codestream-server-api-15.4.0/api_server/docs/integrations/github.md 0000664 0000000 0000000 00000001745 14607772627 0025434 0 ustar 00root root 0000000 0000000 # Integration with Github
## Github Cloud (github.com)
### SSO / OAuth
Github.com SSO support for CodeStream implements the standard OAuth procedure
which requires a dedicated OAuth application on Github.com with the appropriate
settings.
1. Login to Github.com
1. `Settings > Developer Settings > OAuth Apps > Create New App`
1. Set the **Authorization Callback URL** to for your API's URL (similar to
`https://api.codestream.us/no-auth/provider-token/github`)
1. Generate a client secret and add both the Client ID and secret to the
secrets database.
1. Add the credentials to the CodeStream server's config file in the
`intergrations` section.
```
{
"integrations": {
"github": {
"cloud": {
"appClientId": "***************************",
"appClientSecret": "*************************",
"disabled": false
}
}
}
}
```
codestream-server-api-15.4.0/api_server/docs/integrations/gitlab.md 0000664 0000000 0000000 00000002060 14607772627 0025403 0 ustar 00root root 0000000 0000000 # Gitlab Integration
## Gitlab Cloud (gitlab.com)
### Gitlab / OAuth
The Gitlab.com integration for CodeStream implements the standard OAuth
procedure which requires a dedicated OAuth application on Gitlib.com with the
appropriate settings.
1. Login to Gitlab.com
1. `Edit Profile (User Settings) > Applications`
1. Set the **Redirect URL (callback URL)** to for your API's URL (similar to
`https://api.codestream.us/no-auth/provider-token/gitlab`)
1. Select the Confidential, Expire AccessTokens and api Scopes checkboxes
(all three).
1. Create the application. Take note of the **Application ID** and
**Secret**.
1. Add the credentials to the CodeStream server's config file in the
`intergrations` section.
```
{
"integrations": {
"gitlab": {
"cloud": {
"appClientId": "***************************",
"appClientSecret": "*************************",
"disabled": false
}
}
}
}
```
codestream-server-api-15.4.0/api_server/docs/integrations/images/ 0000775 0000000 0000000 00000000000 14607772627 0025066 5 ustar 00root root 0000000 0000000 azure-app-reg-branding-and-props.png 0000664 0000000 0000000 00002631466 14607772627 0033701 0 ustar 00root root 0000000 0000000 codestream-server-api-15.4.0/api_server/docs/integrations/images PNG
IHDR &
d DǏ@ BiCCPICC Profile (c``H,(aa``+)
rwRR` \\T Qk .ȬSrXh_K^4zZ qJrAQ c\^R b "E@G3@t{
a r )@ [' I<
w o%VhʢG`*x%(108 N!*30X1 g#4xvܴ ( o,iF6O?20b`[] 4@! ^D eXIfMM * > F( i N x &
d ASCII Screenshotd pHYs % %IR$ iTXtXML:com.adobe.xmp
2660
3110
Screenshot
T iDOT 2 ( 2 2 @ IDATx`TUڇw@)*`w
fC"bֵg"v%.
If?3 d&$a;ssl<獕:<Ŭowڀ%ʌ G`~Y E7%,QmQռZ-g͛K9w~?M?TܾnGEU ܿ CcGDw_cƌVZٍ7hO=TAԹ+y]wJ+dwy=Cx
K\" ;v5{l?<ǖ-[Z۶mz͛7O6̎?x釟0oi,.~Fjf}>
NѾ}{kM6~?;D @
Q`W.+} uF @;YӮڮZ,kᩧZmb,ߞn"צ w%nݯ`3&^hUpnlΜ9;曛ڻƍgKj~CO-t
63g{л_'6~G{뭷o4v֡C+..w}79D @ .e.~]
SJ& N`B\B @ x SCz湉Ԕvvh :r%Wߴ|j-mB\p) i*hB4Uƿ?Wn*N?7Ö&;:ǎk]vY k/͛ۺEY[5y.go@~y聇좋.ӯjOs=秗:zN{
u_?:n &@ @ Ts ,&;X|
Ry6o[ڬ ~Xt|x<}6uyZ~ꫯn_|=#zlw{챇͚5n: 4OO?r^OAj@ H'F Yk;`_lozXuYUi NVRR_
x~РmݖZ S[: }' {>쳔SCL}n@ @PL(+U !5>Mw\!0!3s@ @n~$$\q>x;H mu_ۧwimwPmA{ri/T=S n}l@M|S& nv6 6tZL}킋.5 :xM0akvMN:ҁAYMg2ظ7ǥ?cvy\p~zםw7hy _=uE3 ,
#9-w @
xa(^ 4hgsji;v˔3u?jNOYX;7P
FTUn&;wnCGh`P6Ve5~?l-j`2ŮZ~Ǐ7
RSw}sG ڍĢ[oīƹnN:j6
$S0P03.={ԩSoIeoV;{n)\'r>oG @
YBz͡}>\I H&}eI5]ؗ# `dXI>T >U!17}̑ut5~~g)Y{ɷ6 w*|S&6ؤ lVLs= W_~N>g&@ @ Us 4@m'켏r7O߭Y˸/L KfL
ٌ4(u__4`
[o[jdOnwq&k,m˶nj.!w}gw_5}
ZP0S}J m@ @ B 0P\ޱpmժ Q \
R{@ @ȘPE\#O+|@➁G$
Q07Ϫ.8r 3;^x=qd=@ѪU~˵F/cBqqP'N
J-׃?؏Hh<]t =z;8t)0Ap
SNۧ~p"sڷoӕ&fΜZCeVYeyH?٩ Ni]=RYwuE~ ֻwo[{ "uG&RZ:!#kRI+n&h R8hU ugsLPû}hT?ǯ[oٴiZ]T~RˮR/O?-~63Ꚇc{7m>4iiFham cn6h= }Q[c5R [5\mmm k[uUmM7&x≅FvHUMu-j:Թ=6G wqEK gԽҫi[mB{=]oZ7:C)Sf:gw'Zµ1c *,N4oU? @&d+z NXe. tm\+ܟ
hv?\K]NpM}{?* m#<ҔTmSj Dc
#3Vv D=Yv5&ԽgLgUȦ 9OPʢ&l&Ite]G xYd) wiuf\3iƷ+J˖-ڑ5ڄ+u.uXۗ_~>V{sG&2v%#Y}7[mzf[g
vm=SPUzi\uhϵ]/ >7@ @ 0a˘Ɗk(dwN& ,N !*UCo.K"OJ\x7Evֳčf;#x ~Y؍7$pU(àY:HHOUs=+lǍkvgV눬˫6|~qoNf2vNsT/ܶzo?nvٵ^kݺw6_ nv曽MvO_#i~ +/ům֩U_z%;w[Sh˯_|Mviyuxglv3}m5, Xm>? _ͧ j ֥ug[V[4vs=篳:6 > |cV
yMuW^nNjRpNl{jC}'ٵkW9DJ40OI?/{߽Q}]xӍSGkL$d{Oa\~ks;æ;֍Pk ?2vi~Z<@c.vWCO{ƌ W93矯o{쵇%*hߪj YB ,]8'9 P*8V?h͟9 Sғt|,\{svﷁm#F_us?LmaDlF>AZ|~
^]n.~vz>|MJiP
A4Ԗm4АXm|
;L46.uQ 5Hʸq O>d}&jʶ~{~ :qr@=1YtTIuӠ$EmǏ_^=>HChp7x>4z@OcAet]T_
>}ѡC_u`4/B
^뒒XASLDU*Z ?J &42lCfmfῗpx?W_&:M {TL(6 ACڿz{ J3f?i}kz4WmE7
AAΔ8( olW^eO?6@ @ lLF)D5A4lrY֚! @^ (09aqDP,
ܘ vϋY";!Le|7m췛{?#_/ܾ`xF5huߤ&5ao|\5k}l5hPj+j&,HLPLa λ4"R(jr=̋tM>K.w9(5=tȠQRO=s*jcB=ϭN8}]?:oy |cwXL_ tAFuʥj<2ƿ30!FBHI# ~۴|c]z饩, ~{~yٞ{Q}םws'Z?ߚhp𮠓O>AKe2xsn_~~}v%SV[me\{MJ M^Yge>lm.Bm!la2x߉VU;( @ @ BGrs`cU@ UݺVξUj{W5 Q5ݚ[!ì]9gWS\BrkqsY6s˚`%t`j4hNB9|G4Շ!'}plWS`2<:(a9Z/{{ ;:ƫ=+
8s!jӾ#,;kp
LPxmDm*ʼP4Ldᚩy|ж
}ʊ -:W B(qt;nrTAVs[lϴUT@ npt~ ewjذa̴
^Pg2Uxeuq}]sMXAk']}t (v
ҽ ꯢl ;Ｓy=o3xAAdL+E@ ( .0.a!nqs z ]_;Aoѣ Mmo?c{>>ܶ
oP(TdcGg pQGّG 7c >;@ W값_vɧt9;i?Os9Ǵ*5j~p-M7SC˯`GcǎZ4:>K40Aǟx_w?D
}O?F_ } !}Μ9֫W/|y HШ=mڵI&F+ξu]5Bxԅ}w5huY`w~efCh}5!*uxm~ұ_O/^swUG#G2>_ qSN#(}E]d7>+#&}Q0* h qXXdoyRwy~١CUR+f9 }syyg
YղE;f6D40a1!sG8d{,fo6 ~.r{Wm~ģ^*U:twz_nvWrUW=;|x(]w}O>d(!*S d#@`B6J *_{OK&`@>vs5͵i jnZ=^I4oW#[ZEi q~
_BX%tV<+Ayw9X g<=cBuf߿_z)xBYQXj礬
pu.:.؞k` ֯yYm::a(ճ
LPk]'
\ YK4ڽFWUd OuW :kvzDFLIk|L#+V
kDA_& F{^kY+_=k9(F ՞j^µV;l3ePGt&%k]9ei+GAo},. Jbԩ
^#BzzGuWѽ [ `Zvt!(+K/YՊ䒥it^|3 C~[G ذ( :Nk;
UO 2& !GȯjS@ g } to~:1Uvۺt5#k: u=IY0®N[PRM&;گ9vT~dfTۥ[߁R ;rH?Kt7G &QO>v7N~{[S ݧ[n~0J{F
:_?o@VK/Ov qk~$C 5m`O?bX JϞ=펻 G e"M\kV#-EG,oӏ?{!_˶ү >,v>3}E?jz`c]h$x=u2MԭEndLhߪLf ( ,@Q3}k#w=5t-ӮwVFrW6JPVQ u5O| P |65hFW`
Ns |j:^謮qFe6 8wW'ku>jRԙ:uWGy#Fد>s?
BPF\umjs,))5:z
LI\)Z:w!l<_#+I긭j%\FrW;YWv=[L %_QyM#GoGWp@RZ+c_QG5x:kf5z Z 2Su _@>E`e, JCW DKw^
X~ +Q5PLnאP@ B|>lGjD=˶bŮQ]+p]nT FO'_0eNm$=kQ}9dL {=?SdW']5o~vح 7ji5s-
ˣ XZpyEyK.#=?(kp}7SX=h`s=~TL߭2-7߲$5k{5ΫaYBvwTֆZc\}~Z/k=_ J{{h`a>}
:N8)A>vnu~(=۷o8lw=rH[
uM ov\}Ä_hȌl#Bь ;϶9<{݊LMl3&<͂*_{"koט Ѭ?h
Q~ghf3g /+'sCD >'02@ @ ZL Y|}];/-zljsgZɎ]x5നubjwx6Ø = 3?`}GWW_AѢvFG; Gn~O>Ğx6lK`2j@9:N2C|B
eXHuK. 2h#u8WQTG;g ^Bgl4Ȓ:|@ U_K]S\hІ)˴l[(e
f:NQ=}:G] B 2z1c%;爐MgMBVSQk}to*C4cysnj㭳
LifQDzQ=:+˂&rj:JzE ' L9M&kGy$Z]}j^w>Z/@ @ lLV*?uukPC&Nc1B @ ѾtȂmۍV/5'AHNkVg¾t=lЮOnB*n]M)n;j/%Qx1vWԺR-?Trd{GkW^yB~v!Qm`a;nM4
53=g5*FWqZÆ
8 ?B67? Z)W֣;ֈ;CtVCJ .Hydڠ:' PF4ծ1wrL /z SNaZE(wt{@]~L/ﺻ~QC]ӿ/#}guf&0!HOLC#Gnv?RR|~^ de\&dr=oޘ 8M|kdjv-8la =|=][ o'-5ҕ^50!wBoU @&* @Y|]vwϕ?~hͿz^|k}Gu ^m龙>Qދm椗{/KtO?ԏ ڵ q3ڑ=tVglaԻ5OzlK`B]RLD;Vkx %止ߣE U4po] ܹe{uu ,:j{v
L_|EͶn;իBm:Gk(B4g>: ;EӧON
5h0i,!Ѽh`»k0ک_A'دLP:g1> Q`2p/]V;Ｓd{H%{4dD>P&lgTh~_4D@/( 5]p=C=j{~k[e @M&$Sc%נnDJ+p
QK@ Z U!r 3&34p[}s=R[ƻl G+1
6vPh~ ;*VX19
>l^xamU0u0y:]| g /"u]` 4?-@y?LJOf\кj ֨%jx.jP#噧11E L.~WSz O?O>?P:u||c}n]LxO UHG;n>zEtP
ᯍ{>ms-0A&ԈA?j&jtqo)0C+{vdꐟ^\#}{}NLP {V_su7gmvuY-&z[=\S] 'le}k LPNkwaj-AtEu w]h`B4(w?m ͬ)0>Ux#
# @NXƧܷJXڍ2"cYyuc[r[ѵ%hZslX|k_:ֵ A9:B~t 5uv&][˫DGSuӈ~ϒ F,?( m`:! s1agΜ)DWճ
LжGFlvϡ|WVm!B2٨
p6
t!g7| >t;P 0mڴJCzM#m^kv*F|uT4jTLũZ+c%!#|P [n9{@R&4td{ҵkWe]|B
5TAB-X }V{~@@>[\ 5]~g@ @ Qu\`aJ.0V @ H&Y]=(rlX#f+֎<6k\5s!ʜб}7>/kL*5+U ;u/>.4xzQm}u$t? 6tM
%=JCM7/ 8pr)~zؐa~t;]ve~VL5 NJcQG $N: _jhpĢ&DGO :HhgMN:7~ /uES&/A;jٻ :k$LfA:xM0fwy_z)BB=_^}uvigÚP4/XX8 ;d ^~3U60A+=+0Damײ>5c&(U#=:' 25 &k0imYQ߇Ч Fn"= ({^==m`BUX# 0>jl Y taFԆ
EE=*Z7eoYs|hk֪Zȵk Əb^z9!9OܪU+?ߏ,MNO@A
ꐯ`us~w}g{챇[|f\cZW`R j(녂 ~݃_mZkez֠EO*=R((gM9٣iePE2D 2 uT?! u PP~/(5B$lʤQvp|m}*@ E@ 1@#"fz?k:a[mmw_寿Fk8M7em % wAۑ{}>׃Gg1Ah;I㔓O_~Y}۷]cr Scmf7J><9ȣ}BX_|{ﾗJ;J2ydߠ :j ~V^ep%7I~!L)YC(_jWb _52рBmm (=!0a Qf5
n+/b驎RG(T N?^"0A Խ-?O.7=?R6eP0OG;b_tE\KC_ .vsw_
9k{Wi*k:P+,hvYT*\rP{"kYY}jE5Cuh뭷;ϟ=s>U/ߝO|bzJ gN~tm`BU ; h. *`W|k Y_ɊۯbFr^T2麶ˊ3mޏvv93m$`b65srRjN:5cȑ/ ¿nj
mD:٪m# JKK}t+H lK`¶nk
駟4#s:}j3L碎
m`PvO?Z{v0ah'Nn:gkՙY9ORgzƍC&]vꪫ@
Y] a{m_<>8#Kc^ԎA`ETm:/Z cǎNݽ֠jgk}@PCpM^z%_\cZWGwhCV
UN3G^].3>PX ~/ lj^DߛﮮO=xM7UMUv}'h\g@ VlsXICݟj{۔~юoYuj *IWWC+$G_2-AYTU{hufm# amwkN#]!G[g)cGe1A#Ĝڽ/7^h$JvK)_)Rək x'_x>@3-[o=裾_ݺwKm
LpۊWۅj`=Ĕq d$¦
Lбn] =b˷Y^ظӚVm-33gP[n
nV~ɮ{KC_S=ҽnvu }q&r{E3}|3Tc^ȥޗ\P6?ӽD*|>+9=
X&]G_|Ej 0g~sαVZ!60!ߪ @&Ū @e-,^G,^i1iVܺ HɻUjc֩o/[0541] :WҭJ,>oJE欿>h߶&/MꨳFWauJW1c؏?|vY ?a_G y2hڻNj+6`7*LaTqW
\ڷoEU LгYmjOTT=P?~4|:g0d߳ A^F*묳~:ڑ9ȯz_;M Ҷː\Cv:V]ݳ&gܢ5?S`sW[Bd',@x?i\uGE:XjT{FWJȰWlėp0d15В=\~8Fs|Q?ea=k}Wuױo&=ﯮ(}+<~~x+0A#Ν;oॺʢ|! 6p{SY)f̘StJ\k;
& 06#tV6y@ ]!s|Uc n$G\J$(`aU tK7튖p55pk
4Z <5uxkK[ŭ%b`cGe1AuP!X^}dgy;_u5s옱veU[Q6/Ӈh`#&2?lFy5L>VKSG|5އi!D] G#rgN`wy#}j20ځ>(@#G {1E秦-^{WK8Ͻ\jM _ma
}Mu2jՎ7 w_܁ zX
\~.\\PR B6}Tc=f_;fC?Uw.moUg) P 5۰@ Q`
-\Ёkj)w.ki Nh+-qcr >{Fl6u^נ?qeWPc(_^6lքgrQ~
8ȴ >tV :VilK` U]̡I|w- %
s(t =^ ++qzAzT U#&(
Eu5P?_Ne P'|eHRW`BW+L/Y4@ʜ9sRw}5菊~ R*&; 9pyf7BeLp 2&ٕ: 2&?(OG܋& n;>7_g<6q?6q(}_~ 7g0 7ۗo_ozh"keF -Y5h]vXk"3XV[meG qD ng} θkaj UC ξa{F
7üW^.ba0˿}]z饾~z rYgٮ}t-~%]GIPC<6|\? sjKlIĘclֳ6ȔA)zUjT~iYgUFy}u'y?Y79ʓv2w\ZNZ?n^v3=Ƹ7F^v'>? StC^H)0#:_J4U`¶nkW]}vlG T%]Ǿ}V
Kߵ ۽+ү3a~:Sq@?wp=rӧxs7J {~>h=>p#0ǟx \ dzw`?t o̙5ǔA߉VEc@ E\X@ rH &q5j5
<oVXbmװ2'eeV?ӦZǵW>AqS>l'o/9_q vM;?b?:謮 mW^չ[hN{W og Y^Y q !z0e9*sCcF!רstieJPQwywW{:D}M x-H40H] ^A*:h9c|ht4+AM뇎=9Z -SFWwWzZyF=:z+ Buy嗫
d~ zD&k꼮(2x RS)))_T ' 3 L8qrGAQu *Y4(0EA t @ IDATR+=ɴ|
^WK\2RkQIbq#ј@QM@WWQÞ5rPFLkDt#eZ7:/Mtu´A9_;t :ڇu^⣇*)E SC)>=r}v:ꡇ\ P,k~ttMS*MF҃?߾TUt/+5eOù%Rl r'r:~}~kwCܗu cQarVe/A @ &D5F @ h7k-U
N'c\y|\k6.ǻ9ݷW-\*';'\{ouWk}7CAE_~N6]}mԎNΪ:sgSԮ3fvlɇutԮڮAZ_۩svP]8ٔ\u8w_i]&>~e{ U=9X} .cB&& { `mڹԽ JUQ"\( Vz̘:ny-IU{f}= `q ?jFQ%(UW% @V&dJ XKA`נP@A,\ϴ7.{uu]A土lXX렚̼^͒ ?}͚OSd{nY@ @ @Lhz@ hve= (rKD
!hn D9-fIgf:9̓% (m '3$>|S_ÇRR@ @ @ wr7c @ C`,mǵ++ -Lp>bWYlM/-ӼVbnÄ^.Uۯw Abӗ_p!X˖-g^=V}v &4yC 1֭ǰDQQkpHFeɘА @ M`M6u 뉆:HN å?z (WB20HU6>#;Ӿz}U>x"@z٪U+5kչ+ @&B@ @K=mG|D @ @ @Lh| ,6=F @ @ @ @ XL(DH^|h;A鲚S=)S{@ @ @ @ @ ^7e K }}`B<^a17d \Ƅ~ȘPW# 4 M@ @ @ @ @ 0
JzK%?feq bnzrYaG @ @ @ @ @ LhBl Pؗ& HX\ V6 }9= 0aqs\@ @ @ @ @ %A¾ .cB,mGwۀ])1/-G @ @ @ @ hJRc! ,i&u%J ]`B?
R{@ @ @ @ @ @`b@ K })cz u ɘP# 4 M@ @ @ @ @ ,
zJz
K$q ȘPד# ,&3j .}h" (c~'MyA @ @ @ @ @ @ @ @ @ @ @ j-AE ZSV^^6G @ @ @ @ @ @ @ @ @ @ @ L84,t m` )!
ĺ:,HTZeeM)cJK-!0V7" u>& l`1</ P@0"=6hp_X =6<& 1OYy9< @n=$J+** lH$|
jc! (c°DQQUT̷Im L cw d#RRb, Ƹ p1!@A @ @ @ @ @ @ @ @ @ @ @`iu18 &OX,feK Yĺu 3&bE6]Ƅ!~3c@ @ @ @ @ @ @ @ @ @ @ rXD+t Lr'$|RN# @].0ah" [QQ&MeK)&ԅr@ @ @ @ @ @ @ @ @ @ @ v 6l
4b/6? !R2(Q\TlY>0ty9c1 K@{aD$O$FƄ:X@ @ @ @ @ @ @ @ @ @ @ @ǐDV S @([Wee+W61) PEEVT\d,iĺ&!AUA x2 , '4o"3'ĺ T`& >}MQ@ @ %Ybϖ$# 4@}愆oC/ֵdPBbɓAP@ @ ̟7l w@ @ @ @ @ @ @ @ @ Kbk2Y C|ƄD"nS t &4u R!0w<9I@ @ @ @ @ @ @ @ @Z.\C+Q*]ƄgKKM1 @ 0!H &
JX&M2& &cBnח@ @ 0!K(VC @ @ @ @ @ @ @ @ :0[ɠ̅#O k//v|D @ @`LX4?F @ @ @ @ @ @ @ @ W *0%䉣m`iSYYy{8s@ @ F 0QX) ,y 00& Q6>8&LX
LN@ hb! #ׁ ݺ+cY<^iSA[")S q @^&t-ʘ&MTƄA^»ը1 &v y ̘ʘ L(<E @A
PJ# @ u`BI҄%\`_6eL1! ,a&,aA @ @ @ @ @ @ @ @ &x,A.cT 0n ,=&,=ך3E @ @ @ @ @ @ @ @ &R6>TȘxA @ h@]! R%ׁ ݺL.H'tu˗" 4 o@ @ @ @ @ @ @ @ @`A.,!aɔl` ʧLY2g LXl @ @ @ @ @ @ @ @
\ JzV^v
\ L(# y'@`B]* "߁ ]`-^)S C\Ƅ@x& @P(Wz" @ w`BɠD,fb}c
@# \#|9_?f6jm>ե@ @ @ @ @ @ >J.2mҤ1.cB%?dLh[# o<ؚWu<=2 qlػ
:vPCj
5.g P_|LP3gܹsR͚5f͛L
ꭨtFE[n2)-`ܹs)zs [\Ɔ_uIS>) @Ƕ|S={W㦮+0R~mܛ6r禜ȧ4yʱ2-lΜ>Xԛz" *ׁ JIb.KhheMX, 2cu.*hZ:_N>w?
7ߺ ە^hEE -g|f,Ym uT!&h9hѓu*k`Ea<Jct%k5o]n9kޢEz
3{ͫԯ}[i[65 ~ݺn`}6:tXVZiEkӺ?e֬6s,eSÉ yMP%-[t iǟڿߘ Дk{֩Cgsف($~>-UﾚVoTF=&m'3'[
kQcuwok[ݣ`ݙfٌ3]v?Ͼ?Ⱦ/+m+/l啓;b/C=@ @ @ @ @ @`eLPPBe"ns+$& qaq7zQO*Y \tѵt7ٻ߯ul;`_{k\S8go { Qcy.2f.]Xe
!0~ ^SfZ _ԑjoJ |"`?A
߶
VF͗:me_uB[yYWGg{1Y1)@`wn`G*
?6LPPM .Ր
Cn?ۆx k併
ٞ{ljW*0Emc wcUm2k:_ey7lOoԽ+ y Эd`
iG*/& 6ʗΜozS_z oӾ[k;6/l>0ѮTbKm۶5M6rg9>ɂ8#oM22 |Tkժe5W_W˻_=+sfϙq=SO6u8jZ|XL0Ӎ:o 4GPA27 PZ vjlmy&L7 0~nl #vNvi'v>eJ8|L&Hȧ`|KS}lsOOe̚ovEW|Tv~ Լynon7P5 "oݎ9rX>Ͽ῟~ /O@ @ @ @ @ @ R t(v#܈P&K]m)?r Myslaϲi~7X6ǩ)0m6vڲ˶vv?γ ~|>,Y7 twƿg>"`9s옠;e-٣zvnwm, u+`]US;o wi6ϽS@ 10bo Uֲt}[.ZuPn;-_]G4\ hL [imWWJ[V3LȊ@ 5GOh 69T!
S0@>եe }-g>*f͊퐃-I͋N/`E_B JЋRUTTQzPzG 6DzPF"%{=}v{v#egfggg3[{'N۷nKrHmڴiMi&t܁JzA;I.^Hg8yƌ"E[P@ @ @ @ @ @ @ @ @ @ @ @ 0\wxRj ,"E[,Soy_IJx|Wx=}zPҥ H-S>mJ KS#[ڵ=9htJP9bݻwGt]>u7>TWI&4|>5kn<ڊYsӟoV QpS qB*%!{g|s-/ɰx*G\ {AŨo.6G^|[DK6;E!OZWh
-&( a @
I Ocs2@m4 [;vr6=km;w1AߨS*U,'95x yr碈!aʷ{M6K~fM~/d*b $6&-d V$ $ P[,OĄĞh/epWpU;̮!+y+$uޙ0?0g 90!%oonܱsM1[W
ՔTq#|h+ZhjA +_%X}fjҥT$KmT*ʩسIƍJLLz. ?i,O9d]x{S[/!R0Qԉ#|1g>;~ 9ժvEq? Oެ qܯhϿr6#2ULjtvFZnR3Ҧ#
!Tu9l@ @ @ @ @ @ @ @ @ @ @ @ 1 0|k||p+M D&%&&1\ p$L'G7nPW;ͳu۴~@[*U*ʜ93 ho #̻idiPL%ÍbΔ1ݽ+PFM9"L`#Qƌ u);!:HR_a22`lٲRXq1A߄ -&dϕ}aTz$n.șӨn=g@lBq?.]In])aW1Wg E@|ҷϕ3tY?7{ vs|j֨F
\rRgr#ԒK5 =v0YS?JưoԭE^̗7$ξp2?a_|=eKS)RlϿ'NN!:ݻ/J1g)F)} &N)}TFWJϊHY'fuV aFq0zyԩ:*$X ӧ87U"*BltxYJ0,IܹsI3K\С9F2" aӯtK.4jHŷ0cO_[]([;f=*yԎ۲QkV??/2?1x_gv[(kPgKqwId{DW FpTըUKވZaC
;?{gW3xNvK_yWD^s߿Μ=K۶ϲ 3uPX/߿N_X)1{,F# ]HbH
C Yd^x$fߺ6l0f6Cm,+/VT,=^ؠޯO
yKA0v4:rRF@ @ @ @ @ @ @ @ @ @ @ &eX
K)(TDLb0셁Gw/UtzOzGR/`{=TByba:9&4}叹1 rlYJʔJD >>zH2۷0#NNƑ7OnjaCzDqQ3a
1T`AJe*SE82-'(yV\w@2Te7Ԥ
Q\(Cʡ|nݢC {w1r
) → false ### context `contains` node? Does this context contain this node? Is the context a parent of this node? Expects: * context: document or node * node: node Returns: true or false slick.contains(
', 'gi', { valueNames: ['between', 'left', 'match', 'right'] }); /* -> [ {name: 'between', value: 'Here is ', start: 0, end: 8}, {name: 'left', value: '- ,
', start: 8, end: 13}, {name: 'match', value: '
', start: 27, end: 33}, {name: 'between', value: ' example', start: 33, end: 41} ] */ // Omitting unneeded parts with null valueNames, and using escapeChar str = '...{1}\\{{function(x,y){return y+x;}}'; XRegExp.matchRecursive(str, '{', '}', 'g', { valueNames: ['literal', null, 'value', null], escapeChar: '\\' }); /* -> [ {name: 'literal', value: '...', start: 0, end: 3}, {name: 'value', value: '1', start: 4, end: 5}, {name: 'literal', value: '\\{', start: 6, end: 8}, {name: 'value', value: 'function(x,y){return y+x;}', start: 9, end: 35} ] */ // Sticky mode via flag y str = '<1><<<2>>><3>4<5>'; XRegExp.matchRecursive(str, '<', '>', 'gy'); // -> ['1', '<<2>>', '3'] ~~~ `XRegExp.matchRecursive` throws an error if it sees an unbalanced delimiter in the target string. ### XRegExp Prototype Methods In browsers, first include the script: ~~~ html ~~~ New XRegExp regexes then gain a collection of useful methods: `apply`, `call`, `forEach`, `globalize`, `xexec`, and `xtest`. ~~~ js // To demonstrate the call method, let's first create the function we'll be using... function filter(array, fn) { var res = []; array.forEach(function (el) {if (fn.call(null, el)) res.push(el);}); return res; } // Now we can filter arrays using functions and regexes filter(['a', 'ba', 'ab', 'b'], XRegExp('^a')); // -> ['a', 'ab'] ~~~ Native `RegExp` objects copied by `XRegExp` are augmented with any `XRegExp.prototype` methods. The following lines therefore work equivalently: ~~~ js XRegExp('[a-z]', 'ig').xexec('abc'); XRegExp(/[a-z]/ig).xexec('abc'); XRegExp.globalize(/[a-z]/i).xexec('abc'); ~~~ ## Installation and usage In browsers: ~~~ html ~~~ Or, to bundle XRegExp with all of its addons: ~~~ html ~~~ Using [npm](http://npmjs.org/): ~~~ bash npm install xregexp ~~~ In [Node.js](http://nodejs.org/) and [CommonJS module](http://wiki.commonjs.org/wiki/Modules) loaders: ~~~ js var XRegExp = require('xregexp').XRegExp; ~~~ ### Running tests on the server with npm ~~~ bash npm install -g qunit # needed to run the tests npm test # in the xregexp root ~~~ If XRegExp was not installed using npm, just open `tests/index.html` in your browser. ## &c **Lookbehind:** A [collection of short functions](https://gist.github.com/2387872) is available that makes it easy to simulate infinite-length leading lookbehind. ## Changelog * Releases: [Version history](http://xregexp.com/history/). * Upcoming: [Milestones](https://github.com/slevithan/XRegExp/issues/milestones), [Roadmap](https://github.com/slevithan/XRegExp/wiki/Roadmap). ## About XRegExp and addons copyright 2007-2012 by [Steven Levithan](http://stevenlevithan.com/). Tools: Unicode range generators by [Mathias Bynens](http://mathiasbynens.be/). Source file concatenator by [Bjarke Walling](http://twitter.com/walling). Prior art: `XRegExp.build` inspired by [Lea Verou](http://lea.verou.me/)'s [RegExp.create](http://lea.verou.me/2011/03/create-complex-regexps-more-easily/). `XRegExp.union` inspired by [Ruby](http://www.ruby-lang.org/). XRegExp's syntax extensions come from Perl, .NET, etc. All code released under the [MIT License](http://mit-license.org/). Fork me to show support, fix, and extend.
=========================================
END OF xregexp@2.0.0 NOTICES AND INFORMATION
%% xregexp@4.2.4 NOTICES AND INFORMATION BEGIN HERE:
=========================================
The MIT License Copyright (c) 2007-present Steven Levithan an
', start: 13, end: 27}, {name: 'right', value: '