• koop-core (the main koop module)
• output-geoservices
• featureserver
• winnow
• koop-logger
• cache-memory

This move will ease development, testing, and publishing. The diagram below illustrates the dependency relationships between the monorepo packages:

Migrating

It’s important to note that the latest Koop has been published under the @koopjs NPM organization. You will need to update your koop applications by:

> npm uninstall koop

> npm install @koopjs/koop-core

Then replace any imports in your code-base. For example:

- const Koop = require('koop')
+ const Koop = require('@koopjs/koop-core')

FeatureServer and Winnow have also moved to the @koopjs NPM organization. If you happen to be using those repositories outside of Koop you’ll need to update to @koopjs/featureserver and @koopjs/winnow. Note that the older repositories have been marked as deprecated and have no explicit maintenance plan.

Contributing

The new monorepo includes tooling to help standardize contributions and make semantic releases to NPM. This includes the requirement of (1) conventional commit messages, (2) code coverage, and (3) semantice release control with changesets. Take a look at the contribution guidelines for more information.

1) Move Koop’s core-dependencies to a monorepo

Packages will include:

koop-core koop-logger koop-cache-memory koop-output-geoservices feature-server winnow

The monorepo approach helps alleviate existing pain points that include:

• Deployment of dependency’s version bumps
• e2e tests for Koop that integrate lastest versions of all packages
• location for FeatureServer conformance testing
• easier tracking/tagging of versions

Note: Package versions will use individually-based version numbers, continuous with version numbers they currently use in independent repos. However, we plan to npm publish all packages under the @koopjs npm org.

2) Support use of the koop logger in Feature-Server and Winnow

These dependencies currently use console.log while other Koop core-dependencies and many plugins leverage the Koop logger. Using koop logger will allow better management of log messages (debug, warn, info, error), help ensure log visibility, and provide formatting required by some deployments.

3) Remove legacy output-geoservices routes (routes without rest/services)

Early versions of output-geoservices did not include routes with the rest/services fragment of the URL path. However, some ArcGIS clients require service URLs to include it to work properly. An additional set of routes with rest/services were added to address this issue, but the old routes without that fragment were maintained to avoid a breaking change. While we avoided a breaking change, it has lead to confusion on which routes to use and question about why some routes won’t work with clients like AGO.

Note: This change will break any existing implementations that leverage routes without the rest/services fragment.

4) Add code coverage tooling to all monorepo packages

5) Remove generic datasets provider and ship in separate plugin

Koop ships with a set of generic “datasets” provider with custom endpoints that allow users to add and retrieve ad hoc JSON to the koop-cache. It’s likely not used very often, and could be a source of memory problems if abused. This provider should be separated and moved to its own plugin repository and therefore installed only when needed by Koop developers.

6) Stretch goal: Remove “hosts” parameter from the provider specification and output-geoservices routes

The existing Koop-provider specification allows for the configuration of two route parameters, hosts and id. In the early development of Koop, hosts was meant to contain information to help target a remote API (maybe a service hostname or route), while id was meant to contain information the help target a specific resource on a remote API. However, such definitions are not always applicable to a given provider, so in reality these are just two optional and generic route parameters. In summary, having two generic route parameters is often unnecessary, creates route-building complications, and adds usage confusion (e.g., what do I use hosts for?). A single generic parameter is sufficient, as it can be a delimited string that hold multiple pieces of information.

We will:

• remove support for providers with an enabled hosts parameter from koop-core
• reject the registration of providers that enable the hosts parameter and log messages that provide a link to migration documents
• update the Koop documentation so that the provider spec no longer includes hosts, and includes steps to migrate old provider to the new specification.

New serve options

After updating the CLI version to v1.1 and checking the documentation for the serve command, you will see two new options --ssl-cert and --ssl-key. Note that these two options only work for Koop plugin (provider, output, auth, etc.) projects. If you use these two options in an app project, the values will be ignored.

$ koop serve [path]

run a koop server for the current project

Positionals:
  path  server file path                                                [string]

Options:
  --port      port number of the server                 [number] [default: 8080]
  --data      path to a GeoJSON data file for testing Koop plugin
                                         [string] [default: "test/data.geojson"]
  --debug     enable nodejs inspector for debugging                    [boolean]
  --watch     enable auto-restart on file change                       [boolean]
  --ssl-cert  path to the SSL certificate file                          [string]
  --ssl-key   path to the SSL key file                                  [string]

These options is used to provide the path to the SSL certificate and key files. If both paths are provided, the command will starts a HTTPS server, instead of a HTTP one.

For example, if I have the certificate file cert.pem and the key file key.pem in the project directory, I can run the server command

$ koop serve --ssl-cert cert.pem --ssl-key key.pem

and the server will start with the following message

Server listening at https://localhost:3000

Note that the URL protocol now is https, instead of http. Such URL is ready to be used in the client that requires HTTPS links.

Under the hood

The server command uses the native node moudle https to create the HTTPS server. You can see the exact implementation in the office guideline How to create an https server?.

What about production use? What about app?

As the whole blog post is talking about the dev server, you may ask “What about the production?”. Since Koop is a dedicated ET(L) server, it does not incldue the HTTPS feature. Implementing HTTPS feature directly in nodejs is not scalable and should not be used for production.

The common industrial practice is to use a proxy server (like nginx) or the service from your cloud provider.

Listing existing plugins

Sometimes you would like to know the plugins already registered in the Koop app. In the past, this could only be done by manually checking the dependency list in the package.json file and the code in src/plugins.js. Now you can simple use the new list command to print a list of all existing plugins:

$ koop list

and the output is a nicely formatted table:

2 plugins are found.

#  Name           Type      Is local plugin?
-  -------------  --------  ----------------
1  test-output    output    true
2  test-provider  provider  true

You can also further filter the list by adding a specific plugin type:

$ koop list provider

and it will result in a filtered table:

1 plugin is found.

#  Name           Type      Is local plugin?
-  -------------  --------  ----------------
1  test-provider  provider  true

Under the hood, a plugin list is maintained within the koop.json file and will be updated when a plugin is added or removed using the CLI. It keeps track of the state of the Koop app and make it easier to get the current statistics of plugins.

Removing a plugin

Remvoing an existing plugin from the Koop app used to be a tedious and error-prone task. It was needed to remove the plugin code and configuration from multiple files, which could be confusing for those who were not familiar with the project structure.

Since the Koop app boileplate is stable, we have automated the plugin removal with the new remove command. It reverses the operation done by the add command and works for the plugin from both the npm or a local directory.

For example, if the plugin is added from npm with

$ koop add provider koop-provider-csv

then the plugin can be removed with

$ koop remove koop-provider-csv

If the plugin is added from a local directory with

$ koop add provider my-private-provider --local

then the plugin can be removed with

$ koop remove my-private-provider

The command will attempt to do the following things for you:

• remove plugin source code
• remove plugin test code
• remove plugin configuration
• update koop configuration
• remove dependencies

Validating a plugin

Each Koop plugin type has a specification on the module exports. The new command validate can be used to verify whether the current Koop plugin project follows the specification. By running the command,

$ koop validate

it will load the module and checks the existence of all necessary export properties. The command will print the result of validation. If the validation passes,

The plugin is valid.

or if there is any problem,

The plugin is not valid.

#  Property       Error
-  -------------  -----------------------------------
1  type           the value is empty
2  Model          the "getData" function is not added

Compatibility

The Koop CLI v1.0 is compatible with a Koop project (app or plugin) created with v0.x. But if you are creating a project, it is recommaned to use the latest version to reduce chance of error.

More

We hope that these commands will save you time in the development. For more information, please check the details in the command documentation. If you have any question about Koop CLI, please feel free to submit an issue.

Provider’s can supply the Geoservices output-plugin with non-WGS84 geospatial data

Prior to this release, the Geoservices output-plugin expected provider’s to supply any geospatial data with WGS84 coordinates. Data using non-WGS84 CRSs had to either (1) convert their data in advance to WGS84, or (2) convert their data to WGS84 on the fly in their provider or in a provider after function. Converting on the fly introduces inefficiencies unless clients request data be output in the WGS84 CRS, because the data has to be reprojected twice; first, in the provider, to WGS84, and second, in the Geoservices output-plugin, to the target CRS.

The Geoservices output-plugin uses the Winnow library for post-processing provider GeoJSON, including reprojection. It now supports source data GeoJSON in non-WGS84 CRS. A provider simply has to either (1) set the GeoJSON crs property for the data’s CRS, or (2) modify the request’s query property so that it includes an inputCrs property. See the “Does GeoJSON produced by providers need to use the WGS84 CRS?” in the FAQs for additional details and references to examples.

It’s important to reiterate that support for GeoJSON in non-WGS84 CRSs depends on the output-plugin. While Koop’s default output-plugin, Geoservices (FeatureServer) now supports it, other output plugins (e.g., Vector Tiles) still expect geospatial data that uses WGS84. These output plugins could be updated to use Winnow to reproject data as necessary - PRs are welcome.

Reprojection can be requested with CRS WKID

The Geoservices output-plugin has long supported reprojection of geospatial data by way of the outSR query parameter. Koop’s limitation however, was that the value of outSR had to be the WKT representation of the CRS for all but the most common systems (WGS84 or Web Mercator). This was problematic because many clients would use a WKID as the value of outSR and Koop would not recognize it. With the release of Geoservices 2.2.1, the output-plugin is able to use the WKID reference for most CRSs. In the event that the WKID is not found, clients can still send a WKT string.

The before function

A before function registered with a provider will execute prior to the provider’s getData method and can be used to gate access or modify the Express request object. In the example below, we use the before to reject the request for a given id:

const koop = new Koop()
const socrata = require('@koopjs/provider-socrata')

koop.register(socrata, {
  before: (request, callback) => {

    const { params: { id } } = request

    // Reject any requests with id 'xyz'
    if (id === 'xyz') {
      const error = new Error('Forbidden')
      error.code = 403
      return callback(error)
    }
    callback()
  }
})

Another example use case of provider transformation functions is adding support for additonal output spatial references. While Koop’s default output supports the outSR parameter, it only does so for spatial references that the proj4 library keeps in memory. To get output in any other spatial reference, outSR must be a valid WKT. However, many clients only send outSR as a WKID. You can use either transformation function to convert WKIDs to their WKT equivalent and assign that value to the outSR parameter..

const koop = new Koop()
const socrata = require('@koopjs/provider-socrata')

koop.register(socrata, {
  before: (request, callback) => {

    const { query: { outSR } } = request

    if (outSR === 2285) request.query.outSR = `PROJCS["NAD83/WashingtonNorth(ftUS)",GEOGCS["NAD83",DATUM["North_American_Datum_1983",SPHEROID["GRS1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY["EPSG","6269"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4269"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["standard_parallel_1",48.73333333333333],PARAMETER["standard_parallel_2",47.5],PARAMETER["latitude_of_origin",47],PARAMETER["central_meridian",-120.8333333333333],PARAMETER["false_easting",1640416.667],PARAMETER["false_northing",0],UNIT["USsurveyfoot",0.3048006096012192,AUTHORITY["EPSG","9003"]],AXIS["X",EAST],AXIS["Y",NORTH],AUTHORITY["EPSG","2285"]]`

    callback()
  }
})

Note that in the examples above, we were able to effectively modify the functionality of the Socrata provider without having to edit the provider code base.

The after function

An after function registered with a provider will execute after the provider’s getData method and can be used modify the provider-generated GeoJSON or the Express.js request object. The after function can be extremely helpful if your source data is not WGS84. Since Koop currently expects all data coming out of getData to have spatial reference WGS84, you can use the after function to reproject it before it gets passed on to Koop outputs:

const koop = new Koop()
const reproject = require('reproject')
const proj4 = require('proj4')
const github = require('@koopjs/provider-github')

koop.register(github, {
  after: (request, geojson, callback) => {

    const fromSR = `PROJCS["NAD83/WashingtonNorth(ftUS)",GEOGCS["NAD83",DATUM["North_American_Datum_1983",SPHEROID["GRS1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],TOWGS84[0,0,0,0,0,0,0],AUTHORITY["EPSG","6269"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4269"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["standard_parallel_1",48.73333333333333],PARAMETER["standard_parallel_2",47.5],PARAMETER["latitude_of_origin",47],PARAMETER["central_meridian",-120.8333333333333],PARAMETER["false_easting",1640416.667],PARAMETER["false_northing",0],UNIT["USsurveyfoot",0.3048006096012192,AUTHORITY["EPSG","9003"]],AXIS["X",EAST],AXIS["Y",NORTH],AUTHORITY["EPSG","2285"]]`

    try {
      const wgs84Data = reproject(fromSR, proj4.wgs84, geojson)
      callback(null, wg484Data)
    } catch (err) {
      callback(err)
    }
  }
})

In the after function above, we use the reproject library to convert geojson arriving with geometry in the EPSG:2285 coordinate system to WGS84, then pass it onto to the callback.

To learn more about modify provider behavior with transformation functions, see the usage docs.

Watch mode

One annoying thing of using the serve command for an app under development is that it doesn’t know code change. If you want to test your new code, the app has to be restarted manually.

With the new watch option, the serve command can watch any file change in the local directory and restart the app automatically on change. The feature leverages the nodemon module for file monitoring and app restarting.

Try it with

koop serve --watch

Debug mode

In previous releases, console message is the only way to debug an app running with serve command. This creates a lot of inconvenience when developing a complex project.

With the new debug, the server command can also invoke the Node.js inspector for the running app. The inspector listens at the default hostname and port (127.0.0.1:9229).

Try it with

koop serve --debug

Yarn support

npm is the default npm package manager used by the CLI. In this release, the support to yarn is added.

To start a new project using yarn, you can use the --npm-client option in new command.

koop new app my-test-app --npm-client=yarn

To change the package manager of an existing project, you can update the koop.json file and update the npmClient value.

{
  "npmClient": "yarn"
}

More

Alongside these new features, we continue to refine Koop project templates by removing unnecessary code and adding more tests. The project created by the new CLI will include more robust code and a cleaner structure.

We hope you will find this release useful. Happy coding :)