The visual test suite has grown larger over the last year, which is an awesome thing for test coverage, preventing regressions, finding and fixing bugs, and understanding features.

But we can do better. Here are several ideas for improving the visual tests to make them easier to work with and increase developer happiness:

1. Move them out of core and include in core as a submodule

This would allow us to be more aggressive at adding more tests without further bloating the mapnik core github repo with large images. Things I'd like to add that I've been holding back on due to concern over size are:

• Adding visual tests for the svg_renderer (7acf106)
• Adding visual test for the grid_renderer that use images instead of json (so that the visual result will be much easier to check)

2. Run them on travis and upload the results to S3 bucket for viewing

Then when we have pull requests that change visual tests a developer can link to a public url for the comparison HTML instead of commit the image changes to github.

3. Make tests standalone

Adding a new test should be as simple as adding a new XML in a specific directory. Expected test fixtures should be generated automatically. We should not need to add an entry for the test in visual_tests/test.py. By default map.zoom_all() should be called and if a test needs to be run with a custom bounding box this could be controlled via an XML Parameter in the test.xml (that the test running looks for):

    <Parameters>
        <Parameter name="bbox">-180,-90,180,90</Parameter>
    </Parameters>

Alternatively many tests could just be ported to not need a custom bounding box (for instance the tests using CSV data could use <Parameter name="extent">-180,-90,180,90</Parameter> in the Datasource.

If a test needs to be disabled or ignored temporarily we could do this like:

    <Parameters>
        <Parameter name="status">off</Parameter>
    </Parameters>

4. Port the test runner to C++ and add an option to run with threads

This would enable a couple very useful things:

• The test runner, if supporting threads, could also be used to ensure a given test case does not crash under threaded load produces the right output still under threaded load
• As pure C++ it would make it much easier to run tests through profiling tools and to run them through valgrind. It surely it possible to get this working through python, but much more of a hassle.
• Most of all: then we would not need the python bindings to be installed which would be great because I'd like to move the python bindings out of core into their own repo (like node-mapnik we'd have python-mapnik).

Ideally the code instead ./benchmark could be refactored and shared so that we use similar approaches for both benchmarks and visual tests.