Switch test suite to use tasty#2848
Conversation
|
Looks good to me! I'll happily continue this for publish and docs when I get a moment. |
230df88 to
63cb34a
Compare
|
👍 I like it |
|
I don't know what's happening there, but as long as the test suite can't be run properly on Windows this is not ready to go in... |
|
Appveyor doesn't seem to complain? If you merge with |
|
|
This doesn't have any immediate effects apart from making the output of the docs tests slightly nicer. It does, however, get us one closer to switching over to a nicer test runner such as tasty's (see #2848 for more details).
This doesn't have any immediate effects apart from making the output of the docs tests slightly nicer. It does, however, get us one step closer to switching over to a nicer test runner such as tasty's (see #2848 for more details).
Still missing docs and publish tests, since these use a custom test runner mechanism
Instead of using the silently package, which causes trouble in combination with tasty, since it silences part of its output as well.
63cb34a to
5708971
Compare
|
Okay, so after a long time of letting this sit... I decided to look into it again, and it turns out that the problems on Windows can be fixed by not using I've added a bit of documentation on how to run the individual tests to |
hdgarrood
left a comment
There was a problem hiding this comment.
I'm not aware of anything else silently was supposed to silence; as long that the output is reasonable (which I'm sure it is) I'm very much in favour of this 👍
|
Thanks :) |
|
Eh, can we reconsider this? Test output is a lot worse with this. Specifically, failing test messages are entirely muted, so we don't get anything more than a red |
|
From one the recent CI runs: @garyb and me also see the failing test output just fine (that's how we found the missing error spans) I'm totally fine with switching back to pure hspec if someone does the work of pulling the different test suites together under a single invocation of |
|
Ah, I'm seeing the failure messages now. Still, this seems to be worse than |
|
Since this is merged I've been using the ability to only run the I've previously searched through the docs for hspec, but it seems like the cmdline pattern syntax only allows to restrict the tests to be run to source files not individual tests. Also I'm not sure in what sense the test output is a lot worse with this, since it print the test names just as hspec. |
Tasty and hspec are both testing frameworks: they are meant to satisfy the same need, and they're not really intended to be used together. See e.g. this comment from the tasty-hspec Haddocks: > hspec and tasty serve similar purposes; consider using one or the > other. > > However, in a pinch, this module allows you to run an hspec Spec as a > tasty TestTree. I'd like to switch back to using hspec throughout for two reasons: 1. With hspec, failures are summarised at the end of the test output. Tasty reports failures inline, which can be very annoying, especially in CI, as you then have to search through the logs to find the failure. Displaying failures at the end seems like a much more sensible default to me. 2. We get to take advantage of HasCallStack in hspec-expectations to help narrow down where a failure is coming from. This is particularly helpful in tests like the ones in TestMake.hs where we run something like ``someAction `shouldReturn` someResult`` more than once in a particular test, as without the source location from HasCallStack it's not immediately clear which assertion is failing. (This would have been useful in purescript#4053). In purescript#2848, when tasty was first introduced, one of the main benefits we were getting was that it would let us run individual tests via CLI options. This is still possible with this PR: hspec has a `--match` CLI option for this. There's no `--accept` flag for golden tests, but instead you can set an `HSPEC_ACCEPT` environment variable, or simply delete the files you want to regenerate.
This is a suggestion, feel free to shoot it down
This switches the test suite to use tasty as its test runner.
The primary benefit this has right now, is that we can run individual tests by running eg:
stack test --test-arguments="-p ide". We can also switch off things like colored output with environment variables, which is nice in a CI setting.I've seen some strange artifacts with Powershell and the compiler test suite:
getConsoleCursorInfo: invalid argument (Invalid handle.)I can avoid the error with setting
--color neverbut it's not really a solution. (I decided I'll check what happens on AppVeyor before researching further)Still missing docs and publish tests, since these use a custom test runner mechanism and I couldn't figure out how to make them use hspec in the limited time I invested.