Skip to content

Show the names of tests that fail sandbox violation#5585

Merged
ceedubs merged 1 commit intounisonweb:trunkfrom
ceedubs:name-test-sandbox-violators
Feb 20, 2025
Merged

Show the names of tests that fail sandbox violation#5585
ceedubs merged 1 commit intounisonweb:trunkfrom
ceedubs:name-test-sandbox-violators

Conversation

@ceedubs
Copy link
Contributor

@ceedubs ceedubs commented Feb 20, 2025

Previously if a test violated the sandbox such as with Debug.trace, the output
just said pure code can't perform I/O but didn't actually report any failing
tests or even name the test that violated the sandbox.

With this change it will at least add a line like:

Error while evaluating test `bar.test`

This should help a lot in tracking down the sandbox violation.

This resolves the worst of #4685, but it doesn't do anything to address the confusing behavior of test watches being exempt from the sandbox and caching tests that would fail when running test on an empty cache.

Previously if a test violated the sandbox such as with `Debug.trace`, the output
just said `pure code can't perform I/O` but didn't actually report any failing
tests or even name the test that violated the sandbox.

With this change it will at least add a line like:

```
Error while evaluating test `bar.test`
```

This should help a lot in tracking down the sandbox violation.

This resolves the worst of [unisonweb#4685](unisonweb#4685), but it doesn't do anything to address the confusing behavior of test watches being exempt from the sandbox and caching tests that would fail when running `test` on an empty cache.
@ceedubs ceedubs force-pushed the name-test-sandbox-violators branch from ed2e63a to d9b9384 Compare February 20, 2025 20:47
@dolio
Copy link
Contributor

dolio commented Feb 20, 2025

I'm not sure what to do about the latter problem.

I think the idea is that when you're adding tests with watch expressions, you're writing code running on your own machine, so there's more trust. But test just runs all tests from something you might have downloaded, so we take some measures to filter out shenanigans.

I don't know that we want to remove the latter, but I guess we could try to add the same restrictions to the former for things that are getting added as tests.

@ceedubs
Copy link
Contributor Author

ceedubs commented Feb 20, 2025

@dolio the answer might be to just allow Debug.trace in the test sandbox. It won't do much harm beyond spamming your terminal a bit. It's the most likely culprit for hiding in a test because it doesn't declare {IO}, so the type system doesn't make you coerce your way out of it.

There is some potentially surprising behavior with the traces not showing up once results are cached. But that probably isn't such a big deal.

@ceedubs ceedubs merged commit 510b184 into unisonweb:trunk Feb 20, 2025
17 checks passed
@ceedubs ceedubs deleted the name-test-sandbox-violators branch February 20, 2025 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants