Skip to content
This repository was archived by the owner on Aug 31, 2021. It is now read-only.

Commit f6c7e17

Browse files
livecodealilivecodeali
committed
[[ ParserTestRunner ]] Add LCS parser test runner and tests
This patch adds a parser test runner that uses the format of the current LCB compiler frontend tests and the java ffi dsl tool tests.
1 parent 057f3b5 commit f6c7e17

9 files changed

+474
-42
lines changed

‎docs/development/testing.md‎

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,12 @@
22

33
Tests are small programs that check that a particular, specific function works correctly. They are run automatically to check whether LiveCode works properly. They're really useful for ensuring that changes to one part of LiveCode don't break other things!
44

5-
The main LiveCode engine repository contains four sets of tests ("test suites"):
5+
The main LiveCode engine repository contains the following sets of tests ("test suites"):
66

77
* **LiveCode Script tests:** script-only stacks that are run using the LiveCode standalone engine. They test features of the LiveCode Script language.
88
* **LiveCode Builder tests:** LCB modules that are run using the **lc-run** tool. They test features of the LCB core language and standard library.
99
* **LiveCode Builder Compiler Frontend tests:** Fragments of LCB code which are run through the compiler and check that the compile succeeds, or emits the correct warnings or errors.
10+
* **LiveCode Script parser tests:** Fragments of LCS code which are run through the parser and check that the compile succeeds, or emits the correct warnings or errors.
1011
* **C++ tests:** low-level tests written in C++ using [Google Test](https://github.com/google/googletest). These perform low-level checks for things that can't be tested any other way.
1112

1213
## Running the Tests
@@ -163,6 +164,52 @@ When compiled, lc-compile will emit an error on the 'put' line because tInnerVar
163164

164165
To help debug compiler tests, set the LCC_VERBOSE environment variable to 1 before running the compiler test. This will cause the compiler testrunner to emit diagnostic information, including the full output of the compile command which is being run.
165166

167+
### LiveCode Script Parser Tests
168+
169+
The syntax for LiveCode Script parser tests is the same as that of the
170+
LCB compiler frontent tests above. LCS parser test files all have the
171+
extension '.parsertest'.
172+
173+
Expected errors are referred to by their name in the parse errors
174+
enumeration. For example the following tests the variable shadowing
175+
parse error "local: name shadows another variable or constant":
176+
177+
%TEST ShadowVariable
178+
local sVar
179+
local %{BEFORE_SHADOW}sVar
180+
%EXPECT PASS
181+
%ERROR PE_LOCAL_SHADOW AT BEFORE_SHADOW
182+
%ENDTEST
183+
184+
The directive `%SET` can be used to specify the value of global properties
185+
used when running the test. In particular, it can be used to set the
186+
value of the `explicitvariables` property. If the `explicitvariables`
187+
property is not set then the test will be run with it set to `true` and
188+
to `false`, and the test will fail if the result differs. For example:
189+
190+
%TEST CommentedContinuation
191+
on compiler_test
192+
-- comment \
193+
with%{SYNTAX} continuation character
194+
end compiler_test
195+
%EXPECT PASS
196+
%ERROR PE_EXPRESSION_NOTLITERAL AT SYNTAX
197+
%ENDTEST
198+
199+
will fail with "error: test ambiguity with explicit vars".
200+
201+
%TEST CommentedContinuationExplicit
202+
%SET explicitvariables true
203+
on compiler_test
204+
-- comment \
205+
with %{SYNTAX}continuation character
206+
end compiler_test
207+
%EXPECT PASS
208+
%ERROR PE_EXPRESSION_NOTLITERAL AT SYNTAX
209+
%ENDTEST
210+
211+
will succeed.
212+
166213
### C++ tests with Google Test
167214

168215
In general, C++ tests should only be used for things that cannot be tested any other way.

‎tests/Makefile‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,14 @@ LCC_CMD = $(LCS_ENGINE) -ui $(LCC_COMPILERTESTRUNNER) run
4545

4646
LCC_LOG = _compiler_test_suite.log
4747

48+
########## LiveCode Script Parser test parameters
49+
50+
LCP_COMPILERTESTRUNNER ?= $(top_srcdir)/tests/_parsertestrunner.livecodescript
51+
52+
LCP_CMD = $(LCS_ENGINE) -ui $(LCP_COMPILERTESTRUNNER) run
53+
54+
LCP_LOG = _parser_test_suite.log
55+
4856
########## lc-run test params
4957

5058
LC_RUN_TESTS ?= $(top_srcdir)/tests/_lcruntests.livecodescript
@@ -68,7 +76,7 @@ LC_COMPILE_FFI_JAVA_LOG = _lc-compile-ffi-java_test_suite.log
6876

6977
.DEFAULT: check
7078

71-
check: lcs-check lcb-check compiler-check lc-compile-ffi-java-check
79+
check: lcs-check lcb-check compiler-check lc-compile-ffi-java-check lcs-parser-check
7280

7381
clean:
7482
-rm -rf $(TEST_DIR) $(LCS_LOG) $(LCB_LOG)
@@ -151,6 +159,17 @@ compiler-check: $(LC_COMPILE)
151159
cmd="$(LCC_CMD)"; \
152160
echo "$$cmd" $(_PRINT_RULE); \
153161
$$cmd
162+
163+
################################################################
164+
# LCS parser tests
165+
################################################################
166+
167+
lcs-parser-check:
168+
@rm -f $(LCP_LOG)
169+
export LCC_VERBOSE="$(LCC_VERBOSE)"; \
170+
cmd="$(LCP_CMD)"; \
171+
echo "$$cmd" $(_PRINT_RULE); \
172+
$$cmd
154173

155174
################################################################
156175
# Basic LCB runner test

‎tests/_compilertestrunnerbehavior.livecodescript‎

Lines changed: 50 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -199,16 +199,16 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
199199
if the result is not empty then
200200
throw format("can't open file '%s'", pScriptFile)
201201
end if
202-
202+
203203
local tScript
204204
read from file pScriptFile until end
205205
put it into tScript
206-
206+
207207
close file pScriptFile
208-
208+
209209
-- First extract the test block we are wanting to run
210210
local tLine, tLineNumber, tTestLineNumber, tState
211-
local tName, tCode, tExpectation, tExpectationReason, tAssertions, tPositions
211+
local tName, tCode, tExpectation, tExpectationReason, tAssertions, tPositions, tSet, tSetCount
212212
put empty into tLine
213213
put 0 into tLineNumber
214214
put 0 into tTestLineNumber
@@ -219,26 +219,28 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
219219
put empty into tExpectationReason
220220
put empty into tAssertions
221221
put empty into tPositions
222+
put empty into tSet
223+
put 0 into tSetCount
222224
repeat for each line tLine in tScript
223225
add 1 to tLineNumber
224-
226+
225227
local tIsDirective
226228
put false into tIsDirective
227229
if tState is "code" then
228230
if word 1 of tLine begins with "%" and \
229231
not (word 1 of tLine begins with "%{") then
230-
put true into tIsDirective
232+
put true into tIsDirective
231233
end if
232234
else
233235
if word 1 of tLine begins with "%" then
234236
put true into tIsDirective
235237
end if
236238
end if
237-
239+
238240
if tIsDirective then
239241
local tToken
240242
put word 1 of tLine into tToken
241-
243+
242244
if tToken is "%%" then
243245
-- We don't allow comment directives inside code blocks
244246
if tState is "code" then
@@ -251,7 +253,7 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
251253
if tState is not empty then
252254
reportCompilerTestError pScriptFile, tLineNumber, "%TEST directive not allowed inside test block"
253255
end if
254-
256+
255257
-- We've started a new test clause, so record the name and move to
256258
-- code state.
257259
reportCompilerTestDiag format("found test '%s' at line %d", word 2 of tLine, tLineNumber)
@@ -261,23 +263,24 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
261263
put empty into tExpectation
262264
put empty into tAssertions
263265
put empty into tPositions
264-
put "code" into tState
266+
put "set" into tState
267+
put 0 into tSetCount
265268
else if tToken is "%EXPECT" then
266269
-- We only allow %EXPECT directives after code blocks
267270
if tState is not "code" then
268271
reportCompilerTestError pScriptFile, tLineNumber, "%EXPECT directive only allowed after code block"
269272
end if
270-
273+
271274
-- If the expectation isn't one of PASS, FAIL, XFAIL, SKIP then it
272275
-- is an error
273276
if word 2 of tLine is not among the items of "PASS,FAIL,SKIP" then
274277
reportCompilerTestError pScriptFile, tLineNumber, format("invalid expectation '%s'", word 2 of tLine)
275278
end if
276-
279+
277280
if word 3 of tLine contains "#" then
278281
reportCompilerTestError pScriptFile, tLineNumber, format("expectation reason cannot contain '#'")
279282
end if
280-
283+
281284
-- Record the expectation, and move to assertions state.
282285
put word 2 of tLine into tExpectation
283286
put word 3 of tLine into tExpectationReason
@@ -290,40 +293,40 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
290293
if tState is not "assertions" then
291294
reportCompilerTestError pScriptFile, tLineNumber, "assertion directive outside of assertion block"
292295
end if
293-
296+
294297
if tToken is "%SUCCESS" then
295298
if the number of words in tLine is not 1 then
296299
reportCompilerTestError pScriptFile, tLineNumber, "incorrect syntax, expected: %SUCCESS"
297300
end if
298-
301+
299302
reportCompilerTestDiag format("found assert success at line %d", tLineNumber)
300303
put "success" & return after tAssertions
301304
else if tToken is "%WARNING" then
302305
if (the number of words in tLine is 4 and \
303-
word 3 of tLine is "AT") then
304-
306+
word 3 of tLine is "AT") then
307+
305308
-- Check that the position exists
306309
if tPositions[word 4 of tLine] is empty then
307310
reportCompilerTestError pScriptFile, tLineNumber, format("unknown position '%s'", word 4 of tLine)
308311
end if
309-
312+
310313
reportCompilerTestDiag format("found assert warning %s for position %s (with '%s') at line %d", word 2 of tLine, word 4 of tLine, word 6 of tLine, tLineNumber)
311-
314+
312315
put "warning", word 2 of tLine, word 4 of tLine & return after tAssertions
313316
else
314317
reportCompilerTestError pScriptFile, tLineNumber, "incorrect syntax, expected: %WARNING <id> AT <position>"
315318
end if
316319
else if tToken is "%ERROR" then
317320
if (the number of words in tLine is 4 and \
318-
word 3 of tLine is "AT") then
319-
321+
word 3 of tLine is "AT") then
322+
320323
-- Check that the position exists
321324
if tPositions[word 4 of tLine] is empty then
322325
reportCompilerTestError pScriptFile, tLineNumber, format("unknown position '%s'", word 4 of tLine)
323326
end if
324-
327+
325328
reportCompilerTestDiag format("found assert error %s for position %s at line %d", word 2 of tLine, word 4 of tLine, tLineNumber)
326-
329+
327330
put "error", word 2 of tLine, word 4 of tLine & return after tAssertions
328331
else
329332
reportCompilerTestError pScriptFile, tLineNumber, "incorrect syntax, expected: %ERROR <id> AT <position>"
@@ -335,24 +338,30 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
335338
if tState is not "assertions" then
336339
reportCompilerTestError pScriptFile, tLineNumber, "premature %ENDTEST directive"
337340
end if
338-
341+
339342
if tAssertions is empty then
340343
reportCompilerTestError pScriptFile, tLineNumber, "no assertions specified"
341344
end if
342-
345+
343346
if tAssertions contains "%SUCCESS" and \
344-
tAssertions contains "%ERROR" then
347+
tAssertions contains "%ERROR" then
345348
reportCompilerTestError pScriptFile, tLineNumber, "both %SUCCESS and %ERROR not allowed in the same set of assertions"
346349
end if
347-
350+
348351
-- If this is the test we are looking for, we are done
349352
if tName is pTest then
350353
exit repeat
351354
end if
352-
355+
353356
-- Move back to initial state
354357
put empty into tName
355358
put empty into tState
359+
else if tToken is "%SET" then
360+
if tState is not "set" then
361+
reportCompilerTestError pScriptFile, tLineNumber, "set directive must precede code"
362+
end if
363+
add 1 to tSetCount
364+
put word 3 of tLine into tSet[word 2 of tLine]
356365
else
357366
-- We've encountered an illegal directive
358367
reportCompilerTestError pScriptFile, tLineNumber, format("unknown directive '%s'", tToken)
@@ -362,10 +371,11 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
362371
if word 1 to -1 of tLine is not empty then
363372
reportCompilerTestError pScriptFile, tLineNumber, "junk outside of test clause"
364373
end if
365-
else if tState is "code" then
374+
else if tState is "code" or tState is "set" then
375+
put "code" into tState
366376
-- If we are in a code block, then accumulate the code line after
367377
-- processing for position directives %{...}.
368-
378+
369379
-- We loop through the line, searching for %{...}, and accumulating the
370380
-- code in between as we go.
371381
local tPositionSkip
@@ -388,25 +398,25 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
388398
reportCompilerTestError pScriptFile, tLineNumber, "unterminated named position"
389399
end if
390400
add tPositionOffset to tPositionEndOffset
391-
401+
392402
-- Check that the name is not empty
393403
local tPositionName
394404
put char tPositionOffset + 2 to tPositionEndOffset - 1 of tLine into tPositionName
395405
if tPositionName is empty then
396406
reportCompilerTestError pScriptFile, tLineNumber, "empty named position"
397407
end if
398-
408+
399409
-- Check that the position hasn't already been defined
400410
if tPositions[tPositionName] is not empty then
401411
reportCompilerTestError pScriptFile, tLineNumber, "named position already defined"
402412
end if
403-
413+
404414
reportCompilerTestDiag format("found position '%s' at line %d", tPositionName, tLineNumber)
405-
put tLineNumber - tTestLineNumber + 1 into tPositions[tPositionName]
406-
415+
put tLineNumber - tTestLineNumber - tSetCount + 1 into tPositions[tPositionName]
416+
407417
put tPositionEndOffset + 1 into tPositionSkip
408418
end repeat
409-
419+
410420
-- Add the newline to the code
411421
put return after tCode
412422
else if tState is "assertions" then
@@ -416,20 +426,21 @@ private command processCompilerTest pInfo, pScriptFile, pTest, @rCompilerTest
416426
end if
417427
end if
418428
end repeat
419-
429+
420430
if tName is empty then
421431
reportCompilerTestError pScriptFile, tLineNumber, format("test '%s' not found in file", pTest)
422432
end if
423-
433+
424434
-- We now have the code for the test, the expectation and a list of
425435
-- assertions.
426-
436+
427437
put tName into rCompilerTest["name"]
428438
put tCode into rCompilerTest["code"]
429439
put tPositions into rCompilerTest["positions"]
430440
put tExpectation into rCompilerTest["expectation"]
431441
put tExpectation into rCompilerTest["reason"]
432442
put tAssertions into rCompilerTest["assertions"]
443+
put tSet into rCompilerTest["set"]
433444
end processCompilerTest
434445

435446
private command reportCompilerTestError pFile, pLine, pMessage

0 commit comments

Comments
 (0)