git @ Cat's Eye Technologies Falderal / 83efaa5
Merge pull request #10 from catseye/develop-0.12 Develop 0.12 Chris Pressey authored 1 year, 8 months ago GitHub committed 1 year, 8 months ago
22 changed file(s) with 794 addition(s) and 486 deletion(s). Raw diff Collapse all Expand all
00 History
11 =======
22
3 Version 0.11 "Dan Ryan Expressway" (current released version):
3 Version 0.12 "Schoenhofen Brewery":
4
5 * When expanding variables in declaration strings, use
6 `string.replace` instead of `re.sub` so that backslash-escaping
7 is not perfomed on the replacement string. (Thanks to
8 James Holderness for bug report and patch.)
9 * In "freestyle" format, lines beginning with `<= `, `<== `, or
10 `<=== ` can supply a section of test input text (Thanks to
11 James Holderness for feature suggestion and patch.)
12 * Falderal is now responsible for quoting all shell metacharacters
13 in the substituion text of `%(...)` variables when command line
14 templates are expanded, so that no quotes need be placed around
15 the variable in the template. (Again, thanks to James Holderness
16 for spotting the bug in the code that led to this improvement.)
17 * `py-falderal` can now run under both Python 2 and Python 3
18 (tested with 2.7, 3.4, and 3.5.)
19 * The doctests that were previously distributed throughout the
20 source code have been converted to a suite of unit tests in
21 the `falderal.tests` module. The `-t` command-line option has
22 been removed; to run internal tests, run this module instead.
23
24 Version 0.11 "Dan Ryan Expressway":
425
526 * Renamed the internal tests to have descriptive names instead of
627 numbers.
1334 block needs no prefixes at all, and it is simply treated as one
1435 verbatim test body.
1536
16 Version 0.10 "Shedd Aquarium" (previous released version):
37 Version 0.10 "Shedd Aquarium":
1738
1839 * Removed the old, bitrotting Haskell implementation.
1940 * Removed deprecated `%(test-text)` and `%(test-file)` variables
123123 Development
124124 -----------
125125
126 Falderal development is
127 [hosted on Bitbucket](https://bitbucket.org/catseye/falderal/) with a
128 [git mirror of the repository on Github](https://github.com/catseye/Falderal).
126 The git repository for the Falderal distribution can be found on GitHub at
127 [https://github.com/catseye/Falderal](https://github.com/catseye/Falderal).
129128
130 Official release distfiles are available on the
129 Official release distfiles are available via the
131130 [Falderal project page](http://catseye.tc/node/Falderal) at
132131 [Cat's Eye Technologies](http://catseye.tc/).
133132
134133 Projects using Falderal
135134 -----------------------
136135
136 * [ALPACA](http://catseye.tc/node/ALPACA)
137 * [Castile](http://catseye.tc/node/Castile)
138 * [Equipage](http://catseye.tc/node/Equipage)
137139 * [Exanoke](http://catseye.tc/node/Exanoke)
138140 * [Flobnar](http://catseye.tc/node/Flobnar)
139141 * [Hev](http://catseye.tc/node/Hev)
142144 * [Pail](http://catseye.tc/node/Pail)
143145 * [Pixley](http://catseye.tc/node/Pixley)
144146 * [PL-{GOTO}.NET](http://catseye.tc/node/PL-{GOTO}.NET)
147 * [Quylthulg](http://catseye.tc/node/Quylthulg)
145148 * [Robin](http://catseye.tc/node/Robin)
146 * [Quylthulg](http://catseye.tc/node/Quylthulg)
149 * [Samovar](http://catseye.tc/node/Samovar)
150 * [SixtyPical](http://catseye.tc/node/SixtyPical)
151 * [Tamsin](http://catseye.tc/node/Tamsin)
147152 * [Velo](http://catseye.tc/node/Velo)
148153 * [Yolk](http://catseye.tc/node/Yolk)
149154 * [Xoomonk](http://catseye.tc/node/Xoomonk)
55 Status
66 ------
77
8 This document is a *draft*. It is nominally "version 0.11" because it
9 describes something that version 0.11 of `py-falderal` mostly implements.
8 This document is a *draft*. It is nominally "version 0.12" because it
9 describes something that version 0.12 of `py-falderal` mostly implements.
1010 We will deign to note which sections of this document the current released
1111 version of `py-falderal` implements, and which it does not. However,
1212 this document is a work in progress, subject to change, and subject to get
8888 * `??> `: expected error text
8989 * `???> `: expected error text
9090
91 In addition, the following introducers may be used to mark a section
92 of test input text on the first of the final lines (but may not be
93 used to end a block):
94
95 * `<= `: test input text
96 * `<== `: test input text
97 * `<=== `: test input text
98
9199 If a block is identified as a freestyle block, all lines preceding the
92 final lines with one of these introducers, are interpreted as having
93 no introducer at all (even if they begin with `| ` or some other sequence
94 already mentioned) and are used as the test body block.
100 first final line appearing with one of these introducers, are interpreted
101 as having no introducer at all (even if they begin with `| ` or some other
102 sequence already mentioned) and are used as the test body block.
95103
96104 Lines without introducers are called _intervening text_.
97105 Lines of intervening text are classified as either blank or non-blank. A
153161 text of the test and produces its output, to be compared with the expected
154162 output.
155163
164 When a variable is expanded, it is Falderal's responsibility to escape any
165 shell metacharacters that occur as part of the substitution text so that
166 it appears to the shell as a single token, and cannot otherwise affect the
167 meaning of the instantiated shell command. It is not necessary for a
168 variable to be enclosed in any kind of quotes in the command line template.
169 It is also not possible for the substitution text to e.g. expand to two
170 arguments, or to form a pipe or redirection specifier, etc.
171
172 The exact method by which the Falderal implementation escapes or quotes
173 the substitution text so that it appears unambiguously as a single argument
174 is left up to the implementation.
175
156176 ##### `%(test-body-file)` #####
157177
158178 The variable `%(test-body-file)` will be replaced by the name of a file which
162182 ##### `%(test-body-text)` #####
163183
164184 The variable `%(test-body-text)` will be replaced by the actual text of the
165 test body. It is assumed that `%(test-body-text)` will appear inside single
166 quotes in the command string, so any single quotes in the text of the test will
167 be escaped by the Falderal implementation by preceding them with backslashes.
185 test body.
168186
169187 ##### `%(test-input-file)` #####
170188
175193 ##### `%(test-input-text)` #####
176194
177195 The variable `%(test-input-text)` will be replaced by the actual text of the
178 test input. It is assumed that `%(test-input-text)` will appear inside single
179 quotes in the command string, so any single quotes in the text of the test will
180 be escaped by the Falderal implementation by preceding them with backslashes.
196 test input.
181197
182198 If neither of the variables `%(test-body-file)` nor `%(test-body-text)` appear
183199 in the command string, the test body text will be provided on the standard
264280 thing to test
265281 ???> error to expect
266282
283 thing to test
284 <=== input to give it
285 ===> output to expect
286
287 thing to test
288 <=== input to give it
289 ???> error to expect
290
267291 Invalid examples:
268292
269293 | thing to test
273297 + input to give it
274298 = output to expect
275299
276 ...test input must be preceded by a test body (if this is the first test.)
300 ...test input must be preceded by a test body, if this is the first test.
301
302 <=== input to give it
303 ???> output to expect
304
305 ...test input must be preceded by a test body always, in freestyle format.
277306
278307 ? error to expect
279308
2626 parser.add_option("-d", "--dump",
2727 action="store_true", default=False,
2828 help="print out info about parsed tests, don't run them")
29 parser.add_option("-t", "--test",
30 action="store_true", default=False,
31 help="run internal tests only and exit")
3229 parser.add_option("-v", "--verbose",
3330 action="store_true", default=False,
3431 help="print out info about each test as it is run")
3532
3633 (options, args) = parser.parse_args(args[1:])
37
38 if options.test:
39 import doctest
40 import falderal.objects
41 (failure_count, test_count) = \
42 doctest.testmod(falderal.objects,
43 optionflags=doctest.NORMALIZE_WHITESPACE)
44 if failure_count > 0:
45 return 1
46 else:
47 return 0
4834
4935 # load Documents and create Falderal Tests from them
5036 documents = []
8167 return 1
8268
8369 if options.dump:
84 print "Functionalities:"
70 print("Functionalities:")
8571 for name in functionalities:
86 print " " + name
72 print(" " + name)
8773 for implementation in functionalities[name].implementations:
88 print " +-" + str(implementation)
89 print "Tests:"
74 print(" +-" + str(implementation))
75 print("Tests:")
9076 for test in tests:
91 print " " + str(test)
77 print(" " + str(test))
9278 return 0
9379
9480 # run tests
9884 try:
9985 for test in tests:
10086 if options.verbose:
101 print str(test)
87 print(str(test))
10288 these_results = test.run(options=options)
10389 if options.verbose:
10490 for result in these_results:
115101 if options.verbose:
116102 for key in dup_check:
117103 if len(dup_check[key]) != 1:
118 print "WARNING: test/impl combination %s was run %d times %r" % (
104 print("WARNING: test/impl combination %s was run %d times %r" % (
119105 key, len(dup_check[key]), dup_check[key]
120 )
106 ))
121107
122108 # report on results
123109 for result in results:
125111 num_results = len(results)
126112 num_failures = len([x for x in results if not x.is_successful()])
127113 if not all_ran:
128 print '**************************************************************'
129 print '** TESTING TERMINATED PREMATURELY -- NOT ALL TESTS WERE RUN **'
130 print '**************************************************************'
114 print('**************************************************************')
115 print('** TESTING TERMINATED PREMATURELY -- NOT ALL TESTS WERE RUN **')
116 print('**************************************************************')
131117
132 print '--------------------------------'
133 print 'Total test runs: %d, failures: %d' % (num_results, num_failures)
134 print '--------------------------------'
118 print('--------------------------------')
119 print('Total test runs: %d, failures: %d' % (num_results, num_failures))
120 print('--------------------------------')
135121
136122 if num_failures == 0:
137123 return 0
33 from subprocess import Popen, PIPE
44 from tempfile import mkstemp
55
6 # Note: the __unicode__ method of all the classes defined herein should
6 # Python 2/3
7 try:
8 unicode = unicode
9 except NameError:
10 unicode = str
11
12 try:
13 from shlex import quote as shlex_quote
14 except ImportError:
15 from pipes import quote as shlex_quote
16
17 # Note: the __str__ method of all the classes defined herein should
718 # produce a short, human-readable summary of the contents of the object,
819 # suitable for displaying in the test results but not necessarily
920 # complete. __repr__ should produce something complete, when it is
4960 def __repr__(self):
5061 return '%s(%r)' % (self.__class__.__name__, self.text)
5162
63 def __eq__(self, other):
64 return self.__class__ == other.__class__ and self.text == other.text
65
5266
5367 class OutputOutcome(Outcome):
54 def __unicode__(self):
68 def __str__(self):
5569 return u'output:\n' + self.text
5670
5771
5872 class ErrorOutcome(Outcome):
59 def __unicode__(self):
73 def __str__(self):
6074 return u'error:\n' + self.text
6175
6276
118132 def short_description(self):
119133 return 'expected %r, got %r' % (self.test.expectation, self.actual)
120134
135 def fmt(self, field, contents):
136 if str == unicode: # Python 3
137 if isinstance(contents, bytes):
138 contents = contents.decode('utf-8')
139 s = field + contents
140 print(s)
141 else: # Python 2
142 s = field + contents
143 print(s)
144
121145 def report(self):
122 print "FAILED : " + self.format_text_block(self.test.description)
123 print "Location: " + self.test.body_block.location()
124 print "Function: " + self.format_text_block(self.test.functionality.name)
125 print "Impl : " + self.format_text_block(self.implementation)
126 print "Body : " + self.format_text_block(self.test.body)
127 #if input is not None:
128 #print "Input : " + self.format_text_block(self.test.input)
129 print "Expected: " + self.format_text_block(self.test.expectation)
130 print "Actual : " + self.format_text_block(self.actual)
131 print
146 self.fmt("FAILED : ", self.format_text_block(self.test.description))
147 self.fmt("Location: ", self.test.body_block.location())
148 self.fmt("Function: ", self.format_text_block(self.test.functionality.name))
149 self.fmt("Impl : ", self.format_text_block(self.implementation))
150 self.fmt("Body : ", self.format_text_block(self.test.body))
151 self.fmt("Expected: ", self.format_text_block(self.test.expectation))
152 self.fmt("Actual : ", self.format_text_block(self.actual))
153 print("")
132154
133155 def is_successful(self):
134156 return False
143165 class Block(object):
144166 """A segment of a Falderal-formatted file.
145167
146 >>> b = Block()
147 >>> b.append(u'line 1')
148 >>> b.append(u'line 2')
149 >>> print b.text()
150 line 1
151 line 2
152 >>> print b.text(seperator='')
153 line 1line 2
154 >>> print b.deconstruct()
155 [('', [u'line 1', u'line 2'])]
156
157 >>> b = Block()
158 >>> b.append(u'-> This is a pragma.')
159 >>> b.append(u"| This is some test input.")
160 >>> b.append(u"| It extends over two lines.")
161 >>> b.append(u'? Expected Error')
162 >>> b.append(u'Plain text')
163 >>> b.append(u'More plain text')
164 >>> b.append(u'| Test with input')
165 >>> b.append(u'+ input-for-test')
166 >>> b.append(u'= Expected result on output')
167 >>> b.append(u'= which extends over two lines')
168 >>> print [pair[0] for pair in b.deconstruct()]
169 [u'->', u'| ', u'? ', '', u'| ', u'+ ', u'= ']
170
171 >>> b = Block()
172 >>> b.append(u'-> This is a pragma.')
173 >>> b.append(u'-> which extends over two lines')
174 >>> print b.classify(ParseState())
175 Pragma(line_num=1)
176
177 >>> f = Functionality('foo')
178 >>> b = Block()
179 >>> b.append(u'| Test body here.')
180 >>> b.append(u'= Expected result here.')
181 >>> print b.classify(ParseState(current_functionality=f))
182 Test(body_block=Block(line_num=1), input_block=None,
183 expectation=OutputOutcome(u'Expected result here.'),
184 functionality=Functionality('foo'), desc_block=None,
185 body=u'Test body here.', input=None)
186
187 >>> b = Block()
188 >>> b.append(u'| Test body here.')
189 >>> b.append(u'? Expected error here.')
190 >>> print b.classify(ParseState(current_functionality=f))
191 Test(body_block=Block(line_num=1), input_block=None,
192 expectation=ErrorOutcome(u'Expected error here.'),
193 functionality=Functionality('foo'), desc_block=None,
194 body=u'Test body here.', input=None)
195
196168 """
197169
198170 FREESTYLE_MAP = {
171 u'<= ': u'+ ',
172 u'<== ': u'+ ',
173 u'<=== ': u'+ ',
199174 u'=> ': u'= ',
200175 u'==> ': u'= ',
201176 u'===> ': u'= ',
203178 u'??> ': u'? ',
204179 u'???> ': u'? ',
205180 }
206 FREESTYLE_PREFIXES = FREESTYLE_MAP.keys()
181 FREESTYLE_PREFIXES = list(FREESTYLE_MAP.keys())
207182 PREFIXES = FREESTYLE_PREFIXES + [
208183 u'| ',
209184 u'+ ',
235210 self.__class__.__name__, self.line_num, filename_repr
236211 )
237212
238 def __unicode__(self):
213 def __str__(self):
239214 return unicode(repr(self))
240215
241216 def location(self):
437412 self.lines.append(line)
438413
439414 def parse_lines_to_blocks(self):
440 r"""Parse the lines of the Document into Blocks.
441
442 >>> d = Document()
443 >>> d.append(u'This is a test file.')
444 >>> d.append(u' -> This is a pragma.')
445 >>> d.append(u'')
446 >>> d.append(u" | This is some test input.\n")
447 >>> d.append(u" | It extends over two lines.")
448 >>> d.append(u' ? Expected Error')
449 >>> d.append(u'')
450 >>> d.append(u' | Test with input')
451 >>> d.append(u' + input-for-test')
452 >>> d.append(u' = Expected result on output')
453 >>> blocks = d.parse_lines_to_blocks()
454 >>> [block.lines for block in blocks if isinstance(block, InterveningText)]
455 [[u'This is a test file.'], [u''], [u'']]
456 >>> [b.__class__.__name__ for b in blocks]
457 ['InterveningText', 'Block', 'InterveningText', 'Block', 'InterveningText', 'Block']
458 >>> [b.line_num for b in blocks]
459 [1, 2, 3, 4, 7, 8]
415 """Parse the lines of the Document into Blocks.
460416
461417 """
462418 indent = None
528484 return tests
529485
530486 def extract_tests(self, functionalities):
531 r"""Extract all Tests from this Document.
532
533 >>> functionalities = {}
534 >>> d = Document()
535 >>> d.append(u"This is a text file.")
536 >>> d.append(u'It contains NO tests.')
537 >>> d.extract_tests(functionalities)
538 []
539
540 >>> d = Document()
541 >>> d.append(u'This is a test file.')
542 >>> d.append(u' -> Tests for functionality "Parse Thing"')
543 >>> d.append(u'')
544 >>> d.append(u" | This is some test body.")
545 >>> d.append(u' = Expected result')
546 >>> d.extract_tests(functionalities)
547 [Test(body_block=Block(line_num=4), input_block=None,
548 expectation=OutputOutcome(u'Expected result'),
549 functionality=Functionality(u'Parse Thing'),
550 desc_block=InterveningText(line_num=1),
551 body=u'This is some test body.', input=None)]
552
553 >>> d = Document()
554 >>> d.append(u'This is a test file.')
555 >>> d.append(u' -> Tests for functionality "Parse Thing"')
556 >>> d.append(u'')
557 >>> d.append(u" | This is some test body.")
558 >>> d.append(u" | It extends over two lines.")
559 >>> d.append(u' ? Expected Error')
560 >>> d.append(u'')
561 >>> d.append(u' | Test with input')
562 >>> d.append(u' + input-for-test')
563 >>> d.append(u' = Expected result on output')
564 >>> d.append(u'')
565 >>> d.append(u' + Other input-for-test')
566 >>> d.append(u' = Other Expected result on output')
567 >>> d.append(u'')
568 >>> d.append(u' -> Tests for functionality "Run Thing"')
569 >>> d.append(u'')
570 >>> d.append(u" | Thing")
571 >>> d.append(u' ? Oops')
572 >>> tests = d.extract_tests(functionalities)
573 >>> [t.body for t in tests]
574 [u'This is some test body.\nIt extends over two lines.',
575 u'Test with input', u'Test with input', u'Thing']
576 >>> [t.input_block for t in tests]
577 [None, Block(line_num=8), Block(line_num=12), None]
578 >>> tests[1].input_block.text()
579 u'input-for-test'
580 >>> tests[2].input_block.text()
581 u'Other input-for-test'
582 >>> [t.expectation for t in tests]
583 [ErrorOutcome(u'Expected Error'),
584 OutputOutcome(u'Expected result on output'),
585 OutputOutcome(u'Other Expected result on output'),
586 ErrorOutcome(u'Oops')]
587 >>> [t.functionality.name for t in tests]
588 [u'Parse Thing', u'Parse Thing', u'Parse Thing', u'Run Thing']
589 >>> sorted(functionalities.keys())
590 [u'Parse Thing', u'Run Thing']
591
592 >>> d = Document()
593 >>> d.append(u" | This is some test body.")
594 >>> d.append(u' = Expected')
595 >>> d.extract_tests({})
596 Traceback (most recent call last):
597 ...
598 FalderalSyntaxError: line 1: functionality under test not specified
599
600 >>> d = Document()
601 >>> d.append(u'This is a test file.')
602 >>> d.append(u' ? Expected Error')
603 >>> d.extract_tests({})
604 Traceback (most recent call last):
605 ...
606 FalderalSyntaxError: line 2: expectation must be preceded by test body or test input
607
608 >>> d = Document()
609 >>> d.append(u' -> Hello, this is pragma')
610 >>> d.append(u' = Expected')
611 >>> d.extract_tests({})
612 Traceback (most recent call last):
613 ...
614 FalderalSyntaxError: line 1: incorrectly formatted test block
615
616 >>> d = Document()
617 >>> d.append(u' | This is test')
618 >>> d.append(u'This is text')
619 >>> d.extract_tests({})
620 Traceback (most recent call last):
621 ...
622 FalderalSyntaxError: line 1: test body must be followed by expectation or test input
623
624 >>> d = Document()
625 >>> d.append(u' -> Hello, this is pragma')
626 >>> d.append(u' + Input to where exactly?')
627 >>> d.extract_tests({})
628 Traceback (most recent call last):
629 ...
630 FalderalSyntaxError: line 1: incorrectly formatted test block
631
632 >>> d = Document()
633 >>> funs = {}
634 >>> d.append(u' -> Functionality "Parse Stuff" is implemented by '
635 ... u'shell command "parse"')
636 >>> d.append(u'')
637 >>> d.append(u' -> Functionality "Parse Stuff" is')
638 >>> d.append(u' -> implemented by shell command "pxxxy"')
639 >>> tests = d.extract_tests(funs)
640 >>> len(funs.keys())
641 1
642 >>> [i for i in funs["Parse Stuff"].implementations]
643 [ShellImplementation(u'parse'), ShellImplementation(u'pxxxy')]
487 """Extract all Tests from this Document.
644488
645489 """
646490 blocks = self.parse_lines_to_blocks()
665509 def __repr__(self):
666510 return "Functionality(%r)" % self.name
667511
668 def __unicode__(self):
512 def __str__(self):
669513 return unicode(repr(self))
670514
671515 def add_implementation(self, implementation):
700544 def __repr__(self):
701545 return '%s(%r)' % (self.__class__.__name__, self.callable)
702546
703 def __unicode__(self):
547 def __str__(self):
704548 return u'callable "%r"' % self.callable
705549
706550 def run(self, body=None, input=None):
718562 def __repr__(self):
719563 return '%s(%r)' % (self.__class__.__name__, self.command)
720564
721 def __unicode__(self):
565 def __str__(self):
722566 return u'shell command "%s"' % self.command
723567
568 def __eq__(self, other):
569 return self.__class__ == other.__class__ and self.command == other.command
570
571 def subst(self, command, var_name, value):
572 """Replace all occurrences of `var_name` in `command` with
573 `value`, but make sure `value` is properly shell-escaped first."""
574 return command.replace(var_name, shlex_quote(value))
575
724576 def run(self, body=None, input=None):
725 r"""
726 >>> i = ShellImplementation('cat')
727 >>> i.run(body=u'text')
728 OutputOutcome(u'text')
729
730 >>> i = ShellImplementation('cat fhofhofhf')
731 >>> i.run(body=u'text')
732 ErrorOutcome(u'cat: fhofhofhf: No such file or directory')
733
734 >>> i = ShellImplementation('cat %(test-body-file)')
735 >>> i.run(body=u'text')
736 OutputOutcome(u'text')
737
738 >>> i = ShellImplementation("echo '%(test-body-text)'")
739 >>> i.run(body=u'text')
740 OutputOutcome(u'text')
741
742 >>> i = ShellImplementation('cat >%(output-file)')
743 >>> i.run(body=u'text')
744 OutputOutcome(u'text')
745
746 >>> i = ShellImplementation("echo '%(test-body-text)' '%(test-input-text)'")
747 >>> i.run(body=u'text', input=u'zzrk')
748 OutputOutcome(u'text zzrk')
749
750 Here the body is sent to cat's stdin, but cat ignores it.
751
752 >>> i = ShellImplementation('cat >%(output-file) <%(test-input-file)')
753 >>> i.run(body=u'text', input=u'zzrk')
754 OutputOutcome(u'zzrk')
755
756 """
757 # expand variables in the command
577 # first, expand all known variables in the command, using subst().
758578 test_filename = None
759579 output_filename = None
760580 command = self.command
772592 file.close()
773593 os.close(fd)
774594 # replace all occurrences in command
775 command = re.sub(r'\%\(test-body-file\)', test_filename, command)
595 command = self.subst(command, '%(test-body-file)', test_filename)
776596 command_contained_test_body_file = True
777597
778598 if '%(test-body-text)' in self.command:
779 # escape all single quotes in body
780 body = re.sub(r"'", r"\'", body)
781599 # replace all occurrences in command
782 command = re.sub(r'\%\(test-body-text\)', body, command)
600 command = self.subst(command, '%(test-body-text)', body)
783601 command_contained_test_body_text = True
784602
785603 if '%(test-input-file)' in self.command:
791609 file.close()
792610 os.close(fd)
793611 # replace all occurrences in command
794 command = re.sub(r'\%\(test-input-file\)', test_input_filename, command)
612 command = self.subst(command, '%(test-input-file)', test_input_filename)
795613 command_contained_test_input_file = True
796614
797615 if '%(test-input-text)' in self.command:
798 # escape all single quotes in input
799 body = re.sub(r"'", r"\'", body)
800616 # replace all occurrences in command
801 command = re.sub(r'\%\(test-input-text\)', input, command)
617 command = self.subst(command, '%(test-input-text)', input)
802618 command_contained_test_input_text = True
803619
804620 if '%(output-file)' in self.command:
806622 fd, output_filename = mkstemp()
807623 os.close(fd)
808624 # replace all occurrences in command
809 command = re.sub(r'\%\(output-file\)', output_filename, command)
625 command = self.subst(command, '%(output-file)', output_filename)
810626
811627 # subshell the command and return the output
812628 pipe = Popen(command, shell=True,
845661 return result
846662
847663 def normalize_output(self, text):
848 text = text.decode('UTF-8', errors='ignore')
664 try:
665 text = text.decode('UTF-8', errors='ignore')
666 except AttributeError:
667 pass
849668 text = re.sub(r'\r\n', '\n', text)
850669 return text.strip('\r\n')
851670
862681 a body and/or input may be passed alone.
863682
864683 TODO: maybe write a helper function for that instead.
865
866 >>> b = Block()
867 >>> b.append(u'foo')
868 >>> b.append(u'bar')
869 >>> i = Block()
870 >>> i.append(u'green')
871 >>> t = Test(body_block=b, input_block=i)
872 >>> print t.body
873 foo
874 bar
875 >>> print t.input
876 green
877684
878685 """
879686 def __init__(self, body_block=None, input_block=None, expectation=None,
907714 """Returns a list of Results, one for each implementation of
908715 the functionality being tested.
909716
910 >>> f = Functionality('Cat File')
911 >>> f.add_implementation(CallableImplementation(lambda x, y: x))
912 >>> t = Test(body=u'foo', expectation=OutputOutcome(u'foo'),
913 ... functionality=f)
914 >>> [r.short_description() for r in t.run()]
915 ['success']
916
917 >>> f = Functionality('Cat File')
918 >>> f.add_implementation(CallableImplementation(lambda x, y: x))
919 >>> t = Test(body=u'foo', expectation=OutputOutcome(u'bar'),
920 ... functionality=f)
921 >>> [r.short_description() for r in t.run()]
922 ["expected OutputOutcome(u'bar'), got OutputOutcome(u'foo')"]
923
924 >>> f = Functionality('Cat File')
925 >>> f.add_implementation(CallableImplementation(lambda x, y: x))
926 >>> t = Test(body=u'foo', expectation=ErrorOutcome(u'foo'),
927 ... functionality=f)
928 >>> [r.short_description() for r in t.run()]
929 ["expected ErrorOutcome(u'foo'), got OutputOutcome(u'foo')"]
930
931 >>> f = Functionality('Cat File')
932 >>> def e(x, y):
933 ... raise ValueError(x)
934 >>> f.add_implementation(CallableImplementation(e))
935 >>> t = Test(body=u'foo', expectation=ErrorOutcome(u'foo'),
936 ... functionality=f)
937 >>> [r.short_description() for r in t.run()]
938 ['success']
939
940 >>> f = Functionality('Cat File')
941 >>> def e(x, y):
942 ... raise ValueError(x)
943 >>> f.add_implementation(CallableImplementation(e))
944 >>> t = Test(body=u'foo', expectation=ErrorOutcome(u'bar'),
945 ... functionality=f)
946 >>> [r.short_description() for r in t.run()]
947 ["expected ErrorOutcome(u'bar'), got ErrorOutcome(u'foo')"]
948
949 >>> f = Functionality('Cat File')
950 >>> def e(x, y):
951 ... raise ValueError(x)
952 >>> f.add_implementation(CallableImplementation(e))
953 >>> t = Test(body=u'foo', expectation=OutputOutcome(u'foo'),
954 ... functionality=f)
955 >>> [r.short_description() for r in t.run()]
956 ["expected OutputOutcome(u'foo'), got ErrorOutcome(u'foo')"]
957
958 >>> f = Functionality('Cat File with Input')
959 >>> f.add_implementation(CallableImplementation(lambda x, y: x + y))
960 >>> t = Test(body=u'foo', input=u'bar', expectation=OutputOutcome(u'foobar'),
961 ... functionality=f)
962 >>> [r.short_description() for r in t.run()]
963 ['success']
964
965 A functionality can have multiple implementations. We test them all.
966
967 >>> f = Functionality('Cat File')
968 >>> def c1(body, input):
969 ... return body
970 >>> def c2(body, input):
971 ... return body + '...'
972 >>> def c3(body, input):
973 ... raise ValueError(body)
974 >>> for c in (c1, c2, c3):
975 ... f.add_implementation(CallableImplementation(c))
976 >>> t = Test(body=u'foo', expectation=OutputOutcome(u'foo'),
977 ... functionality=f)
978 >>> [r.short_description() for r in t.run()]
979 ['success', "expected OutputOutcome(u'foo'), got OutputOutcome(u'foo...')",
980 "expected OutputOutcome(u'foo'), got ErrorOutcome(u'foo')"]
981
982717 """
983718 results = []
984719 for implementation in self.functionality.implementations:
0 # Note: these are unit tests for py-falderal itself,
1 # not tests that Falderal can understand.
2
3 import unittest
4 from unittest import TestCase
5
6 from falderal.objects import (
7 Block, Pragma,
8 ParseState, InterveningText,
9 Document,
10 Functionality, ShellImplementation,
11 Test, OutputOutcome, ErrorOutcome,
12 FalderalSyntaxError,
13 )
14
15
16 class BlockTestCase(TestCase):
17 def test_block(self):
18 b = Block()
19 b.append(u'line 1')
20 b.append(u'line 2')
21 self.assertEqual(b.text(), "line 1\nline 2")
22 self.assertEqual(b.text(seperator=''), "line 1line 2")
23 self.assertEqual(
24 b.deconstruct(),
25 [('', [u'line 1', u'line 2'])]
26 )
27
28 def test_deconstruct_block(self):
29 b = Block()
30 b.append(u'-> This is a pragma.')
31 b.append(u"| This is some test input.")
32 b.append(u"| It extends over two lines.")
33 b.append(u'? Expected Error')
34 b.append(u'Plain text')
35 b.append(u'More plain text')
36 b.append(u'| Test with input')
37 b.append(u'+ input-for-test')
38 b.append(u'= Expected result on output')
39 b.append(u'= which extends over two lines')
40 self.assertEqual(
41 [pair[0] for pair in b.deconstruct()],
42 [u'->', u'| ', u'? ', '', u'| ', u'+ ', u'= ']
43 )
44
45 def test_classify_block_pragma(self):
46 b = Block()
47 b.append(u'-> This is a pragma.')
48 b.append(u'-> which extends over two lines')
49 result = b.classify(ParseState())
50 self.assertIsInstance(result, Pragma)
51 self.assertEqual(result.lines, [u' This is a pragma.', u' which extends over two lines'])
52 self.assertEqual(result.line_num, 1)
53
54 def test_classify_block_success_test(self):
55 f = Functionality('foo')
56 b = Block()
57 b.append(u'| Test body here.')
58 b.append(u'= Expected result here.')
59 result = b.classify(ParseState(current_functionality=f))
60 self.assertIsInstance(result, Test)
61 self.assertEqual(result.body_block.lines, [u'Test body here.'])
62 self.assertEqual(result.body_block.line_num, 1)
63 self.assertEqual(result.body_block.filename, None)
64 self.assertEqual(result.input_block, None)
65 self.assertEqual(result.expectation, OutputOutcome(u'Expected result here.'))
66 self.assertEqual(result.functionality, f)
67 self.assertEqual(result.desc_block, None)
68 self.assertEqual(result.body, u'Test body here.')
69 self.assertEqual(result.input, None)
70
71 def test_classify_block_error_test(self):
72 f = Functionality('foo')
73 b = Block()
74 b.append(u'| Test body here.')
75 b.append(u'? Expected error here.')
76 result = b.classify(ParseState(current_functionality=f))
77 self.assertIsInstance(result, Test)
78 self.assertEqual(result.body_block.lines, [u'Test body here.'])
79 self.assertEqual(result.body_block.line_num, 1)
80 self.assertEqual(result.body_block.filename, None)
81 self.assertEqual(result.input_block, None)
82 self.assertEqual(result.expectation, ErrorOutcome(u'Expected error here.'))
83 self.assertEqual(result.functionality, f)
84 self.assertEqual(result.desc_block, None)
85 self.assertEqual(result.body, u'Test body here.')
86 self.assertEqual(result.input, None)
87
88
89 class DocumentTestCase(TestCase):
90 def test_document(self):
91 d = Document()
92 d.append(u'This is a test file.')
93 d.append(u' -> This is a pragma.')
94 d.append(u'')
95 d.append(u" | This is some test input.\n")
96 d.append(u" | It extends over two lines.")
97 d.append(u' ? Expected Error')
98 d.append(u'')
99 d.append(u' | Test with input')
100 d.append(u' + input-for-test')
101 d.append(u' = Expected result on output')
102 blocks = d.parse_lines_to_blocks()
103 self.assertEqual(
104 [block.lines for block in blocks if isinstance(block, InterveningText)],
105 [[u'This is a test file.'], [u''], [u'']]
106 )
107 self.assertEqual(
108 [b.__class__.__name__ for b in blocks],
109 ['InterveningText', 'Block', 'InterveningText', 'Block', 'InterveningText', 'Block']
110 )
111 self.assertEqual(
112 [b.line_num for b in blocks],
113 [1, 2, 3, 4, 7, 8]
114 )
115
116 def test_extract_tests_empty(self):
117 d = Document()
118 d.append(u"This is a text file.")
119 d.append(u'It contains NO tests.')
120 functionalities = {}
121 self.assertEqual(d.extract_tests(functionalities), [])
122
123 def test_extract_tests_basic(self):
124 d = Document()
125 d.append(u'This is a test file.')
126 d.append(u' -> Tests for functionality "Parse Thing"')
127 d.append(u'')
128 d.append(u" | This is some test body.")
129 d.append(u' = Expected result')
130 functionalities = {}
131 tests = d.extract_tests(functionalities)
132 self.assertEqual(len(tests), 1)
133 result = tests[0]
134 self.assertIsInstance(result, Test)
135 self.assertEqual(result.body_block.lines, [u'This is some test body.'])
136 self.assertEqual(result.body_block.line_num, 4)
137 self.assertEqual(result.body_block.filename, None)
138 self.assertEqual(result.input_block, None)
139 self.assertEqual(result.expectation, OutputOutcome(u'Expected result'))
140 self.assertEqual(result.functionality, functionalities['Parse Thing'])
141 self.assertEqual(result.desc_block.__class__, InterveningText)
142 self.assertEqual(result.desc_block.lines, [u'This is a test file.'])
143 self.assertEqual(result.body, u'This is some test body.')
144 self.assertEqual(result.input, None)
145
146 def test_extract_tests_more(self):
147 d = Document()
148 d.append(u'This is a test file.')
149 d.append(u' -> Tests for functionality "Parse Thing"')
150 d.append(u'')
151 d.append(u" | This is some test body.")
152 d.append(u" | It extends over two lines.")
153 d.append(u' ? Expected Error')
154 d.append(u'')
155 d.append(u' | Test with input')
156 d.append(u' + input-for-test')
157 d.append(u' = Expected result on output')
158 d.append(u'')
159 d.append(u' + Other input-for-test')
160 d.append(u' = Other Expected result on output')
161 d.append(u'')
162 d.append(u' -> Tests for functionality "Run Thing"')
163 d.append(u'')
164 d.append(u" | Thing")
165 d.append(u' ? Oops')
166 functionalities = {}
167 tests = d.extract_tests(functionalities)
168 self.assertEqual(
169 [t.body for t in tests],
170 [u'This is some test body.\nIt extends over two lines.',
171 u'Test with input', u'Test with input', u'Thing']
172 )
173 self.assertEqual(
174 [t.input_block.__class__ for t in tests],
175 [None.__class__, Block, Block, None.__class__]
176 )
177 self.assertEqual(
178 [t.input_block.text() for t in tests if t.input_block is not None],
179 [u'input-for-test', u'Other input-for-test']
180 )
181 self.assertEqual(
182 [t.expectation for t in tests],
183 [ErrorOutcome(u'Expected Error'),
184 OutputOutcome(u'Expected result on output'),
185 OutputOutcome(u'Other Expected result on output'),
186 ErrorOutcome(u'Oops')]
187 )
188 self.assertEqual(
189 [t.functionality.name for t in tests],
190 [u'Parse Thing', u'Parse Thing', u'Parse Thing', u'Run Thing']
191 )
192 self.assertEqual(
193 sorted(functionalities.keys()),
194 [u'Parse Thing', u'Run Thing']
195 )
196
197 def test_no_functionality_under_test(self):
198 d = Document()
199 d.append(u" | This is some test body.")
200 d.append(u' = Expected')
201 with self.assertRaises(FalderalSyntaxError) as ar:
202 d.extract_tests({})
203 self.assertEqual(str(ar.exception), "line 1: functionality under test not specified")
204
205 def test_expectation_in_bad_place(self):
206 d = Document()
207 d.append(u'This is a test file.')
208 d.append(u' ? Expected Error')
209 with self.assertRaises(FalderalSyntaxError) as ar:
210 d.extract_tests({})
211 self.assertEqual(str(ar.exception), "line 2: expectation must be preceded by test body or test input")
212
213 def test_badly_formatted_test_block(self):
214 d = Document()
215 d.append(u' -> Hello, this is pragma')
216 d.append(u' = Expected')
217 with self.assertRaises(FalderalSyntaxError) as ar:
218 d.extract_tests({})
219 self.assertEqual(str(ar.exception), "line 1: incorrectly formatted test block")
220
221 def test_body_not_followed_by_anything_sensible(self):
222 d = Document()
223 d.append(u' | This is test')
224 d.append(u'This is text')
225 with self.assertRaises(FalderalSyntaxError) as ar:
226 d.extract_tests({})
227 self.assertEqual(str(ar.exception), "line 1: test body must be followed by expectation or test input")
228
229 def test_another_badly_formatted_block(self):
230 d = Document()
231 d.append(u' -> Hello, this is pragma')
232 d.append(u' + Input to where exactly?')
233 with self.assertRaises(FalderalSyntaxError) as ar:
234 d.extract_tests({})
235 self.assertEqual(str(ar.exception), "line 1: incorrectly formatted test block")
236
237 def test_parse_functionalities(self):
238 d = Document()
239 funs = {}
240 d.append(u' -> Functionality "Parse Stuff" is implemented by '
241 u'shell command "parse"')
242 d.append(u'')
243 d.append(u' -> Functionality "Parse Stuff" is')
244 d.append(u' -> implemented by shell command "pxxxy"')
245 tests = d.extract_tests(funs)
246 self.assertEqual(list(funs.keys()), ['Parse Stuff'])
247 self.assertEqual(
248 [i for i in funs["Parse Stuff"].implementations],
249 [ShellImplementation(u'parse'), ShellImplementation(u'pxxxy')]
250 )
251
252
253 class ShellImplementationTestCase(TestCase):
254 def test_cat(self):
255 i = ShellImplementation('cat')
256 self.assertEqual(i.run(body=u'text'), OutputOutcome(u'text'))
257
258 def test_cat_file(self):
259 i = ShellImplementation('cat fhofhofhf')
260 self.assertEqual(i.run(body=u'text'), ErrorOutcome(u'cat: fhofhofhf: No such file or directory'))
261
262 def test_cat_test_body_file(self):
263 i = ShellImplementation('cat %(test-body-file)')
264 self.assertEqual(i.run(body=u'text'), OutputOutcome(u'text'))
265
266 def test_cat_test_body_text(self):
267 i = ShellImplementation("echo '%(test-body-text)'")
268 self.assertEqual(i.run(body=u'text'), OutputOutcome(u'text'))
269
270 def test_cat_output_file(self):
271 i = ShellImplementation('cat >%(output-file)')
272 self.assertEqual(i.run(body=u'text'), OutputOutcome(u'text'))
273
274 def test_echo(self):
275 i = ShellImplementation("echo '%(test-body-text)' '%(test-input-text)'")
276 self.assertEqual(i.run(body=u'text', input=u'zzrk'), OutputOutcome(u'text zzrk'))
277
278 def test_cat_stdin(self):
279 # Here the body is sent to cat's stdin, but cat ignores it.
280 i = ShellImplementation('cat >%(output-file) <%(test-input-file)')
281 self.assertEqual(i.run(body=u'text', input=u'zzrk'), OutputOutcome(u'zzrk'))
282
283
284 def TestTestCase(TestCase):
285 def test_test_contents(self):
286 b = Block()
287 b.append(u'foo')
288 b.append(u'bar')
289 i = Block()
290 i.append(u'green')
291 t = Test(body_block=b, input_block=i)
292 self.assertEqual(t.body, "foo\nbar")
293 self.assertEqual(t.input, "green")
294
295 def test_tests_1(self):
296 f = Functionality('Cat File')
297 f.add_implementation(CallableImplementation(lambda x, y: x))
298 t = Test(body=u'foo', expectation=OutputOutcome(u'foo'), functionality=f)
299 self.assertEqual(
300 [r.short_description() for r in t.run()]
301 ['success']
302 )
303
304 def test_tests_2(self):
305 f = Functionality('Cat File')
306 f.add_implementation(CallableImplementation(lambda x, y: x))
307 t = Test(body=u'foo', expectation=OutputOutcome(u'bar'),
308 functionality=f)
309 self.assertEqual(
310 [r.short_description() for r in t.run()],
311 ["expected OutputOutcome(u'bar'), got OutputOutcome(u'foo')"]
312 )
313
314 def test_tests_3(self):
315 f = Functionality('Cat File')
316 f.add_implementation(CallableImplementation(lambda x, y: x))
317 t = Test(body=u'foo', expectation=ErrorOutcome(u'foo'),
318 functionality=f)
319 self.assertEqual(
320 [r.short_description() for r in t.run()],
321 ["expected ErrorOutcome(u'foo'), got OutputOutcome(u'foo')"]
322 )
323
324 def test_tests_4(self):
325 f = Functionality('Cat File')
326 def e(x, y):
327 raise ValueError(x)
328 f.add_implementation(CallableImplementation(e))
329 t = Test(body=u'foo', expectation=ErrorOutcome(u'foo'),
330 functionality=f)
331 self.assertEqual(
332 [r.short_description() for r in t.run()],
333 ['success']
334 )
335
336 def test_tests_5(self):
337 f = Functionality('Cat File')
338 def e(x, y):
339 raise ValueError(x)
340 f.add_implementation(CallableImplementation(e))
341 t = Test(body=u'foo', expectation=ErrorOutcome(u'bar'),
342 functionality=f)
343 self.assertEqual(
344 [r.short_description() for r in t.run()],
345 ["expected ErrorOutcome(u'bar'), got ErrorOutcome(u'foo')"]
346 )
347
348 def test_tests_6(self):
349 f = Functionality('Cat File')
350 def e(x, y):
351 raise ValueError(x)
352 f.add_implementation(CallableImplementation(e))
353 t = Test(body=u'foo', expectation=OutputOutcome(u'foo'),
354 functionality=f)
355 self.assertEqual(
356 [r.short_description() for r in t.run()],
357 ["expected OutputOutcome(u'foo'), got ErrorOutcome(u'foo')"]
358 )
359
360 def test_tests_7(self):
361 f = Functionality('Cat File with Input')
362 f.add_implementation(CallableImplementation(lambda x, y: x + y))
363 t = Test(body=u'foo', input=u'bar', expectation=OutputOutcome(u'foobar'),
364 functionality=f)
365 self.assertEqual(
366 [r.short_description() for r in t.run()],
367 ['success']
368 )
369
370 def test_functionality_with_multiple_implementations(self):
371 # A functionality can have multiple implementations. We test them all.
372
373 f = Functionality('Cat File')
374 def c1(body, input):
375 return body
376 def c2(body, input):
377 return body + '...'
378 def c3(body, input):
379 raise ValueError(body)
380 for c in (c1, c2, c3):
381 f.add_implementation(CallableImplementation(c))
382 t = Test(body=u'foo', expectation=OutputOutcome(u'foo'),
383 functionality=f)
384 self.assertEqual(
385 [r.short_description() for r in t.run()],
386 ['success', "expected OutputOutcome(u'foo'), got OutputOutcome(u'foo...')",
387 "expected OutputOutcome(u'foo'), got ErrorOutcome(u'foo')"]
388 )
389
390
391 if __name__ == '__main__':
392 unittest.main()
11
22 # Really crude test harness for py-falderal itself...
33
4 bin/falderal -v -t || exit 1
4 if [ "x$PYTHON" = "x" ]; then
5 PYTHON="python3"
6 fi
7 FALDERAL="$PYTHON ../bin/falderal"
8
9 PYTHONPATH=src $PYTHON src/falderal/tests.py -v || exit 1
510
611 cd tests
712
813 FIRST_TESTS="
914 test-pass-fail test-no-functionality test-ill-formed test-no-test-body
10 test-var-subst-no-eol
15 test-var-subst test-no-eol
1116 test-utf8 test-crlf
1217 test-bad-indentation
1318 test-input-sections test-shared-body
1621 "
1722 for TEST in ${FIRST_TESTS}; do
1823 echo ${TEST}...
19 ../bin/falderal --cavalier ${TEST}.markdown > ${TEST}.actual 2>&1
24 $FALDERAL --cavalier ${TEST}.markdown > ${TEST}.actual 2>&1
2025 diff -u ${TEST}.expected ${TEST}.actual || exit 1
2126 done
2227
2429 LINTING_TESTS="test-no-tests"
2530 for TEST in ${LINTING_TESTS}; do
2631 echo ${TEST}...
27 ../bin/falderal ${TEST}.markdown > ${TEST}.actual 2>&1
32 $FALDERAL ${TEST}.markdown > ${TEST}.actual 2>&1
2833 diff -u ${TEST}.expected ${TEST}.actual || exit 1
2934 done
3035
3338 "
3439 for TEST in ${TWO_PART_TESTS}; do
3540 echo ${TEST}...
36 ../bin/falderal ${TEST}-a.markdown ${TEST}-b.markdown > ${TEST}.actual 2>&1
41 $FALDERAL ${TEST}-a.markdown ${TEST}-b.markdown > ${TEST}.actual 2>&1
3742 diff -u ${TEST}.expected ${TEST}.actual || exit 1
3843 done
3944
4045 # special tests: -b
4146 TEST=test-substring-error
4247 echo ${TEST}...
43 ../bin/falderal -b ${TEST}.markdown > ${TEST}.actual 2>&1
48 $FALDERAL -b ${TEST}.markdown > ${TEST}.actual 2>&1
4449 diff -u ${TEST}.expected ${TEST}.actual || exit 1
4550
4651 rm -f *.actual
1212 -> shell command "python cat.py -o %(output-file)"
1313
1414 -> Functionality "Cat" is implemented by
15 -> shell command "python echo.py '%(test-body-text)'"
15 -> shell command "python echo.py %(test-body-text)"
55 to be on `stdout`.
66
77 -> Functionality "Error on stdout" is implemented by shell command
8 -> "python fail.py '%(test-body-text)' '' 1"
8 -> "python fail.py %(test-body-text) '' 1"
99
1010 -> Tests for functionality "Error on stdout"
1111
3030 Location: test-freestyle-format.markdown, line 68
3131 Function: Cat
3232 Impl : shell command "python cat.py"
33 Body : meow
33 Body :
34 | meow
35 ? rreoww
3436 Expected: error:
3537 bow
3638 wow
3739 Actual : output:
38 meow
40 | meow
41 ? rreoww
42
43 FAILED :
44
45 The trick of re-using the previous test body with a different
46 test input if the test body is omitted doesn't work with
47 freestyle-format test input sections (i.e., this will fail.)
48
49 Location: test-freestyle-format.markdown, line 111
50 Function: Silly Interpreter
51 Impl : shell command "python silly-interpreter.py %(test-body-file)"
52 Body :
53 Expected: output:
54 zang
55 zing
56 Actual : output:
57
3958
4059 --------------------------------
41 Total test runs: 10, failures: 3
60 Total test runs: 14, failures: 4
4261 --------------------------------
0 Falderal Test: "new" format for tests
0 Falderal Test: "freestyle" format for tests
11 -------------------------------------
22
3 This document tests the alternate format for tests introduced in
4 the Falderal Literate Test Format version 0.11. This format allows
5 the test body to consist entirely of un-prefixed text, as long as
6 it is written in a single indented block, and the final line(s) of
7 the test begin with one of the prefixes `=> ` or `==> ` or `===> `
8 or `?> ` or `??> `or `???> `.
3 This document tests the alternate ("freestyle") format for tests
4 introduced in the Falderal Literate Test Format version 0.11. This
5 format allows the test body to consist entirely of un-prefixed text,
6 as long as it is written in a single indented block, and as long as
7 the final line(s) of the test begin with one of the prefixes
8 `=> ` or `==> ` or `===> ` or `?> ` or `??> `or `???> `.
99
1010 -> Functionality "Cat" is implemented by shell command "python cat.py"
1111
3737
3838 | purr
3939 | prrr
40 | prreow
41 ==> | purr
42 ==> | prrr
43 ==> | prreow
40 = prreow
41 => | purr
42 => | prrr
43 => = prreow
4444
4545 | purr
4646 + prrr
47 + prreow
47 ? prreow
4848 => | purr
4949 => + prrr
50 => + prreow
50 => ? prreow
5151
5252 purr
5353 -> prrr
6464 meow
6565 ??> woof
6666
67 meow
67 | meow
68 ? rreoww
6869 ???> bow
6970 ???> wow
71
72 Freestyle-format tests can also contain input sections.
73
74 -> Functionality "Silly Interpreter" is implemented by
75 -> shell command "python silly-interpreter.py %(test-body-file)"
76
77 -> Tests for functionality "Silly Interpreter"
78
79 read x
80 read y
81 print y
82 print x
83 <= meow
84 <= purr
85 => purr
86 => meow
87
88 read x
89 read y
90 print y
91 print x
92 <== meow
93 <== purr
94 ==> purr
95 ==> meow
96
97 read x
98 read y
99 print y
100 print x
101 <=== meow
102 <=== purr
103 ===> purr
104 ===> meow
105
106 The trick of re-using the previous test body with a different
107 test input if the test body is omitted doesn't work with
108 freestyle-format test input sections (i.e., this will fail.)
109
110 <== zing
111 <== zang
112 ==> zang
113 ==> zing
00 --------------------------------
1 Total test runs: 5, failures: 0
1 Total test runs: 6, failures: 0
22 --------------------------------
3737 + purr
3838 = purr
3939 = meow
40
41 If the input section appears first (i.e. there is no test body),
42 the previous test body is re-used.
43
44 + zing
45 + zang
46 = zang
47 = zing
0 --------------------------------
1 Total test runs: 4, failures: 0
2 --------------------------------
0 Falderal Test: Missing EOL at end of output
1 ----------------------------
2
3 It makes no difference whether there is an EOL at the end of
4 the test output or not.
5
6 -> Functionality "Echo" is implemented by
7 -> shell command "python echo.py %(test-body-text)"
8
9 -> Tests for functionality "Echo"
10
11 | hello
12 = hello
13
14 | hi
15 | hi
16 = hi
17 = hi
18
19 -> Functionality "Echo, no newline" is implemented by
20 -> shell command "python echo.py -n %(test-body-text)"
21
22 -> Tests for functionality "Echo, no newline"
23
24 | hello
25 = hello
26
27 | hi
28 | hi
29 = hi
30 = hi
0 Falderal Test 4
0 Falderal Test: no test body
11 ---------------
22
33 Another Falderal document which is ill-formed.
44 expected text is matched against standard output.
55
66 -> Functionality "Succeed" is implemented by shell command
7 -> "python fail.py '%(test-body-text)' bar 0"
7 -> "python fail.py %(test-body-text) bar 0"
88
99 -> Tests for functionality "Succeed"
1010
1515 output and standard error, it's up to you to redirect them.
1616
1717 -> Functionality "Succeed/All" is implemented by shell command
18 -> "python fail.py '%(test-body-text)' bar 0 2>&1"
18 -> "python fail.py %(test-body-text) bar 0 2>&1"
1919
2020 -> Tests for functionality "Succeed/All"
2121
2727 expected text is matched against standard error.
2828
2929 -> Functionality "Fail" is implemented by shell command
30 -> "python fail.py foo '%(test-body-text)' 1"
30 -> "python fail.py foo %(test-body-text) 1"
3131
3232 -> Tests for functionality "Fail"
3333
3838 output and standard error, it's up to you to redirect them.
3939
4040 -> Functionality "Fail/All" is implemented by shell command
41 -> "python fail.py foo '%(test-body-text)' 1 1>&2"
41 -> "python fail.py foo %(test-body-text) 1 1>&2"
4242
4343 -> Tests for functionality "Fail/All"
4444
66 the expected error text.
77
88 -> Functionality "Fail" is implemented by shell command
9 -> "python fail.py foo '%(test-body-text)' 1"
9 -> "python fail.py foo %(test-body-text) 1"
1010
1111 -> Tests for functionality "Fail"
1212
+0
-3
tests/test-var-subst-no-eol.expected less more
0 --------------------------------
1 Total test runs: 10, failures: 0
2 --------------------------------
+0
-76
tests/test-var-subst-no-eol.markdown less more
0 Falderal Test 5
1 ---------------
2
3 Tests for variable substitution, and missing EOL at end
4 of output.
5
6 Note the use of single quotes around the `%(test-body-text)` variable;
7 without these, shell chaos is likely to result.
8
9 -> Functionality "Echo" is implemented by
10 -> shell command "python echo.py '%(test-body-text)'"
11
12 -> Tests for functionality "Echo"
13
14 | hello
15 = hello
16
17 | hi
18 | hi
19 = hi
20 = hi
21
22 -> Functionality "Echo, no newline" is implemented by
23 -> shell command "python echo.py -n '%(test-body-text)'"
24
25 -> Tests for functionality "Echo, no newline"
26
27 | hello
28 = hello
29
30 | hi
31 | hi
32 = hi
33 = hi
34
35 Note that single quotes needn't be supplied around `%(test-body-file)`
36 or `%(output-file)`.
37
38 -> Functionality "Cat, from file" is implemented by
39 -> shell command "python cat.py -f %(test-body-file)"
40
41 -> Tests for functionality "Cat, from file"
42
43 | hello
44 = hello
45
46 | hi
47 | hi
48 = hi
49 = hi
50
51 -> Functionality "Cat, to file" is implemented by
52 -> shell command "python cat.py -o %(output-file)"
53
54 -> Tests for functionality "Cat, to file"
55
56 | hello
57 = hello
58
59 | hi
60 | hi
61 = hi
62 = hi
63
64 -> Functionality "Cat, to and from file" is implemented by
65 -> shell command "python cat.py -f %(test-body-file) -o %(output-file)"
66
67 -> Tests for functionality "Cat, to and from file"
68
69 | hello
70 = hello
71
72 | hi
73 | hi
74 = hi
75 = hi
0 --------------------------------
1 Total test runs: 13, failures: 0
2 --------------------------------
0 Falderal Test: Variable substitution
1 ------------------------------------
2
3 Tests for variable substitution, and quoting rules during such.
4
5 Note that Falderal is responsible for quoting the substitution text
6 of all `%(...)` variables occurring in a shell command template;
7 it is not necessary to put any quotes around them in the template string.
8
9 -> Functionality "Echo Body" is implemented by shell command
10 -> "python echo.py %(test-body-text)"
11
12 -> Tests for functionality "Echo Body"
13
14 | foo
15 + bar
16 = foo
17
18 Single quotes in the test body text are single escaped.
19
20 | don't
21 + can't
22 = don't
23
24 -> Functionality "Echo Input" is implemented by shell command
25 -> "python echo.py %(test-input-text)"
26
27 -> Tests for functionality "Echo Input"
28
29 | foo
30 + bar
31 = bar
32
33 Single quotes in the test input text are single escaped.
34
35 | don't
36 + can't
37 = can't
38
39 Note that when variables are expanded, backslash sequences in the
40 replacement string ("\n", etc) are not expanded.
41
42 -> Tests for functionality "Echo Body"
43
44 | he\nl\tl\\o
45 = he\nl\tl\\o
46
47 The rule that Falderal is responsible for quoting text substituted
48 into the command template extends to `%(test-body-file)` and
49 `%(test-input-file)` and `%(output-file)` as well.
50
51 -> Functionality "Cat, from file" is implemented by
52 -> shell command "python cat.py -f %(test-body-file)"
53
54 -> Tests for functionality "Cat, from file"
55
56 | hello
57 = hello
58
59 | hi
60 | hi
61 = hi
62 = hi
63
64 -> Functionality "Cat, to file" is implemented by
65 -> shell command "python cat.py -o %(output-file)"
66
67 -> Tests for functionality "Cat, to file"
68
69 | hello
70 = hello
71
72 | hi
73 | hi
74 = hi
75 = hi
76
77 -> Functionality "Cat, to and from file" is implemented by
78 -> shell command "python cat.py -f %(test-body-file) -o %(output-file)"
79
80 -> Tests for functionality "Cat, to and from file"
81
82 | hello
83 = hello
84
85 | hi
86 | hi
87 = hi
88 = hi
89
90 -> Functionality "Cat input, from file" is implemented by
91 -> shell command "python cat.py -f %(test-input-file)"
92
93 -> Tests for functionality "Cat input, from file"
94
95 | hekko
96 + hello
97 = hello
98
99 | hj
100 | hj
101 + hi
102 + hi
103 = hi
104 = hi