git @ Cat's Eye Technologies Falderal / 3996f8a
Import latest version of py-falderal sources. catseye 7 years ago
42 changed file(s) with 1430 addition(s) and 2 deletion(s). Raw diff Collapse all Expand all
0 bin
0 impl/Test.Falderal/bin
11 *.o
22 *.hi
00 syntax: regexp
11
2 ^bin/
2 ^impl/Test.Falderal/bin/
33
44 syntax: glob
55
0 #!/usr/bin/env python
1
2 from os.path import realpath, dirname, join
3 import sys
4 from optparse import OptionParser
5
6 sys.path.insert(0, join(dirname(realpath(sys.argv[0])), '..', 'src'))
7
8 from falderal.driver import main
9
10
11 if __name__ == '__main__':
12 sys.exit(main(sys.argv))
0 `py-falderal`
1 =============
2
3 `py-falderal` is an implementation of Falderal in Python 2.5.x.
4
5 Motivation
6 ----------
7
8 There are a few reasons I had for re-implementing Falderal in Python:
9
10 * The original Falderal implementation grew out of a Haskell-specific hack,
11 and it shows in how it's written.
12
13 * Fewer discrepancies between platforms. In particular, `ghc` for Windows
14 likes to operate in terms of MS-DOS end-of-lines (`CR`, `LF`), but I tend
15 to use it under Cygwin using Unix end-of-lines (`LF`).
16
17 * Smaller install burden: Python sometimes comes bundled with the operating
18 system; Haskell rarely does.
19
20 * Haskell, being lazy, makes it harder to deal with exceptions; unless the
21 Haskell expression is being evaluated both strictly and deeply, exceptions
22 can slip by. For Falderal's purposes, this seems artificial, at best.
23
24 * Relatedly, Python (or CPython, anyway) has better error behavior than
25 Haskell (or `ghc`, anyway); when it crashes, it dumps a backtrace (which I
26 can then analyze), instead of just saying something pithy like `Prelude:
27 empty list` (which I can't.)
28
29 * Any standard worth its salt should probably have more than one
30 implementation, anyway.
31
32 Features
33 --------
34
35 That last point notwithstanding, `py-falderal` implements a slightly different
36 subset of the Falderal test file format than the Haskell implementation does.
37
38 In particular,
39
40 * It mainly implements `shell command` implementations. In practice, partly
41 due to the "strict & deep" evaluation problem mentioned above, that's how
42 I've been using Falderal anyway; also, its approach makes it somewhat more
43 useful for "end-to-end" testing of compilers and interpreters, than for
44 unit-testing individual text-processing functions inside a program.
45 (Technically, it implements `Haskell function` implementations too, but it's
46 something of a hack that uses `ghc -e`.)
47
48 * I plan for it to *only* understand indented Falderal blocks. The idea is
49 that the Falderal tests will almost certainly be embedded in a Markdown
50 document (possibly with Bird-style literate code also embedded therein,)
51 and no extra processing should be required to format something readable
52 from that Markdown. The four-space-indented Falderal blocks come out as
53 preformatted blocks, which is quite good enough. Relatedly,
54
55 * It does no formatting. There are a set of classes for parsing Falderal
56 files (`Document`s) into blocks (`Block`s), and methods for formatting
57 blocks into text. But aside from reporting the results of a test run,
58 they're not used by the utility to format Falderal files. If you want to
59 do extra processing on your Falderal/Markdown file so that Falderal
60 blocks are prettier, you certainly can do that, but you'll need to write
61 your own script which uses these classes, for it is outside the scope of
62 `py-falderal`.
63
64 * I'm not so sure about `-> Functionality "blah" is implemented by shell
65 command "flargh"` anymore. I mean, it should certainly be the case that
66 functionalities can have multiple implementations, but
67 * Perhaps implementations should not be specified in Falderal documents
68 at all -- that approach is more abstract. But it requires them to be
69 specified on the tool's command line, which in practice requires there
70 to be a driver script to run the tests, for a particular implementation;
71 but this is not necessarily a bad thing.
72 * When `falderal` is run on more than one input file, what is the scope
73 of a functionality, and what is the scope of the implementations of a
74 functionality? Currently, in the Falderal semantics, that scope is
75 global, and maybe that is appropriate; but maybe finer-grained control
76 would be nice.
77
78 * `py-falderal` also does not try to optimize the test runs into a block of
79 test runs (which didn't work out so well in the Haskell implementation,
80 for `shell command`s, anyway.)
(New empty file)
Binary diff not shown
0 """"\
1 Usage: falderal [<option>...] <filename.markdown>...
2 """
3
4 from optparse import OptionParser
5 import sys
6
7 from falderal.objects import Document, FalderalSyntaxError
8
9
10 def main(args):
11 parser = OptionParser()
12
13 parser.add_option("-b", "--substring-error",
14 action="store_true", default=False,
15 help="match expected errors as substrings")
16 parser.add_option("-c", "--clear-functionalities",
17 metavar="NAMES", default=None,
18 help="clear all implementations of the specified "
19 "functionalities that were specified in pragmas "
20 "in documents; useful in conjunction with -f. "
21 "Format of the argument is a colon-seperated "
22 "list of functionality names.")
23 parser.add_option("-d", "--dump",
24 action="store_true", default=False)
25 parser.add_option("-f", "--functionalities",
26 metavar="SPEC",
27 help="specify implementations of functionalies, "
28 "over and above what are specified in pragmas "
29 "in documents (not yet implemented)")
30 parser.add_option("-t", "--test",
31 action="store_true", default=False,
32 help="run internal tests and exit")
33
34 (options, args) = parser.parse_args(args[1:])
35
36 # for compatibility with previous versions of falderal
37 if args and args[0] == 'test':
38 args = args[1:]
39
40 if options.test:
41 import doctest
42 import falderal.objects
43 (failure_count, test_count) = \
44 doctest.testmod(falderal.objects,
45 optionflags=doctest.NORMALIZE_WHITESPACE)
46 if failure_count > 0:
47 return 1
48 else:
49 return 0
50
51 try:
52 return test_documents(options, args)
53 except FalderalSyntaxError as e:
54 # if options.show_full_exception: do that, else
55 sys.stderr.write('%s: %s\n' % (e.__class__.__name__, str(e)))
56 return 1
57
58
59 def test_documents(options, args):
60 # load Falderal documents
61 documents = []
62 for filename in args:
63 documents.append(Document.load(filename))
64
65 # collect functionalities
66 # XXX if any implementations are given in the command line,
67 # we can put them in here.
68 functionalities = {}
69
70 # create Falderal Tests
71 tests = []
72 for document in documents:
73 tests += document.parse_blocks_to_tests(functionalities)
74
75 if options.clear_functionalities:
76 for name in options.clear_functionalities.split(':'):
77 n = name.strip()
78 if n in functionalities:
79 functionalities[n].implementations = []
80
81 # XXX lint: check for no tests, or no implementations of a functionality
82 # that is being tested, or a functionality that is not being tested, and
83 # break with an error unless some option to suppress this is present
84
85 if options.dump:
86 print "Functionalities:"
87 for name in functionalities:
88 print " " + name
89 for implementation in functionalities[name].implementations:
90 print " +-" + str(implementation)
91 print "Tests:"
92 for test in tests:
93 print " " + str(test)
94
95 # run tests
96 results = []
97 for test in tests:
98 results += test.run(options=options)
99
100 # report on results
101 for result in results:
102 result.report()
103 num_results = len(results)
104 num_failures = len([x for x in results if not x.is_successful()])
105 print '--------------------------------'
106 print 'Total test runs: %d, failures: %d' % (num_results, num_failures)
107 print '--------------------------------'
108
109 if num_failures == 0:
110 return 0
111 else:
112 return 1
Binary diff not shown
0 import os
1 from os.path import basename
2 import re
3 from subprocess import Popen, PIPE
4 from tempfile import mkstemp
5
6 # Note: the __str__ method of all the classes defined herein should
7 # produce a short, human-readable summary of the contents of the object,
8 # suitable for displaying in the test results but not necessarily
9 # complete. __repr__ should produce something complete, when it is
10 # present. Dumping complete information in a human-readable format
11 # is done by non-magical methods.
12
13
14 ##### Exceptions #####
15
16 class FalderalSyntaxError(ValueError):
17 pass
18
19
20 ##### Options #####
21
22 # If the Falderal objects are used by a command-line driver which
23 # gets an options object from OptParse, it should be duck-type
24 # compatible with objects of this class.
25
26 class Options(object):
27 def __init__(self):
28 self.substring_error = False
29
30
31 DEFAULT_OPTIONS = Options()
32
33
34 ##### Test Results #####
35
36 class RunResult(object):
37 """The result (either expected or actual) of running a test case.
38
39 This is used both for the results of running an implementation of
40 the functionality the test is defined for, and for specifying what
41 the expected result of the test is.
42
43 Note that RunResults are different from TestResults; depending on
44 how the Test is set up, either kind of RunResult may be a Success
45 or Failure.
46
47 """
48 def __init__(self, text):
49 self.text = text
50
51 def __repr__(self):
52 return '%s(%r)' % (self.__class__.__name__, self.text)
53
54
55 class OutputResult(RunResult):
56 def __str__(self):
57 return 'output:\n' + self.text
58
59
60 class ErrorResult(RunResult):
61 def __str__(self):
62 return 'error:\n' + self.text
63
64
65 class TestResult(object):
66 """The result of a test.
67
68 Note that TestResults are different from RunResults; depending on
69 how the Test is set up, either kind of RunResult may be a Success
70 or Failure.
71
72 """
73 def short_description(self):
74 raise NotImplementedError
75
76 def report(self):
77 raise NotImplementedError
78
79 def is_successful(self):
80 raise NotImplementedError
81
82 def format_text_block(self, obj):
83 """If the given text extends over more than one line, precede it
84 with a newline.
85
86 """
87 text = str(obj)
88 if '\n' in text and not text.startswith(('output:', 'error:')):
89 return '\n' + text
90 else:
91 return text
92
93
94 class Success(TestResult):
95 def __init__(self, test, implementation):
96 self.test = test
97 self.implementation = implementation
98
99 def short_description(self):
100 return 'success'
101
102 def report(self):
103 pass
104
105 def is_successful(self):
106 return True
107
108 def __repr__(self):
109 return '%s(%s, %s)' % (self.__class__.__name__, self.test,
110 self.implementation)
111
112
113 class Failure(TestResult):
114 def __init__(self, test, implementation, actual):
115 self.test = test
116 self.implementation = implementation
117 self.actual = actual
118
119 def short_description(self):
120 return 'expected %r, got %r' % (self.test.expectation, self.actual)
121
122 def report(self):
123 print "FAILED : " + self.format_text_block(self.test.description)
124 print "Impl : " + self.format_text_block(self.implementation)
125 print "Input : " + self.format_text_block(self.test.input)
126 print "Expected: " + self.format_text_block(self.test.expectation)
127 print "Actual : " + self.format_text_block(self.actual)
128 print
129
130 def is_successful(self):
131 return False
132
133 def __repr__(self):
134 return '%s(%s, %s, %s)' % (self.__class__.__name__, self.test,
135 self.implementation, self.actual)
136
137
138 ##### Blocks #####
139
140 class Block(object):
141 """A segment of a Falderal-formatted file.
142
143 >>> b = Block('| ')
144 >>> b.append('| test input line 1')
145 >>> b.append('| test input line 2')
146 >>> print b.text(prefix=True)
147 | test input line 1
148 | test input line 2
149 >>> print b.text()
150 test input line 1
151 test input line 2
152 >>> print b.text(seperator='')
153 test input line 1test input line 2
154
155 """
156 def __init__(self, prefix, line_num=None, filename=None):
157 self.prefix = prefix
158 self.lines = []
159 self.line_num = line_num
160 self.filename = filename
161
162 def append(self, line):
163 self.lines.append(line[len(self.prefix):])
164
165 def text(self, prefix=False, seperator='\n'):
166 if not prefix:
167 return seperator.join(self.lines)
168 else:
169 return seperator.join(self.prefix + line for line in self.lines)
170
171
172 class LiterateCode(Block):
173 pass
174
175
176 class Pragma(Block):
177 pass
178
179
180 class TestInput(Block):
181 pass
182
183
184 class ExpectedError(Block):
185 pass
186
187
188 class ExpectedResult(Block):
189 pass
190
191
192 class InterveningMarkdown(Block):
193 pass
194
195
196 ##### Documents #####
197
198 PREFIX = {
199 '| ': TestInput,
200 '? ': ExpectedError,
201 '= ': ExpectedResult,
202 '->': Pragma,
203 '> ': LiterateCode,
204 }
205
206 for prefix in ('| ', '? ', '= ', '->'):
207 PREFIX[' ' + prefix] = PREFIX[prefix]
208
209
210 class Document(object):
211 """An object representing a parsed Falderal file.
212
213 """
214 def __init__(self):
215 self.lines = []
216 self.blocks = None
217
218 @classmethod
219 def load(cls, filename):
220 d = cls()
221 f = open(filename)
222 for line in f:
223 d.append(line)
224 f.close()
225 return d
226
227 def append(self, line):
228 line = line.rstrip('\r\n')
229 self.lines.append(line)
230
231 def parse_lines_to_blocks(self):
232 r"""Parse the lines of the Document into Blocks.
233
234 >>> d = Document()
235 >>> d.append('This is a test file.')
236 >>> d.append('-> This is a pragma.')
237 >>> d.append("| This is some test input.\n")
238 >>> d.append("| It extends over two lines.")
239 >>> d.append('? Expected Error')
240 >>> d.append(' | Indented test')
241 >>> d.append(' = Indented result')
242 >>> d.parse_lines_to_blocks()
243 >>> [b.__class__.__name__ for b in d.blocks]
244 ['InterveningMarkdown', 'Pragma', 'TestInput', 'ExpectedError',
245 'TestInput', 'ExpectedResult']
246 >>> [b.line_num for b in d.blocks]
247 [1, 2, 3, 5, 6, 7]
248
249 """
250 state = '***'
251 blocks = []
252 block = None
253 line_num = 1
254 for line in self.lines:
255 found_prefix = ''
256 for prefix in PREFIX.keys():
257 if line.startswith(prefix):
258 found_prefix = prefix
259 break
260 if found_prefix == state:
261 block.append(line)
262 else:
263 state = found_prefix
264 if block is not None:
265 blocks.append(block)
266 BlockClass = PREFIX.get(state, InterveningMarkdown)
267 block = BlockClass(state, line_num=line_num)
268 block.append(line)
269 line_num += 1
270 if block is not None:
271 blocks.append(block)
272 self.blocks = blocks
273
274 def parse_blocks_to_tests(self, functionalities):
275 r"""Assemble a list of Tests from the blocks in this Document.
276
277 >>> funs = {}
278 >>> d = Document()
279 >>> d.append("This is a text file.")
280 >>> d.append('It contains NO tests.')
281 >>> d.parse_blocks_to_tests(funs)
282 []
283
284 >>> d = Document()
285 >>> d.append('This is a test file.')
286 >>> d.append('-> Tests for functionality "Parse Thing"')
287 >>> d.append("| This is some test input.")
288 >>> d.append("| It extends over two lines.")
289 >>> d.append('? Expected Error')
290 >>> d.append(' | Indented test\n')
291 >>> d.append(' = Indented result')
292 >>> d.append('-> Tests for functionality "Run Thing"')
293 >>> d.append("| Thing")
294 >>> d.append('? Oops')
295 >>> tests = d.parse_blocks_to_tests(funs)
296 >>> [t.input for t in tests]
297 ['This is some test input.\nIt extends over two lines.',
298 'Indented test', 'Thing']
299 >>> [t.expectation for t in tests]
300 [ErrorResult('Expected Error'), OutputResult('Indented result'),
301 ErrorResult('Oops')]
302 >>> [t.functionality.name for t in tests]
303 ['Parse Thing', 'Parse Thing', 'Run Thing']
304 >>> sorted(funs.keys())
305 ['Parse Thing', 'Run Thing']
306
307 >>> d = Document()
308 >>> d.append("| This is some test input.")
309 >>> d.append('= Expected')
310 >>> d.parse_blocks_to_tests({})
311 Traceback (most recent call last):
312 ...
313 FalderalSyntaxError: line 2: functionality under test not specified
314
315 >>> d = Document()
316 >>> d.append('This is a test file.')
317 >>> d.append('? Expected Error')
318 >>> d.parse_blocks_to_tests({})
319 Traceback (most recent call last):
320 ...
321 FalderalSyntaxError: line 2: expectation must be preceded by test input
322
323 >>> d = Document()
324 >>> d.append('-> This is pragma')
325 >>> d.append('= Expected')
326 >>> d.parse_blocks_to_tests({})
327 Traceback (most recent call last):
328 ...
329 FalderalSyntaxError: line 2: expectation must be preceded by test input
330
331 >>> d = Document()
332 >>> d.append('| This is test')
333 >>> d.append('This is text')
334 >>> d.parse_blocks_to_tests({})
335 Traceback (most recent call last):
336 ...
337 FalderalSyntaxError: line 2: test input must be followed by expectation
338
339 >>> d = Document()
340 >>> funs = {}
341 >>> d.append('-> Functionality "Parse Stuff" is implemented by '
342 ... 'shell command "parse"')
343 >>> d.append('')
344 >>> d.append('-> Functionality "Parse Stuff" is')
345 >>> d.append('-> implemented by shell command "pxxxy"')
346 >>> tests = d.parse_blocks_to_tests(funs)
347 >>> len(funs.keys())
348 1
349 >>> [i for i in funs["Parse Stuff"].implementations]
350 [ShellImplementation('parse'), ShellImplementation('pxxxy')]
351
352 """
353 if self.blocks is None:
354 self.parse_lines_to_blocks()
355 tests = []
356 current_functionality = None
357 prev_block = None
358 last_desc_block = None
359 for block in self.blocks:
360 expectation_class = None
361 if isinstance(block, ExpectedError):
362 expectation_class = ErrorResult
363 if isinstance(block, ExpectedResult):
364 expectation_class = OutputResult
365 if expectation_class:
366 if isinstance(prev_block, TestInput):
367 if current_functionality is None:
368 raise FalderalSyntaxError(
369 ("line %d: " % block.line_num) +
370 "functionality under test not specified")
371 test = Test(input=prev_block.text(),
372 expectation=expectation_class(block.text()),
373 functionality=current_functionality,
374 desc_block=last_desc_block)
375 tests.append(test)
376 else:
377 raise FalderalSyntaxError(
378 ("line %d: " % block.line_num) +
379 "expectation must be preceded by test input")
380 else:
381 if isinstance(prev_block, TestInput):
382 raise FalderalSyntaxError(
383 ("line %d: " % block.line_num) +
384 "test input must be followed by expectation")
385 if isinstance(block, Pragma):
386 pragma_text = block.text(seperator=' ')
387 match = re.match(r'^\s*Tests\s+for\s+functionality\s*\"(.*?)\"\s*$', pragma_text)
388 if match:
389 functionality_name = match.group(1)
390 current_functionality = functionalities.setdefault(
391 functionality_name,
392 Functionality(functionality_name)
393 )
394 match = re.match(r'^\s*Functionality\s*\"(.*?)\"\s*is\s+implemented\s+by\s+shell\s+command\s*\"(.*?)\"\s*$', pragma_text)
395 if match:
396 functionality_name = match.group(1)
397 command = match.group(2)
398 functionality = functionalities.setdefault(
399 functionality_name,
400 Functionality(functionality_name)
401 )
402 implementation = ShellImplementation(command)
403 functionality.add_implementation(implementation)
404 match = re.match(r'^\s*Functionality\s*\"(.*?)\"\s*is\s+implemented\s+by\s+Haskell\s+function\s*(.*?)\:(.*?)\s*$', pragma_text)
405 if match:
406 functionality_name = match.group(1)
407 module = match.group(2)
408 function = match.group(3)
409 command = r'ghc -e "do c <- readFile \"%%(test-file)\"; putStrLn $ %s.%s c" %s.hs' % (
410 module, function, module
411 )
412 functionality = functionalities.setdefault(
413 functionality_name,
414 Functionality(functionality_name)
415 )
416 implementation = ShellImplementation(command)
417 functionality.add_implementation(implementation)
418 elif isinstance(block, InterveningMarkdown):
419 if not re.match(r'^\s*$', block.text(seperator=' ')):
420 last_desc_block = block
421 prev_block = block
422 return tests
423
424
425 ##### Functionalities and their Implementations #####
426
427 class Functionality(object):
428 """An object representing a Falderal functionality.
429
430 A functionality can have multiple implementations.
431
432 Each test has exactly one functionality.
433
434 """
435 def __init__(self, name):
436 self.name = name
437 self.implementations = []
438
439 def add_implementation(self, implementation):
440 self.implementations.append(implementation)
441
442
443 class Implementation(object):
444 """An object representing an implementation (something that is
445 used to run a test) in Falderal.
446
447 """
448 def __init__(self):
449 pass
450
451 def run(self, input=None):
452 """Returns the RunResult of running this implementation on the
453 given input.
454
455 """
456 raise NotImplementedError("subclass needs to implement run()")
457
458
459 class CallableImplementation(Implementation):
460 """An implementation which is implemented by a Python callable.
461
462 This is mostly useful for internal tests.
463
464 """
465 def __init__(self, callable):
466 self.callable = callable
467
468 def run(self, input=None):
469 try:
470 result = self.callable(input)
471 return OutputResult(result)
472 except Exception as e:
473 return ErrorResult(str(e))
474
475
476 class ShellImplementation(Implementation):
477 def __init__(self, command):
478 self.command = command
479
480 def run(self, input=None):
481 r"""
482 >>> i = ShellImplementation('cat')
483 >>> i.run('text')
484 OutputResult('text')
485
486 >>> i = ShellImplementation('cat fhofhofhf')
487 >>> i.run('text')
488 ErrorResult('cat: fhofhofhf: No such file or directory')
489
490 >>> i = ShellImplementation('cat %(test-file)')
491 >>> i.run('text')
492 OutputResult('text')
493
494 >>> i = ShellImplementation('echo %(test-text)')
495 >>> i.run('text')
496 OutputResult('text')
497
498 >>> i = ShellImplementation('cat >%(output-file)')
499 >>> i.run('text')
500 OutputResult('text')
501
502 """
503 # expand variables in the command
504 test_filename = None
505 output_filename = None
506 command = self.command
507 if '%(test-file)' in self.command:
508 # choose a temp file name and write the input to that file
509 fd, test_filename = mkstemp(dir='.')
510 test_filename = basename(test_filename)
511 with open(test_filename, 'w') as file:
512 file.write(input)
513 file.close()
514 os.close(fd)
515 # replace all occurrences in command
516 command = re.sub(r'\%\(test-file\)', test_filename, command)
517 input = None
518 if '%(test-text)' in self.command:
519 # escape all single quotes in input
520 input = re.sub(r"'", r"\'", input)
521 # replace all occurrences in command
522 command = re.sub(r'\%\(test-text\)', input, command)
523 input = None
524 if '%(output-file)' in self.command:
525 # choose a temp file name to read output from later
526 fd, output_filename = mkstemp(dir='.')
527 output_filename = basename(output_filename)
528 os.close(fd)
529 # replace all occurrences in command
530 command = re.sub(r'\%\(output-file\)', output_filename, command)
531
532 # subshell the command and return the output
533 pipe = Popen(command, shell=True,
534 stdin=PIPE, stdout=PIPE, stderr=PIPE)
535 outputs = pipe.communicate(input=input)
536 if pipe.returncode == 0:
537 if output_filename is None:
538 output = self.normalize_output(outputs[0])
539 else:
540 f = open(output_filename, 'r')
541 output = f.read()
542 f.close()
543 result = OutputResult(output)
544 else:
545 result = ErrorResult(self.normalize_output(outputs[1]))
546
547 # clean up temporary files
548 for filename in (test_filename, output_filename):
549 if filename is not None:
550 os.unlink(filename)
551 # finis
552 return result
553
554 def normalize_output(self, text):
555 text = re.sub(r'\r\n', '\n', text)
556 return text.strip('\r\n')
557
558 def __repr__(self):
559 return '%s(%r)' % (self.__class__.__name__, self.command)
560
561 def __str__(self):
562 return 'shell command "%s"' % self.command
563
564
565 ##### Tests #####
566
567 class Test(object):
568 """An object representing a Falderal test.
569
570 """
571 def __init__(self, input=None, expectation=None, functionality=None,
572 desc_block=None):
573 self.input = input
574 self.expectation = expectation
575 self.functionality = functionality
576 self.desc_block = desc_block
577
578 def run(self, options=DEFAULT_OPTIONS):
579 """Returns a list of Results, one for each implementation of
580 the functionality being tested.
581
582 >>> f = Functionality('Cat File')
583 >>> f.add_implementation(CallableImplementation(lambda x: x))
584 >>> t = Test(input='foo', expectation=OutputResult('foo'),
585 ... functionality=f)
586 >>> [r.short_description() for r in t.run()]
587 ['success']
588
589 >>> f = Functionality('Cat File')
590 >>> f.add_implementation(CallableImplementation(lambda x: x))
591 >>> t = Test(input='foo', expectation=OutputResult('bar'),
592 ... functionality=f)
593 >>> [r.short_description() for r in t.run()]
594 ["expected OutputResult('bar'), got OutputResult('foo')"]
595
596 >>> f = Functionality('Cat File')
597 >>> f.add_implementation(CallableImplementation(lambda x: x))
598 >>> t = Test(input='foo', expectation=ErrorResult('foo'),
599 ... functionality=f)
600 >>> [r.short_description() for r in t.run()]
601 ["expected ErrorResult('foo'), got OutputResult('foo')"]
602
603 >>> f = Functionality('Cat File')
604 >>> def e(x):
605 ... raise ValueError(x)
606 >>> f.add_implementation(CallableImplementation(e))
607 >>> t = Test(input='foo', expectation=ErrorResult('foo'),
608 ... functionality=f)
609 >>> [r.short_description() for r in t.run()]
610 ['success']
611
612 >>> f = Functionality('Cat File')
613 >>> def e(x):
614 ... raise ValueError(x)
615 >>> f.add_implementation(CallableImplementation(e))
616 >>> t = Test(input='foo', expectation=ErrorResult('bar'),
617 ... functionality=f)
618 >>> [r.short_description() for r in t.run()]
619 ["expected ErrorResult('bar'), got ErrorResult('foo')"]
620
621 >>> f = Functionality('Cat File')
622 >>> def e(x):
623 ... raise ValueError(x)
624 >>> f.add_implementation(CallableImplementation(e))
625 >>> t = Test(input='foo', expectation=OutputResult('foo'),
626 ... functionality=f)
627 >>> [r.short_description() for r in t.run()]
628 ["expected OutputResult('foo'), got ErrorResult('foo')"]
629
630 A functionality can have multiple implementations. We test them all.
631
632 >>> f = Functionality('Cat File')
633 >>> def c1(x):
634 ... return x
635 >>> def c2(x):
636 ... return x + '...'
637 >>> def c3(x):
638 ... raise ValueError(x)
639 >>> for c in (c1, c2, c3):
640 ... f.add_implementation(CallableImplementation(c))
641 >>> t = Test(input='foo', expectation=OutputResult('foo'),
642 ... functionality=f)
643 >>> [r.short_description() for r in t.run()]
644 ['success', "expected OutputResult('foo'), got OutputResult('foo...')",
645 "expected OutputResult('foo'), got ErrorResult('foo')"]
646
647 """
648 results = []
649 for implementation in self.functionality.implementations:
650 result = implementation.run(input=self.input)
651 if self.judge(result, options):
652 results.append(Success(self, implementation))
653 else:
654 results.append(Failure(self, implementation, result))
655 return results
656
657 def judge(self, result, options):
658 if not isinstance(result, self.expectation.__class__):
659 return False
660 if options.substring_error and isinstance(result, ErrorResult):
661 return self.expectation.text in result.text
662 else:
663 return self.expectation.text == result.text
664
665 @property
666 def description(self):
667 if self.desc_block is None:
668 return ''
669 return self.desc_block.text()
Binary diff not shown
0 #!/bin/sh
1
2 # Really crude test harness for py-falderal itself...
3
4 bin/falderal -t || exit 1
5
6 cd tests
7 for TEST in test1 test2 test3 test4 test5 test9 test-utf8 test-crlf; do
8 echo ${TEST}...
9 ../bin/falderal ${TEST}.markdown > ${TEST}.actual 2>&1
10 diff -u ${TEST}.expected ${TEST}.actual || exit 1
11 done
12
13 # two-part tests
14 for TEST in test6 test7 test8; do
15 echo ${TEST}...
16 ../bin/falderal ${TEST}a.markdown ${TEST}b.markdown > ${TEST}.actual 2>&1
17 diff -u ${TEST}.expected ${TEST}.actual || exit 1
18 done
19
20 # special tests: -b
21 TEST=test-substring-error
22 echo ${TEST}...
23 ../bin/falderal -b ${TEST}.markdown > ${TEST}.actual 2>&1
24 diff -u ${TEST}.expected ${TEST}.actual || exit 1
25
26 rm -f *.actual
27 echo 'All tests passed.'
0 import sys
1
2 input = sys.stdin
3 output = sys.stdout
4
5 if len(sys.argv) > 1 and sys.argv[1] == '-f':
6 input = open(sys.argv[2], 'r')
7 sys.argv = sys.argv[2:]
8
9 if len(sys.argv) > 1 and sys.argv[1] == '-o':
10 output = open(sys.argv[2], 'w')
11 sys.argv = sys.argv[2:]
12
13 output.write(input.read())
0 import sys
1
2 for line in sys.stdin:
3 sys.stdout.write(line.strip() + '\r\n')
0 import sys
1
2 if sys.argv[1] == '-n':
3 sys.stdout.write(sys.argv[2])
4 else:
5 sys.stdout.write(sys.argv[1] + '\n')
0 import sys
1
2 sys.stdout.write(sys.argv[1] + '\n')
3 sys.stdout.flush()
4 sys.stderr.write(sys.argv[2] + '\n')
5 sys.exit(int(sys.argv[3]))
0 FAILED :
1
2 No CRLFs in test report (Intentional fail.)
3
4 Impl : shell command "python crlf.py"
5 Input :
6 one
7 two
8 Expected: output:
9 three
10 Actual : output:
11 one
12 two
13
14 --------------------------------
15 Total test runs: 2, failures: 1
16 --------------------------------
0 Falderal Test: CRLF
1 -------------------
2
3 These Falderal tests aim to show that the implementation
4 can compare the result with the expected text, regardless
5 of end-of-line convention (LF or CRLF).
6
7 -> Functionality "CRLF" is implemented by
8 -> shell command "python crlf.py"
9
10 -> Tests for functionality "CRLF"
11
12 Cat CRLFs.
13
14 | my kitty, 'tis of thee
15 | feline felicity
16 | something something
17 = my kitty, 'tis of thee
18 = feline felicity
19 = something something
20
21 No CRLFs in test report (Intentional fail.)
22
23 | one
24 | two
25 = three
0 --------------------------------
1 Total test runs: 3, failures: 0
2 --------------------------------
0 Falderal Test: substring-error
1 ------------------------------
2
3 When the `-b` option is passed to `falderal`, it considers
4 an error-expecting test successful if the error text which
5 was produced simply contains (rather than totally equals)
6 the expected error text.
7
8 -> Functionality "Fail" is implemented by shell command
9 -> "python fail.py foo '%(test-text)' 1"
10
11 -> Tests for functionality "Fail"
12
13 | this is the error message
14 ? this is the error message
15
16 | (file tmpgZ5W7e3, line 123): this is the error message
17 ? this is the error message
18
19 | (file tmpgZ5W7e3, line 123): an error occurred on this line.
20 | It is a very pretty error, consisting of several nested
21 | sub-errors, of which I know very little. (ref #371282)
22 ? an error occurred on this line.
23 ? It is a very pretty error, consisting of several nested
24 ? sub-errors, of which I know very little.
0 FAILED :
1
2 Cat dogs, too. (Intentional fail.)
3
4 Impl : shell command "python cat.py"
5 Input : n ← ★
6 Expected: output:
7 m ← ★
8 Actual : output:
9 n ← ★
10
11 FAILED :
12
13 Cat (file) dogs, too. (Intentional fail.)
14
15 Impl : shell command "python cat.py -f %(test-file) -o %(output-file)"
16 Input : n ← ★
17 Expected: output:
18 m ← ★
19 Actual : output:
20 n ← ★
21
22 --------------------------------
23 Total test runs: 4, failures: 2
24 --------------------------------
0 -> encoding: UTF-8
1
2 Falderal Test: UTF-8
3 --------------------
4
5 This is an example Falderal document which contains Unicode
6 characters, encoded in UTF-8 (this is the assumed encoding
7 of all Falderal documents which go beyond mere ASCII.)
8
9 -> Functionality "Cat" is implemented by
10 -> shell command "python cat.py"
11
12 -> Tests for functionality "Cat"
13
14 Cat cats.
15
16 | n ← ★
17 = n ← ★
18
19 Cat dogs, too. (Intentional fail.)
20
21 | n ← ★
22 = m ← ★
23
24 -> Functionality "Cat (file)" is implemented by
25 -> shell command "python cat.py -f %(test-file) -o %(output-file)"
26
27 -> Tests for functionality "Cat (file)"
28
29 Cat (file) cats.
30
31 | n ← ★
32 = n ← ★
33
34 Cat (file) dogs, too. (Intentional fail.)
35
36 | n ← ★
37 = m ← ★
0 FAILED :
1
2 Cat dogs, too. (Intentional fail.)
3
4 Impl : shell command "python cat.py"
5 Input : meow
6 Expected: output:
7 woof
8 Actual : output:
9 meow
10
11 FAILED :
12
13 Cat dogs, too. (Intentional fail.)
14
15 Impl : shell command "python cat.py"
16 Input :
17 purr
18 prrr
19 prreow
20 Expected: output:
21 woof
22 woof
23 awoooo
24 Actual : output:
25 purr
26 prrr
27 prreow
28
29 --------------------------------
30 Total test runs: 4, failures: 2
31 --------------------------------
0 Falderal Test 1
1 ---------------
2
3 This is an example Falderal document which contains some
4 intentionally failing tests. It is intended to test that
5 your Falderal implementation is, itself, not producing
6 incorrect output.
7
8 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py"
10
11 -> Tests for functionality "Cat"
12
13 Cat cats.
14
15 | meow
16 = meow
17
18 | purr
19 | prrr
20 | prreow
21 = purr
22 = prrr
23 = prreow
24
25 Cat dogs, too. (Intentional fail.)
26
27 | meow
28 = woof
29
30 | purr
31 | prrr
32 | prreow
33 = woof
34 = woof
35 = awoooo
0 FalderalSyntaxError: line 13: functionality under test not specified
0 Falderal Test 2
1 ---------------
2
3 Since no functionality is specified for these tests,
4 `py-falderal` should exit with an exception.
5
6 -> Functionality "Cat" is implemented by
7 -> shell command "python cat.py"
8
9 Cat cats.
10
11 | meow
12 = meow
13
14 | purr
15 | prrr
16 | prreow
17 = purr
18 = prrr
19 = prreow
0 FalderalSyntaxError: line 14: test input must be followed by expectation
0 Falderal Test 3
1 ---------------
2
3 A Falderal document which is ill-formed.
4
5 -> Functionality "Cat" is implemented by
6 -> shell command "python cat.py"
7
8 -> Tests for functionality "Cat"
9
10 Cat cats.
11
12 | meow
13
14 Oops, I didn't include an expectation.
0 FalderalSyntaxError: line 13: expectation must be preceded by test input
0 Falderal Test 4
1 ---------------
2
3 Another Falderal document which is ill-formed.
4
5 -> Functionality "Cat" is implemented by
6 -> shell command "python cat.py"
7
8 -> Tests for functionality "Cat"
9
10 Cat cats.
11
12 = meow
13
14 Oops, I didn't include a test body.
0 --------------------------------
1 Total test runs: 10, failures: 0
2 --------------------------------
0 Falderal Test 5
1 ---------------
2
3 Tests for variable substitution, and missing EOL at end
4 of output.
5
6 Note the use of single quotes around the `%(test-text)` variable;
7 without these, shell chaos is likely to result.
8
9 -> Functionality "Echo" is implemented by
10 -> shell command "python echo.py '%(test-text)'"
11
12 -> Tests for functionality "Echo"
13
14 | hello
15 = hello
16
17 | hi
18 | hi
19 = hi
20 = hi
21
22 -> Functionality "Echo, no newline" is implemented by
23 -> shell command "python echo.py -n '%(test-text)'"
24
25 -> Tests for functionality "Echo, no newline"
26
27 | hello
28 = hello
29
30 | hi
31 | hi
32 = hi
33 = hi
34
35 Note that single quotes needn't be supplied around `%(test-file)`
36 or `%(output-file)`.
37
38 -> Functionality "Cat, from file" is implemented by
39 -> shell command "python cat.py -f %(test-file)"
40
41 -> Tests for functionality "Cat, from file"
42
43 | hello
44 = hello
45
46 | hi
47 | hi
48 = hi
49 = hi
50
51 -> Functionality "Cat, to file" is implemented by
52 -> shell command "python cat.py -o %(output-file)"
53
54 -> Tests for functionality "Cat, to file"
55
56 | hello
57 = hello
58
59 | hi
60 | hi
61 = hi
62 = hi
63
64 -> Functionality "Cat, to and from file" is implemented by
65 -> shell command "python cat.py -f %(test-file) -o %(output-file)"
66
67 -> Tests for functionality "Cat, to and from file"
68
69 | hello
70 = hello
71
72 | hi
73 | hi
74 = hi
75 = hi
0 FalderalSyntaxError: line 13: functionality under test not specified
0 Falderal Test 6a
1 ----------------
2
3 This is a two-file test (the other file is 6b) which shows
4 that even if you says "Tests for functionality x" in this
5 file, that tests-for meaning does not "leak" into the next
6 file.
7
8 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py"
10
11 -> Tests for functionality "Cat"
12
13 Cat cats.
14
15 | meow
16 = meow
0 Falderal Test 6b
1 ----------------
2
3 This is a two-file test (the other file is 6a) which shows
4 that even if you says "Tests for functionality x" in the other
5 file, that tests-for meaning does not "leak" into this file.
6
7 Cat cats.
8
9 | purr
10 | prrr
11 | prreow
12 = purr
13 = prrr
14 = prreow
0 --------------------------------
1 Total test runs: 4, failures: 0
2 --------------------------------
0 Falderal Test 7a
1 ----------------
2
3 This is a two-file test (the other file is 7b) which shows
4 that all implementations of a functionality apply to all
5 tests for that functionality, even when they're not in
6 the same file.
7
8 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py"
10
11 -> Tests for functionality "Cat"
12
13 Cat cats.
14
15 | meow
16 = meow
0 Falderal Test 7b
1 ----------------
2
3 This is a two-file test (the other file is 7a) which shows
4 that all implementations of a functionality apply to all
5 tests for that functionality, even when they're not in
6 the same file.
7
8 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py -f %(test-file)"
10
11 -> Tests for functionality "Cat"
12
13 Cat totally, like, cats.
14
15 | purr
16 | prrr
17 | prreow
18 = purr
19 = prrr
20 = prreow
0 --------------------------------
1 Total test runs: 4, failures: 0
2 --------------------------------
0 Falderal Test 8a
1 ----------------
2
3 This is really just a more lopsided version of test 7.
4
5 -> Tests for functionality "Cat"
6
7 Cat totally, like, cats.
8
9 | purr
10 | prrr
11 | prreow
12 = purr
13 = prrr
14 = prreow
0 Falderal Test 8b
1 ----------------
2
3 This is really just a more lopsided version of test 7.
4
5 -> Functionality "Cat" is implemented by
6 -> shell command "python cat.py"
7
8 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py -f %(test-file)"
10
11 -> Functionality "Cat" is implemented by
12 -> shell command "python cat.py -o %(output-file)"
13
14 -> Functionality "Cat" is implemented by
15 -> shell command "python echo.py '%(test-text)'"
0 --------------------------------
1 Total test runs: 4, failures: 0
2 --------------------------------
0 Falderal Test 9
1 ---------------
2
3 When we have a test that expects a successful result, the
4 expected text is matched against standard output.
5
6 -> Functionality "Succeed" is implemented by shell command
7 -> "python fail.py '%(test-text)' bar 0"
8
9 -> Tests for functionality "Succeed"
10
11 | foo
12 = foo
13
14 If you wish to match the expected result against both standard
15 output and standard error, it's up to you to redirect them.
16
17 -> Functionality "Succeed/All" is implemented by shell command
18 -> "python fail.py '%(test-text)' bar 0 2>&1"
19
20 -> Tests for functionality "Succeed/All"
21
22 | foo
23 = foo
24 = bar
25
26 When we have a test that expects an error result, the
27 expected text is matched against standard error.
28
29 -> Functionality "Fail" is implemented by shell command
30 -> "python fail.py foo '%(test-text)' 1"
31
32 -> Tests for functionality "Fail"
33
34 | bar
35 ? bar
36
37 If you wish to match the expected error against both standard
38 output and standard error, it's up to you to redirect them.
39
40 -> Functionality "Fail/All" is implemented by shell command
41 -> "python fail.py foo '%(test-text)' 1 1>&2"
42
43 -> Tests for functionality "Fail/All"
44
45 | bar
46 ? foo
47 ? bar