git @ Cat's Eye Technologies Falderal / 94aa0e8
Merge pull request #12 from catseye/develop-0.14 Develop 0.14 Chris Pressey authored 2 years ago GitHub committed 2 years ago
49 changed file(s) with 1358 addition(s) and 1334 deletion(s). Raw diff Collapse all Expand all
+0
-10
.hgignore less more
0 syntax: regexp
1
2 ^impl/Test.Falderal/bin/
3
4 syntax: glob
5
6 *.o
7 *.hi
8 *.pyc
9 *.actual
+0
-15
.hgtags less more
0 e0f07b3699cd2b5029dc0e619e30f2897e27564a rel_0_1
1 b5b728aeba88946da3e6ae6ad0ffbc3eac53dd6a rel_0_2
2 582e124855570072a933f14a89ba7ad74632610b rel_0_3
3 582e124855570072a933f14a89ba7ad74632610b rel_0_3
4 8bb4fad90030edb2ac4f7b113c3bf0a361f9274a rel_0_3
5 046f8ec1388ab05e986febbc7c9d67c636076f53 rel_0_4
6 cb2caf712180eb0fd3d759f9fcbf5cef19680675 rel_0_5
7 6f9b761233f86c8a416a7b13599ebf9f50f005b3 rel_0_6
8 5ce741f7daf73c8b0a37daa48ea298b12e3f36cb rel_0_7
9 0397d7575a05550ca8d137baeeb5ea984d0379a3 rel_0_8
10 7575a60f38d4fcf1984e892cd301b4b99faf9606 rel_0_9_2013_1029
11 00acc0048e83c3313ad740b90ce807de115bb21c rel_0_9_2014_0427
12 edfd7af3d505a76b08aebc0af537ea7e5fdeb31d rel_0_9_2014_0525
13 0ff6e40b2c7e6464993fb792f3a3be1c94a6dd2e rel_0_10
14 0205fd963df5b9d5d7306ea40696c593eb3e38ce rel_0_11
+0
-290
HISTORY.markdown less more
0 History
1 =======
2
3 Version 0.13 "Merchandise Mart":
4
5 * Use `ArgumentParser` instead of `OptionParser` internally.
6 * `--substring-error` is now the default behaviour, and if this
7 option is given, it will be ignored. After a long time
8 considering the problem, we finally determined that it does
9 not make sense to insist that different implementations
10 produce the exact same error message.
11 * `--verbose` option now prints out commands for each test as it
12 is run.
13 * Added conventional `--version` option.
14 * Planned to be the last 0.x series release before 1.0.
15 * Added `setup.py` so that Falderal can be treated as a Python
16 package.
17
18 Version 0.12 "Schoenhofen Brewery":
19
20 * When expanding variables in declaration strings, use
21 `string.replace` instead of `re.sub` so that backslash-escaping
22 is not perfomed on the replacement string. (Thanks to
23 James Holderness for bug report and patch.)
24 * In "freestyle" format, lines beginning with `<= `, `<== `, or
25 `<=== ` can supply a section of test input text (Thanks to
26 James Holderness for feature suggestion and patch.)
27 * Falderal is now responsible for quoting all shell metacharacters
28 in the substituion text of `%(...)` variables when command line
29 templates are expanded, so that no quotes need be placed around
30 the variable in the template. (Again, thanks to James Holderness
31 for spotting the bug in the code that led to this improvement.)
32 * `py-falderal` can now run under both Python 2 and Python 3
33 (tested with 2.7, 3.4, and 3.5.)
34 * The doctests that were previously distributed throughout the
35 source code have been converted to a suite of unit tests in
36 the `falderal.tests` module. The `-t` command-line option has
37 been removed; to run internal tests, run this module instead.
38
39 Version 0.11 "Dan Ryan Expressway":
40
41 * Renamed the internal tests to have descriptive names instead of
42 numbers.
43 * The first command-line argument being `test` for compatibility with
44 previous versions of `falderal` has been removed.
45 * Massive internal refactor of block-parsing and test-extraction code.
46 * Supports a new format for test blocks, called "freestyle", where if
47 the last few lines of a block are prefixed with `=> ` or `==> ` or
48 `===> ` or `?> ` or `??> ` or `???> `, the preceding part of the
49 block needs no prefixes at all, and it is simply treated as one
50 verbatim test body.
51
52 Version 0.10 "Shedd Aquarium":
53
54 * Removed the old, bitrotting Haskell implementation.
55 * Removed deprecated `%(test-text)` and `%(test-file)` variables
56 (`%(test-body-text)` and `%(test-body-file)` should be used now.)
57 * Added some adapters in the `script/` directory of the repo:
58 * `fa-under-pty` runs a target program under a pseudoterminal
59 * `fa-bin-to-hex` converts 8-bit binary output to hexadecimal codes
60 * `fa-hex-to-bin` converts hexadecimal codes to 8-bit binary output
61 * Added the rule that a test may consist of an input section (`+`) if
62 there was a previous test, and the test body of the previous test will
63 be re-used with the new input in the new test.
64
65 Version 0.9-2014.0525 "Municipal Pier #2, May 25th 2014":
66
67 * Shell implementations assume input and output is encoded UTF-8; any
68 malformed UTF-8 is simply ignored (for better or worse, for now.)
69 Falderal documents are assumed to be encoded in UTF-8; this includes
70 test bodies, expectations, etc. Internally, in the reference
71 implementation, all strings are stored as Unicode.
72
73 Version 0.9-2014.0427 "Municipal Pier #2":
74
75 * For a test of a shell command implementation that expects a failure,
76 if no output was found on standard error, the output of standard output
77 is checked against the failure expectation instead. The primary use case
78 for this is Hugs, which (unlike `ghc`) displays error messages on stdout
79 rather than stderr. "Municipal Pier #2" was the original name of Navy Pier.
80
81 Version 0.9 "Navy Pier":
82
83 * `py-falderal` now insists that blocks are indented four spaces, as the
84 specification says. It ignores blocks that are not indented.
85 * In an error report `py-falderal` reports the location (filename and line
86 number of the beginning of the test text) of each test that failed, as
87 well as the name of the functionality that was being tested.
88 * Several internal renamings of class and variable names for greater clarity.
89 * Trying to standardize on the terminology "test body" for the body of a
90 test (as opposed to its "input", and opposed to it being passed to a shell
91 command line "as text".)
92 * Support for specifying a block of text to be used as input for a test.
93 * Dropped support for "implemented by Haskell function" backwards-compatibility.
94 * Temporary files are no longer always created in the current directory. That
95 was a workaround to support `ghc` from Haskell Platform under Cygwin. They
96 are now created where the system wants them to be created (but you can set
97 the `TMP` environment variable to "." if you really want them to be created
98 in the current directory.)
99 * `py-falderal` now complains and stops early if there were no documents
100 specified, no functionality definitions were found in any of the documents,
101 no implementations were found for some defined functionality, or if no
102 tests were found in any of the documents, unless `--cavalier` is passed on
103 the command line.
104 * In `--verbose` mode, `py-falderal` will issue warnings about any test that
105 has been run twice (i.e. more than run on identical test body text and
106 implementation.)
107 * If `py-falderal` catches a `KeyboardInterrupt` exception while running tests,
108 it shows a report of all the tests that it did manage to run until that
109 point, and a warning message to the effect that not all tests were run.
110
111 Version 0.8 "Ukrainian Village":
112
113 * The reference implementation is now `py-falderal`.
114 * Blocks *must* now be indented four spaces. That Falderal test documents
115 will be embedded in Markdown is generally assumed (although not strictly
116 required) and they generally have the file extension `.markdown` so that
117 other software (e.g. Bitbucket and Github) knows how to pretty-display
118 them.
119 * Various documentation which was on Bitbucket (wiki, issues) and Github
120 (py-falderal issues) have been moved into Markdown documents in the
121 distribution, which have themselves been re-organized.
122 * Added the `-v`/`--verbose` flag and made it and the `-d`/`--dump` flag
123 do what you would expect from them.
124 * The `-f`/`--functionalities` and `-c`/`--clear-functionalities` flags
125 have been *removed*. The rationale is that implementations of
126 functionalities are clumsy to express on the command line (and anyway
127 the syntax would have to be different) while it is not difficult to
128 create a "configuration" Falderal file which specifies *only* the
129 desired implementations of the desired functionalities, and to load
130 it just before loading a "tests" Falderal file which specified the
131 functionalities being tested, but not their implementations.
132
133 Version 0.7 "Ogden Avenue":
134
135 * **Note that this is the final version of Falderal where `Test.Falderal`
136 is the reference implementation.** In subsequent versions,
137 `py-falderal` (written in Python) will be the reference implementation.
138 `Test.Falderal` will still be in this repository (under the `impl`
139 directory) but will no longer be definitive, and will go essentially
140 unmaintained.
141 * Fixed a bug where number of lines in the intermediary results file
142 was sometimes being counted incorrect.
143 * Slightly improved support for running under Cygwin.
144 * Tests that are implemented by a shell command are now run directly
145 from the `falderal` process (instead of creating an intermediary
146 shell script and running it.) This resulted in better performance.
147 * Addition of `-b` command-line option, which considers a test to
148 have passed if the expected exception message is a substring (rather
149 than an exact match) of the actual produced exception message.
150 TODO: write test for this.
151 * Blocks may now be indented four spaces, to allow them to be
152 embedded directly in Markdown files as pre-formatted text.
153 * A script to just build the binary, without installing it, was added.
154
155 Version 0.6 "Streeterville":
156
157 * Variables may be included in the specification of a shell command
158 implementation; these will be expanded before generating the results
159 generator. These variable include `%(test-text)`, `%(test-file)`,
160 and `%(output-file)`.
161 * `falderal` now lets the user specify, on the command line, the
162 implementations for a named functionality. Supplying `-f 'foo:shell
163 command "foo.sh %(test-file)"'` has the same effect as including the
164 pragma `Functionality "foo" is implemented by shell command "foo.sh
165 %(test-file)"` in the Falderal file. Note that this is in addition
166 to the Functionality-definition pragmas given in the Falderal file;
167 to replace them, you must first clear the ones with that name from
168 the file by supplying `-c foo` on the command line.
169 * `falderal` also allows tests for named functionalities to be skipped
170 completely, by passing the name of the functionality to be skipped
171 after a `-k` flag on the command line.
172 * The Markdown formatter now formats Bird-style embedded code with
173 HTML embedded in the Markdown document. This is so that it can
174 be styled independently from, and thus distinguished from, any plain
175 Markdown indented code blocks which may appear in the literate portion
176 of the source code.
177 * In failure reports, the implementation of the functionality of the test
178 that failed is now reported in each failure.
179 * A race condition(?) that could occur when testing multiple implementations
180 of a functionality, of different kinds (Haskell and shell), has been
181 prevented. Both tests were writing to `results.txt` and immediately
182 deleting it, and this would sometimes confuse `falderal` into thinking
183 one had produced no results (perhaps a result of some creative
184 scheduling by `ghc`, although really, I haven't a clue.) Results are
185 now written to different temporary files with different, generated
186 names.
187 * Previously, if the output of a shell command being tested did not end
188 with a newline, the intermediate results file was not being generated
189 correctly, resulting in failures being misreported. This has been
190 recitified.
191 * Previously, if there were tests given in a Falderal file before any
192 Tests-for pragma was specified, those tests would just be ignored.
193 An error message is now issued, and no testing takes place.
194 * Previously, if multiple Falderal files were given on the command
195 line, they were simply concatenated when loaded, the result being
196 that Functionality-definitions from the first file were visible in
197 the second file, and that any Tests-for in effect at the end of the
198 first file would be in effect at the start of the second file.
199 Files are now loaded and processed seperately.
200
201 Version 0.5 "The Loop":
202
203 * The command-line format of `falderal` has changed, for the better.
204 The `test` subcommand no longer requires that the failure reporting
205 style be specified; instead, it defaults to `standard`, and can be
206 changed with a command-line option. There are also command-line
207 options for selecting the programs to run results generators, and to
208 keep generated files around instead of cleaning them up after testing.
209 * The dependency on `ghc` for running Haskell results generators has been
210 removed; these can be run by `runhaskell` now, and are run by
211 `runhaskell` by default.
212 * Failure reporting is now consistent across languages; both Haskell and
213 Bourne shell results generators generate an intermediate format, which
214 `falderal` digests.
215 * A new pragma `encoding:` was added, so that this directive can be
216 embedded in your Falderal document (for the benefit of your text editor)
217 without necessarily appearing in the formatted document.
218 * We began giving release milestones colorful names. The naming
219 convention is to choose names of Chicagoland neigborhoods, suburbs,
220 landmarks, and institutions. Version 0.5 was named after The Loop in
221 recognition of its ability to shuttle test results between `falderal`
222 and the various results generators implemented in different languages.
223 Previous versions of `Test.Falderal` were retroactively given milestone
224 names during this release.
225
226 Version 0.4 "Blackstone Hotel":
227
228 * For ease of installation, the project is presented as a Cabal package.
229 * A driver executable, `falderal`, is built as part of installing the
230 Cabal package. It provides a command-line interface to formatting
231 Falderal files and, in a limited fashion, running the tests in them.
232 * A shell script formatter has been written, enabling testing of shell
233 commands. One caveat is that reporting for these tests is nowhere near
234 as nice as for Haskell functions, but that will change in the next version.
235 * The Functionality-definition pragma has been implemented, making it
236 possible to write tests in a more abstract, implementation-independent
237 fashion.
238 * Falderal files written to work with Falderal 0.3 should still work with
239 Falderal 0.4, but you are encouraged to use the Functionality-definition
240 pragma introduced in 0.4 to make your tests more implementation-independent.
241
242 Version 0.3 "Chicago Board of Trade":
243
244 * The definition of a Falderal Literate Test Format, distinct from the
245 reference implementation of tools for it in Haskell (`Test.Falderal`).
246 This represented a fairly substantial departure from how previous versions
247 of Falderal worked.
248 * The ability to format a Falderal file to different formats, including
249 Markdown and Haskell.
250 * Running tests is now a matter of formatting a Falderal file as a Haskell
251 module and running the function `testModule` in that module.
252
253 Version 0.2 "Dearborn Station":
254
255 * Added a test harness for Falderal itself, in the form of a simple shell
256 script which diffs the output of `Test.Falderal.Demo` against a text file
257 containing the expected content.
258 * Improved formatting of failure reports. Multi-line input text or expected
259 output is always presented starting on its own line.
260 * Tests may be organized into groups; the entire group is preceded by some
261 literal text, but there is no literal text between the tests in the group.
262 When one of these tests fails, the literal text for the group is reported,
263 along with the number of the test within the group.
264 * Fixed a bug where exception text which extended over multiple lines
265 could not be expected correctly.
266
267 Version 0.1 "Haymarket Square":
268
269 * Provision of a framework for writing and running literate tests which may
270 be embedded in literate Haskell source code.
271 * Testing Haskell functions of type `String -> String`. A test specifies input
272 to the function, and may expect a particular output, or that a particular
273 exception is thrown.
274 * Through simple adapters, testing functions of other types such as
275 `(Show a) => String -> a`.
276
277 Prehistory:
278
279 Falderal started life as a Haskell-specific hack that could be embedded
280 in a Bird-style Literate Haskell source file. I took a framework for
281 literate tests I had already written in a project called _Rho_, and used it
282 as the basis of this code.
283
284 I wrote it because I was designing yet another esoteric programming
285 language, and while working on it I realized I was rebuilding yet another
286 ad-hoc unit test suite, like I had done a half-dozen times before. I didn't
287 want to keep doing this for every language I designed, and I realized that
288 literate test suites could serve as documentation as well; the result was
289 Falderal.
+312
-0
HISTORY.md less more
0 History
1 =======
2
3 Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive":
4
5 * Add a pragma "Functionality ... is implemented by
6 shell command ... but only if shell command ... succeeds".
7 If the second shell command given in the pragma cannot be
8 run or exits with a non-zero exit code, the implementation
9 given by the first shell command is not registered for the given
10 functionality. (If no tests at all are run, it is an error.)
11 * `falderal`'s own test suite, `test.sh`, tests running `falderal`
12 under Python 2 if it is available, and under Python 3 if it is
13 available. If neither is available, it aborts. A particular
14 implementation of Python can also be supplied in the `PYTHON`
15 environment variable. Note however that Python 3 is still
16 required to be installed when running the tests (to run the
17 helper scripts.)
18 * The `falderal` command and the helper utilities (such as
19 `fa-under-pty`) run under `python3` by default now.
20 * Some internal unit tests for `falderal` were accidentally
21 deactivated in some previous version; they are now restored.
22 * `py-falderal` is now generally referred to as `falderal`.
23 * Specification edited for clarity and license text simplified.
24
25 Version 0.13 "Merchandise Mart":
26
27 * Use `ArgumentParser` instead of `OptionParser` internally.
28 * `--substring-error` is now the default behaviour, and if this
29 option is given, it will be ignored. After a long time
30 considering the problem, we finally determined that it does
31 not make sense to insist that different implementations
32 produce the exact same error message.
33 * `--verbose` option now prints out commands for each test as it
34 is run.
35 * Added conventional `--version` option.
36 * Planned to be the last 0.x series release before 1.0.
37 * Added `setup.py` so that Falderal can be treated as a Python
38 package.
39
40 Version 0.12 "Schoenhofen Brewery":
41
42 * When expanding variables in declaration strings, use
43 `string.replace` instead of `re.sub` so that backslash-escaping
44 is not perfomed on the replacement string. (Thanks to
45 James Holderness for bug report and patch.)
46 * In "freestyle" format, lines beginning with `<= `, `<== `, or
47 `<=== ` can supply a section of test input text (Thanks to
48 James Holderness for feature suggestion and patch.)
49 * Falderal is now responsible for quoting all shell metacharacters
50 in the substituion text of `%(...)` variables when command line
51 templates are expanded, so that no quotes need be placed around
52 the variable in the template. (Again, thanks to James Holderness
53 for spotting the bug in the code that led to this improvement.)
54 * `py-falderal` can now run under both Python 2 and Python 3
55 (tested with 2.7, 3.4, and 3.5.)
56 * The doctests that were previously distributed throughout the
57 source code have been converted to a suite of unit tests in
58 the `falderal.tests` module. The `-t` command-line option has
59 been removed; to run internal tests, run this module instead.
60
61 Version 0.11 "Dan Ryan Expressway":
62
63 * Renamed the internal tests to have descriptive names instead of
64 numbers.
65 * The first command-line argument being `test` for compatibility with
66 previous versions of `falderal` has been removed.
67 * Massive internal refactor of block-parsing and test-extraction code.
68 * Supports a new format for test blocks, called "freestyle", where if
69 the last few lines of a block are prefixed with `=> ` or `==> ` or
70 `===> ` or `?> ` or `??> ` or `???> `, the preceding part of the
71 block needs no prefixes at all, and it is simply treated as one
72 verbatim test body.
73
74 Version 0.10 "Shedd Aquarium":
75
76 * Removed the old, bitrotting Haskell implementation.
77 * Removed deprecated `%(test-text)` and `%(test-file)` variables
78 (`%(test-body-text)` and `%(test-body-file)` should be used now.)
79 * Added some adapters in the `script/` directory of the repo:
80 * `fa-under-pty` runs a target program under a pseudoterminal
81 * `fa-bin-to-hex` converts 8-bit binary output to hexadecimal codes
82 * `fa-hex-to-bin` converts hexadecimal codes to 8-bit binary output
83 * Added the rule that a test may consist of an input section (`+`) if
84 there was a previous test, and the test body of the previous test will
85 be re-used with the new input in the new test.
86
87 Version 0.9-2014.0525 "Municipal Pier #2, May 25th 2014":
88
89 * Shell implementations assume input and output is encoded UTF-8; any
90 malformed UTF-8 is simply ignored (for better or worse, for now.)
91 Falderal documents are assumed to be encoded in UTF-8; this includes
92 test bodies, expectations, etc. Internally, in the reference
93 implementation, all strings are stored as Unicode.
94
95 Version 0.9-2014.0427 "Municipal Pier #2":
96
97 * For a test of a shell command implementation that expects a failure,
98 if no output was found on standard error, the output of standard output
99 is checked against the failure expectation instead. The primary use case
100 for this is Hugs, which (unlike `ghc`) displays error messages on stdout
101 rather than stderr. "Municipal Pier #2" was the original name of Navy Pier.
102
103 Version 0.9 "Navy Pier":
104
105 * `py-falderal` now insists that blocks are indented four spaces, as the
106 specification says. It ignores blocks that are not indented.
107 * In an error report `py-falderal` reports the location (filename and line
108 number of the beginning of the test text) of each test that failed, as
109 well as the name of the functionality that was being tested.
110 * Several internal renamings of class and variable names for greater clarity.
111 * Trying to standardize on the terminology "test body" for the body of a
112 test (as opposed to its "input", and opposed to it being passed to a shell
113 command line "as text".)
114 * Support for specifying a block of text to be used as input for a test.
115 * Dropped support for "implemented by Haskell function" backwards-compatibility.
116 * Temporary files are no longer always created in the current directory. That
117 was a workaround to support `ghc` from Haskell Platform under Cygwin. They
118 are now created where the system wants them to be created (but you can set
119 the `TMP` environment variable to "." if you really want them to be created
120 in the current directory.)
121 * `py-falderal` now complains and stops early if there were no documents
122 specified, no functionality definitions were found in any of the documents,
123 no implementations were found for some defined functionality, or if no
124 tests were found in any of the documents, unless `--cavalier` is passed on
125 the command line.
126 * In `--verbose` mode, `py-falderal` will issue warnings about any test that
127 has been run twice (i.e. more than run on identical test body text and
128 implementation.)
129 * If `py-falderal` catches a `KeyboardInterrupt` exception while running tests,
130 it shows a report of all the tests that it did manage to run until that
131 point, and a warning message to the effect that not all tests were run.
132
133 Version 0.8 "Ukrainian Village":
134
135 * The reference implementation is now `py-falderal`.
136 * Blocks *must* now be indented four spaces. That Falderal test documents
137 will be embedded in Markdown is generally assumed (although not strictly
138 required) and they generally have the file extension `.markdown` so that
139 other software (e.g. Bitbucket and Github) knows how to pretty-display
140 them.
141 * Various documentation which was on Bitbucket (wiki, issues) and Github
142 (py-falderal issues) have been moved into Markdown documents in the
143 distribution, which have themselves been re-organized.
144 * Added the `-v`/`--verbose` flag and made it and the `-d`/`--dump` flag
145 do what you would expect from them.
146 * The `-f`/`--functionalities` and `-c`/`--clear-functionalities` flags
147 have been *removed*. The rationale is that implementations of
148 functionalities are clumsy to express on the command line (and anyway
149 the syntax would have to be different) while it is not difficult to
150 create a "configuration" Falderal file which specifies *only* the
151 desired implementations of the desired functionalities, and to load
152 it just before loading a "tests" Falderal file which specified the
153 functionalities being tested, but not their implementations.
154
155 Version 0.7 "Ogden Avenue":
156
157 * **Note that this is the final version of Falderal where `Test.Falderal`
158 is the reference implementation.** In subsequent versions,
159 `py-falderal` (written in Python) will be the reference implementation.
160 `Test.Falderal` will still be in this repository (under the `impl`
161 directory) but will no longer be definitive, and will go essentially
162 unmaintained.
163 * Fixed a bug where number of lines in the intermediary results file
164 was sometimes being counted incorrect.
165 * Slightly improved support for running under Cygwin.
166 * Tests that are implemented by a shell command are now run directly
167 from the `falderal` process (instead of creating an intermediary
168 shell script and running it.) This resulted in better performance.
169 * Addition of `-b` command-line option, which considers a test to
170 have passed if the expected exception message is a substring (rather
171 than an exact match) of the actual produced exception message.
172 TODO: write test for this.
173 * Blocks may now be indented four spaces, to allow them to be
174 embedded directly in Markdown files as pre-formatted text.
175 * A script to just build the binary, without installing it, was added.
176
177 Version 0.6 "Streeterville":
178
179 * Variables may be included in the specification of a shell command
180 implementation; these will be expanded before generating the results
181 generator. These variable include `%(test-text)`, `%(test-file)`,
182 and `%(output-file)`.
183 * `falderal` now lets the user specify, on the command line, the
184 implementations for a named functionality. Supplying `-f 'foo:shell
185 command "foo.sh %(test-file)"'` has the same effect as including the
186 pragma `Functionality "foo" is implemented by shell command "foo.sh
187 %(test-file)"` in the Falderal file. Note that this is in addition
188 to the Functionality-definition pragmas given in the Falderal file;
189 to replace them, you must first clear the ones with that name from
190 the file by supplying `-c foo` on the command line.
191 * `falderal` also allows tests for named functionalities to be skipped
192 completely, by passing the name of the functionality to be skipped
193 after a `-k` flag on the command line.
194 * The Markdown formatter now formats Bird-style embedded code with
195 HTML embedded in the Markdown document. This is so that it can
196 be styled independently from, and thus distinguished from, any plain
197 Markdown indented code blocks which may appear in the literate portion
198 of the source code.
199 * In failure reports, the implementation of the functionality of the test
200 that failed is now reported in each failure.
201 * A race condition(?) that could occur when testing multiple implementations
202 of a functionality, of different kinds (Haskell and shell), has been
203 prevented. Both tests were writing to `results.txt` and immediately
204 deleting it, and this would sometimes confuse `falderal` into thinking
205 one had produced no results (perhaps a result of some creative
206 scheduling by `ghc`, although really, I haven't a clue.) Results are
207 now written to different temporary files with different, generated
208 names.
209 * Previously, if the output of a shell command being tested did not end
210 with a newline, the intermediate results file was not being generated
211 correctly, resulting in failures being misreported. This has been
212 recitified.
213 * Previously, if there were tests given in a Falderal file before any
214 Tests-for pragma was specified, those tests would just be ignored.
215 An error message is now issued, and no testing takes place.
216 * Previously, if multiple Falderal files were given on the command
217 line, they were simply concatenated when loaded, the result being
218 that Functionality-definitions from the first file were visible in
219 the second file, and that any Tests-for in effect at the end of the
220 first file would be in effect at the start of the second file.
221 Files are now loaded and processed seperately.
222
223 Version 0.5 "The Loop":
224
225 * The command-line format of `falderal` has changed, for the better.
226 The `test` subcommand no longer requires that the failure reporting
227 style be specified; instead, it defaults to `standard`, and can be
228 changed with a command-line option. There are also command-line
229 options for selecting the programs to run results generators, and to
230 keep generated files around instead of cleaning them up after testing.
231 * The dependency on `ghc` for running Haskell results generators has been
232 removed; these can be run by `runhaskell` now, and are run by
233 `runhaskell` by default.
234 * Failure reporting is now consistent across languages; both Haskell and
235 Bourne shell results generators generate an intermediate format, which
236 `falderal` digests.
237 * A new pragma `encoding:` was added, so that this directive can be
238 embedded in your Falderal document (for the benefit of your text editor)
239 without necessarily appearing in the formatted document.
240 * We began giving release milestones colorful names. The naming
241 convention is to choose names of Chicagoland neigborhoods, suburbs,
242 landmarks, and institutions. Version 0.5 was named after The Loop in
243 recognition of its ability to shuttle test results between `falderal`
244 and the various results generators implemented in different languages.
245 Previous versions of `Test.Falderal` were retroactively given milestone
246 names during this release.
247
248 Version 0.4 "Blackstone Hotel":
249
250 * For ease of installation, the project is presented as a Cabal package.
251 * A driver executable, `falderal`, is built as part of installing the
252 Cabal package. It provides a command-line interface to formatting
253 Falderal files and, in a limited fashion, running the tests in them.
254 * A shell script formatter has been written, enabling testing of shell
255 commands. One caveat is that reporting for these tests is nowhere near
256 as nice as for Haskell functions, but that will change in the next version.
257 * The Functionality-definition pragma has been implemented, making it
258 possible to write tests in a more abstract, implementation-independent
259 fashion.
260 * Falderal files written to work with Falderal 0.3 should still work with
261 Falderal 0.4, but you are encouraged to use the Functionality-definition
262 pragma introduced in 0.4 to make your tests more implementation-independent.
263
264 Version 0.3 "Chicago Board of Trade":
265
266 * The definition of a Falderal Literate Test Format, distinct from the
267 reference implementation of tools for it in Haskell (`Test.Falderal`).
268 This represented a fairly substantial departure from how previous versions
269 of Falderal worked.
270 * The ability to format a Falderal file to different formats, including
271 Markdown and Haskell.
272 * Running tests is now a matter of formatting a Falderal file as a Haskell
273 module and running the function `testModule` in that module.
274
275 Version 0.2 "Dearborn Station":
276
277 * Added a test harness for Falderal itself, in the form of a simple shell
278 script which diffs the output of `Test.Falderal.Demo` against a text file
279 containing the expected content.
280 * Improved formatting of failure reports. Multi-line input text or expected
281 output is always presented starting on its own line.
282 * Tests may be organized into groups; the entire group is preceded by some
283 literal text, but there is no literal text between the tests in the group.
284 When one of these tests fails, the literal text for the group is reported,
285 along with the number of the test within the group.
286 * Fixed a bug where exception text which extended over multiple lines
287 could not be expected correctly.
288
289 Version 0.1 "Haymarket Square":
290
291 * Provision of a framework for writing and running literate tests which may
292 be embedded in literate Haskell source code.
293 * Testing Haskell functions of type `String -> String`. A test specifies input
294 to the function, and may expect a particular output, or that a particular
295 exception is thrown.
296 * Through simple adapters, testing functions of other types such as
297 `(Show a) => String -> a`.
298
299 Prehistory:
300
301 Falderal started life as a Haskell-specific hack that could be embedded
302 in a Bird-style Literate Haskell source file. I took a framework for
303 literate tests I had already written in a project called _Rho_, and used it
304 as the basis of this code.
305
306 I wrote it because I was designing yet another esoteric programming
307 language, and while working on it I realized I was rebuilding yet another
308 ad-hoc unit test suite, like I had done a half-dozen times before. I didn't
309 want to keep doing this for every language I designed, and I realized that
310 literate test suites could serve as documentation as well; the result was
311 Falderal.
+22
-81
LICENSE less more
0 Falderal is distributed under the following BSD-compatible licenses.
0 BSD 3-Clause License
11
2 All documentation is covered by this license, modelled after the
3 "Report on the Programming Language Haskell 98" license:
4
5 -----------------------------------------------------------------------------
6
7 Copyright (c)2011-2016 Chris Pressey, Cat's Eye Technologies.
8 All rights reserved.
9
10 The authors intend this Report to belong to the entire Falderal
11 community, and so we grant permission to copy and distribute it for
12 any purpose, provided that it is reproduced in its entirety,
13 including this Notice. Modified versions of this Report may also be
14 copied and distributed for any purpose, provided that the modified
15 version is clearly presented as such, and that it does not claim to
16 be a definition of the Falderal Literate Test Format.
17
18 -----------------------------------------------------------------------------
19
20 All source code for the implementations of Falderal in this distribution
21 (py-falderal) as well as the internal test suite itself is covered by this
22 BSD 3-clause license:
23
24 -----------------------------------------------------------------------------
25
26 Copyright (c)2011-2016 Cat's Eye Technologies. All rights reserved.
2 Copyright (c) 2011-2022, Chris Pressey, Cat's Eye Technologies.
3 All rights reserved.
274
285 Redistribution and use in source and binary forms, with or without
29 modification, are permitted provided that the following conditions
30 are met:
6 modification, are permitted provided that the following conditions are met:
317
32 1. Redistributions of source code must retain the above copyright
33 notices, this list of conditions and the following disclaimer.
34 2. Redistributions in binary form must reproduce the above copyright
35 notices, this list of conditions, and the following disclaimer in
36 the documentation and/or other materials provided with the
37 distribution.
38 3. Neither the names of the copyright holders nor the names of their
39 contributors may be used to endorse or promote products derived
40 from this software without specific prior written permission.
8 * Redistributions of source code must retain the above copyright notice, this
9 list of conditions and the following disclaimer.
4110
42 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
43 ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
44 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
45 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
46 COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
47 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
48 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
49 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
50 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
51 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
52 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
53 POSSIBILITY OF SUCH DAMAGE.
11 * Redistributions in binary form must reproduce the above copyright notice,
12 this list of conditions and the following disclaimer in the documentation
13 and/or other materials provided with the distribution.
5414
55 -----------------------------------------------------------------------------
15 * Neither the name of the copyright holder nor the names of its
16 contributors may be used to endorse or promote products derived from
17 this software without specific prior written permission.
5618
57 The adapters in the `script` subdirectory are in the public domain,
58 and covered under the following UNLICENSE:
59
60 -----------------------------------------------------------------------------
61
62 This is free and unencumbered software released into the public domain.
63
64 Anyone is free to copy, modify, publish, use, compile, sell, or
65 distribute this software, either in source code form or as a compiled
66 binary, for any purpose, commercial or non-commercial, and by any
67 means.
68
69 In jurisdictions that recognize copyright laws, the author or authors
70 of this software dedicate any and all copyright interest in the
71 software to the public domain. We make this dedication for the benefit
72 of the public at large and to the detriment of our heirs and
73 successors. We intend this dedication to be an overt act of
74 relinquishment in perpetuity of all present and future rights to this
75 software under copyright law.
76
77 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
78 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
79 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
80 IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
81 OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
82 ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
83 OTHER DEALINGS IN THE SOFTWARE.
84
85 For more information, please refer to <http://unlicense.org/>
86
87 -----------------------------------------------------------------------------
19 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
20 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
22 DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
23 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
25 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
26 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
27 OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+0
-182
README.markdown less more
0 Falderal
1 ========
2
3 Version 0.13 "Merchandise Mart"
4
5 This is the reference distribution of Falderal, a file format for literate
6 test suites. What sets Falderal apart from most other test frameworks is
7 that it recognizes that **your tests are for a functionality, not a particular
8 implementation of that functionality**.
9
10 Falderal is particularly suited for:
11
12 * documenting programming languages with prose and examples
13 * testing multiple implementations of a programming language
14 * performing Test-Driven Language Design (TDLD)
15 * being embedded in Markdown documents
16
17 Falderal in three words: "Doctests for DSLs".
18
19 If you're more interested in running the tools than learning about the format
20 itself, skip down to [Implementation](#implementation).
21
22 Motivation
23 ----------
24
25 Say you have a programming language. You probably have a document describing
26 the language. That document probably includes examples. Those examples are
27 probably intended to produce some kind of output. That output is probably
28 expected to be a certain, predicted thing, and if it's not that thing, the
29 implementation is probably considered incorrect.
30
31 So why not write those examples in a format that can be run and tested?
32
33 You could write a bunch of standalone test sources, and store the output you
34 expect from them in a bunch of other files, and write a shell script that runs
35 each program and `diff`s the output with the expected output. But this is a
36 lot of clutter — finding a particular example might not be so easy. Each
37 test source exists in a void, not necessarily logically grouped with other,
38 similar tests. And any text you write describing a test needs to be in the
39 comment syntax of your programming language (if your programming language
40 supports comments) and is also detached from all the other test descriptions.
41
42 You could write unit tests in the unit test framework of your choice, but
43 if your programming language has more than one implementation one day (and
44 you should really consider that possibility) then you might not be able to
45 re-use it so easily for other implementations in other languages.
46
47 In a language like Python, you could write doctests, but that also ties your
48 tests to one implementation of your language. There can be awkward
49 quoting issues with how you embed your test sources inside those embedded
50 strings that comprise your doctests, as well.
51
52 Or... you could write a Markdown document with beautiful yet precise prose
53 describing your wonderful language, alternating with example code (in the
54 form of embedded Falderal tests) clarifying each of the points you are
55 making; then you could use a Falderal-comprehending tool to run each of these
56 tests against any implementation of your language which exists or will exist
57 in the future.
58
59 *And* you could even write this document *before* you even start implementing
60 your language; then when it is all clear "on paper", you have a target at
61 which you can aim while writing your language. As you implement more and more
62 of it, more and more tests in your test suite will pass. This is simply the
63 idea behind Test-Driven Development (TDD) applied to language design, which we
64 will call Test-Driven Language Design (TDLD).
65
66 Features of the Format
67 ----------------------
68
69 Falderal is just a file format; it does not specify exactly what tools must
70 do with the tests they extract. However, it is generally assumed that most
71 tools will want to, at the very least,
72
73 * Run tests from one or more documents.
74 * Report the results, with some given level of detail.
75
76 There is, of course, a reference implementation which does both of these
77 things. It is called py-falderal and it is written in Python 2.7.
78
79 Each Falderal test is for some abstract _functionality_, and each
80 functionality may have multiple concrete _implementations_. Thus the same
81 tests can be run multiple times, once for each implementation of the
82 functionality they test.
83
84 Directives in the Falderal document may assign functionalities to tests,
85 and may define implementations for given functionalities. Implementations
86 may be defined outside of any document, as well. Falderal defines one
87 general kind of implementation, implementation by Bourne shell command, but
88 is not inherently restricted from supporting other kinds of implementations.
89
90 Inherent Limitations
91 --------------------
92
93 Being tests, rather than proofs, Falderal tests cannot tell you that your
94 implementation of a language is correct. If one or more tests fail, that's
95 an indication that your implementation is not correct; but even if all tests
96 pass, you have no guarantee that the implementation doesn't do something
97 contrary to the spec in one of the infinite number of cases that you have not
98 enumerated.
99
100 There is also no way to test that certain programs represent infinite loops,
101 for reasons that should be obvious.
102
103 Contents of this Distribution
104 -----------------------------
105
106 This distribution contains:
107
108 * `doc` — contains documents about Falderal. For the specification of
109 the file format, see
110 [`doc/Falderal_Literate_Test_Format.markdown`](doc/Falderal_Literate_Test_Format.markdown).
111 (Note that this specification should not be expected to remain stable
112 through the 0.x version series.) There are other documents in there too.
113 * `bin/falderal` — the reference implementation of Falderal.
114 See "Implementation", below, for details.
115 * `script` — miscellaneous small tools intended to be used in tests.
116 * `src` — source code for py-falderal.
117 * `tests` — a set of tests for Falderal itself. (Note that these are not
118 written in Falderal, as that would just be too confusing.)
119 * `HISTORY.markdown` — changelog for releases of Falderal.
120 * `TODO.markdown` — areas where Falderal and its implementations could be
121 improved.
122
123 Implementation
124 --------------
125
126 This distribution contains `falderal`, which is the reference implementation
127 of Falderal, written in Python and sometimes referred to as "py-falderal".
128
129 To use it, you can clone this repository and run it as `bin/falderal`
130 from the directory of your clone, or you can put the `bin` directory
131 on your executable search path, and run it as `falderal` anywhere.
132
133 Or you can install it using `pip`:
134
135 pip install -e Falderal==0.13
136
137 (Depending on your needs, you may wish to establish a virtual environment
138 first. Describing how to do so is outside the scope of this document.)
139
140 The implementation is (somewhat) documented in `doc/py-falderal.markdown`.
141
142 Development
143 -----------
144
145 The git repository for the Falderal distribution can be found on GitHub at
146 [https://github.com/catseye/Falderal](https://github.com/catseye/Falderal).
147
148 Official release distfiles are available via the
149 [Falderal project page](http://catseye.tc/node/Falderal) at
150 [Cat's Eye Technologies](http://catseye.tc/).
151
152 Projects using Falderal
153 -----------------------
154
155 * [ALPACA](http://catseye.tc/node/ALPACA)
156 * [Castile](http://catseye.tc/node/Castile)
157 * [Equipage](http://catseye.tc/node/Equipage)
158 * [Exanoke](http://catseye.tc/node/Exanoke)
159 * [Flobnar](http://catseye.tc/node/Flobnar)
160 * [Hev](http://catseye.tc/node/Hev)
161 * [Iphigeneia](http://catseye.tc/node/Iphigeneia)
162 * [Madison](http://catseye.tc/node/Madison)
163 * [Pail](http://catseye.tc/node/Pail)
164 * [Pixley](http://catseye.tc/node/Pixley)
165 * [PL-{GOTO}.NET](http://catseye.tc/node/PL-{GOTO}.NET)
166 * [Quylthulg](http://catseye.tc/node/Quylthulg)
167 * [Robin](http://catseye.tc/node/Robin)
168 * [Samovar](http://catseye.tc/node/Samovar)
169 * [SixtyPical](http://catseye.tc/node/SixtyPical)
170 * [Tamsin](http://catseye.tc/node/Tamsin)
171 * [Velo](http://catseye.tc/node/Velo)
172 * [Yolk](http://catseye.tc/node/Yolk)
173 * [Xoomonk](http://catseye.tc/node/Xoomonk)
174
175 Xoomonk, Madison, Velo, and Exanoke are good examples of how a literate
176 test suite can be useful in both describing a programming language through
177 examples and testing that an implementation of the language does not violate
178 the language specification. They are, in fact, exercises in Test-Driven
179 Language Design (TDLD), where the tests were written as part of designing the
180 language, before any attempt at implementation; the others are more like
181 traditional test suites, written after-the-fact.
+190
-0
README.md less more
0 Falderal
1 ========
2
3 Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive"
4
5 This is the reference distribution of Falderal, a file format for literate
6 test suites. What sets Falderal apart from most other test frameworks is
7 that it recognizes that **your tests are for a functionality, not a particular
8 implementation of that functionality**.
9
10 Falderal is particularly suited for:
11
12 * documenting programming languages with prose and examples
13 * testing multiple implementations of a programming language
14 * performing Test-Driven Language Design (TDLD)
15 * being embedded in Markdown documents
16
17 Falderal in three words: "Doctests for DSLs".
18
19 If you're more interested in running the tools than learning about the format
20 itself, skip down to [Implementation](#implementation).
21
22 Motivation
23 ----------
24
25 Say you have a programming language. You probably have a document describing
26 the language. That document probably includes examples. Those examples are
27 probably intended to produce some kind of output. That output is probably
28 expected to be a certain, predicted thing, and if it's not that thing, the
29 implementation is probably considered incorrect.
30
31 So why not write those examples in a format that can be run and tested?
32
33 You could write a bunch of standalone test sources, and store the output you
34 expect from them in a bunch of other files, and write a shell script that runs
35 each program and `diff`s the output with the expected output. But this is a
36 lot of clutter — finding a particular example might not be so easy. Each
37 test source exists in a void, not necessarily logically grouped with other,
38 similar tests. And any text you write describing a test needs to be in the
39 comment syntax of your programming language (if your programming language
40 supports comments) and is also detached from all the other test descriptions.
41
42 You could write unit tests in the unit test framework of your choice, but
43 if your programming language has more than one implementation one day (and
44 you should really consider that possibility) then you might not be able to
45 re-use it so easily for other implementations in other languages.
46
47 In a language like Python, you could write doctests, but that also ties your
48 tests to one implementation of your language. There can be awkward
49 quoting issues with how you embed your test sources inside those embedded
50 strings that comprise your doctests, as well.
51
52 Or... you could write a Markdown document with beautiful yet precise prose
53 describing your wonderful language, alternating with example code (in the
54 form of embedded Falderal tests) clarifying each of the points you are
55 making; then you could use a Falderal-comprehending tool to run each of these
56 tests against any implementation of your language which exists or will exist
57 in the future.
58
59 *And* you could even write this document *before* you even start implementing
60 your language; then when it is all clear "on paper", you have a target at
61 which you can aim while writing your language. As you implement more and more
62 of it, more and more tests in your test suite will pass. This is simply the
63 idea behind Test-Driven Development (TDD) applied to language design, which we
64 will call Test-Driven Language Design (TDLD).
65
66 Features of the Format
67 ----------------------
68
69 Falderal is just a file format; it does not specify exactly what tools must
70 do with the tests they extract. However, it is generally assumed that most
71 tools will want to, at the very least,
72
73 * Run tests from one or more documents.
74 * Report the results, with some given level of detail.
75
76 There is, of course, a reference implementation which does both of these
77 things. It is called `falderal` and it is written in Python 3.x. More
78 about it can be read in the [`falderal` document](doc/falderal.md).
79
80 Each Falderal test is for some abstract _functionality_, and each
81 functionality may have multiple concrete _implementations_. Thus the same
82 tests can be run multiple times, once for each implementation of the
83 functionality they test.
84
85 Directives in the Falderal document may assign functionalities to tests,
86 and may define implementations for given functionalities. Implementations
87 may be defined outside of any document, as well. Falderal defines one
88 general kind of implementation, implementation by Bourne shell command, but
89 is not inherently restricted from supporting other kinds of implementations.
90
91 Inherent Limitations
92 --------------------
93
94 Being tests, rather than proofs, Falderal tests cannot tell you that your
95 implementation of a language is correct. If one or more tests fail, that's
96 an indication that your implementation is not correct; but even if all tests
97 pass, you have no guarantee that the implementation doesn't do something
98 contrary to the spec in one of the infinite number of cases that you have not
99 enumerated.
100
101 There is also no way to test that certain programs represent infinite loops,
102 for reasons that should be obvious.
103
104 Contents of this Distribution
105 -----------------------------
106
107 This distribution contains:
108
109 * `doc` — contains documents about Falderal. For the specification of
110 the file format, see
111 [`doc/Falderal Literate Test Format.md`](doc/Falderal%20Literate%20Test%20Format.md).
112 (Note that this specification should not be expected to remain stable
113 through the 0.x version series.) There are other documents in there too.
114 * `bin/falderal` — the reference implementation of Falderal.
115 See "Implementation", below, for details.
116 * `script` — miscellaneous small tools intended to be used in tests.
117 * `src` — source code for the `falderal` tool.
118 * `tests` — a set of tests for Falderal itself. (Note that these are not
119 written in Falderal, as that would just be too confusing.)
120 * `HISTORY.md` — changelog for releases of Falderal.
121 * `TODO.md` — areas where Falderal and its implementations could be
122 improved.
123
124 Implementation
125 --------------
126
127 This distribution contains `falderal`, which is the reference implementation
128 of Falderal, written in Python and sometimes referred to as "py-falderal".
129
130 To use it, you can clone this repository and run it as `bin/falderal`
131 from the directory of your clone, or you can put the `bin` directory
132 on your executable search path, and run it as `falderal` anywhere.
133
134 Or you can install it using `pip`:
135
136 pip install -e Falderal==0.14
137
138 (Depending on your needs, you may wish to establish a virtual environment
139 first. Describing how to do so is outside the scope of this document.)
140
141 The implementation is (somewhat) documented in
142 [`doc/falderal.md`](doc/falderal.md).
143
144 The implementation has its own test suite (not written in Falderal, as
145 that might be too confusing). It can be run by running `./test.sh`.
146
147 Development
148 -----------
149
150 The git repository for the Falderal distribution can be found on GitHub at
151 [https://github.com/catseye/Falderal](https://github.com/catseye/Falderal).
152
153 Official release distfiles are available via the
154 [Falderal project page](http://catseye.tc/node/Falderal) at
155 [Cat's Eye Technologies](http://catseye.tc/).
156
157 Projects using Falderal
158 -----------------------
159
160 * [ALPACA](https://catseye.tc/node/ALPACA)
161 * [Castile](https://catseye.tc/node/Castile)
162 * [Equipage](https://catseye.tc/node/Equipage)
163 * [Exanoke](https://catseye.tc/node/Exanoke)
164 * [Flobnar](https://catseye.tc/node/Flobnar)
165 * [Hev](https://catseye.tc/node/Hev)
166 * [hatoucan](https://catseye.tc/node/hatoucan)
167 * [Iphigeneia](https://catseye.tc/node/Iphigeneia)
168 * [Madison](https://catseye.tc/node/Madison)
169 * [Nhohnhehr](https://catseye.tc/node/Nhohnhehr)
170 * [Pail](https://catseye.tc/node/Pail)
171 * [Pixley](https://catseye.tc/node/Pixley)
172 * [PL-{GOTO}.NET](https://catseye.tc/node/PL-{GOTO}.NET)
173 * [Quylthulg](https://catseye.tc/node/Quylthulg)
174 * [Robin](https://catseye.tc/node/Robin)
175 * [Samovar](https://catseye.tc/node/Samovar)
176 * [SixtyPical](https://catseye.tc/node/SixtyPical)
177 * [Tamsin](https://catseye.tc/node/Tamsin)
178 * [Velo](https://catseye.tc/node/Velo)
179 * [Xoomonk](https://catseye.tc/node/Xoomonk)
180 * [Yolk](https://catseye.tc/node/Yolk)
181 * [ZOWIE](https://catseye.tc/node/ZOWIE)
182
183 Xoomonk, Madison, Velo, and Exanoke are good examples of how a literate
184 test suite can be useful in both describing a programming language through
185 examples and testing that an implementation of the language does not violate
186 the language specification. They are, in fact, exercises in Test-Driven
187 Language Design (TDLD), where the tests were written as part of designing the
188 language, before any attempt at implementation; the others are more like
189 traditional test suites, written after-the-fact.
+0
-186
TODO.markdown less more
0 TODO
1 ====
2
3 (collected from the Falderal issue tracker on Bitbucket and the
4 py-falderal issue tracker on github)
5
6 Falderal Literate Test Format
7 -----------------------------
8
9 ### Policy for expecting both errors and output, success and failure
10
11 Policy should be this, by example:
12
13 | test
14 = foo
15
16 Means: I expect this to succeed and to produce `foo` on stdout, and
17 I don't care what's on stderr (or — stderr should be empty?)
18
19 | test
20 = foo
21 ? bar
22
23 Means: I expect this to succeed, to produce `foo` on stdout, and to
24 produce `bar` on stderr.
25
26 | test
27 ? foo
28
29 Means: I expect this to fail, and to produce `foo` on stderr.
30 And to not care about stdout (or expect it to be empty.)
31
32 | test
33 ? foo
34 = bar
35
36 Means: I expect this to fail, to produce `foo` on stderr, and to
37 produce `bar` on stdout.
38
39 In other words, an error expectation may follow an output expectation
40 and vice versa. Error expectations always match stderr, output expectations
41 always match stdout. Which one's first should dictate whether we expect
42 the command to succeed or fail.
43
44 What's after here hasn't been re-edited yet.
45
46 When you have a program that produces output on both stdout and stderr,
47 whether it fails or not, you might want to expect text on both stdout and
48 stderr.
49
50 Currently it expects the text on stdout if it is a `=` expectation, and on
51 stderr if it is a `?` expectation.
52
53 You can't work around this so well by tacking `2>&1` onto the end of the
54 command, because then stderr will always be empty.
55
56 We could, by default, tack `2>&1` on the end ourselves and look only at
57 stdout. This might be the simplest approach.
58
59 We might want to add options that avoid doing that, but if so, what should
60 they be? Should each test be able to configure this? Should a single test
61 be able to have both `=` and `?` expectations, each for each stream?
62
63 This is complicated by the presence of `%(output-file)`; currently, if that
64 is given, stdout is ignored in preference to it (but stderr is still
65 checked, if the command failed. There should probably be a corresponding
66 `%(error-file)` variable.)
67
68 I think the current behaviour could work, with the following policy:
69
70 If the command succeeds, your `=` expectation will be matched against
71 stdout only. If you wish to match against both stdout and stderr in these
72 cases, add `2>&1` to your shell command.
73
74 If the command fails, your `?` expectation will be matched against stderr
75 only. If you wish to match against both stdout and stderr in these cases,
76 add `1>&2` to your shell command.
77
78 Either way, it's still worth investigating whether it's worthwhile to have
79 both `=` and `?` expectations on a single test. (I can't convince myself
80 that stdout and stderr will always be combined deterministically, and
81 having both kinds of expectations would allow non-deterministic combinations
82 of the two to be matched.)
83
84 ### Allow expectations to be transformed during comparison
85
86 It would be nice to allow expectations to be transformed before they are
87 compared to the actual output. The main use case for this that I can think of
88 is to allow the expected output to be "pretty printed" (that is, nicely
89 formatted) in the Falderal file, while the functionality being tested just
90 produces a dump. The nicely formatted expected output should be "crunched"
91 into the same ugly format as the dump.
92
93 This doesn't work as well the other way; although one could compose the
94 functionality being tested with an actual pretty-printer, that would
95 prescribe a certain indentation scheme etc. that the expected output would
96 have to match exactly. It would be rather better if the writer of the tests
97 could format their expected output as they find most aesthetically pleasing
98 in their literate tests, and have that be transformed instead.
99
100 This might be somewhat tricky, however; if the transformation applied is
101 too powerful, it can distort or eliminate the meaning of the test, and erode
102 confidence.
103
104 ### Allow use of patterns in expected output
105
106 Likely by way of regexps. This would be particularly valuable in
107 exception-expecting tests, where we don't care about details such as the
108 line number of the Haskell file at which the exception occurred.
109
110 ### Allow equivalency tests to be defined and run.
111
112 To test functions of type `(Eq a) => String -> a`, you should be able to
113 give give multiple input strings in a set; if the function does not map
114 them all to the same value, that's a test failure.
115
116 Syntax for an equivalency test might look like this:
117
118 | 2+2
119 ==
120 | 3+1
121 ==
122 | 7-3
123
124
125 py-falderal
126 -----------
127
128 ### Test report accumulation
129
130 Multiple runs of `falderal` ought to be able to accumulate their results
131 to a temporary file. No report should be generated if this is selected.
132 At the end, the report can be generated from the file.
133
134 rm -f FILE
135 falderal --record-to FILE tests1.markdown
136 falderal --record-to FILE tests2.markdown
137 falderal --report-from FILE
138
139 ### Support 'weak' testing
140
141 In which we only care about whether the command succeeded or failed.
142 In practice, this could be useful for testing the parser (just test
143 if these forms parse.) Or, if not this, then think of something that
144 would make just testing parsers more useful.
145
146 ### Split InterveningMarkdown blocks to make nice test descriptions
147
148 For example, if we have
149
150 ...test #1...
151 Some text.
152 Heading
153 -------
154 More text
155 ...test #2...
156
157 The description for test #2 should consist of "More text"; possibly also
158 the heading, but not "Some text". This can take place in a pre-processing
159 phase which simply splits every `InterveningMarkdown` block into multiple
160 blocks, at its headers. It should understand both underlined and atx-style
161 headers.
162
163 ### option to colourize test result output
164
165 Using one of the approaches listed here:
166
167 http://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python
168
169 ...py-falderal ought to provide an option (not default, of course, and not
170 if stdout is not a tty) to colorize the output with, of course, pass=green,
171 fail=red.
172
173 But, you'd often want to pipe the output to `less`, which by default makes
174 control characters visible, defeating colourization. But there is a flag
175 to less, `less -R`, which retains colourization. So use that.
176
177 ### Flag invalid sequences of lines as errors
178
179 2011-08-05
180
181 Currently, in `convertLinesToBlocks`, some invalid sequences of lines are
182 ignored. They should be flagged as errors in the test suite file.
183
184 (This was written against Test.Falderal but similar considerations could
185 be made for py-falderal.)
+186
-0
TODO.md less more
0 TODO
1 ====
2
3 (collected from the Falderal issue tracker on Bitbucket and the
4 py-falderal issue tracker on github)
5
6 Falderal Literate Test Format
7 -----------------------------
8
9 ### Policy for expecting both errors and output, success and failure
10
11 Policy should be this, by example:
12
13 | test
14 = foo
15
16 Means: I expect this to succeed and to produce `foo` on stdout, and
17 I don't care what's on stderr (or — stderr should be empty?)
18
19 | test
20 = foo
21 ? bar
22
23 Means: I expect this to succeed, to produce `foo` on stdout, and to
24 produce `bar` on stderr.
25
26 | test
27 ? foo
28
29 Means: I expect this to fail, and to produce `foo` on stderr.
30 And to not care about stdout (or expect it to be empty.)
31
32 | test
33 ? foo
34 = bar
35
36 Means: I expect this to fail, to produce `foo` on stderr, and to
37 produce `bar` on stdout.
38
39 In other words, an error expectation may follow an output expectation
40 and vice versa. Error expectations always match stderr, output expectations
41 always match stdout. Which one's first should dictate whether we expect
42 the command to succeed or fail.
43
44 What's after here hasn't been re-edited yet.
45
46 When you have a program that produces output on both stdout and stderr,
47 whether it fails or not, you might want to expect text on both stdout and
48 stderr.
49
50 Currently it expects the text on stdout if it is a `=` expectation, and on
51 stderr if it is a `?` expectation.
52
53 You can't work around this so well by tacking `2>&1` onto the end of the
54 command, because then stderr will always be empty.
55
56 We could, by default, tack `2>&1` on the end ourselves and look only at
57 stdout. This might be the simplest approach.
58
59 We might want to add options that avoid doing that, but if so, what should
60 they be? Should each test be able to configure this? Should a single test
61 be able to have both `=` and `?` expectations, each for each stream?
62
63 This is complicated by the presence of `%(output-file)`; currently, if that
64 is given, stdout is ignored in preference to it (but stderr is still
65 checked, if the command failed. There should probably be a corresponding
66 `%(error-file)` variable.)
67
68 I think the current behaviour could work, with the following policy:
69
70 If the command succeeds, your `=` expectation will be matched against
71 stdout only. If you wish to match against both stdout and stderr in these
72 cases, add `2>&1` to your shell command.
73
74 If the command fails, your `?` expectation will be matched against stderr
75 only. If you wish to match against both stdout and stderr in these cases,
76 add `1>&2` to your shell command.
77
78 Either way, it's still worth investigating whether it's worthwhile to have
79 both `=` and `?` expectations on a single test. (I can't convince myself
80 that stdout and stderr will always be combined deterministically, and
81 having both kinds of expectations would allow non-deterministic combinations
82 of the two to be matched.)
83
84 ### Allow expectations to be transformed during comparison
85
86 It would be nice to allow expectations to be transformed before they are
87 compared to the actual output. The main use case for this that I can think of
88 is to allow the expected output to be "pretty printed" (that is, nicely
89 formatted) in the Falderal file, while the functionality being tested just
90 produces a dump. The nicely formatted expected output should be "crunched"
91 into the same ugly format as the dump.
92
93 This doesn't work as well the other way; although one could compose the
94 functionality being tested with an actual pretty-printer, that would
95 prescribe a certain indentation scheme etc. that the expected output would
96 have to match exactly. It would be rather better if the writer of the tests
97 could format their expected output as they find most aesthetically pleasing
98 in their literate tests, and have that be transformed instead.
99
100 This might be somewhat tricky, however; if the transformation applied is
101 too powerful, it can distort or eliminate the meaning of the test, and erode
102 confidence.
103
104 ### Allow use of patterns in expected output
105
106 Likely by way of regexps. This would be particularly valuable in
107 exception-expecting tests, where we don't care about details such as the
108 line number of the Haskell file at which the exception occurred.
109
110 ### Allow equivalency tests to be defined and run.
111
112 To test functions of type `(Eq a) => String -> a`, you should be able to
113 give give multiple input strings in a set; if the function does not map
114 them all to the same value, that's a test failure.
115
116 Syntax for an equivalency test might look like this:
117
118 | 2+2
119 ==
120 | 3+1
121 ==
122 | 7-3
123
124
125 py-falderal
126 -----------
127
128 ### Test report accumulation
129
130 Multiple runs of `falderal` ought to be able to accumulate their results
131 to a temporary file. No report should be generated if this is selected.
132 At the end, the report can be generated from the file.
133
134 rm -f FILE
135 falderal --record-to FILE tests1.markdown
136 falderal --record-to FILE tests2.markdown
137 falderal --report-from FILE
138
139 ### Support 'weak' testing
140
141 In which we only care about whether the command succeeded or failed.
142 In practice, this could be useful for testing the parser (just test
143 if these forms parse.) Or, if not this, then think of something that
144 would make just testing parsers more useful.
145
146 ### Split InterveningMarkdown blocks to make nice test descriptions
147
148 For example, if we have
149
150 ...test #1...
151 Some text.
152 Heading
153 -------
154 More text
155 ...test #2...
156
157 The description for test #2 should consist of "More text"; possibly also
158 the heading, but not "Some text". This can take place in a pre-processing
159 phase which simply splits every `InterveningMarkdown` block into multiple
160 blocks, at its headers. It should understand both underlined and atx-style
161 headers.
162
163 ### option to colourize test result output
164
165 Using one of the approaches listed here:
166
167 http://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python
168
169 ...py-falderal ought to provide an option (not default, of course, and not
170 if stdout is not a tty) to colorize the output with, of course, pass=green,
171 fail=red.
172
173 But, you'd often want to pipe the output to `less`, which by default makes
174 control characters visible, defeating colourization. But there is a flag
175 to less, `less -R`, which retains colourization. So use that.
176
177 ### Flag invalid sequences of lines as errors
178
179 2011-08-05
180
181 Currently, in `convertLinesToBlocks`, some invalid sequences of lines are
182 ignored. They should be flagged as errors in the test suite file.
183
184 (This was written against Test.Falderal but similar considerations could
185 be made for py-falderal.)
+1
-1
bin/falderal less more
0 #!/usr/bin/env python
0 #!/usr/bin/env python3
11
22 from os.path import realpath, dirname, join
33 import sys
+361
-0
doc/Falderal Literate Test Format.md less more
0 Falderal Literate Test Format
1 =============================
2
3 This document describes the proposed Falderal Literate Test Format.
4
5 Status
6 ------
7
8 This document is a *draft*. It is nominally version 0.14. It should
9 not be considered to be anything except a draft until it is described
10 as "version 1.0".
11
12 Overview
13 --------
14
15 A Falderal Literate Test Suite is a plain text file (or a set of such
16 files) where some of the lines have a special meaning in the Falderal Literate
17 Test Format. Certain groupings of the lines defined by this format are
18 intended to denote _tests_, while others denote _pragmas_ which specify
19 how those tests should be run. Outside of these groupings, lines have
20 no speical meaning in the Falderal Literate Test Format, other than that they are
21 presumed to contain descriptive text -- more-or-less equivalent to
22 _comments_ in a programming language source code file.
23
24 The plain text file may also be formatted in some other format, such as
25 Markdown. Although this is not required, the format of the lines which
26 do have a special meaning in the Falderal Literate Test Format, has been designed
27 to be compatible with Markdown.
28
29 A tool which claims to understand the Falderal Literate Test Format may
30 choose to extract the tests from such a document, run them, and report which
31 and how many of them passed or failed. That would be a typical application
32 for this format. However, no tool is strictly required to do this with the
33 tests found in a document; a tool may, for example, simply reformat the
34 tests to produce an output file in a different format.
35
36 In the syntax templates below, words in `/slashes/` indicate a variable
37 rather than literal text.
38
39 Syntax
40 ------
41
42 Every grouping of lines that has a special meaning to the Falderal Literate
43 Test Format begins with an indent of four (4) spaces from the
44 leftmost column, is preceded by non-indented text, and is followed by either
45 non-indented text or the end of the file. Such a grouping of lines is
46 called a _block_.
47
48 There are two general formats that any block can be formatted in. In the first,
49 so-called "verbose" format,
50 each indent of 4 spaces is followed immediately on that line by a distinguished
51 sequence of characters, called an _introducer_. The introducers which have
52 meaning to the Falderal Literate Test Format are as follows:
53
54 * `->` (hyphen, greater-than sign): pragma
55 * `| ` (vertical bar, space): test body text
56 * `+ ` (plus sign, space): test input text
57 * `= ` (equals sign, space): expected output text
58 * `? ` (question mark, space): expected error text
59
60 If the same introducer occurs on multiple adjacent lines, the text after
61 each introducer is concatenated to form one coalesced chunk of relevant text.
62 This allows, for example, multi-line text to be given as the body, the input,
63 or the expected output of a test.
64
65 There are some restrictions on the order in which introducers can sensibly
66 occur in a block:
67
68 * Test body text should occur at the start of a block.
69 * Test body test may be optionally followed by test input text.
70 * The first test input text must be immediately preceded by test body text.
71 * Subsequent test input texts need not be preceded by a test body text;
72 in this case, the previously-appearing test body text will be used again.
73 * Test body text must be followed by either test input text, expected
74 output text, or expected error text, with no intervening text.
75 * Either expected output or error text must follow either test body
76 text or test input text, with no intervening text.
77
78 See the sections for these introducers, below, for more details.
79
80 In the other, so-called "freestyle" format, not all lines in a block require
81 introducers. A freestyle format block is indentified as such if one or
82 more of the final lines of the block begin with any of the following
83 introducers:
84
85 * `=> `: expected output text
86 * `==> `: expected output text
87 * `===> `: expected output text
88 * `?> `: expected error text
89 * `??> `: expected error text
90 * `???> `: expected error text
91
92 In addition, the following introducers may be used to mark a section
93 of test input text on the first of the final lines (but may not be
94 used to end a block):
95
96 * `<= `: test input text
97 * `<== `: test input text
98 * `<=== `: test input text
99
100 If a block is identified as a freestyle block, all lines preceding the
101 first final line appearing with one of these introducers, are interpreted
102 as having no introducer at all (even if they begin with `| ` or some other
103 sequence already mentioned) and are used as the test body block.
104
105 Lines without introducers are called _intervening text_.
106 Lines of intervening text are classified as either blank or non-blank. A
107 line is blank if it contains no characters, or if it contains only whitespace.
108 A group of non-blank lines is referred to as a _paragraph_.
109
110 Pragmas
111 -------
112
113 ### encoding ###
114
115 The encoding pragma allows a Falderal file to specify what encoding is
116 used for the characters in it. An implementation of Falderal is not
117 expected to be able to handle any coding other than UTF-8, however,
118 this pragma is included for the benefit of text editors and other tools,
119 to indicate that the document is in fact in UTF-8 encoding.
120
121 Example:
122
123 -> encoding: UTF-8
124
125 ### Functionality-definition ###
126
127 The Functionality-definition pragma allows a Falderal file to define a
128 way in which a functionality being tested is implemented. It has the
129 following syntax:
130
131 -> Functionality /functionality-name/ is implemented by /implementation-type/ /implementation-specifier/
132
133 _implementation-type_ must at present be `shell command`. The format of the
134 _implementation-specifier_ differs according to the _implementation-type_.
135 The _functionality-name_ is arbitrary text enclosed within double quotes,
136 which may be referenced in a later Tests-for pragma.
137
138 Note that the Functionality-definitions given in a Falderal file should
139 not be considered exhaustive, or even requisite, by a tool. The tool may
140 accept additional definitions of a functionality, referencing it by its
141 name, from an external source such as the command line or a configuration
142 file, and may be instructed to ignore certain Functionality-definitions in
143 a Falderal file (if, for example, certain implementation are not currently
144 available or of interest to the user.) Indeed, the functionality referred
145 to by a _functionality-name_ in a Tests-for pragma need not be defined by any
146 Functionality-definition pragma in the same Falderal file, and this
147 situation requires the functionality to be specified to the tool in some
148 other manner, such as the tool looking for the functionality in some other
149 file provided to it.
150
151 #### Shell commands ####
152
153 For shell commands, the _implementation-specifier_ is in the format
154 `"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may
155 be used, so pipes, redirection, etc., are supported. Note that the double
156 quotation mark characters used to enclosed the command have meaning only to
157 the Falderal format — they are not part of the command, are not passed to the
158 shell, and do not require double quotation mark characters that are enclosed
159 by them to be escaped with a backslash.
160
161 Certain subsequences, called _variables_, if present in the command string,
162 will be expanded before execution, and will alter how the command reads the
163 text of the test and produces its output, to be compared with the expected
164 output.
165
166 When a variable is expanded, it is Falderal's responsibility to escape any
167 shell metacharacters that occur as part of the substitution text so that
168 it appears to the shell as a single token, and cannot otherwise affect the
169 meaning of the instantiated shell command. It is not necessary for a
170 variable to be enclosed in any kind of quotes in the command line template.
171 It is also not possible for the substitution text to e.g. expand to two
172 arguments, or to form a pipe or redirection specifier, etc.
173
174 The exact method by which the Falderal implementation escapes or quotes
175 the substitution text so that it appears unambiguously as a single argument
176 is left up to the implementation.
177
178 ##### `%(test-body-file)` #####
179
180 The variable `%(test-body-file)` will be replaced by the name of a file which
181 contains the text of the test body. This may be a temporary file created
182 solely for this purpose by the Falderal implementation.
183
184 ##### `%(test-body-text)` #####
185
186 The variable `%(test-body-text)` will be replaced by the actual text of the
187 test body.
188
189 ##### `%(test-input-file)` #####
190
191 The variable `%(test-input-file)` will be replaced by the name of a file which
192 contains the text of the test input. This may be a temporary file created
193 solely for this purpose by the Falderal implementation.
194
195 ##### `%(test-input-text)` #####
196
197 The variable `%(test-input-text)` will be replaced by the actual text of the
198 test input.
199
200 If neither of the variables `%(test-body-file)` nor `%(test-body-text)` appear
201 in the command string, the test body text will be provided on the standard
202 input of the shell command.
203
204 If neither of the variables `%(test-input-file)` nor `%(test-input-text)` appear
205 in the command string, the test input text will be provided on the standard
206 input of the shell command.
207
208 If both the test body text and the test input text are slated to appear on the
209 input of the shell command, then the behaviour is (presently) undefined.
210
211 ##### `%(output-file)` #####
212
213 The variable `%(output-file)` will be replaced by the name of a file
214 (temporary file) to which the test results will be written. If it does
215 not appear in the command string, the output text will be read from
216 the standard output of the command.
217
218 How shell commands support error output is not yet standardized.
219
220 For example:
221
222 -> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %(test-file) > %(output-file)"
223
224 ### Conditional-functionality-definition ###
225
226 The Conditional-functionality-definition pragma is an extended version
227 of the Functionality-definition pragma, in which the implementation is
228 only associated with functionality under certain conditions. It has the
229 following syntax (shown here across multiple lines for convenience):
230
231 -> Functionality /functionality-name/ is implemented by /implementation-type/ /implementation-specifier/
232 -> but only if /implementation-type/ /implementation-specifier/ succeeds
233
234 When a tool processes this pragma, it should first try executing the
235 second implementation-specifier, in the manner required by the second
236 implementation-type, without any particular input, and ascertain whether
237 it succeeds or fails. If it succeeds, the first implementation-type and
238 implementation-specifier should be registered as a definition of the
239 functionality-name, in the same manner as the Functionality-definition
240 pragma. If it does not succeed, this pragma should not change any
241 any functionality definitions, and should be treated as a no-op.
242
243 ### Tests-for ###
244
245 The Tests-for pragma determines what functionality will be used to run all
246 following tests, until the next Tests-for pragma. It has the following
247 syntax:
248
249 -> Tests for /functionality-name/
250
251 The _functionality-name_ refers to a functionality, which may be specified
252 by a Functionality-definition pragma elsewhere in the Falderal file.
253
254 For example:
255
256 -> Tests for 'Reversing a string'
257
258 Test Body, Test Input and Expected Text
259 ---------------------------------------
260
261 Each section of test body text may or may not be followed by a section of
262 test input text; either way it must then be followed immediately by either
263 and expected output section or expected error section.
264
265 Valid examples in the "verbose" format:
266
267 | thing to test
268 = output to expect
269
270 | thing to test
271 ? error to expect
272
273 | thing to test
274 + input to give it
275 = output to expect
276
277 | thing to test
278 + input to give it
279 ? error to expect
280
281 + different input to give the immediately previously defined test body
282 ? different error to expect
283
284 Valid examples in the "freestyle" format:
285
286 thing to test
287 => output to expect
288
289 thing to test
290 ==> output to expect
291
292 thing to test
293 ===> output to expect
294
295 thing to test
296 ?> error to expect
297
298 thing to test
299 ??> error to expect
300
301 thing to test
302 ???> error to expect
303
304 thing to test
305 <=== input to give it
306 ===> output to expect
307
308 thing to test
309 <=== input to give it
310 ???> error to expect
311
312 Invalid examples:
313
314 | thing to test
315
316 ...needs an expectation.
317
318 + input to give it
319 = output to expect
320
321 ...test input must be preceded by a test body, if this is the first test.
322
323 <=== input to give it
324 ???> output to expect
325
326 ...test input must be preceded by a test body always, in freestyle format.
327
328 ? error to expect
329
330 ...expectation must be preceded by either test input or test body.
331
332 A test body section may also be preceded by a paragraph of text; the
333 intent of the Falderal Literate Test Format is that this text should
334 describe the test, or rather, the aspect of the behaviour of the system
335 that the test is meant to check. It is therefore reasonable that this
336 text should be displayed along with the contents (test body text and
337 expected output or error) of the test, in, for example, a test result
338 report.
339
340 Discussion
341 ==========
342
343 (This section is non-normative.)
344
345 Typically, a file in Falderal Literate Test Format will also be in
346 Markdown format.
347
348 The format of the lines which comprise the Falderal Literate Test Format
349 was chosen to not conflict with many other common text formats (including
350 but not limited to Bird-style Literate Haskell, and Markdown); thus
351 literate test suites may be embedded in a wide variety of other formats.
352 However, there are inevitably some conflicts with some textual formats;
353 for example, when embedded in C code and many other languages, Falderal
354 entries should be preceded by `/*` and followed by `*/`, to ensure that
355 they are regarded as comments. Also, reStructuredText uses the `|` line
356 prefix to denote preformatted plain text.
357
358 The format of pragmas was chosen such that they could be read literately,
359 and as such, a formatting tool could format them in the output document
360 with little if any change.
+0
-340
doc/Falderal_Literate_Test_Format.markdown less more
0 Falderal Literate Test Format
1 =============================
2
3 This document describes the proposed Falderal Literate Test Format.
4
5 Status
6 ------
7
8 This document is a *draft*. It is nominally "version 0.12" because it
9 describes something that version 0.12 of `py-falderal` mostly implements.
10 We will deign to note which sections of this document the current released
11 version of `py-falderal` implements, and which it does not. However,
12 this document is a work in progress, subject to change, and subject to get
13 out of sync with `py-falderal`. It should not be considered to be
14 anything except a draft until it is described as "version 1.0".
15
16 Overview
17 --------
18
19 A Falderal Literate Test Suite is a plain text file where some of the lines
20 have special meaning to the Falderal Literate Test Format. Certain
21 groupings of the lines defined by this format are intended to denote tests,
22 while others denote pragmas which specify how those tests should be run.
23 Outside of these groupings, lines have no meaning to the Falderal Literate
24 Test Format other than that they are presumed to be mainly descriptive text.
25
26 The plain text file may also be formatted in some other format, such as
27 Markdown, although this is not required.
28
29 A tool which claims to understand the Falderal Literate Test Format may
30 choose to extract the tests from such a document, run them, and report which
31 and how many of them passed or failed. That would be a typical application
32 for this format. However, no tool is strictly required to do this with the
33 tests found in a document; a tool may, for example, simply reformat the
34 tests to produce an output file in a different format.
35
36 In the syntax templates below, words in `/slashes/` indicate a variable
37 rather than literal text.
38
39 Syntax
40 ------
41
42 Each grouping of lines which has special meaning to the Falderal Literate
43 Test Format always begins with an indent of four (4) spaces from the
44 leftmost column, preceded by non-indented text, and followed by either
45 non-indented text or the end of the file. Such a grouping of lines is
46 called a _block_.
47
48 There are two general formats to any block. In the first, "verbose" format,
49 each indent of 4 spaces is followed immediately on that line by distinguished
50 sequence of characters, called an _introducer_. The introducers which have
51 meaning to the Falderal Literate Test Format are as follows:
52
53 * `->` (hyphen, greater-than sign): pragma
54 * `| ` (vertical bar, space): test body text
55 * `+ ` (plus sign, space): test input text
56 * `= ` (equals sign, space): expected output text
57 * `? ` (question mark, space): expected error text
58
59 If the same introducer occurs on multiple adjacent lines, the text after
60 each introducer is concatenated to form one chunk of relevant text. This
61 allows, for example, multi-line text to be given as the body, the input,
62 or the expected output of a test.
63
64 There are some restrictions on the order in which introducers can sensibly
65 occur in a block:
66
67 * Test body text should occur at the start of a block.
68 * Test body test may be optionally followed by test input text.
69 * The first test input text must be immediately preceded by test body text.
70 * Subsequent test input texts need not be preceded by a test body text;
71 in this case, the previously-appearing test body text will be used again.
72 * Test body text must be followed by either test input text, expected
73 output text, or expected error text, with no intervening text.
74 * Either expected output or error text must follow either test body
75 text or test input text, with no intervening text.
76
77 See the sections for these introducers, below, for more details.
78
79 In the other, "freestyle" format, not all lines in a block require
80 introducers. A freestyle format block is indentified as such if one or
81 more of the final lines of the block begin with any of the following
82 introducers:
83
84 * `=> `: expected output text
85 * `==> `: expected output text
86 * `===> `: expected output text
87 * `?> `: expected error text
88 * `??> `: expected error text
89 * `???> `: expected error text
90
91 In addition, the following introducers may be used to mark a section
92 of test input text on the first of the final lines (but may not be
93 used to end a block):
94
95 * `<= `: test input text
96 * `<== `: test input text
97 * `<=== `: test input text
98
99 If a block is identified as a freestyle block, all lines preceding the
100 first final line appearing with one of these introducers, are interpreted
101 as having no introducer at all (even if they begin with `| ` or some other
102 sequence already mentioned) and are used as the test body block.
103
104 Lines without introducers are called _intervening text_.
105 Lines of intervening text are classified as either blank or non-blank. A
106 line is blank if it contains no characters, or if it contains only whitespace.
107 A group of non-blank lines is referred to as a _paragraph_.
108
109 Pragmas
110 -------
111
112 ### encoding ###
113
114 The encoding pragma allows a Falderal file to specify what encoding is
115 used for the characters in it. An implementation of Falderal is not
116 expected to be able to handle any coding other than UTF-8, however,
117 this pragma is included for the benefit of text editors and other tools,
118 to indicate that the document is in fact in UTF-8 encoding.
119
120 Example:
121
122 -> encoding: UTF-8
123
124 ### Functionality-definition ###
125
126 The Functionality-definition pragma allows a Falderal file to describe
127 ways in which a functionality being tested is implemented. It has the
128 following syntax:
129
130 -> Functionality /functionality-name/ is implemented by /functionality-type/ /functionality-specifier/
131
132 _functionality-type_ must at present be `shell command`. The format of the
133 _functionality-specifier_ differs according to the functionality-type.
134 The _functionality-name_ is arbitrary text enclosed within double quotes,
135 which may be referenced in a later Tests-for pragma.
136
137 Note that the Functionality-definitions given in a Falderal file should
138 not be considered exhaustive, or even requisite, by a tool. The tool may
139 accept additional definitions of a functionality, referencing it by its
140 name, from an external source such as the command line or a configuration
141 file, and may be instructed to ignore certain Functionality-definitions in
142 a Falderal file (if, for example, certain implementation are not currently
143 available or of interest to the user.) Indeed, the functionality referred
144 to by a _functionality-name_ in a Tests-for pragma need not be defined by any
145 Functionality-definition pragma in the same Falderal file, and this
146 situation requires the functionality to be specified to the tool in some
147 other manner.
148
149 #### Shell commands ####
150
151 For shell commands, the _functionality-specifier_ is in the format
152 `"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may
153 be used, so pipes, redirection, etc., are supported. Note that the double
154 quotation mark characters used to enclosed the command have meaning only to
155 the Falderal format — they are not part of the command, are not passed to the
156 shell, and do not require double quotation mark characters that are enclosed
157 by them to be escaped with a backslash.
158
159 Certain subsequences, called _variables_, if present in the command string,
160 will be expanded before execution, and will alter how the command reads the
161 text of the test and produces its output, to be compared with the expected
162 output.
163
164 When a variable is expanded, it is Falderal's responsibility to escape any
165 shell metacharacters that occur as part of the substitution text so that
166 it appears to the shell as a single token, and cannot otherwise affect the
167 meaning of the instantiated shell command. It is not necessary for a
168 variable to be enclosed in any kind of quotes in the command line template.
169 It is also not possible for the substitution text to e.g. expand to two
170 arguments, or to form a pipe or redirection specifier, etc.
171
172 The exact method by which the Falderal implementation escapes or quotes
173 the substitution text so that it appears unambiguously as a single argument
174 is left up to the implementation.
175
176 ##### `%(test-body-file)` #####
177
178 The variable `%(test-body-file)` will be replaced by the name of a file which
179 contains the text of the test body. This may be a temporary file created
180 solely for this purpose by the Falderal implementation.
181
182 ##### `%(test-body-text)` #####
183
184 The variable `%(test-body-text)` will be replaced by the actual text of the
185 test body.
186
187 ##### `%(test-input-file)` #####
188
189 The variable `%(test-input-file)` will be replaced by the name of a file which
190 contains the text of the test input. This may be a temporary file created
191 solely for this purpose by the Falderal implementation.
192
193 ##### `%(test-input-text)` #####
194
195 The variable `%(test-input-text)` will be replaced by the actual text of the
196 test input.
197
198 If neither of the variables `%(test-body-file)` nor `%(test-body-text)` appear
199 in the command string, the test body text will be provided on the standard
200 input of the shell command.
201
202 If neither of the variables `%(test-input-file)` nor `%(test-input-text)` appear
203 in the command string, the test input text will be provided on the standard
204 input of the shell command.
205
206 If both the test body text and the test input text are slated to appear on the
207 input of the shell command, then the behaviour is (presently) undefined.
208
209 ##### `%(output-file)` #####
210
211 The variable `%(output-file)` will be replaced by the name of a file
212 (temporary file) to which the test results will be written. If it does
213 not appear in the command string, the output text will be read from
214 the standard output of the command.
215
216 How shell commands support error output is not yet standardized.
217
218 For example:
219
220 -> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %(test-file) > %(output-file)"
221
222 ### Tests-for ###
223
224 The Tests-for pragma determines what functionality will be used to run all
225 following tests, until the next Tests-for pragma. It has the following
226 syntax:
227
228 -> Tests for /functionality-name/
229
230 The _functionality-name_ refers to a functionality, which may be specified
231 by a Functionality-definition pragma elsewhere in the Falderal file.
232
233 For example:
234
235 -> Tests for 'Reversing a string'
236
237 Test Body, Test Input and Expected Text
238 ---------------------------------------
239
240 Each section of test body text may or may not be followed by a section of
241 test input text; either way it must then be followed immediately by either
242 and expected output section or expected error section.
243
244 Valid examples in the "verbose" format:
245
246 | thing to test
247 = output to expect
248
249 | thing to test
250 ? error to expect
251
252 | thing to test
253 + input to give it
254 = output to expect
255
256 | thing to test
257 + input to give it
258 ? error to expect
259
260 + different input to give the immediately previously defined test body
261 ? different error to expect
262
263 Valid examples in the "freestyle" format:
264
265 thing to test
266 => output to expect
267
268 thing to test
269 ==> output to expect
270
271 thing to test
272 ===> output to expect
273
274 thing to test
275 ?> error to expect
276
277 thing to test
278 ??> error to expect
279
280 thing to test
281 ???> error to expect
282
283 thing to test
284 <=== input to give it
285 ===> output to expect
286
287 thing to test
288 <=== input to give it
289 ???> error to expect
290
291 Invalid examples:
292
293 | thing to test
294
295 ...needs an expectation.
296
297 + input to give it
298 = output to expect
299
300 ...test input must be preceded by a test body, if this is the first test.
301
302 <=== input to give it
303 ???> output to expect
304
305 ...test input must be preceded by a test body always, in freestyle format.
306
307 ? error to expect
308
309 ...expectation must be preceded by either test input or test body.
310
311 A test body section may also be preceded by a paragraph of text; the
312 intent of the Falderal Literate Test Format is that this text should
313 describe the test, or rather, the aspect of the behaviour of the system
314 that the test is meant to check. It is therefore reasonable that this
315 text should be displayed along with the contents (test body text and
316 expected output or error) of the test, in, for example, a test result
317 report.
318
319 Discussion
320 ==========
321
322 (This section is non-normative.)
323
324 Typically, a file in Falderal Literate Test Format will also be in
325 Markdown format.
326
327 The format of the lines which comprise the Falderal Literate Test Format
328 was chosen to not conflict with many other common text formats (including
329 but not limited to Bird-style Literate Haskell, and Markdown); thus
330 literate test suites may be embedded in a wide variety of other formats.
331 However, there are inevitably some conflicts with some textual formats;
332 for example, when embedded in C code and many other languages, Falderal
333 entries should be preceded by `/*` and followed by `*/`, to ensure that
334 they are regarded as comments. Also, reStructuredText uses the `|` line
335 prefix to denote preformatted plain text.
336
337 The format of pragmas was chosen such that they could be read literately,
338 and as such, a formatting tool could format them in the output document
339 with little if any change.
+13
-0
doc/LICENSE less more
0 Falderal's documentation is covered by the following BSD-compatible license,
1 modelled after the "Report on the Programming Language Haskell 98" license:
2
3 Copyright (c)2011-2022 Chris Pressey, Cat's Eye Technologies.
4 All rights reserved.
5
6 The authors intend this Report to belong to the entire Falderal
7 community, and so we grant permission to copy and distribute it for
8 any purpose, provided that it is reproduced in its entirety,
9 including this Notice. Modified versions of this Report may also be
10 copied and distributed for any purpose, provided that the modified
11 version is clearly presented as such, and that it does not claim to
12 be a definition of the Falderal Literate Test Format.
+33
-0
doc/falderal.md less more
0 `falderal`
1 =============
2
3 `falderal` is an implementation of Falderal in Python 3.x. It should
4 also function under Python 2.7.x, but as 2.7.x is a legacy version of Python,
5 such support should not be relied upon.
6
7 `falderal` implements the
8 [Falderal Literate Test Format](Falderal%20Literate%20Test%20Format.md) as
9 a runner of the tests embedded in a Falderal Test Suite.
10
11 `falderal` accepts multiple input files. The Functionality-definition
12 pragmas in these files apply to all the Tests-for pragmas in all of the
13 files. This permits a somewhat modular usage, where the inclusion (or
14 omission) of a file from the list of files on the command line, can
15 determine what definitions of a functionality will be tested (or not tested).
16
17 ### History
18
19 Historically, the first implementation of the Falderal Literate Test Format
20 was in Haskell. It followed an early version of the Falderal Literate Test Format
21 which looked somewhat different than it does today (at version 0.14).
22
23 Subsequently, an implementation in Python was written, and called `py-falderal`.
24
25 This document described some of the differences between `py-falderal` and that
26 original implementation. But since much of that information is merely of
27 historical interest, it has been stripped from this document. It is still
28 available in version control history and/or archived distfiles of previous
29 versions of the Falderal distribution.
30
31 In addition, the `py-falderal` tool is now generally referred to as simply
32 `falderal`.
+0
-88
doc/py-falderal.markdown less more
0 `py-falderal`
1 =============
2
3 `py-falderal` is an implementation of Falderal in Python 2.5.x.
4
5 Motivation
6 ----------
7
8 The original reference implementation of Falderal was written in Haskell.
9 There are a few reasons I had for re-implementing it in Python:
10
11 * The original Falderal implementation grew out of a Haskell-specific hack,
12 and it showed in how it was written.
13
14 * Fewer discrepancies between platforms. In particular, `ghc` for Windows
15 likes to operate in terms of MS-DOS end-of-lines (`CR`, `LF`), but I tend
16 to use it under Cygwin using Unix end-of-lines (`LF`).
17
18 * Smaller install burden: Python sometimes comes bundled with the operating
19 system; Haskell rarely does.
20
21 * Haskell, being lazy, makes it harder to deal with exceptions; unless the
22 Haskell expression is being evaluated both strictly and deeply, exceptions
23 can slip by. Haskell's facilities for forcing an expression to be
24 evaluated strictly and deeply are rather hacky (last I checked, the most
25 portable way to do so was to use `show`.) For Falderal's purposes, this
26 limitation seems artificial, at best.
27
28 * Relatedly, Python (or CPython, anyway) has better error behavior than
29 Haskell (or `ghc`, anyway); when it crashes, it dumps a backtrace (which I
30 can then analyze), instead of just saying something pithy like `Prelude:
31 empty list` (which I can't.)
32
33 Features
34 --------
35
36 `py-falderal` implements a slightly different subset of the
37 [Falderal Literate Test Format](Falderal_Literate_Test_Format.markdown) than
38 the Haskell implementation did.
39
40 In particular,
41
42 * It mainly implements `shell command` implementations. In practice, partly
43 due to the "strict & deep" evaluation problem mentioned above, that's how
44 I've been using Falderal anyway; also, its approach makes it somewhat more
45 useful for "end-to-end" testing of compilers and interpreters, than for
46 unit-testing individual text-processing functions inside a program.
47 (Technically, it implements `Haskell function` implementations too, but it's
48 something of a hack that uses `ghc -e`.)
49
50 * I plan for it to *only* understand indented Falderal blocks. The idea is
51 that the Falderal tests will almost certainly be embedded in a Markdown
52 document (possibly with Bird-style literate code also embedded therein,)
53 and no extra processing should be required to format something readable
54 from that Markdown. The four-space-indented Falderal blocks come out as
55 preformatted blocks, which is quite good enough. Relatedly,
56
57 * It does no formatting. There are a set of classes for parsing Falderal
58 files (`Document`s) into blocks (`Block`s), and methods for formatting
59 blocks into text. But aside from reporting the results of a test run,
60 they're not used by the utility to format Falderal files. If you want to
61 do extra processing on your Falderal/Markdown file so that Falderal
62 blocks are prettier, you certainly can do that, but you'll need to write
63 your own script which uses these classes, for it is outside the scope of
64 `py-falderal`.
65
66 * I'm not so sure about `-> Functionality "blah" is implemented by shell
67 command "flargh"` anymore. I mean, it should certainly be the case that
68 functionalities can have multiple implementations, but
69
70 * Perhaps implementations should not be specified in Falderal documents
71 at all — that approach is more abstract. But it requires them to be
72 specified on the tool's command line, which in practice requires there
73 to be a driver script to run the tests, for a particular implementation;
74 but this is not necessarily a bad thing. (The effective practice has
75 been, actually, to write out a bit of "configuration Markdown" with
76 that "Functionality is implemented by" snippet in it, and to load that
77 before loading the main tests. Rather than to configure the functionality
78 implementation(s) on the command line itself.)
79 * When `falderal` is run on more than one input file, what is the scope
80 of a functionality, and what is the scope of the implementations of a
81 functionality? Currently, in the Falderal semantics, that scope is
82 global, and maybe that is appropriate; but maybe finer-grained control
83 would be nice.
84
85 * `py-falderal` also does not try to optimize the test runs into a block of
86 test runs (which didn't work out so well in the Haskell implementation,
87 for `shell command`s, anyway.)
+25
-0
pyproject.toml less more
0 [build-system]
1 requires = ["setuptools>=61.0"]
2 build-backend = "setuptools.build_meta"
3
4 [project]
5 name = "Falderal"
6 version = "0.14"
7 authors = [
8 { name="Chris Pressey", email="packages@catseye.tc" },
9 ]
10 urls = { website="https://catseye.tc/node/Falderal" }
11 description = "Falderal: literate testing for programming languages"
12 readme = "README.md"
13 license = { file="LICENSE" }
14 classifiers = [
15 "License :: OSI Approved :: BSD License",
16 "Operating System :: OS Independent",
17 "Programming Language :: Python :: 2.7",
18 "Programming Language :: Python :: 3",
19 "Topic :: Software Development :: Testing",
20 ]
21 requires-python = ">=2.7, >=3.5"
22
23 [project.scripts]
24 falderal = "falderal.driver:sys_main"
+1
-1
script/fa-bin-to-hex less more
0 #!/usr/bin/env python
0 #!/usr/bin/env python3
11
22 # converts raw bytes on input to hex couples on input.
33
+1
-1
script/fa-hex-to-bin less more
0 #!/usr/bin/env python
0 #!/usr/bin/env python3
11
22 # converts hex couples on input to raw bytes on output.
33
+1
-1
script/fa-under-pty less more
0 #!/usr/bin/env python
0 #!/usr/bin/env python3
11
22 import pty
33 import sys
+0
-28
setup.py less more
0 import setuptools
1
2
3 with open("README.markdown", "r") as fh:
4 long_description = fh.read()
5
6 setuptools.setup(
7 name='Falderal',
8 version='0.13',
9 description='Definition of, and tools for using, the Falderal literate testing format',
10 long_description=long_description,
11 long_description_content_type="text/markdown",
12 author='Chris Pressey',
13 author_email='packages@catseye.tc',
14 url='https://catseye.tc/node/Falderal',
15 packages=['falderal'],
16 package_dir={'': 'src'},
17 scripts=['bin/falderal'],
18 classifiers=[
19 "Development Status :: 4 - Beta",
20 "Intended Audience :: Developers",
21 "License :: OSI Approved :: BSD License",
22 "Operating System :: OS Independent",
23 "Programming Language :: Python :: 2.7",
24 "Programming Language :: Python :: 3",
25 "Topic :: Software Development :: Testing",
26 ],
27 )
+8
-3
src/falderal/driver.py less more
77 from falderal.objects import Document, FalderalSyntaxError
88
99
10 ##### Exceptions #####
10 # ###### -*- Exceptions -*- ###### #
1111
1212 class FalderalLintingError(ValueError):
1313 pass
1414
1515
16 ##### Main #####
16 # ###### -*-*- Main -*-*- ###### #
1717
1818 def main(args):
1919 argparser = ArgumentParser()
3131 argparser.add_argument("-v", "--verbose",
3232 action="store_true", default=False,
3333 help="print out info about each test as it is run")
34 argparser.add_argument('--version', action='version', version="%(prog)s 0.13")
34 argparser.add_argument('--version', action='version', version="%(prog)s 0.14")
3535
3636 options = argparser.parse_args(args[1:])
3737
5959 )
6060 for name in functionalities:
6161 functionality = functionalities[name]
62 functionality.filter_out_unavailable_implementations()
6263 if not functionality.implementations:
6364 raise FalderalLintingError(
6465 "No implementations were found for the functionality '%s'" %
131132 return 0
132133 else:
133134 return 1
135
136
137 def sys_main():
138 return main(sys.argv)
+34
-2
src/falderal/objects.py less more
3737 class Options(object):
3838 def __init__(self):
3939 self.substring_error = False
40 self.verbose = False
4041
4142
4243 DEFAULT_OPTIONS = Options()
5859 self.text = text
5960
6061 def __repr__(self):
61 return '%s(%r)' % (self.__class__.__name__, self.text)
62 reprtext = repr(self.text)
63 reprtext = 'u' + reprtext if not reprtext.startswith('u') else reprtext
64 return '%s(%s)' % (self.__class__.__name__, reprtext)
6265
6366 def __eq__(self, other):
6467 return self.__class__ == other.__class__ and self.text == other.text
357360 functionality_name,
358361 Functionality(functionality_name)
359362 )
363 match = re.match(r'^\s*Functionality\s*\"(.*?)\"\s*is\s+implemented\s+by\s+shell\s+command\s*\"(.*?)\"\s*but\s+only\s+if\s+shell\s+command\s*\"(.*?)\"\s*succeeds\s*$', pragma_text)
364 if match:
365 functionality_name = match.group(1)
366 command = match.group(2)
367 gating_command = match.group(3)
368 functionality = state.functionalities.setdefault(
369 functionality_name,
370 Functionality(functionality_name)
371 )
372 implementation = ShellImplementation(command, gating_command=gating_command)
373 functionality.add_implementation(implementation)
360374 match = re.match(r'^\s*Functionality\s*\"(.*?)\"\s*is\s+implemented\s+by\s+shell\s+command\s*\"(.*?)\"\s*$', pragma_text)
361375 if match:
362376 functionality_name = match.group(1)
494508
495509 ##### Functionalities and their Implementations #####
496510
511
497512 class Functionality(object):
498513 """An object representing a Falderal functionality.
499514
515530 def add_implementation(self, implementation):
516531 self.implementations.append(implementation)
517532
533 def filter_out_unavailable_implementations(self):
534 self.implementations = [i for i in self.implementations if i.is_available()]
535
518536
519537 class Implementation(object):
520538 """An object representing an implementation (something that is
523541 """
524542 def __init__(self):
525543 pass
544
545 def is_available(self):
546 return True
526547
527548 def run(self, body=None, input=None, verbose=False):
528549 """Returns the RunResult of running this implementation on the
556577
557578
558579 class ShellImplementation(Implementation):
559 def __init__(self, command):
580 def __init__(self, command, gating_command=None):
560581 self.command = command
582 self.gating_command = gating_command
561583
562584 def __repr__(self):
563585 return '%s(%r)' % (self.__class__.__name__, self.command)
567589
568590 def __eq__(self, other):
569591 return self.__class__ == other.__class__ and self.command == other.command
592
593 def is_available(self):
594 if not self.gating_command:
595 return True
596 pipe = Popen(
597 self.gating_command, shell=True,
598 stdin=PIPE, stdout=PIPE, stderr=PIPE
599 )
600 outputs = pipe.communicate()
601 return pipe.returncode == 0
570602
571603 def subst(self, command, var_name, value):
572604 """Replace all occurrences of `var_name` in `command` with
+47
-10
src/falderal/tests.py less more
0 # Note: these are unit tests for py-falderal itself,
0 # Note: these are unit tests for the `falderal` tool itself,
11 # not tests that Falderal can understand.
22
33 import unittest
77 Block, Pragma,
88 ParseState, InterveningText,
99 Document,
10 Functionality, ShellImplementation,
10 Functionality, ShellImplementation, CallableImplementation,
1111 Test, OutputOutcome, ErrorOutcome,
1212 FalderalSyntaxError,
1313 )
242242 d.append(u'')
243243 d.append(u' -> Functionality "Parse Stuff" is')
244244 d.append(u' -> implemented by shell command "pxxxy"')
245 tests = d.extract_tests(funs)
245 d.extract_tests(funs)
246246 self.assertEqual(list(funs.keys()), ['Parse Stuff'])
247247 self.assertEqual(
248248 [i for i in funs["Parse Stuff"].implementations],
249249 [ShellImplementation(u'parse'), ShellImplementation(u'pxxxy')]
250 )
251
252 def test_parse_functionalities_with_gated_implementations(self):
253 d = Document()
254 funs = {}
255 d.append(u' -> Functionality "Parse Stuff" is implemented by '
256 u'shell command "parse"')
257 d.append(u'')
258 d.append(u' -> Functionality "Parse Stuff" is')
259 d.append(u' -> implemented by shell command "pxxxy"')
260 d.append(u' -> but only if shell command "command -v pxxxy"')
261 d.append(u' -> succeeds')
262 d.extract_tests(funs)
263 self.assertEqual(list(funs.keys()), ['Parse Stuff'])
264 self.assertEqual(
265 funs["Parse Stuff"].implementations,
266 [ShellImplementation(u'parse'), ShellImplementation(u'pxxxy')]
267 )
268 implementations = funs["Parse Stuff"].implementations
269 self.assertEqual(implementations[0].gating_command, None)
270 self.assertEqual(implementations[1].gating_command, 'command -v pxxxy')
271 # TODO: mock is_available
272 funs["Parse Stuff"].filter_out_unavailable_implementations()
273 self.assertEqual(
274 funs["Parse Stuff"].implementations,
275 [ShellImplementation(u'parse')]
250276 )
251277
252278
281307 self.assertEqual(i.run(body=u'text', input=u'zzrk'), OutputOutcome(u'zzrk'))
282308
283309
284 def TestTestCase(TestCase):
310 class TestTestCase(TestCase):
285311 def test_test_contents(self):
286312 b = Block()
287313 b.append(u'foo')
297323 f.add_implementation(CallableImplementation(lambda x, y: x))
298324 t = Test(body=u'foo', expectation=OutputOutcome(u'foo'), functionality=f)
299325 self.assertEqual(
300 [r.short_description() for r in t.run()]
326 [r.short_description() for r in t.run()],
301327 ['success']
302328 )
303329
304330 def test_tests_2(self):
305331 f = Functionality('Cat File')
306332 f.add_implementation(CallableImplementation(lambda x, y: x))
307 t = Test(body=u'foo', expectation=OutputOutcome(u'bar'),
308 functionality=f)
333 t = Test(
334 body=u'foo', expectation=OutputOutcome(u'bar'), functionality=f
335 )
309336 self.assertEqual(
310337 [r.short_description() for r in t.run()],
311338 ["expected OutputOutcome(u'bar'), got OutputOutcome(u'foo')"]
323350
324351 def test_tests_4(self):
325352 f = Functionality('Cat File')
353
326354 def e(x, y):
327355 raise ValueError(x)
328356 f.add_implementation(CallableImplementation(e))
335363
336364 def test_tests_5(self):
337365 f = Functionality('Cat File')
366
338367 def e(x, y):
339368 raise ValueError(x)
340369 f.add_implementation(CallableImplementation(e))
347376
348377 def test_tests_6(self):
349378 f = Functionality('Cat File')
379
350380 def e(x, y):
351381 raise ValueError(x)
352382 f.add_implementation(CallableImplementation(e))
371401 # A functionality can have multiple implementations. We test them all.
372402
373403 f = Functionality('Cat File')
404
374405 def c1(body, input):
375406 return body
407
376408 def c2(body, input):
377409 return body + '...'
410
378411 def c3(body, input):
379412 raise ValueError(body)
413
380414 for c in (c1, c2, c3):
381415 f.add_implementation(CallableImplementation(c))
382416 t = Test(body=u'foo', expectation=OutputOutcome(u'foo'),
383417 functionality=f)
384418 self.assertEqual(
385419 [r.short_description() for r in t.run()],
386 ['success', "expected OutputOutcome(u'foo'), got OutputOutcome(u'foo...')",
387 "expected OutputOutcome(u'foo'), got ErrorOutcome(u'foo')"]
388 )
420 [
421 "success",
422 "expected OutputOutcome(u'foo'), got OutputOutcome(u'foo...')",
423 "expected OutputOutcome(u'foo'), got ErrorOutcome(u'foo')"
424 ]
425 )
389426
390427
391428 if __name__ == '__main__':
+74
-46
test.sh less more
00 #!/bin/sh
11
2 # Really crude test harness for py-falderal itself...
2 # Rather crude test harness for `falderal` itself.
33
4 if [ "x$PYTHON" = "x" ]; then
5 PYTHON="python3"
4 run_falderal_integration_tests() {
5 FALDERAL="$1"
6
7 FIRST_TESTS="
8 test-pass-fail test-no-functionality test-ill-formed test-no-test-body
9 test-var-subst test-no-eol
10 test-utf8 test-crlf
11 test-bad-indentation
12 test-input-sections test-shared-body
13 test-stdout-stderr test-err-no-stderr
14 test-freestyle-format
15 "
16 for TEST in ${FIRST_TESTS}; do
17 echo ${TEST}...
18 $FALDERAL --cavalier ${TEST}.markdown > ${TEST}.actual 2>&1
19 diff -u ${TEST}.expected ${TEST}.actual || exit 1
20 done
21
22 # tests for linting
23 LINTING_TESTS="test-no-tests"
24 for TEST in ${LINTING_TESTS}; do
25 echo ${TEST}...
26 $FALDERAL ${TEST}.markdown > ${TEST}.actual 2>&1
27 diff -u ${TEST}.expected ${TEST}.actual || exit 1
28 done
29
30 TWO_PART_TESTS="
31 test-no-functionality-leak test-implementations-global test-appliances
32 "
33 for TEST in ${TWO_PART_TESTS}; do
34 echo ${TEST}...
35 $FALDERAL ${TEST}-a.markdown ${TEST}-b.markdown > ${TEST}.actual 2>&1
36 diff -u ${TEST}.expected ${TEST}.actual || exit 1
37 done
38
39 # special tests: -b
40 TEST=test-substring-error
41 echo ${TEST}...
42 $FALDERAL -b ${TEST}.markdown > ${TEST}.actual 2>&1
43 diff -u ${TEST}.expected ${TEST}.actual || exit 1
44 }
45
46 run_falderal_tests() {
47 PYTHON="$1"
48 PYTHONPATH=src $PYTHON src/falderal/tests.py -v || exit 1
49 ( cd tests && run_falderal_integration_tests "$PYTHON ../bin/falderal" )
50 }
51
52 ### Main ###
53
54 if [ "x$PYTHON" != "x" ]; then
55 if command -v "$PYTHON" > /dev/null 2>&1; then
56 run_falderal_tests "$PYTHON" || exit 1
57 else
58 echo "$PYTHON not found on executable search path. Aborting."
59 exit 1
60 fi
61 else
62 MISSING=""
63 if command -v python2 > /dev/null 2>&1; then
64 run_falderal_tests "python2" || exit 1
65 else
66 MISSING="${MISSING}2"
67 fi
68 if command -v python3 > /dev/null 2>&1; then
69 run_falderal_tests "python3" || exit 1
70 else
71 MISSING="${MISSING}3"
72 fi
73 if [ "x${MISSING}" = "x23" ]; then
74 echo "Neither python2 nor python3 found on executable search path. Aborting."
75 exit 1
76 fi
677 fi
7 FALDERAL="$PYTHON ../bin/falderal"
8
9 PYTHONPATH=src $PYTHON src/falderal/tests.py -v || exit 1
10
11 cd tests
12
13 FIRST_TESTS="
14 test-pass-fail test-no-functionality test-ill-formed test-no-test-body
15 test-var-subst test-no-eol
16 test-utf8 test-crlf
17 test-bad-indentation
18 test-input-sections test-shared-body
19 test-stdout-stderr test-err-no-stderr
20 test-freestyle-format
21 "
22 for TEST in ${FIRST_TESTS}; do
23 echo ${TEST}...
24 $FALDERAL --cavalier ${TEST}.markdown > ${TEST}.actual 2>&1
25 diff -u ${TEST}.expected ${TEST}.actual || exit 1
26 done
27
28 # tests for linting
29 LINTING_TESTS="test-no-tests"
30 for TEST in ${LINTING_TESTS}; do
31 echo ${TEST}...
32 $FALDERAL ${TEST}.markdown > ${TEST}.actual 2>&1
33 diff -u ${TEST}.expected ${TEST}.actual || exit 1
34 done
35
36 TWO_PART_TESTS="
37 test-no-functionality-leak test-implementations-global test-appliances
38 "
39 for TEST in ${TWO_PART_TESTS}; do
40 echo ${TEST}...
41 $FALDERAL ${TEST}-a.markdown ${TEST}-b.markdown > ${TEST}.actual 2>&1
42 diff -u ${TEST}.expected ${TEST}.actual || exit 1
43 done
44
45 # special tests: -b
46 TEST=test-substring-error
47 echo ${TEST}...
48 $FALDERAL -b ${TEST}.markdown > ${TEST}.actual 2>&1
49 diff -u ${TEST}.expected ${TEST}.actual || exit 1
5078
5179 rm -f *.actual
5280 echo 'All tests passed.'
+2
-2
tests/silly-interpreter.py less more
55 for line in program:
66 if line.startswith('print '):
77 var = line[6]
8 print vars.get(var, '')
8 print(vars.get(var, ''))
99 if line.startswith('read '):
1010 var = line[5]
11 vars[var] = raw_input('')
11 vars[var] = input('')
+4
-4
tests/test-appliances-b.markdown less more
33 This is really just a more lopsided version of test 7.
44
55 -> Functionality "Cat" is implemented by
6 -> shell command "python cat.py"
6 -> shell command "python3 cat.py"
77
88 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py -f %(test-body-file)"
9 -> shell command "python3 cat.py -f %(test-body-file)"
1010
1111 -> Functionality "Cat" is implemented by
12 -> shell command "python cat.py -o %(output-file)"
12 -> shell command "python3 cat.py -o %(output-file)"
1313
1414 -> Functionality "Cat" is implemented by
15 -> shell command "python echo.py %(test-body-text)"
15 -> shell command "python3 echo.py %(test-body-text)"
+1
-1
tests/test-bad-indentation.markdown less more
00 Falderal Test: Bad indentation
11 ------------------------------
22
3 Since these blocks are not indented correctly, `py-falderal` should not
3 Since these blocks are not indented correctly, `falderal` should not
44 load tests from them. (Ideally it should produce warnings though.)
55
66 -> Functionality "Cat" is implemented by
+1
-1
tests/test-crlf.expected less more
33
44 Location: test-crlf.markdown, line 24
55 Function: CRLF
6 Impl : shell command "python crlf.py"
6 Impl : shell command "python3 crlf.py"
77 Body :
88 one
99 two
+1
-1
tests/test-crlf.markdown less more
55 of end-of-line convention (LF or CRLF).
66
77 -> Functionality "CRLF" is implemented by
8 -> shell command "python crlf.py"
8 -> shell command "python3 crlf.py"
99
1010 -> Tests for functionality "CRLF"
1111
+1
-1
tests/test-err-no-stderr.markdown less more
55 to be on `stdout`.
66
77 -> Functionality "Error on stdout" is implemented by shell command
8 -> "python fail.py %(test-body-text) '' 1"
8 -> "python3 fail.py %(test-body-text) '' 1"
99
1010 -> Tests for functionality "Error on stdout"
1111
+4
-4
tests/test-freestyle-format.expected less more
33
44 Location: test-freestyle-format.markdown, line 62
55 Function: Cat
6 Impl : shell command "python cat.py"
6 Impl : shell command "python3 cat.py"
77 Body : meow
88 Expected: error:
99 woof
1616
1717 Location: test-freestyle-format.markdown, line 65
1818 Function: Cat
19 Impl : shell command "python cat.py"
19 Impl : shell command "python3 cat.py"
2020 Body : meow
2121 Expected: error:
2222 woof
2929
3030 Location: test-freestyle-format.markdown, line 68
3131 Function: Cat
32 Impl : shell command "python cat.py"
32 Impl : shell command "python3 cat.py"
3333 Body :
3434 | meow
3535 ? rreoww
4848
4949 Location: test-freestyle-format.markdown, line 111
5050 Function: Silly Interpreter
51 Impl : shell command "python silly-interpreter.py %(test-body-file)"
51 Impl : shell command "python3 silly-interpreter.py %(test-body-file)"
5252 Body :
5353 Expected: output:
5454 zang
+2
-2
tests/test-freestyle-format.markdown less more
77 the final line(s) of the test begin with one of the prefixes
88 `=> ` or `==> ` or `===> ` or `?> ` or `??> `or `???> `.
99
10 -> Functionality "Cat" is implemented by shell command "python cat.py"
10 -> Functionality "Cat" is implemented by shell command "python3 cat.py"
1111
1212 -> Tests for functionality "Cat"
1313
7272 Freestyle-format tests can also contain input sections.
7373
7474 -> Functionality "Silly Interpreter" is implemented by
75 -> shell command "python silly-interpreter.py %(test-body-file)"
75 -> shell command "python3 silly-interpreter.py %(test-body-file)"
7676
7777 -> Tests for functionality "Silly Interpreter"
7878
+1
-1
tests/test-ill-formed.markdown less more
33 A Falderal document which is ill-formed.
44
55 -> Functionality "Cat" is implemented by
6 -> shell command "python cat.py"
6 -> shell command "python3 cat.py"
77
88 -> Tests for functionality "Cat"
99
+1
-1
tests/test-implementations-global-a.markdown less more
66 the same file.
77
88 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py"
9 -> shell command "python3 cat.py"
1010
1111 -> Tests for functionality "Cat"
1212
+1
-1
tests/test-implementations-global-b.markdown less more
66 the same file.
77
88 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py -f %(test-body-file)"
9 -> shell command "python3 cat.py -f %(test-body-file)"
1010
1111 -> Tests for functionality "Cat"
1212
+1
-1
tests/test-input-sections.markdown less more
33 Test tests that have test input sections.
44
55 -> Functionality "Silly Interpreter" is implemented by
6 -> shell command "python silly-interpreter.py %(test-body-file)"
6 -> shell command "python3 silly-interpreter.py %(test-body-file)"
77
88 -> Tests for functionality "Silly Interpreter"
99
+2
-2
tests/test-no-eol.markdown less more
44 the test output or not.
55
66 -> Functionality "Echo" is implemented by
7 -> shell command "python echo.py %(test-body-text)"
7 -> shell command "python3 echo.py %(test-body-text)"
88
99 -> Tests for functionality "Echo"
1010
1717 = hi
1818
1919 -> Functionality "Echo, no newline" is implemented by
20 -> shell command "python echo.py -n %(test-body-text)"
20 -> shell command "python3 echo.py -n %(test-body-text)"
2121
2222 -> Tests for functionality "Echo, no newline"
2323
+1
-1
tests/test-no-functionality-leak-a.markdown less more
66 file.
77
88 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py"
9 -> shell command "python3 cat.py"
1010
1111 -> Tests for functionality "Cat"
1212
+2
-2
tests/test-no-functionality.markdown less more
11 -------------------------------
22
33 Since no functionality is specified for these tests,
4 `py-falderal` should exit with an exception.
4 `falderal` should exit with an exception.
55
66 -> Functionality "Cat" is implemented by
7 -> shell command "python cat.py"
7 -> shell command "python3 cat.py"
88
99 Cat cats.
1010
+1
-1
tests/test-no-test-body.markdown less more
33 Another Falderal document which is ill-formed.
44
55 -> Functionality "Cat" is implemented by
6 -> shell command "python cat.py"
6 -> shell command "python3 cat.py"
77
88 -> Tests for functionality "Cat"
99
+2
-2
tests/test-no-tests.markdown less more
00 Falderal Test for No Tests Found
11 --------------------------------
22
3 Since these blocks are not indented correctly, `py-falderal` should not
3 Since these blocks are not indented correctly, `falderal` should not
44 load tests from them. (Ideally it should produce warnings though.)
55
66 -> Functionality "Cat" is implemented by
7 -> shell command "python cat.py"
7 -> shell command "python3 cat.py"
88
99 Cat cats.
1010
+2
-2
tests/test-pass-fail.expected less more
33
44 Location: test-pass-fail.markdown, line 28
55 Function: Cat
6 Impl : shell command "python cat.py"
6 Impl : shell command "python3 cat.py"
77 Body : meow
88 Expected: output:
99 woof
1616
1717 Location: test-pass-fail.markdown, line 31
1818 Function: Cat
19 Impl : shell command "python cat.py"
19 Impl : shell command "python3 cat.py"
2020 Body :
2121 purr
2222 prrr
+1
-1
tests/test-pass-fail.markdown less more
66 incorrect output.
77
88 -> Functionality "Cat" is implemented by
9 -> shell command "python cat.py"
9 -> shell command "python3 cat.py"
1010
1111 -> Tests for functionality "Cat"
1212
+1
-1
tests/test-shared-body.markdown less more
33 Test that tests with input sections can re-use the previous test's test body.
44
55 -> Functionality "Silly Interpreter" is implemented by
6 -> shell command "python silly-interpreter.py %(test-body-file)"
6 -> shell command "python3 silly-interpreter.py %(test-body-file)"
77
88 -> Tests for functionality "Silly Interpreter"
99
+4
-4
tests/test-stdout-stderr.markdown less more
44 expected text is matched against standard output.
55
66 -> Functionality "Succeed" is implemented by shell command
7 -> "python fail.py %(test-body-text) bar 0"
7 -> "python3 fail.py %(test-body-text) bar 0"
88
99 -> Tests for functionality "Succeed"
1010
1515 output and standard error, it's up to you to redirect them.
1616
1717 -> Functionality "Succeed/All" is implemented by shell command
18 -> "python fail.py %(test-body-text) bar 0 2>&1"
18 -> "python3 fail.py %(test-body-text) bar 0 2>&1"
1919
2020 -> Tests for functionality "Succeed/All"
2121
2727 expected text is matched against standard error.
2828
2929 -> Functionality "Fail" is implemented by shell command
30 -> "python fail.py foo %(test-body-text) 1"
30 -> "python3 fail.py foo %(test-body-text) 1"
3131
3232 -> Tests for functionality "Fail"
3333
3838 output and standard error, it's up to you to redirect them.
3939
4040 -> Functionality "Fail/All" is implemented by shell command
41 -> "python fail.py foo %(test-body-text) 1 1>&2"
41 -> "python3 fail.py foo %(test-body-text) 1 1>&2"
4242
4343 -> Tests for functionality "Fail/All"
4444
+1
-1
tests/test-substring-error.markdown less more
66 the expected error text.
77
88 -> Functionality "Fail" is implemented by shell command
9 -> "python fail.py foo %(test-body-text) 1"
9 -> "python3 fail.py foo %(test-body-text) 1"
1010
1111 -> Tests for functionality "Fail"
1212
+3
-3
tests/test-utf8.expected less more
33
44 Location: test-utf8.markdown, line 22
55 Function: Cat
6 Impl : shell command "python cat.py"
6 Impl : shell command "python3 cat.py"
77 Body : n ← ★
88 Expected: output:
99 m ← ★
1616
1717 Location: test-utf8.markdown, line 37
1818 Function: Cat (file)
19 Impl : shell command "python cat.py -f %(test-body-file) -o %(output-file)"
19 Impl : shell command "python3 cat.py -f %(test-body-file) -o %(output-file)"
2020 Body : n ← ★
2121 Expected: output:
2222 m ← ★
2929
3030 Location: test-utf8.markdown, line 53
3131 Function: Cat (input)
32 Impl : shell command "python cat.py -f %(test-input-file) -o %(output-file)"
32 Impl : shell command "python3 cat.py -f %(test-input-file) -o %(output-file)"
3333 Body : m ← ★
3434 Expected: output:
3535 m ← ★
+3
-3
tests/test-utf8.markdown less more
77 of all Falderal documents which go beyond mere ASCII.)
88
99 -> Functionality "Cat" is implemented by
10 -> shell command "python cat.py"
10 -> shell command "python3 cat.py"
1111
1212 -> Tests for functionality "Cat"
1313
2222 = m ← ★
2323
2424 -> Functionality "Cat (file)" is implemented by
25 -> shell command "python cat.py -f %(test-body-file) -o %(output-file)"
25 -> shell command "python3 cat.py -f %(test-body-file) -o %(output-file)"
2626
2727 -> Tests for functionality "Cat (file)"
2828
3737 = m ← ★
3838
3939 -> Functionality "Cat (input)" is implemented by
40 -> shell command "python cat.py -f %(test-input-file) -o %(output-file)"
40 -> shell command "python3 cat.py -f %(test-input-file) -o %(output-file)"
4141
4242 -> Tests for functionality "Cat (input)"
4343
+6
-6
tests/test-var-subst.markdown less more
77 it is not necessary to put any quotes around them in the template string.
88
99 -> Functionality "Echo Body" is implemented by shell command
10 -> "python echo.py %(test-body-text)"
10 -> "python3 echo.py %(test-body-text)"
1111
1212 -> Tests for functionality "Echo Body"
1313
2222 = don't
2323
2424 -> Functionality "Echo Input" is implemented by shell command
25 -> "python echo.py %(test-input-text)"
25 -> "python3 echo.py %(test-input-text)"
2626
2727 -> Tests for functionality "Echo Input"
2828
4949 `%(test-input-file)` and `%(output-file)` as well.
5050
5151 -> Functionality "Cat, from file" is implemented by
52 -> shell command "python cat.py -f %(test-body-file)"
52 -> shell command "python3 cat.py -f %(test-body-file)"
5353
5454 -> Tests for functionality "Cat, from file"
5555
6262 = hi
6363
6464 -> Functionality "Cat, to file" is implemented by
65 -> shell command "python cat.py -o %(output-file)"
65 -> shell command "python3 cat.py -o %(output-file)"
6666
6767 -> Tests for functionality "Cat, to file"
6868
7575 = hi
7676
7777 -> Functionality "Cat, to and from file" is implemented by
78 -> shell command "python cat.py -f %(test-body-file) -o %(output-file)"
78 -> shell command "python3 cat.py -f %(test-body-file) -o %(output-file)"
7979
8080 -> Tests for functionality "Cat, to and from file"
8181
8888 = hi
8989
9090 -> Functionality "Cat input, from file" is implemented by
91 -> shell command "python cat.py -f %(test-input-file)"
91 -> shell command "python3 cat.py -f %(test-input-file)"
9292
9393 -> Tests for functionality "Cat input, from file"
9494