Use .md for file extension for Markdown files. Other cleanups.
Chris Pressey
4 years ago
0 | e0f07b3699cd2b5029dc0e619e30f2897e27564a rel_0_1 | |
1 | b5b728aeba88946da3e6ae6ad0ffbc3eac53dd6a rel_0_2 | |
2 | 582e124855570072a933f14a89ba7ad74632610b rel_0_3 | |
3 | 582e124855570072a933f14a89ba7ad74632610b rel_0_3 | |
4 | 8bb4fad90030edb2ac4f7b113c3bf0a361f9274a rel_0_3 | |
5 | 046f8ec1388ab05e986febbc7c9d67c636076f53 rel_0_4 | |
6 | cb2caf712180eb0fd3d759f9fcbf5cef19680675 rel_0_5 | |
7 | 6f9b761233f86c8a416a7b13599ebf9f50f005b3 rel_0_6 | |
8 | 5ce741f7daf73c8b0a37daa48ea298b12e3f36cb rel_0_7 | |
9 | 0397d7575a05550ca8d137baeeb5ea984d0379a3 rel_0_8 | |
10 | 7575a60f38d4fcf1984e892cd301b4b99faf9606 rel_0_9_2013_1029 | |
11 | 00acc0048e83c3313ad740b90ce807de115bb21c rel_0_9_2014_0427 | |
12 | edfd7af3d505a76b08aebc0af537ea7e5fdeb31d rel_0_9_2014_0525 | |
13 | 0ff6e40b2c7e6464993fb792f3a3be1c94a6dd2e rel_0_10 | |
14 | 0205fd963df5b9d5d7306ea40696c593eb3e38ce rel_0_11 |
0 | History | |
1 | ======= | |
2 | ||
3 | Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive": | |
4 | ||
5 | * (planned) Add a pragma "Functionality ... is available if | |
6 | shell command ... can be successfully run". If the command | |
7 | cannot be successfully run, disable tests for the given | |
8 | functionality. (If no tests at all are run, it is an error.) | |
9 | * (planned) Add a command-line option to require that the | |
10 | version number of Falderal is at least a given version number. | |
11 | * `falderal`'s own test suite, `test.sh`, tests running `falderal` | |
12 | under Python 2 if it is available, and under Python 3 if it is | |
13 | available. If neither is available, it aborts. A particular | |
14 | implementation of Python can also be supplied in the `PYTHON` | |
15 | environment variable. | |
16 | ||
17 | Version 0.13 "Merchandise Mart": | |
18 | ||
19 | * Use `ArgumentParser` instead of `OptionParser` internally. | |
20 | * `--substring-error` is now the default behaviour, and if this | |
21 | option is given, it will be ignored. After a long time | |
22 | considering the problem, we finally determined that it does | |
23 | not make sense to insist that different implementations | |
24 | produce the exact same error message. | |
25 | * `--verbose` option now prints out commands for each test as it | |
26 | is run. | |
27 | * Added conventional `--version` option. | |
28 | * Planned to be the last 0.x series release before 1.0. | |
29 | * Added `setup.py` so that Falderal can be treated as a Python | |
30 | package. | |
31 | ||
32 | Version 0.12 "Schoenhofen Brewery": | |
33 | ||
34 | * When expanding variables in declaration strings, use | |
35 | `string.replace` instead of `re.sub` so that backslash-escaping | |
36 | is not perfomed on the replacement string. (Thanks to | |
37 | James Holderness for bug report and patch.) | |
38 | * In "freestyle" format, lines beginning with `<= `, `<== `, or | |
39 | `<=== ` can supply a section of test input text (Thanks to | |
40 | James Holderness for feature suggestion and patch.) | |
41 | * Falderal is now responsible for quoting all shell metacharacters | |
42 | in the substituion text of `%(...)` variables when command line | |
43 | templates are expanded, so that no quotes need be placed around | |
44 | the variable in the template. (Again, thanks to James Holderness | |
45 | for spotting the bug in the code that led to this improvement.) | |
46 | * `py-falderal` can now run under both Python 2 and Python 3 | |
47 | (tested with 2.7, 3.4, and 3.5.) | |
48 | * The doctests that were previously distributed throughout the | |
49 | source code have been converted to a suite of unit tests in | |
50 | the `falderal.tests` module. The `-t` command-line option has | |
51 | been removed; to run internal tests, run this module instead. | |
52 | ||
53 | Version 0.11 "Dan Ryan Expressway": | |
54 | ||
55 | * Renamed the internal tests to have descriptive names instead of | |
56 | numbers. | |
57 | * The first command-line argument being `test` for compatibility with | |
58 | previous versions of `falderal` has been removed. | |
59 | * Massive internal refactor of block-parsing and test-extraction code. | |
60 | * Supports a new format for test blocks, called "freestyle", where if | |
61 | the last few lines of a block are prefixed with `=> ` or `==> ` or | |
62 | `===> ` or `?> ` or `??> ` or `???> `, the preceding part of the | |
63 | block needs no prefixes at all, and it is simply treated as one | |
64 | verbatim test body. | |
65 | ||
66 | Version 0.10 "Shedd Aquarium": | |
67 | ||
68 | * Removed the old, bitrotting Haskell implementation. | |
69 | * Removed deprecated `%(test-text)` and `%(test-file)` variables | |
70 | (`%(test-body-text)` and `%(test-body-file)` should be used now.) | |
71 | * Added some adapters in the `script/` directory of the repo: | |
72 | * `fa-under-pty` runs a target program under a pseudoterminal | |
73 | * `fa-bin-to-hex` converts 8-bit binary output to hexadecimal codes | |
74 | * `fa-hex-to-bin` converts hexadecimal codes to 8-bit binary output | |
75 | * Added the rule that a test may consist of an input section (`+`) if | |
76 | there was a previous test, and the test body of the previous test will | |
77 | be re-used with the new input in the new test. | |
78 | ||
79 | Version 0.9-2014.0525 "Municipal Pier #2, May 25th 2014": | |
80 | ||
81 | * Shell implementations assume input and output is encoded UTF-8; any | |
82 | malformed UTF-8 is simply ignored (for better or worse, for now.) | |
83 | Falderal documents are assumed to be encoded in UTF-8; this includes | |
84 | test bodies, expectations, etc. Internally, in the reference | |
85 | implementation, all strings are stored as Unicode. | |
86 | ||
87 | Version 0.9-2014.0427 "Municipal Pier #2": | |
88 | ||
89 | * For a test of a shell command implementation that expects a failure, | |
90 | if no output was found on standard error, the output of standard output | |
91 | is checked against the failure expectation instead. The primary use case | |
92 | for this is Hugs, which (unlike `ghc`) displays error messages on stdout | |
93 | rather than stderr. "Municipal Pier #2" was the original name of Navy Pier. | |
94 | ||
95 | Version 0.9 "Navy Pier": | |
96 | ||
97 | * `py-falderal` now insists that blocks are indented four spaces, as the | |
98 | specification says. It ignores blocks that are not indented. | |
99 | * In an error report `py-falderal` reports the location (filename and line | |
100 | number of the beginning of the test text) of each test that failed, as | |
101 | well as the name of the functionality that was being tested. | |
102 | * Several internal renamings of class and variable names for greater clarity. | |
103 | * Trying to standardize on the terminology "test body" for the body of a | |
104 | test (as opposed to its "input", and opposed to it being passed to a shell | |
105 | command line "as text".) | |
106 | * Support for specifying a block of text to be used as input for a test. | |
107 | * Dropped support for "implemented by Haskell function" backwards-compatibility. | |
108 | * Temporary files are no longer always created in the current directory. That | |
109 | was a workaround to support `ghc` from Haskell Platform under Cygwin. They | |
110 | are now created where the system wants them to be created (but you can set | |
111 | the `TMP` environment variable to "." if you really want them to be created | |
112 | in the current directory.) | |
113 | * `py-falderal` now complains and stops early if there were no documents | |
114 | specified, no functionality definitions were found in any of the documents, | |
115 | no implementations were found for some defined functionality, or if no | |
116 | tests were found in any of the documents, unless `--cavalier` is passed on | |
117 | the command line. | |
118 | * In `--verbose` mode, `py-falderal` will issue warnings about any test that | |
119 | has been run twice (i.e. more than run on identical test body text and | |
120 | implementation.) | |
121 | * If `py-falderal` catches a `KeyboardInterrupt` exception while running tests, | |
122 | it shows a report of all the tests that it did manage to run until that | |
123 | point, and a warning message to the effect that not all tests were run. | |
124 | ||
125 | Version 0.8 "Ukrainian Village": | |
126 | ||
127 | * The reference implementation is now `py-falderal`. | |
128 | * Blocks *must* now be indented four spaces. That Falderal test documents | |
129 | will be embedded in Markdown is generally assumed (although not strictly | |
130 | required) and they generally have the file extension `.markdown` so that | |
131 | other software (e.g. Bitbucket and Github) knows how to pretty-display | |
132 | them. | |
133 | * Various documentation which was on Bitbucket (wiki, issues) and Github | |
134 | (py-falderal issues) have been moved into Markdown documents in the | |
135 | distribution, which have themselves been re-organized. | |
136 | * Added the `-v`/`--verbose` flag and made it and the `-d`/`--dump` flag | |
137 | do what you would expect from them. | |
138 | * The `-f`/`--functionalities` and `-c`/`--clear-functionalities` flags | |
139 | have been *removed*. The rationale is that implementations of | |
140 | functionalities are clumsy to express on the command line (and anyway | |
141 | the syntax would have to be different) while it is not difficult to | |
142 | create a "configuration" Falderal file which specifies *only* the | |
143 | desired implementations of the desired functionalities, and to load | |
144 | it just before loading a "tests" Falderal file which specified the | |
145 | functionalities being tested, but not their implementations. | |
146 | ||
147 | Version 0.7 "Ogden Avenue": | |
148 | ||
149 | * **Note that this is the final version of Falderal where `Test.Falderal` | |
150 | is the reference implementation.** In subsequent versions, | |
151 | `py-falderal` (written in Python) will be the reference implementation. | |
152 | `Test.Falderal` will still be in this repository (under the `impl` | |
153 | directory) but will no longer be definitive, and will go essentially | |
154 | unmaintained. | |
155 | * Fixed a bug where number of lines in the intermediary results file | |
156 | was sometimes being counted incorrect. | |
157 | * Slightly improved support for running under Cygwin. | |
158 | * Tests that are implemented by a shell command are now run directly | |
159 | from the `falderal` process (instead of creating an intermediary | |
160 | shell script and running it.) This resulted in better performance. | |
161 | * Addition of `-b` command-line option, which considers a test to | |
162 | have passed if the expected exception message is a substring (rather | |
163 | than an exact match) of the actual produced exception message. | |
164 | TODO: write test for this. | |
165 | * Blocks may now be indented four spaces, to allow them to be | |
166 | embedded directly in Markdown files as pre-formatted text. | |
167 | * A script to just build the binary, without installing it, was added. | |
168 | ||
169 | Version 0.6 "Streeterville": | |
170 | ||
171 | * Variables may be included in the specification of a shell command | |
172 | implementation; these will be expanded before generating the results | |
173 | generator. These variable include `%(test-text)`, `%(test-file)`, | |
174 | and `%(output-file)`. | |
175 | * `falderal` now lets the user specify, on the command line, the | |
176 | implementations for a named functionality. Supplying `-f 'foo:shell | |
177 | command "foo.sh %(test-file)"'` has the same effect as including the | |
178 | pragma `Functionality "foo" is implemented by shell command "foo.sh | |
179 | %(test-file)"` in the Falderal file. Note that this is in addition | |
180 | to the Functionality-definition pragmas given in the Falderal file; | |
181 | to replace them, you must first clear the ones with that name from | |
182 | the file by supplying `-c foo` on the command line. | |
183 | * `falderal` also allows tests for named functionalities to be skipped | |
184 | completely, by passing the name of the functionality to be skipped | |
185 | after a `-k` flag on the command line. | |
186 | * The Markdown formatter now formats Bird-style embedded code with | |
187 | HTML embedded in the Markdown document. This is so that it can | |
188 | be styled independently from, and thus distinguished from, any plain | |
189 | Markdown indented code blocks which may appear in the literate portion | |
190 | of the source code. | |
191 | * In failure reports, the implementation of the functionality of the test | |
192 | that failed is now reported in each failure. | |
193 | * A race condition(?) that could occur when testing multiple implementations | |
194 | of a functionality, of different kinds (Haskell and shell), has been | |
195 | prevented. Both tests were writing to `results.txt` and immediately | |
196 | deleting it, and this would sometimes confuse `falderal` into thinking | |
197 | one had produced no results (perhaps a result of some creative | |
198 | scheduling by `ghc`, although really, I haven't a clue.) Results are | |
199 | now written to different temporary files with different, generated | |
200 | names. | |
201 | * Previously, if the output of a shell command being tested did not end | |
202 | with a newline, the intermediate results file was not being generated | |
203 | correctly, resulting in failures being misreported. This has been | |
204 | recitified. | |
205 | * Previously, if there were tests given in a Falderal file before any | |
206 | Tests-for pragma was specified, those tests would just be ignored. | |
207 | An error message is now issued, and no testing takes place. | |
208 | * Previously, if multiple Falderal files were given on the command | |
209 | line, they were simply concatenated when loaded, the result being | |
210 | that Functionality-definitions from the first file were visible in | |
211 | the second file, and that any Tests-for in effect at the end of the | |
212 | first file would be in effect at the start of the second file. | |
213 | Files are now loaded and processed seperately. | |
214 | ||
215 | Version 0.5 "The Loop": | |
216 | ||
217 | * The command-line format of `falderal` has changed, for the better. | |
218 | The `test` subcommand no longer requires that the failure reporting | |
219 | style be specified; instead, it defaults to `standard`, and can be | |
220 | changed with a command-line option. There are also command-line | |
221 | options for selecting the programs to run results generators, and to | |
222 | keep generated files around instead of cleaning them up after testing. | |
223 | * The dependency on `ghc` for running Haskell results generators has been | |
224 | removed; these can be run by `runhaskell` now, and are run by | |
225 | `runhaskell` by default. | |
226 | * Failure reporting is now consistent across languages; both Haskell and | |
227 | Bourne shell results generators generate an intermediate format, which | |
228 | `falderal` digests. | |
229 | * A new pragma `encoding:` was added, so that this directive can be | |
230 | embedded in your Falderal document (for the benefit of your text editor) | |
231 | without necessarily appearing in the formatted document. | |
232 | * We began giving release milestones colorful names. The naming | |
233 | convention is to choose names of Chicagoland neigborhoods, suburbs, | |
234 | landmarks, and institutions. Version 0.5 was named after The Loop in | |
235 | recognition of its ability to shuttle test results between `falderal` | |
236 | and the various results generators implemented in different languages. | |
237 | Previous versions of `Test.Falderal` were retroactively given milestone | |
238 | names during this release. | |
239 | ||
240 | Version 0.4 "Blackstone Hotel": | |
241 | ||
242 | * For ease of installation, the project is presented as a Cabal package. | |
243 | * A driver executable, `falderal`, is built as part of installing the | |
244 | Cabal package. It provides a command-line interface to formatting | |
245 | Falderal files and, in a limited fashion, running the tests in them. | |
246 | * A shell script formatter has been written, enabling testing of shell | |
247 | commands. One caveat is that reporting for these tests is nowhere near | |
248 | as nice as for Haskell functions, but that will change in the next version. | |
249 | * The Functionality-definition pragma has been implemented, making it | |
250 | possible to write tests in a more abstract, implementation-independent | |
251 | fashion. | |
252 | * Falderal files written to work with Falderal 0.3 should still work with | |
253 | Falderal 0.4, but you are encouraged to use the Functionality-definition | |
254 | pragma introduced in 0.4 to make your tests more implementation-independent. | |
255 | ||
256 | Version 0.3 "Chicago Board of Trade": | |
257 | ||
258 | * The definition of a Falderal Literate Test Format, distinct from the | |
259 | reference implementation of tools for it in Haskell (`Test.Falderal`). | |
260 | This represented a fairly substantial departure from how previous versions | |
261 | of Falderal worked. | |
262 | * The ability to format a Falderal file to different formats, including | |
263 | Markdown and Haskell. | |
264 | * Running tests is now a matter of formatting a Falderal file as a Haskell | |
265 | module and running the function `testModule` in that module. | |
266 | ||
267 | Version 0.2 "Dearborn Station": | |
268 | ||
269 | * Added a test harness for Falderal itself, in the form of a simple shell | |
270 | script which diffs the output of `Test.Falderal.Demo` against a text file | |
271 | containing the expected content. | |
272 | * Improved formatting of failure reports. Multi-line input text or expected | |
273 | output is always presented starting on its own line. | |
274 | * Tests may be organized into groups; the entire group is preceded by some | |
275 | literal text, but there is no literal text between the tests in the group. | |
276 | When one of these tests fails, the literal text for the group is reported, | |
277 | along with the number of the test within the group. | |
278 | * Fixed a bug where exception text which extended over multiple lines | |
279 | could not be expected correctly. | |
280 | ||
281 | Version 0.1 "Haymarket Square": | |
282 | ||
283 | * Provision of a framework for writing and running literate tests which may | |
284 | be embedded in literate Haskell source code. | |
285 | * Testing Haskell functions of type `String -> String`. A test specifies input | |
286 | to the function, and may expect a particular output, or that a particular | |
287 | exception is thrown. | |
288 | * Through simple adapters, testing functions of other types such as | |
289 | `(Show a) => String -> a`. | |
290 | ||
291 | Prehistory: | |
292 | ||
293 | Falderal started life as a Haskell-specific hack that could be embedded | |
294 | in a Bird-style Literate Haskell source file. I took a framework for | |
295 | literate tests I had already written in a project called _Rho_, and used it | |
296 | as the basis of this code. | |
297 | ||
298 | I wrote it because I was designing yet another esoteric programming | |
299 | language, and while working on it I realized I was rebuilding yet another | |
300 | ad-hoc unit test suite, like I had done a half-dozen times before. I didn't | |
301 | want to keep doing this for every language I designed, and I realized that | |
302 | literate test suites could serve as documentation as well; the result was | |
303 | Falderal. |
0 | History | |
1 | ======= | |
2 | ||
3 | Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive": | |
4 | ||
5 | * (planned) Add a pragma "Functionality ... is available if | |
6 | shell command ... can be successfully run". If the command | |
7 | cannot be successfully run, disable tests for the given | |
8 | functionality. (If no tests at all are run, it is an error.) | |
9 | * (planned) Add a command-line option to require that the | |
10 | version number of Falderal is at least a given version number. | |
11 | * `falderal`'s own test suite, `test.sh`, tests running `falderal` | |
12 | under Python 2 if it is available, and under Python 3 if it is | |
13 | available. If neither is available, it aborts. A particular | |
14 | implementation of Python can also be supplied in the `PYTHON` | |
15 | environment variable. | |
16 | ||
17 | Version 0.13 "Merchandise Mart": | |
18 | ||
19 | * Use `ArgumentParser` instead of `OptionParser` internally. | |
20 | * `--substring-error` is now the default behaviour, and if this | |
21 | option is given, it will be ignored. After a long time | |
22 | considering the problem, we finally determined that it does | |
23 | not make sense to insist that different implementations | |
24 | produce the exact same error message. | |
25 | * `--verbose` option now prints out commands for each test as it | |
26 | is run. | |
27 | * Added conventional `--version` option. | |
28 | * Planned to be the last 0.x series release before 1.0. | |
29 | * Added `setup.py` so that Falderal can be treated as a Python | |
30 | package. | |
31 | ||
32 | Version 0.12 "Schoenhofen Brewery": | |
33 | ||
34 | * When expanding variables in declaration strings, use | |
35 | `string.replace` instead of `re.sub` so that backslash-escaping | |
36 | is not perfomed on the replacement string. (Thanks to | |
37 | James Holderness for bug report and patch.) | |
38 | * In "freestyle" format, lines beginning with `<= `, `<== `, or | |
39 | `<=== ` can supply a section of test input text (Thanks to | |
40 | James Holderness for feature suggestion and patch.) | |
41 | * Falderal is now responsible for quoting all shell metacharacters | |
42 | in the substituion text of `%(...)` variables when command line | |
43 | templates are expanded, so that no quotes need be placed around | |
44 | the variable in the template. (Again, thanks to James Holderness | |
45 | for spotting the bug in the code that led to this improvement.) | |
46 | * `py-falderal` can now run under both Python 2 and Python 3 | |
47 | (tested with 2.7, 3.4, and 3.5.) | |
48 | * The doctests that were previously distributed throughout the | |
49 | source code have been converted to a suite of unit tests in | |
50 | the `falderal.tests` module. The `-t` command-line option has | |
51 | been removed; to run internal tests, run this module instead. | |
52 | ||
53 | Version 0.11 "Dan Ryan Expressway": | |
54 | ||
55 | * Renamed the internal tests to have descriptive names instead of | |
56 | numbers. | |
57 | * The first command-line argument being `test` for compatibility with | |
58 | previous versions of `falderal` has been removed. | |
59 | * Massive internal refactor of block-parsing and test-extraction code. | |
60 | * Supports a new format for test blocks, called "freestyle", where if | |
61 | the last few lines of a block are prefixed with `=> ` or `==> ` or | |
62 | `===> ` or `?> ` or `??> ` or `???> `, the preceding part of the | |
63 | block needs no prefixes at all, and it is simply treated as one | |
64 | verbatim test body. | |
65 | ||
66 | Version 0.10 "Shedd Aquarium": | |
67 | ||
68 | * Removed the old, bitrotting Haskell implementation. | |
69 | * Removed deprecated `%(test-text)` and `%(test-file)` variables | |
70 | (`%(test-body-text)` and `%(test-body-file)` should be used now.) | |
71 | * Added some adapters in the `script/` directory of the repo: | |
72 | * `fa-under-pty` runs a target program under a pseudoterminal | |
73 | * `fa-bin-to-hex` converts 8-bit binary output to hexadecimal codes | |
74 | * `fa-hex-to-bin` converts hexadecimal codes to 8-bit binary output | |
75 | * Added the rule that a test may consist of an input section (`+`) if | |
76 | there was a previous test, and the test body of the previous test will | |
77 | be re-used with the new input in the new test. | |
78 | ||
79 | Version 0.9-2014.0525 "Municipal Pier #2, May 25th 2014": | |
80 | ||
81 | * Shell implementations assume input and output is encoded UTF-8; any | |
82 | malformed UTF-8 is simply ignored (for better or worse, for now.) | |
83 | Falderal documents are assumed to be encoded in UTF-8; this includes | |
84 | test bodies, expectations, etc. Internally, in the reference | |
85 | implementation, all strings are stored as Unicode. | |
86 | ||
87 | Version 0.9-2014.0427 "Municipal Pier #2": | |
88 | ||
89 | * For a test of a shell command implementation that expects a failure, | |
90 | if no output was found on standard error, the output of standard output | |
91 | is checked against the failure expectation instead. The primary use case | |
92 | for this is Hugs, which (unlike `ghc`) displays error messages on stdout | |
93 | rather than stderr. "Municipal Pier #2" was the original name of Navy Pier. | |
94 | ||
95 | Version 0.9 "Navy Pier": | |
96 | ||
97 | * `py-falderal` now insists that blocks are indented four spaces, as the | |
98 | specification says. It ignores blocks that are not indented. | |
99 | * In an error report `py-falderal` reports the location (filename and line | |
100 | number of the beginning of the test text) of each test that failed, as | |
101 | well as the name of the functionality that was being tested. | |
102 | * Several internal renamings of class and variable names for greater clarity. | |
103 | * Trying to standardize on the terminology "test body" for the body of a | |
104 | test (as opposed to its "input", and opposed to it being passed to a shell | |
105 | command line "as text".) | |
106 | * Support for specifying a block of text to be used as input for a test. | |
107 | * Dropped support for "implemented by Haskell function" backwards-compatibility. | |
108 | * Temporary files are no longer always created in the current directory. That | |
109 | was a workaround to support `ghc` from Haskell Platform under Cygwin. They | |
110 | are now created where the system wants them to be created (but you can set | |
111 | the `TMP` environment variable to "." if you really want them to be created | |
112 | in the current directory.) | |
113 | * `py-falderal` now complains and stops early if there were no documents | |
114 | specified, no functionality definitions were found in any of the documents, | |
115 | no implementations were found for some defined functionality, or if no | |
116 | tests were found in any of the documents, unless `--cavalier` is passed on | |
117 | the command line. | |
118 | * In `--verbose` mode, `py-falderal` will issue warnings about any test that | |
119 | has been run twice (i.e. more than run on identical test body text and | |
120 | implementation.) | |
121 | * If `py-falderal` catches a `KeyboardInterrupt` exception while running tests, | |
122 | it shows a report of all the tests that it did manage to run until that | |
123 | point, and a warning message to the effect that not all tests were run. | |
124 | ||
125 | Version 0.8 "Ukrainian Village": | |
126 | ||
127 | * The reference implementation is now `py-falderal`. | |
128 | * Blocks *must* now be indented four spaces. That Falderal test documents | |
129 | will be embedded in Markdown is generally assumed (although not strictly | |
130 | required) and they generally have the file extension `.markdown` so that | |
131 | other software (e.g. Bitbucket and Github) knows how to pretty-display | |
132 | them. | |
133 | * Various documentation which was on Bitbucket (wiki, issues) and Github | |
134 | (py-falderal issues) have been moved into Markdown documents in the | |
135 | distribution, which have themselves been re-organized. | |
136 | * Added the `-v`/`--verbose` flag and made it and the `-d`/`--dump` flag | |
137 | do what you would expect from them. | |
138 | * The `-f`/`--functionalities` and `-c`/`--clear-functionalities` flags | |
139 | have been *removed*. The rationale is that implementations of | |
140 | functionalities are clumsy to express on the command line (and anyway | |
141 | the syntax would have to be different) while it is not difficult to | |
142 | create a "configuration" Falderal file which specifies *only* the | |
143 | desired implementations of the desired functionalities, and to load | |
144 | it just before loading a "tests" Falderal file which specified the | |
145 | functionalities being tested, but not their implementations. | |
146 | ||
147 | Version 0.7 "Ogden Avenue": | |
148 | ||
149 | * **Note that this is the final version of Falderal where `Test.Falderal` | |
150 | is the reference implementation.** In subsequent versions, | |
151 | `py-falderal` (written in Python) will be the reference implementation. | |
152 | `Test.Falderal` will still be in this repository (under the `impl` | |
153 | directory) but will no longer be definitive, and will go essentially | |
154 | unmaintained. | |
155 | * Fixed a bug where number of lines in the intermediary results file | |
156 | was sometimes being counted incorrect. | |
157 | * Slightly improved support for running under Cygwin. | |
158 | * Tests that are implemented by a shell command are now run directly | |
159 | from the `falderal` process (instead of creating an intermediary | |
160 | shell script and running it.) This resulted in better performance. | |
161 | * Addition of `-b` command-line option, which considers a test to | |
162 | have passed if the expected exception message is a substring (rather | |
163 | than an exact match) of the actual produced exception message. | |
164 | TODO: write test for this. | |
165 | * Blocks may now be indented four spaces, to allow them to be | |
166 | embedded directly in Markdown files as pre-formatted text. | |
167 | * A script to just build the binary, without installing it, was added. | |
168 | ||
169 | Version 0.6 "Streeterville": | |
170 | ||
171 | * Variables may be included in the specification of a shell command | |
172 | implementation; these will be expanded before generating the results | |
173 | generator. These variable include `%(test-text)`, `%(test-file)`, | |
174 | and `%(output-file)`. | |
175 | * `falderal` now lets the user specify, on the command line, the | |
176 | implementations for a named functionality. Supplying `-f 'foo:shell | |
177 | command "foo.sh %(test-file)"'` has the same effect as including the | |
178 | pragma `Functionality "foo" is implemented by shell command "foo.sh | |
179 | %(test-file)"` in the Falderal file. Note that this is in addition | |
180 | to the Functionality-definition pragmas given in the Falderal file; | |
181 | to replace them, you must first clear the ones with that name from | |
182 | the file by supplying `-c foo` on the command line. | |
183 | * `falderal` also allows tests for named functionalities to be skipped | |
184 | completely, by passing the name of the functionality to be skipped | |
185 | after a `-k` flag on the command line. | |
186 | * The Markdown formatter now formats Bird-style embedded code with | |
187 | HTML embedded in the Markdown document. This is so that it can | |
188 | be styled independently from, and thus distinguished from, any plain | |
189 | Markdown indented code blocks which may appear in the literate portion | |
190 | of the source code. | |
191 | * In failure reports, the implementation of the functionality of the test | |
192 | that failed is now reported in each failure. | |
193 | * A race condition(?) that could occur when testing multiple implementations | |
194 | of a functionality, of different kinds (Haskell and shell), has been | |
195 | prevented. Both tests were writing to `results.txt` and immediately | |
196 | deleting it, and this would sometimes confuse `falderal` into thinking | |
197 | one had produced no results (perhaps a result of some creative | |
198 | scheduling by `ghc`, although really, I haven't a clue.) Results are | |
199 | now written to different temporary files with different, generated | |
200 | names. | |
201 | * Previously, if the output of a shell command being tested did not end | |
202 | with a newline, the intermediate results file was not being generated | |
203 | correctly, resulting in failures being misreported. This has been | |
204 | recitified. | |
205 | * Previously, if there were tests given in a Falderal file before any | |
206 | Tests-for pragma was specified, those tests would just be ignored. | |
207 | An error message is now issued, and no testing takes place. | |
208 | * Previously, if multiple Falderal files were given on the command | |
209 | line, they were simply concatenated when loaded, the result being | |
210 | that Functionality-definitions from the first file were visible in | |
211 | the second file, and that any Tests-for in effect at the end of the | |
212 | first file would be in effect at the start of the second file. | |
213 | Files are now loaded and processed seperately. | |
214 | ||
215 | Version 0.5 "The Loop": | |
216 | ||
217 | * The command-line format of `falderal` has changed, for the better. | |
218 | The `test` subcommand no longer requires that the failure reporting | |
219 | style be specified; instead, it defaults to `standard`, and can be | |
220 | changed with a command-line option. There are also command-line | |
221 | options for selecting the programs to run results generators, and to | |
222 | keep generated files around instead of cleaning them up after testing. | |
223 | * The dependency on `ghc` for running Haskell results generators has been | |
224 | removed; these can be run by `runhaskell` now, and are run by | |
225 | `runhaskell` by default. | |
226 | * Failure reporting is now consistent across languages; both Haskell and | |
227 | Bourne shell results generators generate an intermediate format, which | |
228 | `falderal` digests. | |
229 | * A new pragma `encoding:` was added, so that this directive can be | |
230 | embedded in your Falderal document (for the benefit of your text editor) | |
231 | without necessarily appearing in the formatted document. | |
232 | * We began giving release milestones colorful names. The naming | |
233 | convention is to choose names of Chicagoland neigborhoods, suburbs, | |
234 | landmarks, and institutions. Version 0.5 was named after The Loop in | |
235 | recognition of its ability to shuttle test results between `falderal` | |
236 | and the various results generators implemented in different languages. | |
237 | Previous versions of `Test.Falderal` were retroactively given milestone | |
238 | names during this release. | |
239 | ||
240 | Version 0.4 "Blackstone Hotel": | |
241 | ||
242 | * For ease of installation, the project is presented as a Cabal package. | |
243 | * A driver executable, `falderal`, is built as part of installing the | |
244 | Cabal package. It provides a command-line interface to formatting | |
245 | Falderal files and, in a limited fashion, running the tests in them. | |
246 | * A shell script formatter has been written, enabling testing of shell | |
247 | commands. One caveat is that reporting for these tests is nowhere near | |
248 | as nice as for Haskell functions, but that will change in the next version. | |
249 | * The Functionality-definition pragma has been implemented, making it | |
250 | possible to write tests in a more abstract, implementation-independent | |
251 | fashion. | |
252 | * Falderal files written to work with Falderal 0.3 should still work with | |
253 | Falderal 0.4, but you are encouraged to use the Functionality-definition | |
254 | pragma introduced in 0.4 to make your tests more implementation-independent. | |
255 | ||
256 | Version 0.3 "Chicago Board of Trade": | |
257 | ||
258 | * The definition of a Falderal Literate Test Format, distinct from the | |
259 | reference implementation of tools for it in Haskell (`Test.Falderal`). | |
260 | This represented a fairly substantial departure from how previous versions | |
261 | of Falderal worked. | |
262 | * The ability to format a Falderal file to different formats, including | |
263 | Markdown and Haskell. | |
264 | * Running tests is now a matter of formatting a Falderal file as a Haskell | |
265 | module and running the function `testModule` in that module. | |
266 | ||
267 | Version 0.2 "Dearborn Station": | |
268 | ||
269 | * Added a test harness for Falderal itself, in the form of a simple shell | |
270 | script which diffs the output of `Test.Falderal.Demo` against a text file | |
271 | containing the expected content. | |
272 | * Improved formatting of failure reports. Multi-line input text or expected | |
273 | output is always presented starting on its own line. | |
274 | * Tests may be organized into groups; the entire group is preceded by some | |
275 | literal text, but there is no literal text between the tests in the group. | |
276 | When one of these tests fails, the literal text for the group is reported, | |
277 | along with the number of the test within the group. | |
278 | * Fixed a bug where exception text which extended over multiple lines | |
279 | could not be expected correctly. | |
280 | ||
281 | Version 0.1 "Haymarket Square": | |
282 | ||
283 | * Provision of a framework for writing and running literate tests which may | |
284 | be embedded in literate Haskell source code. | |
285 | * Testing Haskell functions of type `String -> String`. A test specifies input | |
286 | to the function, and may expect a particular output, or that a particular | |
287 | exception is thrown. | |
288 | * Through simple adapters, testing functions of other types such as | |
289 | `(Show a) => String -> a`. | |
290 | ||
291 | Prehistory: | |
292 | ||
293 | Falderal started life as a Haskell-specific hack that could be embedded | |
294 | in a Bird-style Literate Haskell source file. I took a framework for | |
295 | literate tests I had already written in a project called _Rho_, and used it | |
296 | as the basis of this code. | |
297 | ||
298 | I wrote it because I was designing yet another esoteric programming | |
299 | language, and while working on it I realized I was rebuilding yet another | |
300 | ad-hoc unit test suite, like I had done a half-dozen times before. I didn't | |
301 | want to keep doing this for every language I designed, and I realized that | |
302 | literate test suites could serve as documentation as well; the result was | |
303 | Falderal. |
0 | Falderal | |
1 | ======== | |
2 | ||
3 | Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive" | |
4 | ||
5 | This is the reference distribution of Falderal, a file format for literate | |
6 | test suites. What sets Falderal apart from most other test frameworks is | |
7 | that it recognizes that **your tests are for a functionality, not a particular | |
8 | implementation of that functionality**. | |
9 | ||
10 | Falderal is particularly suited for: | |
11 | ||
12 | * documenting programming languages with prose and examples | |
13 | * testing multiple implementations of a programming language | |
14 | * performing Test-Driven Language Design (TDLD) | |
15 | * being embedded in Markdown documents | |
16 | ||
17 | Falderal in three words: "Doctests for DSLs". | |
18 | ||
19 | If you're more interested in running the tools than learning about the format | |
20 | itself, skip down to [Implementation](#implementation). | |
21 | ||
22 | Motivation | |
23 | ---------- | |
24 | ||
25 | Say you have a programming language. You probably have a document describing | |
26 | the language. That document probably includes examples. Those examples are | |
27 | probably intended to produce some kind of output. That output is probably | |
28 | expected to be a certain, predicted thing, and if it's not that thing, the | |
29 | implementation is probably considered incorrect. | |
30 | ||
31 | So why not write those examples in a format that can be run and tested? | |
32 | ||
33 | You could write a bunch of standalone test sources, and store the output you | |
34 | expect from them in a bunch of other files, and write a shell script that runs | |
35 | each program and `diff`s the output with the expected output. But this is a | |
36 | lot of clutter — finding a particular example might not be so easy. Each | |
37 | test source exists in a void, not necessarily logically grouped with other, | |
38 | similar tests. And any text you write describing a test needs to be in the | |
39 | comment syntax of your programming language (if your programming language | |
40 | supports comments) and is also detached from all the other test descriptions. | |
41 | ||
42 | You could write unit tests in the unit test framework of your choice, but | |
43 | if your programming language has more than one implementation one day (and | |
44 | you should really consider that possibility) then you might not be able to | |
45 | re-use it so easily for other implementations in other languages. | |
46 | ||
47 | In a language like Python, you could write doctests, but that also ties your | |
48 | tests to one implementation of your language. There can be awkward | |
49 | quoting issues with how you embed your test sources inside those embedded | |
50 | strings that comprise your doctests, as well. | |
51 | ||
52 | Or... you could write a Markdown document with beautiful yet precise prose | |
53 | describing your wonderful language, alternating with example code (in the | |
54 | form of embedded Falderal tests) clarifying each of the points you are | |
55 | making; then you could use a Falderal-comprehending tool to run each of these | |
56 | tests against any implementation of your language which exists or will exist | |
57 | in the future. | |
58 | ||
59 | *And* you could even write this document *before* you even start implementing | |
60 | your language; then when it is all clear "on paper", you have a target at | |
61 | which you can aim while writing your language. As you implement more and more | |
62 | of it, more and more tests in your test suite will pass. This is simply the | |
63 | idea behind Test-Driven Development (TDD) applied to language design, which we | |
64 | will call Test-Driven Language Design (TDLD). | |
65 | ||
66 | Features of the Format | |
67 | ---------------------- | |
68 | ||
69 | Falderal is just a file format; it does not specify exactly what tools must | |
70 | do with the tests they extract. However, it is generally assumed that most | |
71 | tools will want to, at the very least, | |
72 | ||
73 | * Run tests from one or more documents. | |
74 | * Report the results, with some given level of detail. | |
75 | ||
76 | There is, of course, a reference implementation which does both of these | |
77 | things. It is called py-falderal and it is written in Python 2.7. | |
78 | ||
79 | Each Falderal test is for some abstract _functionality_, and each | |
80 | functionality may have multiple concrete _implementations_. Thus the same | |
81 | tests can be run multiple times, once for each implementation of the | |
82 | functionality they test. | |
83 | ||
84 | Directives in the Falderal document may assign functionalities to tests, | |
85 | and may define implementations for given functionalities. Implementations | |
86 | may be defined outside of any document, as well. Falderal defines one | |
87 | general kind of implementation, implementation by Bourne shell command, but | |
88 | is not inherently restricted from supporting other kinds of implementations. | |
89 | ||
90 | Inherent Limitations | |
91 | -------------------- | |
92 | ||
93 | Being tests, rather than proofs, Falderal tests cannot tell you that your | |
94 | implementation of a language is correct. If one or more tests fail, that's | |
95 | an indication that your implementation is not correct; but even if all tests | |
96 | pass, you have no guarantee that the implementation doesn't do something | |
97 | contrary to the spec in one of the infinite number of cases that you have not | |
98 | enumerated. | |
99 | ||
100 | There is also no way to test that certain programs represent infinite loops, | |
101 | for reasons that should be obvious. | |
102 | ||
103 | Contents of this Distribution | |
104 | ----------------------------- | |
105 | ||
106 | This distribution contains: | |
107 | ||
108 | * `doc` — contains documents about Falderal. For the specification of | |
109 | the file format, see | |
110 | [`doc/Falderal_Literate_Test_Format.markdown`](doc/Falderal_Literate_Test_Format.markdown). | |
111 | (Note that this specification should not be expected to remain stable | |
112 | through the 0.x version series.) There are other documents in there too. | |
113 | * `bin/falderal` — the reference implementation of Falderal. | |
114 | See "Implementation", below, for details. | |
115 | * `script` — miscellaneous small tools intended to be used in tests. | |
116 | * `src` — source code for py-falderal. | |
117 | * `tests` — a set of tests for Falderal itself. (Note that these are not | |
118 | written in Falderal, as that would just be too confusing.) | |
119 | * `HISTORY.markdown` — changelog for releases of Falderal. | |
120 | * `TODO.markdown` — areas where Falderal and its implementations could be | |
121 | improved. | |
122 | ||
123 | Implementation | |
124 | -------------- | |
125 | ||
126 | This distribution contains `falderal`, which is the reference implementation | |
127 | of Falderal, written in Python and sometimes referred to as "py-falderal". | |
128 | ||
129 | To use it, you can clone this repository and run it as `bin/falderal` | |
130 | from the directory of your clone, or you can put the `bin` directory | |
131 | on your executable search path, and run it as `falderal` anywhere. | |
132 | ||
133 | Or you can install it using `pip`: | |
134 | ||
135 | pip install -e Falderal==0.13 | |
136 | ||
137 | (Depending on your needs, you may wish to establish a virtual environment | |
138 | first. Describing how to do so is outside the scope of this document.) | |
139 | ||
140 | The implementation is (somewhat) documented in `doc/py-falderal.markdown`. | |
141 | ||
142 | Development | |
143 | ----------- | |
144 | ||
145 | The git repository for the Falderal distribution can be found on GitHub at | |
146 | [https://github.com/catseye/Falderal](https://github.com/catseye/Falderal). | |
147 | ||
148 | Official release distfiles are available via the | |
149 | [Falderal project page](http://catseye.tc/node/Falderal) at | |
150 | [Cat's Eye Technologies](http://catseye.tc/). | |
151 | ||
152 | Projects using Falderal | |
153 | ----------------------- | |
154 | ||
155 | * [ALPACA](https://catseye.tc/node/ALPACA) | |
156 | * [Castile](https://catseye.tc/node/Castile) | |
157 | * [Equipage](https://catseye.tc/node/Equipage) | |
158 | * [Exanoke](https://catseye.tc/node/Exanoke) | |
159 | * [Flobnar](https://catseye.tc/node/Flobnar) | |
160 | * [Hev](https://catseye.tc/node/Hev) | |
161 | * [hatoucan](https://catseye.tc/node/hatoucan) | |
162 | * [Iphigeneia](https://catseye.tc/node/Iphigeneia) | |
163 | * [Madison](https://catseye.tc/node/Madison) | |
164 | * [Nhohnhehr](https://catseye.tc/node/Nhohnhehr) | |
165 | * [Pail](https://catseye.tc/node/Pail) | |
166 | * [Pixley](https://catseye.tc/node/Pixley) | |
167 | * [PL-{GOTO}.NET](https://catseye.tc/node/PL-{GOTO}.NET) | |
168 | * [Quylthulg](https://catseye.tc/node/Quylthulg) | |
169 | * [Robin](https://catseye.tc/node/Robin) | |
170 | * [Samovar](https://catseye.tc/node/Samovar) | |
171 | * [SixtyPical](https://catseye.tc/node/SixtyPical) | |
172 | * [Tamsin](https://catseye.tc/node/Tamsin) | |
173 | * [Velo](https://catseye.tc/node/Velo) | |
174 | * [Xoomonk](https://catseye.tc/node/Xoomonk) | |
175 | * [Yolk](https://catseye.tc/node/Yolk) | |
176 | * [ZOWIE](https://catseye.tc/node/ZOWIE) | |
177 | ||
178 | Xoomonk, Madison, Velo, and Exanoke are good examples of how a literate | |
179 | test suite can be useful in both describing a programming language through | |
180 | examples and testing that an implementation of the language does not violate | |
181 | the language specification. They are, in fact, exercises in Test-Driven | |
182 | Language Design (TDLD), where the tests were written as part of designing the | |
183 | language, before any attempt at implementation; the others are more like | |
184 | traditional test suites, written after-the-fact. |
0 | Falderal | |
1 | ======== | |
2 | ||
3 | Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive" | |
4 | ||
5 | This is the reference distribution of Falderal, a file format for literate | |
6 | test suites. What sets Falderal apart from most other test frameworks is | |
7 | that it recognizes that **your tests are for a functionality, not a particular | |
8 | implementation of that functionality**. | |
9 | ||
10 | Falderal is particularly suited for: | |
11 | ||
12 | * documenting programming languages with prose and examples | |
13 | * testing multiple implementations of a programming language | |
14 | * performing Test-Driven Language Design (TDLD) | |
15 | * being embedded in Markdown documents | |
16 | ||
17 | Falderal in three words: "Doctests for DSLs". | |
18 | ||
19 | If you're more interested in running the tools than learning about the format | |
20 | itself, skip down to [Implementation](#implementation). | |
21 | ||
22 | Motivation | |
23 | ---------- | |
24 | ||
25 | Say you have a programming language. You probably have a document describing | |
26 | the language. That document probably includes examples. Those examples are | |
27 | probably intended to produce some kind of output. That output is probably | |
28 | expected to be a certain, predicted thing, and if it's not that thing, the | |
29 | implementation is probably considered incorrect. | |
30 | ||
31 | So why not write those examples in a format that can be run and tested? | |
32 | ||
33 | You could write a bunch of standalone test sources, and store the output you | |
34 | expect from them in a bunch of other files, and write a shell script that runs | |
35 | each program and `diff`s the output with the expected output. But this is a | |
36 | lot of clutter — finding a particular example might not be so easy. Each | |
37 | test source exists in a void, not necessarily logically grouped with other, | |
38 | similar tests. And any text you write describing a test needs to be in the | |
39 | comment syntax of your programming language (if your programming language | |
40 | supports comments) and is also detached from all the other test descriptions. | |
41 | ||
42 | You could write unit tests in the unit test framework of your choice, but | |
43 | if your programming language has more than one implementation one day (and | |
44 | you should really consider that possibility) then you might not be able to | |
45 | re-use it so easily for other implementations in other languages. | |
46 | ||
47 | In a language like Python, you could write doctests, but that also ties your | |
48 | tests to one implementation of your language. There can be awkward | |
49 | quoting issues with how you embed your test sources inside those embedded | |
50 | strings that comprise your doctests, as well. | |
51 | ||
52 | Or... you could write a Markdown document with beautiful yet precise prose | |
53 | describing your wonderful language, alternating with example code (in the | |
54 | form of embedded Falderal tests) clarifying each of the points you are | |
55 | making; then you could use a Falderal-comprehending tool to run each of these | |
56 | tests against any implementation of your language which exists or will exist | |
57 | in the future. | |
58 | ||
59 | *And* you could even write this document *before* you even start implementing | |
60 | your language; then when it is all clear "on paper", you have a target at | |
61 | which you can aim while writing your language. As you implement more and more | |
62 | of it, more and more tests in your test suite will pass. This is simply the | |
63 | idea behind Test-Driven Development (TDD) applied to language design, which we | |
64 | will call Test-Driven Language Design (TDLD). | |
65 | ||
66 | Features of the Format | |
67 | ---------------------- | |
68 | ||
69 | Falderal is just a file format; it does not specify exactly what tools must | |
70 | do with the tests they extract. However, it is generally assumed that most | |
71 | tools will want to, at the very least, | |
72 | ||
73 | * Run tests from one or more documents. | |
74 | * Report the results, with some given level of detail. | |
75 | ||
76 | There is, of course, a reference implementation which does both of these | |
77 | things. It is called py-falderal and it is written in Python 2.7. | |
78 | ||
79 | Each Falderal test is for some abstract _functionality_, and each | |
80 | functionality may have multiple concrete _implementations_. Thus the same | |
81 | tests can be run multiple times, once for each implementation of the | |
82 | functionality they test. | |
83 | ||
84 | Directives in the Falderal document may assign functionalities to tests, | |
85 | and may define implementations for given functionalities. Implementations | |
86 | may be defined outside of any document, as well. Falderal defines one | |
87 | general kind of implementation, implementation by Bourne shell command, but | |
88 | is not inherently restricted from supporting other kinds of implementations. | |
89 | ||
90 | Inherent Limitations | |
91 | -------------------- | |
92 | ||
93 | Being tests, rather than proofs, Falderal tests cannot tell you that your | |
94 | implementation of a language is correct. If one or more tests fail, that's | |
95 | an indication that your implementation is not correct; but even if all tests | |
96 | pass, you have no guarantee that the implementation doesn't do something | |
97 | contrary to the spec in one of the infinite number of cases that you have not | |
98 | enumerated. | |
99 | ||
100 | There is also no way to test that certain programs represent infinite loops, | |
101 | for reasons that should be obvious. | |
102 | ||
103 | Contents of this Distribution | |
104 | ----------------------------- | |
105 | ||
106 | This distribution contains: | |
107 | ||
108 | * `doc` — contains documents about Falderal. For the specification of | |
109 | the file format, see | |
110 | [`doc/Falderal_Literate_Test_Format.md`](doc/Falderal_Literate_Test_Format.md). | |
111 | (Note that this specification should not be expected to remain stable | |
112 | through the 0.x version series.) There are other documents in there too. | |
113 | * `bin/falderal` — the reference implementation of Falderal. | |
114 | See "Implementation", below, for details. | |
115 | * `script` — miscellaneous small tools intended to be used in tests. | |
116 | * `src` — source code for py-falderal. | |
117 | * `tests` — a set of tests for Falderal itself. (Note that these are not | |
118 | written in Falderal, as that would just be too confusing.) | |
119 | * `HISTORY.md` — changelog for releases of Falderal. | |
120 | * `TODO.md` — areas where Falderal and its implementations could be | |
121 | improved. | |
122 | ||
123 | Implementation | |
124 | -------------- | |
125 | ||
126 | This distribution contains `falderal`, which is the reference implementation | |
127 | of Falderal, written in Python and sometimes referred to as "py-falderal". | |
128 | ||
129 | To use it, you can clone this repository and run it as `bin/falderal` | |
130 | from the directory of your clone, or you can put the `bin` directory | |
131 | on your executable search path, and run it as `falderal` anywhere. | |
132 | ||
133 | Or you can install it using `pip`: | |
134 | ||
135 | pip install -e Falderal==0.14 | |
136 | ||
137 | (Depending on your needs, you may wish to establish a virtual environment | |
138 | first. Describing how to do so is outside the scope of this document.) | |
139 | ||
140 | The implementation is (somewhat) documented in | |
141 | [`doc/py-falderal.md`](doc/py-falderal.md). | |
142 | ||
143 | The implementation has its own test suite (not written in Falderal, as | |
144 | that might be too confusing). It can be run by running `./test.sh`. | |
145 | ||
146 | Development | |
147 | ----------- | |
148 | ||
149 | The git repository for the Falderal distribution can be found on GitHub at | |
150 | [https://github.com/catseye/Falderal](https://github.com/catseye/Falderal). | |
151 | ||
152 | Official release distfiles are available via the | |
153 | [Falderal project page](http://catseye.tc/node/Falderal) at | |
154 | [Cat's Eye Technologies](http://catseye.tc/). | |
155 | ||
156 | Projects using Falderal | |
157 | ----------------------- | |
158 | ||
159 | * [ALPACA](https://catseye.tc/node/ALPACA) | |
160 | * [Castile](https://catseye.tc/node/Castile) | |
161 | * [Equipage](https://catseye.tc/node/Equipage) | |
162 | * [Exanoke](https://catseye.tc/node/Exanoke) | |
163 | * [Flobnar](https://catseye.tc/node/Flobnar) | |
164 | * [Hev](https://catseye.tc/node/Hev) | |
165 | * [hatoucan](https://catseye.tc/node/hatoucan) | |
166 | * [Iphigeneia](https://catseye.tc/node/Iphigeneia) | |
167 | * [Madison](https://catseye.tc/node/Madison) | |
168 | * [Nhohnhehr](https://catseye.tc/node/Nhohnhehr) | |
169 | * [Pail](https://catseye.tc/node/Pail) | |
170 | * [Pixley](https://catseye.tc/node/Pixley) | |
171 | * [PL-{GOTO}.NET](https://catseye.tc/node/PL-{GOTO}.NET) | |
172 | * [Quylthulg](https://catseye.tc/node/Quylthulg) | |
173 | * [Robin](https://catseye.tc/node/Robin) | |
174 | * [Samovar](https://catseye.tc/node/Samovar) | |
175 | * [SixtyPical](https://catseye.tc/node/SixtyPical) | |
176 | * [Tamsin](https://catseye.tc/node/Tamsin) | |
177 | * [Velo](https://catseye.tc/node/Velo) | |
178 | * [Xoomonk](https://catseye.tc/node/Xoomonk) | |
179 | * [Yolk](https://catseye.tc/node/Yolk) | |
180 | * [ZOWIE](https://catseye.tc/node/ZOWIE) | |
181 | ||
182 | Xoomonk, Madison, Velo, and Exanoke are good examples of how a literate | |
183 | test suite can be useful in both describing a programming language through | |
184 | examples and testing that an implementation of the language does not violate | |
185 | the language specification. They are, in fact, exercises in Test-Driven | |
186 | Language Design (TDLD), where the tests were written as part of designing the | |
187 | language, before any attempt at implementation; the others are more like | |
188 | traditional test suites, written after-the-fact. |
0 | TODO | |
1 | ==== | |
2 | ||
3 | (collected from the Falderal issue tracker on Bitbucket and the | |
4 | py-falderal issue tracker on github) | |
5 | ||
6 | Falderal Literate Test Format | |
7 | ----------------------------- | |
8 | ||
9 | ### Policy for expecting both errors and output, success and failure | |
10 | ||
11 | Policy should be this, by example: | |
12 | ||
13 | | test | |
14 | = foo | |
15 | ||
16 | Means: I expect this to succeed and to produce `foo` on stdout, and | |
17 | I don't care what's on stderr (or — stderr should be empty?) | |
18 | ||
19 | | test | |
20 | = foo | |
21 | ? bar | |
22 | ||
23 | Means: I expect this to succeed, to produce `foo` on stdout, and to | |
24 | produce `bar` on stderr. | |
25 | ||
26 | | test | |
27 | ? foo | |
28 | ||
29 | Means: I expect this to fail, and to produce `foo` on stderr. | |
30 | And to not care about stdout (or expect it to be empty.) | |
31 | ||
32 | | test | |
33 | ? foo | |
34 | = bar | |
35 | ||
36 | Means: I expect this to fail, to produce `foo` on stderr, and to | |
37 | produce `bar` on stdout. | |
38 | ||
39 | In other words, an error expectation may follow an output expectation | |
40 | and vice versa. Error expectations always match stderr, output expectations | |
41 | always match stdout. Which one's first should dictate whether we expect | |
42 | the command to succeed or fail. | |
43 | ||
44 | What's after here hasn't been re-edited yet. | |
45 | ||
46 | When you have a program that produces output on both stdout and stderr, | |
47 | whether it fails or not, you might want to expect text on both stdout and | |
48 | stderr. | |
49 | ||
50 | Currently it expects the text on stdout if it is a `=` expectation, and on | |
51 | stderr if it is a `?` expectation. | |
52 | ||
53 | You can't work around this so well by tacking `2>&1` onto the end of the | |
54 | command, because then stderr will always be empty. | |
55 | ||
56 | We could, by default, tack `2>&1` on the end ourselves and look only at | |
57 | stdout. This might be the simplest approach. | |
58 | ||
59 | We might want to add options that avoid doing that, but if so, what should | |
60 | they be? Should each test be able to configure this? Should a single test | |
61 | be able to have both `=` and `?` expectations, each for each stream? | |
62 | ||
63 | This is complicated by the presence of `%(output-file)`; currently, if that | |
64 | is given, stdout is ignored in preference to it (but stderr is still | |
65 | checked, if the command failed. There should probably be a corresponding | |
66 | `%(error-file)` variable.) | |
67 | ||
68 | I think the current behaviour could work, with the following policy: | |
69 | ||
70 | If the command succeeds, your `=` expectation will be matched against | |
71 | stdout only. If you wish to match against both stdout and stderr in these | |
72 | cases, add `2>&1` to your shell command. | |
73 | ||
74 | If the command fails, your `?` expectation will be matched against stderr | |
75 | only. If you wish to match against both stdout and stderr in these cases, | |
76 | add `1>&2` to your shell command. | |
77 | ||
78 | Either way, it's still worth investigating whether it's worthwhile to have | |
79 | both `=` and `?` expectations on a single test. (I can't convince myself | |
80 | that stdout and stderr will always be combined deterministically, and | |
81 | having both kinds of expectations would allow non-deterministic combinations | |
82 | of the two to be matched.) | |
83 | ||
84 | ### Allow expectations to be transformed during comparison | |
85 | ||
86 | It would be nice to allow expectations to be transformed before they are | |
87 | compared to the actual output. The main use case for this that I can think of | |
88 | is to allow the expected output to be "pretty printed" (that is, nicely | |
89 | formatted) in the Falderal file, while the functionality being tested just | |
90 | produces a dump. The nicely formatted expected output should be "crunched" | |
91 | into the same ugly format as the dump. | |
92 | ||
93 | This doesn't work as well the other way; although one could compose the | |
94 | functionality being tested with an actual pretty-printer, that would | |
95 | prescribe a certain indentation scheme etc. that the expected output would | |
96 | have to match exactly. It would be rather better if the writer of the tests | |
97 | could format their expected output as they find most aesthetically pleasing | |
98 | in their literate tests, and have that be transformed instead. | |
99 | ||
100 | This might be somewhat tricky, however; if the transformation applied is | |
101 | too powerful, it can distort or eliminate the meaning of the test, and erode | |
102 | confidence. | |
103 | ||
104 | ### Allow use of patterns in expected output | |
105 | ||
106 | Likely by way of regexps. This would be particularly valuable in | |
107 | exception-expecting tests, where we don't care about details such as the | |
108 | line number of the Haskell file at which the exception occurred. | |
109 | ||
110 | ### Allow equivalency tests to be defined and run. | |
111 | ||
112 | To test functions of type `(Eq a) => String -> a`, you should be able to | |
113 | give give multiple input strings in a set; if the function does not map | |
114 | them all to the same value, that's a test failure. | |
115 | ||
116 | Syntax for an equivalency test might look like this: | |
117 | ||
118 | | 2+2 | |
119 | == | |
120 | | 3+1 | |
121 | == | |
122 | | 7-3 | |
123 | ||
124 | ||
125 | py-falderal | |
126 | ----------- | |
127 | ||
128 | ### Test report accumulation | |
129 | ||
130 | Multiple runs of `falderal` ought to be able to accumulate their results | |
131 | to a temporary file. No report should be generated if this is selected. | |
132 | At the end, the report can be generated from the file. | |
133 | ||
134 | rm -f FILE | |
135 | falderal --record-to FILE tests1.markdown | |
136 | falderal --record-to FILE tests2.markdown | |
137 | falderal --report-from FILE | |
138 | ||
139 | ### Support 'weak' testing | |
140 | ||
141 | In which we only care about whether the command succeeded or failed. | |
142 | In practice, this could be useful for testing the parser (just test | |
143 | if these forms parse.) Or, if not this, then think of something that | |
144 | would make just testing parsers more useful. | |
145 | ||
146 | ### Split InterveningMarkdown blocks to make nice test descriptions | |
147 | ||
148 | For example, if we have | |
149 | ||
150 | ...test #1... | |
151 | Some text. | |
152 | Heading | |
153 | ------- | |
154 | More text | |
155 | ...test #2... | |
156 | ||
157 | The description for test #2 should consist of "More text"; possibly also | |
158 | the heading, but not "Some text". This can take place in a pre-processing | |
159 | phase which simply splits every `InterveningMarkdown` block into multiple | |
160 | blocks, at its headers. It should understand both underlined and atx-style | |
161 | headers. | |
162 | ||
163 | ### option to colourize test result output | |
164 | ||
165 | Using one of the approaches listed here: | |
166 | ||
167 | http://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python | |
168 | ||
169 | ...py-falderal ought to provide an option (not default, of course, and not | |
170 | if stdout is not a tty) to colorize the output with, of course, pass=green, | |
171 | fail=red. | |
172 | ||
173 | But, you'd often want to pipe the output to `less`, which by default makes | |
174 | control characters visible, defeating colourization. But there is a flag | |
175 | to less, `less -R`, which retains colourization. So use that. | |
176 | ||
177 | ### Flag invalid sequences of lines as errors | |
178 | ||
179 | 2011-08-05 | |
180 | ||
181 | Currently, in `convertLinesToBlocks`, some invalid sequences of lines are | |
182 | ignored. They should be flagged as errors in the test suite file. | |
183 | ||
184 | (This was written against Test.Falderal but similar considerations could | |
185 | be made for py-falderal.) |
0 | TODO | |
1 | ==== | |
2 | ||
3 | (collected from the Falderal issue tracker on Bitbucket and the | |
4 | py-falderal issue tracker on github) | |
5 | ||
6 | Falderal Literate Test Format | |
7 | ----------------------------- | |
8 | ||
9 | ### Policy for expecting both errors and output, success and failure | |
10 | ||
11 | Policy should be this, by example: | |
12 | ||
13 | | test | |
14 | = foo | |
15 | ||
16 | Means: I expect this to succeed and to produce `foo` on stdout, and | |
17 | I don't care what's on stderr (or — stderr should be empty?) | |
18 | ||
19 | | test | |
20 | = foo | |
21 | ? bar | |
22 | ||
23 | Means: I expect this to succeed, to produce `foo` on stdout, and to | |
24 | produce `bar` on stderr. | |
25 | ||
26 | | test | |
27 | ? foo | |
28 | ||
29 | Means: I expect this to fail, and to produce `foo` on stderr. | |
30 | And to not care about stdout (or expect it to be empty.) | |
31 | ||
32 | | test | |
33 | ? foo | |
34 | = bar | |
35 | ||
36 | Means: I expect this to fail, to produce `foo` on stderr, and to | |
37 | produce `bar` on stdout. | |
38 | ||
39 | In other words, an error expectation may follow an output expectation | |
40 | and vice versa. Error expectations always match stderr, output expectations | |
41 | always match stdout. Which one's first should dictate whether we expect | |
42 | the command to succeed or fail. | |
43 | ||
44 | What's after here hasn't been re-edited yet. | |
45 | ||
46 | When you have a program that produces output on both stdout and stderr, | |
47 | whether it fails or not, you might want to expect text on both stdout and | |
48 | stderr. | |
49 | ||
50 | Currently it expects the text on stdout if it is a `=` expectation, and on | |
51 | stderr if it is a `?` expectation. | |
52 | ||
53 | You can't work around this so well by tacking `2>&1` onto the end of the | |
54 | command, because then stderr will always be empty. | |
55 | ||
56 | We could, by default, tack `2>&1` on the end ourselves and look only at | |
57 | stdout. This might be the simplest approach. | |
58 | ||
59 | We might want to add options that avoid doing that, but if so, what should | |
60 | they be? Should each test be able to configure this? Should a single test | |
61 | be able to have both `=` and `?` expectations, each for each stream? | |
62 | ||
63 | This is complicated by the presence of `%(output-file)`; currently, if that | |
64 | is given, stdout is ignored in preference to it (but stderr is still | |
65 | checked, if the command failed. There should probably be a corresponding | |
66 | `%(error-file)` variable.) | |
67 | ||
68 | I think the current behaviour could work, with the following policy: | |
69 | ||
70 | If the command succeeds, your `=` expectation will be matched against | |
71 | stdout only. If you wish to match against both stdout and stderr in these | |
72 | cases, add `2>&1` to your shell command. | |
73 | ||
74 | If the command fails, your `?` expectation will be matched against stderr | |
75 | only. If you wish to match against both stdout and stderr in these cases, | |
76 | add `1>&2` to your shell command. | |
77 | ||
78 | Either way, it's still worth investigating whether it's worthwhile to have | |
79 | both `=` and `?` expectations on a single test. (I can't convince myself | |
80 | that stdout and stderr will always be combined deterministically, and | |
81 | having both kinds of expectations would allow non-deterministic combinations | |
82 | of the two to be matched.) | |
83 | ||
84 | ### Allow expectations to be transformed during comparison | |
85 | ||
86 | It would be nice to allow expectations to be transformed before they are | |
87 | compared to the actual output. The main use case for this that I can think of | |
88 | is to allow the expected output to be "pretty printed" (that is, nicely | |
89 | formatted) in the Falderal file, while the functionality being tested just | |
90 | produces a dump. The nicely formatted expected output should be "crunched" | |
91 | into the same ugly format as the dump. | |
92 | ||
93 | This doesn't work as well the other way; although one could compose the | |
94 | functionality being tested with an actual pretty-printer, that would | |
95 | prescribe a certain indentation scheme etc. that the expected output would | |
96 | have to match exactly. It would be rather better if the writer of the tests | |
97 | could format their expected output as they find most aesthetically pleasing | |
98 | in their literate tests, and have that be transformed instead. | |
99 | ||
100 | This might be somewhat tricky, however; if the transformation applied is | |
101 | too powerful, it can distort or eliminate the meaning of the test, and erode | |
102 | confidence. | |
103 | ||
104 | ### Allow use of patterns in expected output | |
105 | ||
106 | Likely by way of regexps. This would be particularly valuable in | |
107 | exception-expecting tests, where we don't care about details such as the | |
108 | line number of the Haskell file at which the exception occurred. | |
109 | ||
110 | ### Allow equivalency tests to be defined and run. | |
111 | ||
112 | To test functions of type `(Eq a) => String -> a`, you should be able to | |
113 | give give multiple input strings in a set; if the function does not map | |
114 | them all to the same value, that's a test failure. | |
115 | ||
116 | Syntax for an equivalency test might look like this: | |
117 | ||
118 | | 2+2 | |
119 | == | |
120 | | 3+1 | |
121 | == | |
122 | | 7-3 | |
123 | ||
124 | ||
125 | py-falderal | |
126 | ----------- | |
127 | ||
128 | ### Test report accumulation | |
129 | ||
130 | Multiple runs of `falderal` ought to be able to accumulate their results | |
131 | to a temporary file. No report should be generated if this is selected. | |
132 | At the end, the report can be generated from the file. | |
133 | ||
134 | rm -f FILE | |
135 | falderal --record-to FILE tests1.markdown | |
136 | falderal --record-to FILE tests2.markdown | |
137 | falderal --report-from FILE | |
138 | ||
139 | ### Support 'weak' testing | |
140 | ||
141 | In which we only care about whether the command succeeded or failed. | |
142 | In practice, this could be useful for testing the parser (just test | |
143 | if these forms parse.) Or, if not this, then think of something that | |
144 | would make just testing parsers more useful. | |
145 | ||
146 | ### Split InterveningMarkdown blocks to make nice test descriptions | |
147 | ||
148 | For example, if we have | |
149 | ||
150 | ...test #1... | |
151 | Some text. | |
152 | Heading | |
153 | ------- | |
154 | More text | |
155 | ...test #2... | |
156 | ||
157 | The description for test #2 should consist of "More text"; possibly also | |
158 | the heading, but not "Some text". This can take place in a pre-processing | |
159 | phase which simply splits every `InterveningMarkdown` block into multiple | |
160 | blocks, at its headers. It should understand both underlined and atx-style | |
161 | headers. | |
162 | ||
163 | ### option to colourize test result output | |
164 | ||
165 | Using one of the approaches listed here: | |
166 | ||
167 | http://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python | |
168 | ||
169 | ...py-falderal ought to provide an option (not default, of course, and not | |
170 | if stdout is not a tty) to colorize the output with, of course, pass=green, | |
171 | fail=red. | |
172 | ||
173 | But, you'd often want to pipe the output to `less`, which by default makes | |
174 | control characters visible, defeating colourization. But there is a flag | |
175 | to less, `less -R`, which retains colourization. So use that. | |
176 | ||
177 | ### Flag invalid sequences of lines as errors | |
178 | ||
179 | 2011-08-05 | |
180 | ||
181 | Currently, in `convertLinesToBlocks`, some invalid sequences of lines are | |
182 | ignored. They should be flagged as errors in the test suite file. | |
183 | ||
184 | (This was written against Test.Falderal but similar considerations could | |
185 | be made for py-falderal.) |
0 | Falderal Literate Test Format | |
1 | ============================= | |
2 | ||
3 | This document describes the proposed Falderal Literate Test Format. | |
4 | ||
5 | Status | |
6 | ------ | |
7 | ||
8 | This document is a *draft*. It is nominally "version 0.12" because it | |
9 | describes something that version 0.12 of `py-falderal` mostly implements. | |
10 | We will deign to note which sections of this document the current released | |
11 | version of `py-falderal` implements, and which it does not. However, | |
12 | this document is a work in progress, subject to change, and subject to get | |
13 | out of sync with `py-falderal`. It should not be considered to be | |
14 | anything except a draft until it is described as "version 1.0". | |
15 | ||
16 | Overview | |
17 | -------- | |
18 | ||
19 | A Falderal Literate Test Suite is a plain text file where some of the lines | |
20 | have special meaning to the Falderal Literate Test Format. Certain | |
21 | groupings of the lines defined by this format are intended to denote tests, | |
22 | while others denote pragmas which specify how those tests should be run. | |
23 | Outside of these groupings, lines have no meaning to the Falderal Literate | |
24 | Test Format other than that they are presumed to be mainly descriptive text. | |
25 | ||
26 | The plain text file may also be formatted in some other format, such as | |
27 | Markdown, although this is not required. | |
28 | ||
29 | A tool which claims to understand the Falderal Literate Test Format may | |
30 | choose to extract the tests from such a document, run them, and report which | |
31 | and how many of them passed or failed. That would be a typical application | |
32 | for this format. However, no tool is strictly required to do this with the | |
33 | tests found in a document; a tool may, for example, simply reformat the | |
34 | tests to produce an output file in a different format. | |
35 | ||
36 | In the syntax templates below, words in `/slashes/` indicate a variable | |
37 | rather than literal text. | |
38 | ||
39 | Syntax | |
40 | ------ | |
41 | ||
42 | Each grouping of lines which has special meaning to the Falderal Literate | |
43 | Test Format always begins with an indent of four (4) spaces from the | |
44 | leftmost column, preceded by non-indented text, and followed by either | |
45 | non-indented text or the end of the file. Such a grouping of lines is | |
46 | called a _block_. | |
47 | ||
48 | There are two general formats to any block. In the first, "verbose" format, | |
49 | each indent of 4 spaces is followed immediately on that line by distinguished | |
50 | sequence of characters, called an _introducer_. The introducers which have | |
51 | meaning to the Falderal Literate Test Format are as follows: | |
52 | ||
53 | * `->` (hyphen, greater-than sign): pragma | |
54 | * `| ` (vertical bar, space): test body text | |
55 | * `+ ` (plus sign, space): test input text | |
56 | * `= ` (equals sign, space): expected output text | |
57 | * `? ` (question mark, space): expected error text | |
58 | ||
59 | If the same introducer occurs on multiple adjacent lines, the text after | |
60 | each introducer is concatenated to form one chunk of relevant text. This | |
61 | allows, for example, multi-line text to be given as the body, the input, | |
62 | or the expected output of a test. | |
63 | ||
64 | There are some restrictions on the order in which introducers can sensibly | |
65 | occur in a block: | |
66 | ||
67 | * Test body text should occur at the start of a block. | |
68 | * Test body test may be optionally followed by test input text. | |
69 | * The first test input text must be immediately preceded by test body text. | |
70 | * Subsequent test input texts need not be preceded by a test body text; | |
71 | in this case, the previously-appearing test body text will be used again. | |
72 | * Test body text must be followed by either test input text, expected | |
73 | output text, or expected error text, with no intervening text. | |
74 | * Either expected output or error text must follow either test body | |
75 | text or test input text, with no intervening text. | |
76 | ||
77 | See the sections for these introducers, below, for more details. | |
78 | ||
79 | In the other, "freestyle" format, not all lines in a block require | |
80 | introducers. A freestyle format block is indentified as such if one or | |
81 | more of the final lines of the block begin with any of the following | |
82 | introducers: | |
83 | ||
84 | * `=> `: expected output text | |
85 | * `==> `: expected output text | |
86 | * `===> `: expected output text | |
87 | * `?> `: expected error text | |
88 | * `??> `: expected error text | |
89 | * `???> `: expected error text | |
90 | ||
91 | In addition, the following introducers may be used to mark a section | |
92 | of test input text on the first of the final lines (but may not be | |
93 | used to end a block): | |
94 | ||
95 | * `<= `: test input text | |
96 | * `<== `: test input text | |
97 | * `<=== `: test input text | |
98 | ||
99 | If a block is identified as a freestyle block, all lines preceding the | |
100 | first final line appearing with one of these introducers, are interpreted | |
101 | as having no introducer at all (even if they begin with `| ` or some other | |
102 | sequence already mentioned) and are used as the test body block. | |
103 | ||
104 | Lines without introducers are called _intervening text_. | |
105 | Lines of intervening text are classified as either blank or non-blank. A | |
106 | line is blank if it contains no characters, or if it contains only whitespace. | |
107 | A group of non-blank lines is referred to as a _paragraph_. | |
108 | ||
109 | Pragmas | |
110 | ------- | |
111 | ||
112 | ### encoding ### | |
113 | ||
114 | The encoding pragma allows a Falderal file to specify what encoding is | |
115 | used for the characters in it. An implementation of Falderal is not | |
116 | expected to be able to handle any coding other than UTF-8, however, | |
117 | this pragma is included for the benefit of text editors and other tools, | |
118 | to indicate that the document is in fact in UTF-8 encoding. | |
119 | ||
120 | Example: | |
121 | ||
122 | -> encoding: UTF-8 | |
123 | ||
124 | ### Functionality-definition ### | |
125 | ||
126 | The Functionality-definition pragma allows a Falderal file to describe | |
127 | ways in which a functionality being tested is implemented. It has the | |
128 | following syntax: | |
129 | ||
130 | -> Functionality /functionality-name/ is implemented by /functionality-type/ /functionality-specifier/ | |
131 | ||
132 | _functionality-type_ must at present be `shell command`. The format of the | |
133 | _functionality-specifier_ differs according to the functionality-type. | |
134 | The _functionality-name_ is arbitrary text enclosed within double quotes, | |
135 | which may be referenced in a later Tests-for pragma. | |
136 | ||
137 | Note that the Functionality-definitions given in a Falderal file should | |
138 | not be considered exhaustive, or even requisite, by a tool. The tool may | |
139 | accept additional definitions of a functionality, referencing it by its | |
140 | name, from an external source such as the command line or a configuration | |
141 | file, and may be instructed to ignore certain Functionality-definitions in | |
142 | a Falderal file (if, for example, certain implementation are not currently | |
143 | available or of interest to the user.) Indeed, the functionality referred | |
144 | to by a _functionality-name_ in a Tests-for pragma need not be defined by any | |
145 | Functionality-definition pragma in the same Falderal file, and this | |
146 | situation requires the functionality to be specified to the tool in some | |
147 | other manner. | |
148 | ||
149 | #### Shell commands #### | |
150 | ||
151 | For shell commands, the _functionality-specifier_ is in the format | |
152 | `"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may | |
153 | be used, so pipes, redirection, etc., are supported. Note that the double | |
154 | quotation mark characters used to enclosed the command have meaning only to | |
155 | the Falderal format — they are not part of the command, are not passed to the | |
156 | shell, and do not require double quotation mark characters that are enclosed | |
157 | by them to be escaped with a backslash. | |
158 | ||
159 | Certain subsequences, called _variables_, if present in the command string, | |
160 | will be expanded before execution, and will alter how the command reads the | |
161 | text of the test and produces its output, to be compared with the expected | |
162 | output. | |
163 | ||
164 | When a variable is expanded, it is Falderal's responsibility to escape any | |
165 | shell metacharacters that occur as part of the substitution text so that | |
166 | it appears to the shell as a single token, and cannot otherwise affect the | |
167 | meaning of the instantiated shell command. It is not necessary for a | |
168 | variable to be enclosed in any kind of quotes in the command line template. | |
169 | It is also not possible for the substitution text to e.g. expand to two | |
170 | arguments, or to form a pipe or redirection specifier, etc. | |
171 | ||
172 | The exact method by which the Falderal implementation escapes or quotes | |
173 | the substitution text so that it appears unambiguously as a single argument | |
174 | is left up to the implementation. | |
175 | ||
176 | ##### `%(test-body-file)` ##### | |
177 | ||
178 | The variable `%(test-body-file)` will be replaced by the name of a file which | |
179 | contains the text of the test body. This may be a temporary file created | |
180 | solely for this purpose by the Falderal implementation. | |
181 | ||
182 | ##### `%(test-body-text)` ##### | |
183 | ||
184 | The variable `%(test-body-text)` will be replaced by the actual text of the | |
185 | test body. | |
186 | ||
187 | ##### `%(test-input-file)` ##### | |
188 | ||
189 | The variable `%(test-input-file)` will be replaced by the name of a file which | |
190 | contains the text of the test input. This may be a temporary file created | |
191 | solely for this purpose by the Falderal implementation. | |
192 | ||
193 | ##### `%(test-input-text)` ##### | |
194 | ||
195 | The variable `%(test-input-text)` will be replaced by the actual text of the | |
196 | test input. | |
197 | ||
198 | If neither of the variables `%(test-body-file)` nor `%(test-body-text)` appear | |
199 | in the command string, the test body text will be provided on the standard | |
200 | input of the shell command. | |
201 | ||
202 | If neither of the variables `%(test-input-file)` nor `%(test-input-text)` appear | |
203 | in the command string, the test input text will be provided on the standard | |
204 | input of the shell command. | |
205 | ||
206 | If both the test body text and the test input text are slated to appear on the | |
207 | input of the shell command, then the behaviour is (presently) undefined. | |
208 | ||
209 | ##### `%(output-file)` ##### | |
210 | ||
211 | The variable `%(output-file)` will be replaced by the name of a file | |
212 | (temporary file) to which the test results will be written. If it does | |
213 | not appear in the command string, the output text will be read from | |
214 | the standard output of the command. | |
215 | ||
216 | How shell commands support error output is not yet standardized. | |
217 | ||
218 | For example: | |
219 | ||
220 | -> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %(test-file) > %(output-file)" | |
221 | ||
222 | ### Tests-for ### | |
223 | ||
224 | The Tests-for pragma determines what functionality will be used to run all | |
225 | following tests, until the next Tests-for pragma. It has the following | |
226 | syntax: | |
227 | ||
228 | -> Tests for /functionality-name/ | |
229 | ||
230 | The _functionality-name_ refers to a functionality, which may be specified | |
231 | by a Functionality-definition pragma elsewhere in the Falderal file. | |
232 | ||
233 | For example: | |
234 | ||
235 | -> Tests for 'Reversing a string' | |
236 | ||
237 | Test Body, Test Input and Expected Text | |
238 | --------------------------------------- | |
239 | ||
240 | Each section of test body text may or may not be followed by a section of | |
241 | test input text; either way it must then be followed immediately by either | |
242 | and expected output section or expected error section. | |
243 | ||
244 | Valid examples in the "verbose" format: | |
245 | ||
246 | | thing to test | |
247 | = output to expect | |
248 | ||
249 | | thing to test | |
250 | ? error to expect | |
251 | ||
252 | | thing to test | |
253 | + input to give it | |
254 | = output to expect | |
255 | ||
256 | | thing to test | |
257 | + input to give it | |
258 | ? error to expect | |
259 | ||
260 | + different input to give the immediately previously defined test body | |
261 | ? different error to expect | |
262 | ||
263 | Valid examples in the "freestyle" format: | |
264 | ||
265 | thing to test | |
266 | => output to expect | |
267 | ||
268 | thing to test | |
269 | ==> output to expect | |
270 | ||
271 | thing to test | |
272 | ===> output to expect | |
273 | ||
274 | thing to test | |
275 | ?> error to expect | |
276 | ||
277 | thing to test | |
278 | ??> error to expect | |
279 | ||
280 | thing to test | |
281 | ???> error to expect | |
282 | ||
283 | thing to test | |
284 | <=== input to give it | |
285 | ===> output to expect | |
286 | ||
287 | thing to test | |
288 | <=== input to give it | |
289 | ???> error to expect | |
290 | ||
291 | Invalid examples: | |
292 | ||
293 | | thing to test | |
294 | ||
295 | ...needs an expectation. | |
296 | ||
297 | + input to give it | |
298 | = output to expect | |
299 | ||
300 | ...test input must be preceded by a test body, if this is the first test. | |
301 | ||
302 | <=== input to give it | |
303 | ???> output to expect | |
304 | ||
305 | ...test input must be preceded by a test body always, in freestyle format. | |
306 | ||
307 | ? error to expect | |
308 | ||
309 | ...expectation must be preceded by either test input or test body. | |
310 | ||
311 | A test body section may also be preceded by a paragraph of text; the | |
312 | intent of the Falderal Literate Test Format is that this text should | |
313 | describe the test, or rather, the aspect of the behaviour of the system | |
314 | that the test is meant to check. It is therefore reasonable that this | |
315 | text should be displayed along with the contents (test body text and | |
316 | expected output or error) of the test, in, for example, a test result | |
317 | report. | |
318 | ||
319 | Discussion | |
320 | ========== | |
321 | ||
322 | (This section is non-normative.) | |
323 | ||
324 | Typically, a file in Falderal Literate Test Format will also be in | |
325 | Markdown format. | |
326 | ||
327 | The format of the lines which comprise the Falderal Literate Test Format | |
328 | was chosen to not conflict with many other common text formats (including | |
329 | but not limited to Bird-style Literate Haskell, and Markdown); thus | |
330 | literate test suites may be embedded in a wide variety of other formats. | |
331 | However, there are inevitably some conflicts with some textual formats; | |
332 | for example, when embedded in C code and many other languages, Falderal | |
333 | entries should be preceded by `/*` and followed by `*/`, to ensure that | |
334 | they are regarded as comments. Also, reStructuredText uses the `|` line | |
335 | prefix to denote preformatted plain text. | |
336 | ||
337 | The format of pragmas was chosen such that they could be read literately, | |
338 | and as such, a formatting tool could format them in the output document | |
339 | with little if any change. |
0 | Falderal Literate Test Format | |
1 | ============================= | |
2 | ||
3 | This document describes the proposed Falderal Literate Test Format. | |
4 | ||
5 | Status | |
6 | ------ | |
7 | ||
8 | This document is a *draft*. It is nominally "version 0.12" because it | |
9 | describes something that version 0.12 of `py-falderal` mostly implements. | |
10 | We will deign to note which sections of this document the current released | |
11 | version of `py-falderal` implements, and which it does not. However, | |
12 | this document is a work in progress, subject to change, and subject to get | |
13 | out of sync with `py-falderal`. It should not be considered to be | |
14 | anything except a draft until it is described as "version 1.0". | |
15 | ||
16 | Overview | |
17 | -------- | |
18 | ||
19 | A Falderal Literate Test Suite is a plain text file where some of the lines | |
20 | have special meaning to the Falderal Literate Test Format. Certain | |
21 | groupings of the lines defined by this format are intended to denote tests, | |
22 | while others denote pragmas which specify how those tests should be run. | |
23 | Outside of these groupings, lines have no meaning to the Falderal Literate | |
24 | Test Format other than that they are presumed to be mainly descriptive text. | |
25 | ||
26 | The plain text file may also be formatted in some other format, such as | |
27 | Markdown, although this is not required. | |
28 | ||
29 | A tool which claims to understand the Falderal Literate Test Format may | |
30 | choose to extract the tests from such a document, run them, and report which | |
31 | and how many of them passed or failed. That would be a typical application | |
32 | for this format. However, no tool is strictly required to do this with the | |
33 | tests found in a document; a tool may, for example, simply reformat the | |
34 | tests to produce an output file in a different format. | |
35 | ||
36 | In the syntax templates below, words in `/slashes/` indicate a variable | |
37 | rather than literal text. | |
38 | ||
39 | Syntax | |
40 | ------ | |
41 | ||
42 | Each grouping of lines which has special meaning to the Falderal Literate | |
43 | Test Format always begins with an indent of four (4) spaces from the | |
44 | leftmost column, preceded by non-indented text, and followed by either | |
45 | non-indented text or the end of the file. Such a grouping of lines is | |
46 | called a _block_. | |
47 | ||
48 | There are two general formats to any block. In the first, "verbose" format, | |
49 | each indent of 4 spaces is followed immediately on that line by distinguished | |
50 | sequence of characters, called an _introducer_. The introducers which have | |
51 | meaning to the Falderal Literate Test Format are as follows: | |
52 | ||
53 | * `->` (hyphen, greater-than sign): pragma | |
54 | * `| ` (vertical bar, space): test body text | |
55 | * `+ ` (plus sign, space): test input text | |
56 | * `= ` (equals sign, space): expected output text | |
57 | * `? ` (question mark, space): expected error text | |
58 | ||
59 | If the same introducer occurs on multiple adjacent lines, the text after | |
60 | each introducer is concatenated to form one chunk of relevant text. This | |
61 | allows, for example, multi-line text to be given as the body, the input, | |
62 | or the expected output of a test. | |
63 | ||
64 | There are some restrictions on the order in which introducers can sensibly | |
65 | occur in a block: | |
66 | ||
67 | * Test body text should occur at the start of a block. | |
68 | * Test body test may be optionally followed by test input text. | |
69 | * The first test input text must be immediately preceded by test body text. | |
70 | * Subsequent test input texts need not be preceded by a test body text; | |
71 | in this case, the previously-appearing test body text will be used again. | |
72 | * Test body text must be followed by either test input text, expected | |
73 | output text, or expected error text, with no intervening text. | |
74 | * Either expected output or error text must follow either test body | |
75 | text or test input text, with no intervening text. | |
76 | ||
77 | See the sections for these introducers, below, for more details. | |
78 | ||
79 | In the other, "freestyle" format, not all lines in a block require | |
80 | introducers. A freestyle format block is indentified as such if one or | |
81 | more of the final lines of the block begin with any of the following | |
82 | introducers: | |
83 | ||
84 | * `=> `: expected output text | |
85 | * `==> `: expected output text | |
86 | * `===> `: expected output text | |
87 | * `?> `: expected error text | |
88 | * `??> `: expected error text | |
89 | * `???> `: expected error text | |
90 | ||
91 | In addition, the following introducers may be used to mark a section | |
92 | of test input text on the first of the final lines (but may not be | |
93 | used to end a block): | |
94 | ||
95 | * `<= `: test input text | |
96 | * `<== `: test input text | |
97 | * `<=== `: test input text | |
98 | ||
99 | If a block is identified as a freestyle block, all lines preceding the | |
100 | first final line appearing with one of these introducers, are interpreted | |
101 | as having no introducer at all (even if they begin with `| ` or some other | |
102 | sequence already mentioned) and are used as the test body block. | |
103 | ||
104 | Lines without introducers are called _intervening text_. | |
105 | Lines of intervening text are classified as either blank or non-blank. A | |
106 | line is blank if it contains no characters, or if it contains only whitespace. | |
107 | A group of non-blank lines is referred to as a _paragraph_. | |
108 | ||
109 | Pragmas | |
110 | ------- | |
111 | ||
112 | ### encoding ### | |
113 | ||
114 | The encoding pragma allows a Falderal file to specify what encoding is | |
115 | used for the characters in it. An implementation of Falderal is not | |
116 | expected to be able to handle any coding other than UTF-8, however, | |
117 | this pragma is included for the benefit of text editors and other tools, | |
118 | to indicate that the document is in fact in UTF-8 encoding. | |
119 | ||
120 | Example: | |
121 | ||
122 | -> encoding: UTF-8 | |
123 | ||
124 | ### Functionality-definition ### | |
125 | ||
126 | The Functionality-definition pragma allows a Falderal file to describe | |
127 | ways in which a functionality being tested is implemented. It has the | |
128 | following syntax: | |
129 | ||
130 | -> Functionality /functionality-name/ is implemented by /functionality-type/ /functionality-specifier/ | |
131 | ||
132 | _functionality-type_ must at present be `shell command`. The format of the | |
133 | _functionality-specifier_ differs according to the functionality-type. | |
134 | The _functionality-name_ is arbitrary text enclosed within double quotes, | |
135 | which may be referenced in a later Tests-for pragma. | |
136 | ||
137 | Note that the Functionality-definitions given in a Falderal file should | |
138 | not be considered exhaustive, or even requisite, by a tool. The tool may | |
139 | accept additional definitions of a functionality, referencing it by its | |
140 | name, from an external source such as the command line or a configuration | |
141 | file, and may be instructed to ignore certain Functionality-definitions in | |
142 | a Falderal file (if, for example, certain implementation are not currently | |
143 | available or of interest to the user.) Indeed, the functionality referred | |
144 | to by a _functionality-name_ in a Tests-for pragma need not be defined by any | |
145 | Functionality-definition pragma in the same Falderal file, and this | |
146 | situation requires the functionality to be specified to the tool in some | |
147 | other manner. | |
148 | ||
149 | #### Shell commands #### | |
150 | ||
151 | For shell commands, the _functionality-specifier_ is in the format | |
152 | `"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may | |
153 | be used, so pipes, redirection, etc., are supported. Note that the double | |
154 | quotation mark characters used to enclosed the command have meaning only to | |
155 | the Falderal format — they are not part of the command, are not passed to the | |
156 | shell, and do not require double quotation mark characters that are enclosed | |
157 | by them to be escaped with a backslash. | |
158 | ||
159 | Certain subsequences, called _variables_, if present in the command string, | |
160 | will be expanded before execution, and will alter how the command reads the | |
161 | text of the test and produces its output, to be compared with the expected | |
162 | output. | |
163 | ||
164 | When a variable is expanded, it is Falderal's responsibility to escape any | |
165 | shell metacharacters that occur as part of the substitution text so that | |
166 | it appears to the shell as a single token, and cannot otherwise affect the | |
167 | meaning of the instantiated shell command. It is not necessary for a | |
168 | variable to be enclosed in any kind of quotes in the command line template. | |
169 | It is also not possible for the substitution text to e.g. expand to two | |
170 | arguments, or to form a pipe or redirection specifier, etc. | |
171 | ||
172 | The exact method by which the Falderal implementation escapes or quotes | |
173 | the substitution text so that it appears unambiguously as a single argument | |
174 | is left up to the implementation. | |
175 | ||
176 | ##### `%(test-body-file)` ##### | |
177 | ||
178 | The variable `%(test-body-file)` will be replaced by the name of a file which | |
179 | contains the text of the test body. This may be a temporary file created | |
180 | solely for this purpose by the Falderal implementation. | |
181 | ||
182 | ##### `%(test-body-text)` ##### | |
183 | ||
184 | The variable `%(test-body-text)` will be replaced by the actual text of the | |
185 | test body. | |
186 | ||
187 | ##### `%(test-input-file)` ##### | |
188 | ||
189 | The variable `%(test-input-file)` will be replaced by the name of a file which | |
190 | contains the text of the test input. This may be a temporary file created | |
191 | solely for this purpose by the Falderal implementation. | |
192 | ||
193 | ##### `%(test-input-text)` ##### | |
194 | ||
195 | The variable `%(test-input-text)` will be replaced by the actual text of the | |
196 | test input. | |
197 | ||
198 | If neither of the variables `%(test-body-file)` nor `%(test-body-text)` appear | |
199 | in the command string, the test body text will be provided on the standard | |
200 | input of the shell command. | |
201 | ||
202 | If neither of the variables `%(test-input-file)` nor `%(test-input-text)` appear | |
203 | in the command string, the test input text will be provided on the standard | |
204 | input of the shell command. | |
205 | ||
206 | If both the test body text and the test input text are slated to appear on the | |
207 | input of the shell command, then the behaviour is (presently) undefined. | |
208 | ||
209 | ##### `%(output-file)` ##### | |
210 | ||
211 | The variable `%(output-file)` will be replaced by the name of a file | |
212 | (temporary file) to which the test results will be written. If it does | |
213 | not appear in the command string, the output text will be read from | |
214 | the standard output of the command. | |
215 | ||
216 | How shell commands support error output is not yet standardized. | |
217 | ||
218 | For example: | |
219 | ||
220 | -> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %(test-file) > %(output-file)" | |
221 | ||
222 | ### Tests-for ### | |
223 | ||
224 | The Tests-for pragma determines what functionality will be used to run all | |
225 | following tests, until the next Tests-for pragma. It has the following | |
226 | syntax: | |
227 | ||
228 | -> Tests for /functionality-name/ | |
229 | ||
230 | The _functionality-name_ refers to a functionality, which may be specified | |
231 | by a Functionality-definition pragma elsewhere in the Falderal file. | |
232 | ||
233 | For example: | |
234 | ||
235 | -> Tests for 'Reversing a string' | |
236 | ||
237 | Test Body, Test Input and Expected Text | |
238 | --------------------------------------- | |
239 | ||
240 | Each section of test body text may or may not be followed by a section of | |
241 | test input text; either way it must then be followed immediately by either | |
242 | and expected output section or expected error section. | |
243 | ||
244 | Valid examples in the "verbose" format: | |
245 | ||
246 | | thing to test | |
247 | = output to expect | |
248 | ||
249 | | thing to test | |
250 | ? error to expect | |
251 | ||
252 | | thing to test | |
253 | + input to give it | |
254 | = output to expect | |
255 | ||
256 | | thing to test | |
257 | + input to give it | |
258 | ? error to expect | |
259 | ||
260 | + different input to give the immediately previously defined test body | |
261 | ? different error to expect | |
262 | ||
263 | Valid examples in the "freestyle" format: | |
264 | ||
265 | thing to test | |
266 | => output to expect | |
267 | ||
268 | thing to test | |
269 | ==> output to expect | |
270 | ||
271 | thing to test | |
272 | ===> output to expect | |
273 | ||
274 | thing to test | |
275 | ?> error to expect | |
276 | ||
277 | thing to test | |
278 | ??> error to expect | |
279 | ||
280 | thing to test | |
281 | ???> error to expect | |
282 | ||
283 | thing to test | |
284 | <=== input to give it | |
285 | ===> output to expect | |
286 | ||
287 | thing to test | |
288 | <=== input to give it | |
289 | ???> error to expect | |
290 | ||
291 | Invalid examples: | |
292 | ||
293 | | thing to test | |
294 | ||
295 | ...needs an expectation. | |
296 | ||
297 | + input to give it | |
298 | = output to expect | |
299 | ||
300 | ...test input must be preceded by a test body, if this is the first test. | |
301 | ||
302 | <=== input to give it | |
303 | ???> output to expect | |
304 | ||
305 | ...test input must be preceded by a test body always, in freestyle format. | |
306 | ||
307 | ? error to expect | |
308 | ||
309 | ...expectation must be preceded by either test input or test body. | |
310 | ||
311 | A test body section may also be preceded by a paragraph of text; the | |
312 | intent of the Falderal Literate Test Format is that this text should | |
313 | describe the test, or rather, the aspect of the behaviour of the system | |
314 | that the test is meant to check. It is therefore reasonable that this | |
315 | text should be displayed along with the contents (test body text and | |
316 | expected output or error) of the test, in, for example, a test result | |
317 | report. | |
318 | ||
319 | Discussion | |
320 | ========== | |
321 | ||
322 | (This section is non-normative.) | |
323 | ||
324 | Typically, a file in Falderal Literate Test Format will also be in | |
325 | Markdown format. | |
326 | ||
327 | The format of the lines which comprise the Falderal Literate Test Format | |
328 | was chosen to not conflict with many other common text formats (including | |
329 | but not limited to Bird-style Literate Haskell, and Markdown); thus | |
330 | literate test suites may be embedded in a wide variety of other formats. | |
331 | However, there are inevitably some conflicts with some textual formats; | |
332 | for example, when embedded in C code and many other languages, Falderal | |
333 | entries should be preceded by `/*` and followed by `*/`, to ensure that | |
334 | they are regarded as comments. Also, reStructuredText uses the `|` line | |
335 | prefix to denote preformatted plain text. | |
336 | ||
337 | The format of pragmas was chosen such that they could be read literately, | |
338 | and as such, a formatting tool could format them in the output document | |
339 | with little if any change. |
0 | `py-falderal` | |
1 | ============= | |
2 | ||
3 | `py-falderal` is an implementation of Falderal in Python 2.5.x. | |
4 | ||
5 | Motivation | |
6 | ---------- | |
7 | ||
8 | The original reference implementation of Falderal was written in Haskell. | |
9 | There are a few reasons I had for re-implementing it in Python: | |
10 | ||
11 | * The original Falderal implementation grew out of a Haskell-specific hack, | |
12 | and it showed in how it was written. | |
13 | ||
14 | * Fewer discrepancies between platforms. In particular, `ghc` for Windows | |
15 | likes to operate in terms of MS-DOS end-of-lines (`CR`, `LF`), but I tend | |
16 | to use it under Cygwin using Unix end-of-lines (`LF`). | |
17 | ||
18 | * Smaller install burden: Python sometimes comes bundled with the operating | |
19 | system; Haskell rarely does. | |
20 | ||
21 | * Haskell, being lazy, makes it harder to deal with exceptions; unless the | |
22 | Haskell expression is being evaluated both strictly and deeply, exceptions | |
23 | can slip by. Haskell's facilities for forcing an expression to be | |
24 | evaluated strictly and deeply are rather hacky (last I checked, the most | |
25 | portable way to do so was to use `show`.) For Falderal's purposes, this | |
26 | limitation seems artificial, at best. | |
27 | ||
28 | * Relatedly, Python (or CPython, anyway) has better error behavior than | |
29 | Haskell (or `ghc`, anyway); when it crashes, it dumps a backtrace (which I | |
30 | can then analyze), instead of just saying something pithy like `Prelude: | |
31 | empty list` (which I can't.) | |
32 | ||
33 | Features | |
34 | -------- | |
35 | ||
36 | `py-falderal` implements a slightly different subset of the | |
37 | [Falderal Literate Test Format](Falderal_Literate_Test_Format.markdown) than | |
38 | the Haskell implementation did. | |
39 | ||
40 | In particular, | |
41 | ||
42 | * It mainly implements `shell command` implementations. In practice, partly | |
43 | due to the "strict & deep" evaluation problem mentioned above, that's how | |
44 | I've been using Falderal anyway; also, its approach makes it somewhat more | |
45 | useful for "end-to-end" testing of compilers and interpreters, than for | |
46 | unit-testing individual text-processing functions inside a program. | |
47 | (Technically, it implements `Haskell function` implementations too, but it's | |
48 | something of a hack that uses `ghc -e`.) | |
49 | ||
50 | * I plan for it to *only* understand indented Falderal blocks. The idea is | |
51 | that the Falderal tests will almost certainly be embedded in a Markdown | |
52 | document (possibly with Bird-style literate code also embedded therein,) | |
53 | and no extra processing should be required to format something readable | |
54 | from that Markdown. The four-space-indented Falderal blocks come out as | |
55 | preformatted blocks, which is quite good enough. Relatedly, | |
56 | ||
57 | * It does no formatting. There are a set of classes for parsing Falderal | |
58 | files (`Document`s) into blocks (`Block`s), and methods for formatting | |
59 | blocks into text. But aside from reporting the results of a test run, | |
60 | they're not used by the utility to format Falderal files. If you want to | |
61 | do extra processing on your Falderal/Markdown file so that Falderal | |
62 | blocks are prettier, you certainly can do that, but you'll need to write | |
63 | your own script which uses these classes, for it is outside the scope of | |
64 | `py-falderal`. | |
65 | ||
66 | * I'm not so sure about `-> Functionality "blah" is implemented by shell | |
67 | command "flargh"` anymore. I mean, it should certainly be the case that | |
68 | functionalities can have multiple implementations, but | |
69 | ||
70 | * Perhaps implementations should not be specified in Falderal documents | |
71 | at all — that approach is more abstract. But it requires them to be | |
72 | specified on the tool's command line, which in practice requires there | |
73 | to be a driver script to run the tests, for a particular implementation; | |
74 | but this is not necessarily a bad thing. (The effective practice has | |
75 | been, actually, to write out a bit of "configuration Markdown" with | |
76 | that "Functionality is implemented by" snippet in it, and to load that | |
77 | before loading the main tests. Rather than to configure the functionality | |
78 | implementation(s) on the command line itself.) | |
79 | * When `falderal` is run on more than one input file, what is the scope | |
80 | of a functionality, and what is the scope of the implementations of a | |
81 | functionality? Currently, in the Falderal semantics, that scope is | |
82 | global, and maybe that is appropriate; but maybe finer-grained control | |
83 | would be nice. | |
84 | ||
85 | * `py-falderal` also does not try to optimize the test runs into a block of | |
86 | test runs (which didn't work out so well in the Haskell implementation, | |
87 | for `shell command`s, anyway.) |
0 | `py-falderal` | |
1 | ============= | |
2 | ||
3 | `py-falderal` is an implementation of Falderal in Python 2.5.x. | |
4 | ||
5 | Motivation | |
6 | ---------- | |
7 | ||
8 | The original reference implementation of Falderal was written in Haskell. | |
9 | There are a few reasons I had for re-implementing it in Python: | |
10 | ||
11 | * The original Falderal implementation grew out of a Haskell-specific hack, | |
12 | and it showed in how it was written. | |
13 | ||
14 | * Fewer discrepancies between platforms. In particular, `ghc` for Windows | |
15 | likes to operate in terms of MS-DOS end-of-lines (`CR`, `LF`), but I tend | |
16 | to use it under Cygwin using Unix end-of-lines (`LF`). | |
17 | ||
18 | * Smaller install burden: Python sometimes comes bundled with the operating | |
19 | system; Haskell rarely does. | |
20 | ||
21 | * Haskell, being lazy, makes it harder to deal with exceptions; unless the | |
22 | Haskell expression is being evaluated both strictly and deeply, exceptions | |
23 | can slip by. Haskell's facilities for forcing an expression to be | |
24 | evaluated strictly and deeply are rather hacky (last I checked, the most | |
25 | portable way to do so was to use `show`.) For Falderal's purposes, this | |
26 | limitation seems artificial, at best. | |
27 | ||
28 | * Relatedly, Python (or CPython, anyway) has better error behavior than | |
29 | Haskell (or `ghc`, anyway); when it crashes, it dumps a backtrace (which I | |
30 | can then analyze), instead of just saying something pithy like `Prelude: | |
31 | empty list` (which I can't.) | |
32 | ||
33 | Features | |
34 | -------- | |
35 | ||
36 | `py-falderal` implements a slightly different subset of the | |
37 | [Falderal Literate Test Format](Falderal_Literate_Test_Format.markdown) than | |
38 | the Haskell implementation did. | |
39 | ||
40 | In particular, | |
41 | ||
42 | * It mainly implements `shell command` implementations. In practice, partly | |
43 | due to the "strict & deep" evaluation problem mentioned above, that's how | |
44 | I've been using Falderal anyway; also, its approach makes it somewhat more | |
45 | useful for "end-to-end" testing of compilers and interpreters, than for | |
46 | unit-testing individual text-processing functions inside a program. | |
47 | (Technically, it implements `Haskell function` implementations too, but it's | |
48 | something of a hack that uses `ghc -e`.) | |
49 | ||
50 | * I plan for it to *only* understand indented Falderal blocks. The idea is | |
51 | that the Falderal tests will almost certainly be embedded in a Markdown | |
52 | document (possibly with Bird-style literate code also embedded therein,) | |
53 | and no extra processing should be required to format something readable | |
54 | from that Markdown. The four-space-indented Falderal blocks come out as | |
55 | preformatted blocks, which is quite good enough. Relatedly, | |
56 | ||
57 | * It does no formatting. There are a set of classes for parsing Falderal | |
58 | files (`Document`s) into blocks (`Block`s), and methods for formatting | |
59 | blocks into text. But aside from reporting the results of a test run, | |
60 | they're not used by the utility to format Falderal files. If you want to | |
61 | do extra processing on your Falderal/Markdown file so that Falderal | |
62 | blocks are prettier, you certainly can do that, but you'll need to write | |
63 | your own script which uses these classes, for it is outside the scope of | |
64 | `py-falderal`. | |
65 | ||
66 | * I'm not so sure about `-> Functionality "blah" is implemented by shell | |
67 | command "flargh"` anymore. I mean, it should certainly be the case that | |
68 | functionalities can have multiple implementations, but | |
69 | ||
70 | * Perhaps implementations should not be specified in Falderal documents | |
71 | at all — that approach is more abstract. But it requires them to be | |
72 | specified on the tool's command line, which in practice requires there | |
73 | to be a driver script to run the tests, for a particular implementation; | |
74 | but this is not necessarily a bad thing. (The effective practice has | |
75 | been, actually, to write out a bit of "configuration Markdown" with | |
76 | that "Functionality is implemented by" snippet in it, and to load that | |
77 | before loading the main tests. Rather than to configure the functionality | |
78 | implementation(s) on the command line itself.) | |
79 | * When `falderal` is run on more than one input file, what is the scope | |
80 | of a functionality, and what is the scope of the implementations of a | |
81 | functionality? Currently, in the Falderal semantics, that scope is | |
82 | global, and maybe that is appropriate; but maybe finer-grained control | |
83 | would be nice. | |
84 | ||
85 | * `py-falderal` also does not try to optimize the test runs into a block of | |
86 | test runs (which didn't work out so well in the Haskell implementation, | |
87 | for `shell command`s, anyway.) |
0 | 0 | import setuptools |
1 | 1 | |
2 | 2 | |
3 | with open("README.markdown", "r") as fh: | |
3 | with open("README.md", "r") as fh: | |
4 | 4 | long_description = fh.read() |
5 | 5 | |
6 | 6 | setuptools.setup( |
7 | 7 | name='Falderal', |
8 | version='0.13', | |
8 | version='0.14', | |
9 | 9 | description='Definition of, and tools for using, the Falderal literate testing format', |
10 | 10 | long_description=long_description, |
11 | 11 | long_description_content_type="text/markdown", |