git @ Cat's Eye Technologies Falderal / 6dab138
Use .md for file extension for Markdown files. Other cleanups. Chris Pressey 6 months ago
13 changed file(s) with 1109 addition(s) and 1130 deletion(s). Raw diff Collapse all Expand all
+0
-10
.hgignore less more
0 syntax: regexp
1
2 ^impl/Test.Falderal/bin/
3
4 syntax: glob
5
6 *.o
7 *.hi
8 *.pyc
9 *.actual
+0
-15
.hgtags less more
0 e0f07b3699cd2b5029dc0e619e30f2897e27564a rel_0_1
1 b5b728aeba88946da3e6ae6ad0ffbc3eac53dd6a rel_0_2
2 582e124855570072a933f14a89ba7ad74632610b rel_0_3
3 582e124855570072a933f14a89ba7ad74632610b rel_0_3
4 8bb4fad90030edb2ac4f7b113c3bf0a361f9274a rel_0_3
5 046f8ec1388ab05e986febbc7c9d67c636076f53 rel_0_4
6 cb2caf712180eb0fd3d759f9fcbf5cef19680675 rel_0_5
7 6f9b761233f86c8a416a7b13599ebf9f50f005b3 rel_0_6
8 5ce741f7daf73c8b0a37daa48ea298b12e3f36cb rel_0_7
9 0397d7575a05550ca8d137baeeb5ea984d0379a3 rel_0_8
10 7575a60f38d4fcf1984e892cd301b4b99faf9606 rel_0_9_2013_1029
11 00acc0048e83c3313ad740b90ce807de115bb21c rel_0_9_2014_0427
12 edfd7af3d505a76b08aebc0af537ea7e5fdeb31d rel_0_9_2014_0525
13 0ff6e40b2c7e6464993fb792f3a3be1c94a6dd2e rel_0_10
14 0205fd963df5b9d5d7306ea40696c593eb3e38ce rel_0_11
+0
-304
HISTORY.markdown less more
0 History
1 =======
2
3 Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive":
4
5 * (planned) Add a pragma "Functionality ... is available if
6 shell command ... can be successfully run". If the command
7 cannot be successfully run, disable tests for the given
8 functionality. (If no tests at all are run, it is an error.)
9 * (planned) Add a command-line option to require that the
10 version number of Falderal is at least a given version number.
11 * `falderal`'s own test suite, `test.sh`, tests running `falderal`
12 under Python 2 if it is available, and under Python 3 if it is
13 available. If neither is available, it aborts. A particular
14 implementation of Python can also be supplied in the `PYTHON`
15 environment variable.
16
17 Version 0.13 "Merchandise Mart":
18
19 * Use `ArgumentParser` instead of `OptionParser` internally.
20 * `--substring-error` is now the default behaviour, and if this
21 option is given, it will be ignored. After a long time
22 considering the problem, we finally determined that it does
23 not make sense to insist that different implementations
24 produce the exact same error message.
25 * `--verbose` option now prints out commands for each test as it
26 is run.
27 * Added conventional `--version` option.
28 * Planned to be the last 0.x series release before 1.0.
29 * Added `setup.py` so that Falderal can be treated as a Python
30 package.
31
32 Version 0.12 "Schoenhofen Brewery":
33
34 * When expanding variables in declaration strings, use
35 `string.replace` instead of `re.sub` so that backslash-escaping
36 is not perfomed on the replacement string. (Thanks to
37 James Holderness for bug report and patch.)
38 * In "freestyle" format, lines beginning with `<= `, `<== `, or
39 `<=== ` can supply a section of test input text (Thanks to
40 James Holderness for feature suggestion and patch.)
41 * Falderal is now responsible for quoting all shell metacharacters
42 in the substituion text of `%(...)` variables when command line
43 templates are expanded, so that no quotes need be placed around
44 the variable in the template. (Again, thanks to James Holderness
45 for spotting the bug in the code that led to this improvement.)
46 * `py-falderal` can now run under both Python 2 and Python 3
47 (tested with 2.7, 3.4, and 3.5.)
48 * The doctests that were previously distributed throughout the
49 source code have been converted to a suite of unit tests in
50 the `falderal.tests` module. The `-t` command-line option has
51 been removed; to run internal tests, run this module instead.
52
53 Version 0.11 "Dan Ryan Expressway":
54
55 * Renamed the internal tests to have descriptive names instead of
56 numbers.
57 * The first command-line argument being `test` for compatibility with
58 previous versions of `falderal` has been removed.
59 * Massive internal refactor of block-parsing and test-extraction code.
60 * Supports a new format for test blocks, called "freestyle", where if
61 the last few lines of a block are prefixed with `=> ` or `==> ` or
62 `===> ` or `?> ` or `??> ` or `???> `, the preceding part of the
63 block needs no prefixes at all, and it is simply treated as one
64 verbatim test body.
65
66 Version 0.10 "Shedd Aquarium":
67
68 * Removed the old, bitrotting Haskell implementation.
69 * Removed deprecated `%(test-text)` and `%(test-file)` variables
70 (`%(test-body-text)` and `%(test-body-file)` should be used now.)
71 * Added some adapters in the `script/` directory of the repo:
72 * `fa-under-pty` runs a target program under a pseudoterminal
73 * `fa-bin-to-hex` converts 8-bit binary output to hexadecimal codes
74 * `fa-hex-to-bin` converts hexadecimal codes to 8-bit binary output
75 * Added the rule that a test may consist of an input section (`+`) if
76 there was a previous test, and the test body of the previous test will
77 be re-used with the new input in the new test.
78
79 Version 0.9-2014.0525 "Municipal Pier #2, May 25th 2014":
80
81 * Shell implementations assume input and output is encoded UTF-8; any
82 malformed UTF-8 is simply ignored (for better or worse, for now.)
83 Falderal documents are assumed to be encoded in UTF-8; this includes
84 test bodies, expectations, etc. Internally, in the reference
85 implementation, all strings are stored as Unicode.
86
87 Version 0.9-2014.0427 "Municipal Pier #2":
88
89 * For a test of a shell command implementation that expects a failure,
90 if no output was found on standard error, the output of standard output
91 is checked against the failure expectation instead. The primary use case
92 for this is Hugs, which (unlike `ghc`) displays error messages on stdout
93 rather than stderr. "Municipal Pier #2" was the original name of Navy Pier.
94
95 Version 0.9 "Navy Pier":
96
97 * `py-falderal` now insists that blocks are indented four spaces, as the
98 specification says. It ignores blocks that are not indented.
99 * In an error report `py-falderal` reports the location (filename and line
100 number of the beginning of the test text) of each test that failed, as
101 well as the name of the functionality that was being tested.
102 * Several internal renamings of class and variable names for greater clarity.
103 * Trying to standardize on the terminology "test body" for the body of a
104 test (as opposed to its "input", and opposed to it being passed to a shell
105 command line "as text".)
106 * Support for specifying a block of text to be used as input for a test.
107 * Dropped support for "implemented by Haskell function" backwards-compatibility.
108 * Temporary files are no longer always created in the current directory. That
109 was a workaround to support `ghc` from Haskell Platform under Cygwin. They
110 are now created where the system wants them to be created (but you can set
111 the `TMP` environment variable to "." if you really want them to be created
112 in the current directory.)
113 * `py-falderal` now complains and stops early if there were no documents
114 specified, no functionality definitions were found in any of the documents,
115 no implementations were found for some defined functionality, or if no
116 tests were found in any of the documents, unless `--cavalier` is passed on
117 the command line.
118 * In `--verbose` mode, `py-falderal` will issue warnings about any test that
119 has been run twice (i.e. more than run on identical test body text and
120 implementation.)
121 * If `py-falderal` catches a `KeyboardInterrupt` exception while running tests,
122 it shows a report of all the tests that it did manage to run until that
123 point, and a warning message to the effect that not all tests were run.
124
125 Version 0.8 "Ukrainian Village":
126
127 * The reference implementation is now `py-falderal`.
128 * Blocks *must* now be indented four spaces. That Falderal test documents
129 will be embedded in Markdown is generally assumed (although not strictly
130 required) and they generally have the file extension `.markdown` so that
131 other software (e.g. Bitbucket and Github) knows how to pretty-display
132 them.
133 * Various documentation which was on Bitbucket (wiki, issues) and Github
134 (py-falderal issues) have been moved into Markdown documents in the
135 distribution, which have themselves been re-organized.
136 * Added the `-v`/`--verbose` flag and made it and the `-d`/`--dump` flag
137 do what you would expect from them.
138 * The `-f`/`--functionalities` and `-c`/`--clear-functionalities` flags
139 have been *removed*. The rationale is that implementations of
140 functionalities are clumsy to express on the command line (and anyway
141 the syntax would have to be different) while it is not difficult to
142 create a "configuration" Falderal file which specifies *only* the
143 desired implementations of the desired functionalities, and to load
144 it just before loading a "tests" Falderal file which specified the
145 functionalities being tested, but not their implementations.
146
147 Version 0.7 "Ogden Avenue":
148
149 * **Note that this is the final version of Falderal where `Test.Falderal`
150 is the reference implementation.** In subsequent versions,
151 `py-falderal` (written in Python) will be the reference implementation.
152 `Test.Falderal` will still be in this repository (under the `impl`
153 directory) but will no longer be definitive, and will go essentially
154 unmaintained.
155 * Fixed a bug where number of lines in the intermediary results file
156 was sometimes being counted incorrect.
157 * Slightly improved support for running under Cygwin.
158 * Tests that are implemented by a shell command are now run directly
159 from the `falderal` process (instead of creating an intermediary
160 shell script and running it.) This resulted in better performance.
161 * Addition of `-b` command-line option, which considers a test to
162 have passed if the expected exception message is a substring (rather
163 than an exact match) of the actual produced exception message.
164 TODO: write test for this.
165 * Blocks may now be indented four spaces, to allow them to be
166 embedded directly in Markdown files as pre-formatted text.
167 * A script to just build the binary, without installing it, was added.
168
169 Version 0.6 "Streeterville":
170
171 * Variables may be included in the specification of a shell command
172 implementation; these will be expanded before generating the results
173 generator. These variable include `%(test-text)`, `%(test-file)`,
174 and `%(output-file)`.
175 * `falderal` now lets the user specify, on the command line, the
176 implementations for a named functionality. Supplying `-f 'foo:shell
177 command "foo.sh %(test-file)"'` has the same effect as including the
178 pragma `Functionality "foo" is implemented by shell command "foo.sh
179 %(test-file)"` in the Falderal file. Note that this is in addition
180 to the Functionality-definition pragmas given in the Falderal file;
181 to replace them, you must first clear the ones with that name from
182 the file by supplying `-c foo` on the command line.
183 * `falderal` also allows tests for named functionalities to be skipped
184 completely, by passing the name of the functionality to be skipped
185 after a `-k` flag on the command line.
186 * The Markdown formatter now formats Bird-style embedded code with
187 HTML embedded in the Markdown document. This is so that it can
188 be styled independently from, and thus distinguished from, any plain
189 Markdown indented code blocks which may appear in the literate portion
190 of the source code.
191 * In failure reports, the implementation of the functionality of the test
192 that failed is now reported in each failure.
193 * A race condition(?) that could occur when testing multiple implementations
194 of a functionality, of different kinds (Haskell and shell), has been
195 prevented. Both tests were writing to `results.txt` and immediately
196 deleting it, and this would sometimes confuse `falderal` into thinking
197 one had produced no results (perhaps a result of some creative
198 scheduling by `ghc`, although really, I haven't a clue.) Results are
199 now written to different temporary files with different, generated
200 names.
201 * Previously, if the output of a shell command being tested did not end
202 with a newline, the intermediate results file was not being generated
203 correctly, resulting in failures being misreported. This has been
204 recitified.
205 * Previously, if there were tests given in a Falderal file before any
206 Tests-for pragma was specified, those tests would just be ignored.
207 An error message is now issued, and no testing takes place.
208 * Previously, if multiple Falderal files were given on the command
209 line, they were simply concatenated when loaded, the result being
210 that Functionality-definitions from the first file were visible in
211 the second file, and that any Tests-for in effect at the end of the
212 first file would be in effect at the start of the second file.
213 Files are now loaded and processed seperately.
214
215 Version 0.5 "The Loop":
216
217 * The command-line format of `falderal` has changed, for the better.
218 The `test` subcommand no longer requires that the failure reporting
219 style be specified; instead, it defaults to `standard`, and can be
220 changed with a command-line option. There are also command-line
221 options for selecting the programs to run results generators, and to
222 keep generated files around instead of cleaning them up after testing.
223 * The dependency on `ghc` for running Haskell results generators has been
224 removed; these can be run by `runhaskell` now, and are run by
225 `runhaskell` by default.
226 * Failure reporting is now consistent across languages; both Haskell and
227 Bourne shell results generators generate an intermediate format, which
228 `falderal` digests.
229 * A new pragma `encoding:` was added, so that this directive can be
230 embedded in your Falderal document (for the benefit of your text editor)
231 without necessarily appearing in the formatted document.
232 * We began giving release milestones colorful names. The naming
233 convention is to choose names of Chicagoland neigborhoods, suburbs,
234 landmarks, and institutions. Version 0.5 was named after The Loop in
235 recognition of its ability to shuttle test results between `falderal`
236 and the various results generators implemented in different languages.
237 Previous versions of `Test.Falderal` were retroactively given milestone
238 names during this release.
239
240 Version 0.4 "Blackstone Hotel":
241
242 * For ease of installation, the project is presented as a Cabal package.
243 * A driver executable, `falderal`, is built as part of installing the
244 Cabal package. It provides a command-line interface to formatting
245 Falderal files and, in a limited fashion, running the tests in them.
246 * A shell script formatter has been written, enabling testing of shell
247 commands. One caveat is that reporting for these tests is nowhere near
248 as nice as for Haskell functions, but that will change in the next version.
249 * The Functionality-definition pragma has been implemented, making it
250 possible to write tests in a more abstract, implementation-independent
251 fashion.
252 * Falderal files written to work with Falderal 0.3 should still work with
253 Falderal 0.4, but you are encouraged to use the Functionality-definition
254 pragma introduced in 0.4 to make your tests more implementation-independent.
255
256 Version 0.3 "Chicago Board of Trade":
257
258 * The definition of a Falderal Literate Test Format, distinct from the
259 reference implementation of tools for it in Haskell (`Test.Falderal`).
260 This represented a fairly substantial departure from how previous versions
261 of Falderal worked.
262 * The ability to format a Falderal file to different formats, including
263 Markdown and Haskell.
264 * Running tests is now a matter of formatting a Falderal file as a Haskell
265 module and running the function `testModule` in that module.
266
267 Version 0.2 "Dearborn Station":
268
269 * Added a test harness for Falderal itself, in the form of a simple shell
270 script which diffs the output of `Test.Falderal.Demo` against a text file
271 containing the expected content.
272 * Improved formatting of failure reports. Multi-line input text or expected
273 output is always presented starting on its own line.
274 * Tests may be organized into groups; the entire group is preceded by some
275 literal text, but there is no literal text between the tests in the group.
276 When one of these tests fails, the literal text for the group is reported,
277 along with the number of the test within the group.
278 * Fixed a bug where exception text which extended over multiple lines
279 could not be expected correctly.
280
281 Version 0.1 "Haymarket Square":
282
283 * Provision of a framework for writing and running literate tests which may
284 be embedded in literate Haskell source code.
285 * Testing Haskell functions of type `String -> String`. A test specifies input
286 to the function, and may expect a particular output, or that a particular
287 exception is thrown.
288 * Through simple adapters, testing functions of other types such as
289 `(Show a) => String -> a`.
290
291 Prehistory:
292
293 Falderal started life as a Haskell-specific hack that could be embedded
294 in a Bird-style Literate Haskell source file. I took a framework for
295 literate tests I had already written in a project called _Rho_, and used it
296 as the basis of this code.
297
298 I wrote it because I was designing yet another esoteric programming
299 language, and while working on it I realized I was rebuilding yet another
300 ad-hoc unit test suite, like I had done a half-dozen times before. I didn't
301 want to keep doing this for every language I designed, and I realized that
302 literate test suites could serve as documentation as well; the result was
303 Falderal.
0 History
1 =======
2
3 Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive":
4
5 * (planned) Add a pragma "Functionality ... is available if
6 shell command ... can be successfully run". If the command
7 cannot be successfully run, disable tests for the given
8 functionality. (If no tests at all are run, it is an error.)
9 * (planned) Add a command-line option to require that the
10 version number of Falderal is at least a given version number.
11 * `falderal`'s own test suite, `test.sh`, tests running `falderal`
12 under Python 2 if it is available, and under Python 3 if it is
13 available. If neither is available, it aborts. A particular
14 implementation of Python can also be supplied in the `PYTHON`
15 environment variable.
16
17 Version 0.13 "Merchandise Mart":
18
19 * Use `ArgumentParser` instead of `OptionParser` internally.
20 * `--substring-error` is now the default behaviour, and if this
21 option is given, it will be ignored. After a long time
22 considering the problem, we finally determined that it does
23 not make sense to insist that different implementations
24 produce the exact same error message.
25 * `--verbose` option now prints out commands for each test as it
26 is run.
27 * Added conventional `--version` option.
28 * Planned to be the last 0.x series release before 1.0.
29 * Added `setup.py` so that Falderal can be treated as a Python
30 package.
31
32 Version 0.12 "Schoenhofen Brewery":
33
34 * When expanding variables in declaration strings, use
35 `string.replace` instead of `re.sub` so that backslash-escaping
36 is not perfomed on the replacement string. (Thanks to
37 James Holderness for bug report and patch.)
38 * In "freestyle" format, lines beginning with `<= `, `<== `, or
39 `<=== ` can supply a section of test input text (Thanks to
40 James Holderness for feature suggestion and patch.)
41 * Falderal is now responsible for quoting all shell metacharacters
42 in the substituion text of `%(...)` variables when command line
43 templates are expanded, so that no quotes need be placed around
44 the variable in the template. (Again, thanks to James Holderness
45 for spotting the bug in the code that led to this improvement.)
46 * `py-falderal` can now run under both Python 2 and Python 3
47 (tested with 2.7, 3.4, and 3.5.)
48 * The doctests that were previously distributed throughout the
49 source code have been converted to a suite of unit tests in
50 the `falderal.tests` module. The `-t` command-line option has
51 been removed; to run internal tests, run this module instead.
52
53 Version 0.11 "Dan Ryan Expressway":
54
55 * Renamed the internal tests to have descriptive names instead of
56 numbers.
57 * The first command-line argument being `test` for compatibility with
58 previous versions of `falderal` has been removed.
59 * Massive internal refactor of block-parsing and test-extraction code.
60 * Supports a new format for test blocks, called "freestyle", where if
61 the last few lines of a block are prefixed with `=> ` or `==> ` or
62 `===> ` or `?> ` or `??> ` or `???> `, the preceding part of the
63 block needs no prefixes at all, and it is simply treated as one
64 verbatim test body.
65
66 Version 0.10 "Shedd Aquarium":
67
68 * Removed the old, bitrotting Haskell implementation.
69 * Removed deprecated `%(test-text)` and `%(test-file)` variables
70 (`%(test-body-text)` and `%(test-body-file)` should be used now.)
71 * Added some adapters in the `script/` directory of the repo:
72 * `fa-under-pty` runs a target program under a pseudoterminal
73 * `fa-bin-to-hex` converts 8-bit binary output to hexadecimal codes
74 * `fa-hex-to-bin` converts hexadecimal codes to 8-bit binary output
75 * Added the rule that a test may consist of an input section (`+`) if
76 there was a previous test, and the test body of the previous test will
77 be re-used with the new input in the new test.
78
79 Version 0.9-2014.0525 "Municipal Pier #2, May 25th 2014":
80
81 * Shell implementations assume input and output is encoded UTF-8; any
82 malformed UTF-8 is simply ignored (for better or worse, for now.)
83 Falderal documents are assumed to be encoded in UTF-8; this includes
84 test bodies, expectations, etc. Internally, in the reference
85 implementation, all strings are stored as Unicode.
86
87 Version 0.9-2014.0427 "Municipal Pier #2":
88
89 * For a test of a shell command implementation that expects a failure,
90 if no output was found on standard error, the output of standard output
91 is checked against the failure expectation instead. The primary use case
92 for this is Hugs, which (unlike `ghc`) displays error messages on stdout
93 rather than stderr. "Municipal Pier #2" was the original name of Navy Pier.
94
95 Version 0.9 "Navy Pier":
96
97 * `py-falderal` now insists that blocks are indented four spaces, as the
98 specification says. It ignores blocks that are not indented.
99 * In an error report `py-falderal` reports the location (filename and line
100 number of the beginning of the test text) of each test that failed, as
101 well as the name of the functionality that was being tested.
102 * Several internal renamings of class and variable names for greater clarity.
103 * Trying to standardize on the terminology "test body" for the body of a
104 test (as opposed to its "input", and opposed to it being passed to a shell
105 command line "as text".)
106 * Support for specifying a block of text to be used as input for a test.
107 * Dropped support for "implemented by Haskell function" backwards-compatibility.
108 * Temporary files are no longer always created in the current directory. That
109 was a workaround to support `ghc` from Haskell Platform under Cygwin. They
110 are now created where the system wants them to be created (but you can set
111 the `TMP` environment variable to "." if you really want them to be created
112 in the current directory.)
113 * `py-falderal` now complains and stops early if there were no documents
114 specified, no functionality definitions were found in any of the documents,
115 no implementations were found for some defined functionality, or if no
116 tests were found in any of the documents, unless `--cavalier` is passed on
117 the command line.
118 * In `--verbose` mode, `py-falderal` will issue warnings about any test that
119 has been run twice (i.e. more than run on identical test body text and
120 implementation.)
121 * If `py-falderal` catches a `KeyboardInterrupt` exception while running tests,
122 it shows a report of all the tests that it did manage to run until that
123 point, and a warning message to the effect that not all tests were run.
124
125 Version 0.8 "Ukrainian Village":
126
127 * The reference implementation is now `py-falderal`.
128 * Blocks *must* now be indented four spaces. That Falderal test documents
129 will be embedded in Markdown is generally assumed (although not strictly
130 required) and they generally have the file extension `.markdown` so that
131 other software (e.g. Bitbucket and Github) knows how to pretty-display
132 them.
133 * Various documentation which was on Bitbucket (wiki, issues) and Github
134 (py-falderal issues) have been moved into Markdown documents in the
135 distribution, which have themselves been re-organized.
136 * Added the `-v`/`--verbose` flag and made it and the `-d`/`--dump` flag
137 do what you would expect from them.
138 * The `-f`/`--functionalities` and `-c`/`--clear-functionalities` flags
139 have been *removed*. The rationale is that implementations of
140 functionalities are clumsy to express on the command line (and anyway
141 the syntax would have to be different) while it is not difficult to
142 create a "configuration" Falderal file which specifies *only* the
143 desired implementations of the desired functionalities, and to load
144 it just before loading a "tests" Falderal file which specified the
145 functionalities being tested, but not their implementations.
146
147 Version 0.7 "Ogden Avenue":
148
149 * **Note that this is the final version of Falderal where `Test.Falderal`
150 is the reference implementation.** In subsequent versions,
151 `py-falderal` (written in Python) will be the reference implementation.
152 `Test.Falderal` will still be in this repository (under the `impl`
153 directory) but will no longer be definitive, and will go essentially
154 unmaintained.
155 * Fixed a bug where number of lines in the intermediary results file
156 was sometimes being counted incorrect.
157 * Slightly improved support for running under Cygwin.
158 * Tests that are implemented by a shell command are now run directly
159 from the `falderal` process (instead of creating an intermediary
160 shell script and running it.) This resulted in better performance.
161 * Addition of `-b` command-line option, which considers a test to
162 have passed if the expected exception message is a substring (rather
163 than an exact match) of the actual produced exception message.
164 TODO: write test for this.
165 * Blocks may now be indented four spaces, to allow them to be
166 embedded directly in Markdown files as pre-formatted text.
167 * A script to just build the binary, without installing it, was added.
168
169 Version 0.6 "Streeterville":
170
171 * Variables may be included in the specification of a shell command
172 implementation; these will be expanded before generating the results
173 generator. These variable include `%(test-text)`, `%(test-file)`,
174 and `%(output-file)`.
175 * `falderal` now lets the user specify, on the command line, the
176 implementations for a named functionality. Supplying `-f 'foo:shell
177 command "foo.sh %(test-file)"'` has the same effect as including the
178 pragma `Functionality "foo" is implemented by shell command "foo.sh
179 %(test-file)"` in the Falderal file. Note that this is in addition
180 to the Functionality-definition pragmas given in the Falderal file;
181 to replace them, you must first clear the ones with that name from
182 the file by supplying `-c foo` on the command line.
183 * `falderal` also allows tests for named functionalities to be skipped
184 completely, by passing the name of the functionality to be skipped
185 after a `-k` flag on the command line.
186 * The Markdown formatter now formats Bird-style embedded code with
187 HTML embedded in the Markdown document. This is so that it can
188 be styled independently from, and thus distinguished from, any plain
189 Markdown indented code blocks which may appear in the literate portion
190 of the source code.
191 * In failure reports, the implementation of the functionality of the test
192 that failed is now reported in each failure.
193 * A race condition(?) that could occur when testing multiple implementations
194 of a functionality, of different kinds (Haskell and shell), has been
195 prevented. Both tests were writing to `results.txt` and immediately
196 deleting it, and this would sometimes confuse `falderal` into thinking
197 one had produced no results (perhaps a result of some creative
198 scheduling by `ghc`, although really, I haven't a clue.) Results are
199 now written to different temporary files with different, generated
200 names.
201 * Previously, if the output of a shell command being tested did not end
202 with a newline, the intermediate results file was not being generated
203 correctly, resulting in failures being misreported. This has been
204 recitified.
205 * Previously, if there were tests given in a Falderal file before any
206 Tests-for pragma was specified, those tests would just be ignored.
207 An error message is now issued, and no testing takes place.
208 * Previously, if multiple Falderal files were given on the command
209 line, they were simply concatenated when loaded, the result being
210 that Functionality-definitions from the first file were visible in
211 the second file, and that any Tests-for in effect at the end of the
212 first file would be in effect at the start of the second file.
213 Files are now loaded and processed seperately.
214
215 Version 0.5 "The Loop":
216
217 * The command-line format of `falderal` has changed, for the better.
218 The `test` subcommand no longer requires that the failure reporting
219 style be specified; instead, it defaults to `standard`, and can be
220 changed with a command-line option. There are also command-line
221 options for selecting the programs to run results generators, and to
222 keep generated files around instead of cleaning them up after testing.
223 * The dependency on `ghc` for running Haskell results generators has been
224 removed; these can be run by `runhaskell` now, and are run by
225 `runhaskell` by default.
226 * Failure reporting is now consistent across languages; both Haskell and
227 Bourne shell results generators generate an intermediate format, which
228 `falderal` digests.
229 * A new pragma `encoding:` was added, so that this directive can be
230 embedded in your Falderal document (for the benefit of your text editor)
231 without necessarily appearing in the formatted document.
232 * We began giving release milestones colorful names. The naming
233 convention is to choose names of Chicagoland neigborhoods, suburbs,
234 landmarks, and institutions. Version 0.5 was named after The Loop in
235 recognition of its ability to shuttle test results between `falderal`
236 and the various results generators implemented in different languages.
237 Previous versions of `Test.Falderal` were retroactively given milestone
238 names during this release.
239
240 Version 0.4 "Blackstone Hotel":
241
242 * For ease of installation, the project is presented as a Cabal package.
243 * A driver executable, `falderal`, is built as part of installing the
244 Cabal package. It provides a command-line interface to formatting
245 Falderal files and, in a limited fashion, running the tests in them.
246 * A shell script formatter has been written, enabling testing of shell
247 commands. One caveat is that reporting for these tests is nowhere near
248 as nice as for Haskell functions, but that will change in the next version.
249 * The Functionality-definition pragma has been implemented, making it
250 possible to write tests in a more abstract, implementation-independent
251 fashion.
252 * Falderal files written to work with Falderal 0.3 should still work with
253 Falderal 0.4, but you are encouraged to use the Functionality-definition
254 pragma introduced in 0.4 to make your tests more implementation-independent.
255
256 Version 0.3 "Chicago Board of Trade":
257
258 * The definition of a Falderal Literate Test Format, distinct from the
259 reference implementation of tools for it in Haskell (`Test.Falderal`).
260 This represented a fairly substantial departure from how previous versions
261 of Falderal worked.
262 * The ability to format a Falderal file to different formats, including
263 Markdown and Haskell.
264 * Running tests is now a matter of formatting a Falderal file as a Haskell
265 module and running the function `testModule` in that module.
266
267 Version 0.2 "Dearborn Station":
268
269 * Added a test harness for Falderal itself, in the form of a simple shell
270 script which diffs the output of `Test.Falderal.Demo` against a text file
271 containing the expected content.
272 * Improved formatting of failure reports. Multi-line input text or expected
273 output is always presented starting on its own line.
274 * Tests may be organized into groups; the entire group is preceded by some
275 literal text, but there is no literal text between the tests in the group.
276 When one of these tests fails, the literal text for the group is reported,
277 along with the number of the test within the group.
278 * Fixed a bug where exception text which extended over multiple lines
279 could not be expected correctly.
280
281 Version 0.1 "Haymarket Square":
282
283 * Provision of a framework for writing and running literate tests which may
284 be embedded in literate Haskell source code.
285 * Testing Haskell functions of type `String -> String`. A test specifies input
286 to the function, and may expect a particular output, or that a particular
287 exception is thrown.
288 * Through simple adapters, testing functions of other types such as
289 `(Show a) => String -> a`.
290
291 Prehistory:
292
293 Falderal started life as a Haskell-specific hack that could be embedded
294 in a Bird-style Literate Haskell source file. I took a framework for
295 literate tests I had already written in a project called _Rho_, and used it
296 as the basis of this code.
297
298 I wrote it because I was designing yet another esoteric programming
299 language, and while working on it I realized I was rebuilding yet another
300 ad-hoc unit test suite, like I had done a half-dozen times before. I didn't
301 want to keep doing this for every language I designed, and I realized that
302 literate test suites could serve as documentation as well; the result was
303 Falderal.
+0
-185
README.markdown less more
0 Falderal
1 ========
2
3 Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive"
4
5 This is the reference distribution of Falderal, a file format for literate
6 test suites. What sets Falderal apart from most other test frameworks is
7 that it recognizes that **your tests are for a functionality, not a particular
8 implementation of that functionality**.
9
10 Falderal is particularly suited for:
11
12 * documenting programming languages with prose and examples
13 * testing multiple implementations of a programming language
14 * performing Test-Driven Language Design (TDLD)
15 * being embedded in Markdown documents
16
17 Falderal in three words: "Doctests for DSLs".
18
19 If you're more interested in running the tools than learning about the format
20 itself, skip down to [Implementation](#implementation).
21
22 Motivation
23 ----------
24
25 Say you have a programming language. You probably have a document describing
26 the language. That document probably includes examples. Those examples are
27 probably intended to produce some kind of output. That output is probably
28 expected to be a certain, predicted thing, and if it's not that thing, the
29 implementation is probably considered incorrect.
30
31 So why not write those examples in a format that can be run and tested?
32
33 You could write a bunch of standalone test sources, and store the output you
34 expect from them in a bunch of other files, and write a shell script that runs
35 each program and `diff`s the output with the expected output. But this is a
36 lot of clutter — finding a particular example might not be so easy. Each
37 test source exists in a void, not necessarily logically grouped with other,
38 similar tests. And any text you write describing a test needs to be in the
39 comment syntax of your programming language (if your programming language
40 supports comments) and is also detached from all the other test descriptions.
41
42 You could write unit tests in the unit test framework of your choice, but
43 if your programming language has more than one implementation one day (and
44 you should really consider that possibility) then you might not be able to
45 re-use it so easily for other implementations in other languages.
46
47 In a language like Python, you could write doctests, but that also ties your
48 tests to one implementation of your language. There can be awkward
49 quoting issues with how you embed your test sources inside those embedded
50 strings that comprise your doctests, as well.
51
52 Or... you could write a Markdown document with beautiful yet precise prose
53 describing your wonderful language, alternating with example code (in the
54 form of embedded Falderal tests) clarifying each of the points you are
55 making; then you could use a Falderal-comprehending tool to run each of these
56 tests against any implementation of your language which exists or will exist
57 in the future.
58
59 *And* you could even write this document *before* you even start implementing
60 your language; then when it is all clear "on paper", you have a target at
61 which you can aim while writing your language. As you implement more and more
62 of it, more and more tests in your test suite will pass. This is simply the
63 idea behind Test-Driven Development (TDD) applied to language design, which we
64 will call Test-Driven Language Design (TDLD).
65
66 Features of the Format
67 ----------------------
68
69 Falderal is just a file format; it does not specify exactly what tools must
70 do with the tests they extract. However, it is generally assumed that most
71 tools will want to, at the very least,
72
73 * Run tests from one or more documents.
74 * Report the results, with some given level of detail.
75
76 There is, of course, a reference implementation which does both of these
77 things. It is called py-falderal and it is written in Python 2.7.
78
79 Each Falderal test is for some abstract _functionality_, and each
80 functionality may have multiple concrete _implementations_. Thus the same
81 tests can be run multiple times, once for each implementation of the
82 functionality they test.
83
84 Directives in the Falderal document may assign functionalities to tests,
85 and may define implementations for given functionalities. Implementations
86 may be defined outside of any document, as well. Falderal defines one
87 general kind of implementation, implementation by Bourne shell command, but
88 is not inherently restricted from supporting other kinds of implementations.
89
90 Inherent Limitations
91 --------------------
92
93 Being tests, rather than proofs, Falderal tests cannot tell you that your
94 implementation of a language is correct. If one or more tests fail, that's
95 an indication that your implementation is not correct; but even if all tests
96 pass, you have no guarantee that the implementation doesn't do something
97 contrary to the spec in one of the infinite number of cases that you have not
98 enumerated.
99
100 There is also no way to test that certain programs represent infinite loops,
101 for reasons that should be obvious.
102
103 Contents of this Distribution
104 -----------------------------
105
106 This distribution contains:
107
108 * `doc` — contains documents about Falderal. For the specification of
109 the file format, see
110 [`doc/Falderal_Literate_Test_Format.markdown`](doc/Falderal_Literate_Test_Format.markdown).
111 (Note that this specification should not be expected to remain stable
112 through the 0.x version series.) There are other documents in there too.
113 * `bin/falderal` — the reference implementation of Falderal.
114 See "Implementation", below, for details.
115 * `script` — miscellaneous small tools intended to be used in tests.
116 * `src` — source code for py-falderal.
117 * `tests` — a set of tests for Falderal itself. (Note that these are not
118 written in Falderal, as that would just be too confusing.)
119 * `HISTORY.markdown` — changelog for releases of Falderal.
120 * `TODO.markdown` — areas where Falderal and its implementations could be
121 improved.
122
123 Implementation
124 --------------
125
126 This distribution contains `falderal`, which is the reference implementation
127 of Falderal, written in Python and sometimes referred to as "py-falderal".
128
129 To use it, you can clone this repository and run it as `bin/falderal`
130 from the directory of your clone, or you can put the `bin` directory
131 on your executable search path, and run it as `falderal` anywhere.
132
133 Or you can install it using `pip`:
134
135 pip install -e Falderal==0.13
136
137 (Depending on your needs, you may wish to establish a virtual environment
138 first. Describing how to do so is outside the scope of this document.)
139
140 The implementation is (somewhat) documented in `doc/py-falderal.markdown`.
141
142 Development
143 -----------
144
145 The git repository for the Falderal distribution can be found on GitHub at
146 [https://github.com/catseye/Falderal](https://github.com/catseye/Falderal).
147
148 Official release distfiles are available via the
149 [Falderal project page](http://catseye.tc/node/Falderal) at
150 [Cat's Eye Technologies](http://catseye.tc/).
151
152 Projects using Falderal
153 -----------------------
154
155 * [ALPACA](https://catseye.tc/node/ALPACA)
156 * [Castile](https://catseye.tc/node/Castile)
157 * [Equipage](https://catseye.tc/node/Equipage)
158 * [Exanoke](https://catseye.tc/node/Exanoke)
159 * [Flobnar](https://catseye.tc/node/Flobnar)
160 * [Hev](https://catseye.tc/node/Hev)
161 * [hatoucan](https://catseye.tc/node/hatoucan)
162 * [Iphigeneia](https://catseye.tc/node/Iphigeneia)
163 * [Madison](https://catseye.tc/node/Madison)
164 * [Nhohnhehr](https://catseye.tc/node/Nhohnhehr)
165 * [Pail](https://catseye.tc/node/Pail)
166 * [Pixley](https://catseye.tc/node/Pixley)
167 * [PL-{GOTO}.NET](https://catseye.tc/node/PL-{GOTO}.NET)
168 * [Quylthulg](https://catseye.tc/node/Quylthulg)
169 * [Robin](https://catseye.tc/node/Robin)
170 * [Samovar](https://catseye.tc/node/Samovar)
171 * [SixtyPical](https://catseye.tc/node/SixtyPical)
172 * [Tamsin](https://catseye.tc/node/Tamsin)
173 * [Velo](https://catseye.tc/node/Velo)
174 * [Xoomonk](https://catseye.tc/node/Xoomonk)
175 * [Yolk](https://catseye.tc/node/Yolk)
176 * [ZOWIE](https://catseye.tc/node/ZOWIE)
177
178 Xoomonk, Madison, Velo, and Exanoke are good examples of how a literate
179 test suite can be useful in both describing a programming language through
180 examples and testing that an implementation of the language does not violate
181 the language specification. They are, in fact, exercises in Test-Driven
182 Language Design (TDLD), where the tests were written as part of designing the
183 language, before any attempt at implementation; the others are more like
184 traditional test suites, written after-the-fact.
0 Falderal
1 ========
2
3 Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive"
4
5 This is the reference distribution of Falderal, a file format for literate
6 test suites. What sets Falderal apart from most other test frameworks is
7 that it recognizes that **your tests are for a functionality, not a particular
8 implementation of that functionality**.
9
10 Falderal is particularly suited for:
11
12 * documenting programming languages with prose and examples
13 * testing multiple implementations of a programming language
14 * performing Test-Driven Language Design (TDLD)
15 * being embedded in Markdown documents
16
17 Falderal in three words: "Doctests for DSLs".
18
19 If you're more interested in running the tools than learning about the format
20 itself, skip down to [Implementation](#implementation).
21
22 Motivation
23 ----------
24
25 Say you have a programming language. You probably have a document describing
26 the language. That document probably includes examples. Those examples are
27 probably intended to produce some kind of output. That output is probably
28 expected to be a certain, predicted thing, and if it's not that thing, the
29 implementation is probably considered incorrect.
30
31 So why not write those examples in a format that can be run and tested?
32
33 You could write a bunch of standalone test sources, and store the output you
34 expect from them in a bunch of other files, and write a shell script that runs
35 each program and `diff`s the output with the expected output. But this is a
36 lot of clutter — finding a particular example might not be so easy. Each
37 test source exists in a void, not necessarily logically grouped with other,
38 similar tests. And any text you write describing a test needs to be in the
39 comment syntax of your programming language (if your programming language
40 supports comments) and is also detached from all the other test descriptions.
41
42 You could write unit tests in the unit test framework of your choice, but
43 if your programming language has more than one implementation one day (and
44 you should really consider that possibility) then you might not be able to
45 re-use it so easily for other implementations in other languages.
46
47 In a language like Python, you could write doctests, but that also ties your
48 tests to one implementation of your language. There can be awkward
49 quoting issues with how you embed your test sources inside those embedded
50 strings that comprise your doctests, as well.
51
52 Or... you could write a Markdown document with beautiful yet precise prose
53 describing your wonderful language, alternating with example code (in the
54 form of embedded Falderal tests) clarifying each of the points you are
55 making; then you could use a Falderal-comprehending tool to run each of these
56 tests against any implementation of your language which exists or will exist
57 in the future.
58
59 *And* you could even write this document *before* you even start implementing
60 your language; then when it is all clear "on paper", you have a target at
61 which you can aim while writing your language. As you implement more and more
62 of it, more and more tests in your test suite will pass. This is simply the
63 idea behind Test-Driven Development (TDD) applied to language design, which we
64 will call Test-Driven Language Design (TDLD).
65
66 Features of the Format
67 ----------------------
68
69 Falderal is just a file format; it does not specify exactly what tools must
70 do with the tests they extract. However, it is generally assumed that most
71 tools will want to, at the very least,
72
73 * Run tests from one or more documents.
74 * Report the results, with some given level of detail.
75
76 There is, of course, a reference implementation which does both of these
77 things. It is called py-falderal and it is written in Python 2.7.
78
79 Each Falderal test is for some abstract _functionality_, and each
80 functionality may have multiple concrete _implementations_. Thus the same
81 tests can be run multiple times, once for each implementation of the
82 functionality they test.
83
84 Directives in the Falderal document may assign functionalities to tests,
85 and may define implementations for given functionalities. Implementations
86 may be defined outside of any document, as well. Falderal defines one
87 general kind of implementation, implementation by Bourne shell command, but
88 is not inherently restricted from supporting other kinds of implementations.
89
90 Inherent Limitations
91 --------------------
92
93 Being tests, rather than proofs, Falderal tests cannot tell you that your
94 implementation of a language is correct. If one or more tests fail, that's
95 an indication that your implementation is not correct; but even if all tests
96 pass, you have no guarantee that the implementation doesn't do something
97 contrary to the spec in one of the infinite number of cases that you have not
98 enumerated.
99
100 There is also no way to test that certain programs represent infinite loops,
101 for reasons that should be obvious.
102
103 Contents of this Distribution
104 -----------------------------
105
106 This distribution contains:
107
108 * `doc` — contains documents about Falderal. For the specification of
109 the file format, see
110 [`doc/Falderal_Literate_Test_Format.md`](doc/Falderal_Literate_Test_Format.md).
111 (Note that this specification should not be expected to remain stable
112 through the 0.x version series.) There are other documents in there too.
113 * `bin/falderal` — the reference implementation of Falderal.
114 See "Implementation", below, for details.
115 * `script` — miscellaneous small tools intended to be used in tests.
116 * `src` — source code for py-falderal.
117 * `tests` — a set of tests for Falderal itself. (Note that these are not
118 written in Falderal, as that would just be too confusing.)
119 * `HISTORY.md` — changelog for releases of Falderal.
120 * `TODO.md` — areas where Falderal and its implementations could be
121 improved.
122
123 Implementation
124 --------------
125
126 This distribution contains `falderal`, which is the reference implementation
127 of Falderal, written in Python and sometimes referred to as "py-falderal".
128
129 To use it, you can clone this repository and run it as `bin/falderal`
130 from the directory of your clone, or you can put the `bin` directory
131 on your executable search path, and run it as `falderal` anywhere.
132
133 Or you can install it using `pip`:
134
135 pip install -e Falderal==0.14
136
137 (Depending on your needs, you may wish to establish a virtual environment
138 first. Describing how to do so is outside the scope of this document.)
139
140 The implementation is (somewhat) documented in
141 [`doc/py-falderal.md`](doc/py-falderal.md).
142
143 The implementation has its own test suite (not written in Falderal, as
144 that might be too confusing). It can be run by running `./test.sh`.
145
146 Development
147 -----------
148
149 The git repository for the Falderal distribution can be found on GitHub at
150 [https://github.com/catseye/Falderal](https://github.com/catseye/Falderal).
151
152 Official release distfiles are available via the
153 [Falderal project page](http://catseye.tc/node/Falderal) at
154 [Cat's Eye Technologies](http://catseye.tc/).
155
156 Projects using Falderal
157 -----------------------
158
159 * [ALPACA](https://catseye.tc/node/ALPACA)
160 * [Castile](https://catseye.tc/node/Castile)
161 * [Equipage](https://catseye.tc/node/Equipage)
162 * [Exanoke](https://catseye.tc/node/Exanoke)
163 * [Flobnar](https://catseye.tc/node/Flobnar)
164 * [Hev](https://catseye.tc/node/Hev)
165 * [hatoucan](https://catseye.tc/node/hatoucan)
166 * [Iphigeneia](https://catseye.tc/node/Iphigeneia)
167 * [Madison](https://catseye.tc/node/Madison)
168 * [Nhohnhehr](https://catseye.tc/node/Nhohnhehr)
169 * [Pail](https://catseye.tc/node/Pail)
170 * [Pixley](https://catseye.tc/node/Pixley)
171 * [PL-{GOTO}.NET](https://catseye.tc/node/PL-{GOTO}.NET)
172 * [Quylthulg](https://catseye.tc/node/Quylthulg)
173 * [Robin](https://catseye.tc/node/Robin)
174 * [Samovar](https://catseye.tc/node/Samovar)
175 * [SixtyPical](https://catseye.tc/node/SixtyPical)
176 * [Tamsin](https://catseye.tc/node/Tamsin)
177 * [Velo](https://catseye.tc/node/Velo)
178 * [Xoomonk](https://catseye.tc/node/Xoomonk)
179 * [Yolk](https://catseye.tc/node/Yolk)
180 * [ZOWIE](https://catseye.tc/node/ZOWIE)
181
182 Xoomonk, Madison, Velo, and Exanoke are good examples of how a literate
183 test suite can be useful in both describing a programming language through
184 examples and testing that an implementation of the language does not violate
185 the language specification. They are, in fact, exercises in Test-Driven
186 Language Design (TDLD), where the tests were written as part of designing the
187 language, before any attempt at implementation; the others are more like
188 traditional test suites, written after-the-fact.
+0
-186
TODO.markdown less more
0 TODO
1 ====
2
3 (collected from the Falderal issue tracker on Bitbucket and the
4 py-falderal issue tracker on github)
5
6 Falderal Literate Test Format
7 -----------------------------
8
9 ### Policy for expecting both errors and output, success and failure
10
11 Policy should be this, by example:
12
13 | test
14 = foo
15
16 Means: I expect this to succeed and to produce `foo` on stdout, and
17 I don't care what's on stderr (or — stderr should be empty?)
18
19 | test
20 = foo
21 ? bar
22
23 Means: I expect this to succeed, to produce `foo` on stdout, and to
24 produce `bar` on stderr.
25
26 | test
27 ? foo
28
29 Means: I expect this to fail, and to produce `foo` on stderr.
30 And to not care about stdout (or expect it to be empty.)
31
32 | test
33 ? foo
34 = bar
35
36 Means: I expect this to fail, to produce `foo` on stderr, and to
37 produce `bar` on stdout.
38
39 In other words, an error expectation may follow an output expectation
40 and vice versa. Error expectations always match stderr, output expectations
41 always match stdout. Which one's first should dictate whether we expect
42 the command to succeed or fail.
43
44 What's after here hasn't been re-edited yet.
45
46 When you have a program that produces output on both stdout and stderr,
47 whether it fails or not, you might want to expect text on both stdout and
48 stderr.
49
50 Currently it expects the text on stdout if it is a `=` expectation, and on
51 stderr if it is a `?` expectation.
52
53 You can't work around this so well by tacking `2>&1` onto the end of the
54 command, because then stderr will always be empty.
55
56 We could, by default, tack `2>&1` on the end ourselves and look only at
57 stdout. This might be the simplest approach.
58
59 We might want to add options that avoid doing that, but if so, what should
60 they be? Should each test be able to configure this? Should a single test
61 be able to have both `=` and `?` expectations, each for each stream?
62
63 This is complicated by the presence of `%(output-file)`; currently, if that
64 is given, stdout is ignored in preference to it (but stderr is still
65 checked, if the command failed. There should probably be a corresponding
66 `%(error-file)` variable.)
67
68 I think the current behaviour could work, with the following policy:
69
70 If the command succeeds, your `=` expectation will be matched against
71 stdout only. If you wish to match against both stdout and stderr in these
72 cases, add `2>&1` to your shell command.
73
74 If the command fails, your `?` expectation will be matched against stderr
75 only. If you wish to match against both stdout and stderr in these cases,
76 add `1>&2` to your shell command.
77
78 Either way, it's still worth investigating whether it's worthwhile to have
79 both `=` and `?` expectations on a single test. (I can't convince myself
80 that stdout and stderr will always be combined deterministically, and
81 having both kinds of expectations would allow non-deterministic combinations
82 of the two to be matched.)
83
84 ### Allow expectations to be transformed during comparison
85
86 It would be nice to allow expectations to be transformed before they are
87 compared to the actual output. The main use case for this that I can think of
88 is to allow the expected output to be "pretty printed" (that is, nicely
89 formatted) in the Falderal file, while the functionality being tested just
90 produces a dump. The nicely formatted expected output should be "crunched"
91 into the same ugly format as the dump.
92
93 This doesn't work as well the other way; although one could compose the
94 functionality being tested with an actual pretty-printer, that would
95 prescribe a certain indentation scheme etc. that the expected output would
96 have to match exactly. It would be rather better if the writer of the tests
97 could format their expected output as they find most aesthetically pleasing
98 in their literate tests, and have that be transformed instead.
99
100 This might be somewhat tricky, however; if the transformation applied is
101 too powerful, it can distort or eliminate the meaning of the test, and erode
102 confidence.
103
104 ### Allow use of patterns in expected output
105
106 Likely by way of regexps. This would be particularly valuable in
107 exception-expecting tests, where we don't care about details such as the
108 line number of the Haskell file at which the exception occurred.
109
110 ### Allow equivalency tests to be defined and run.
111
112 To test functions of type `(Eq a) => String -> a`, you should be able to
113 give give multiple input strings in a set; if the function does not map
114 them all to the same value, that's a test failure.
115
116 Syntax for an equivalency test might look like this:
117
118 | 2+2
119 ==
120 | 3+1
121 ==
122 | 7-3
123
124
125 py-falderal
126 -----------
127
128 ### Test report accumulation
129
130 Multiple runs of `falderal` ought to be able to accumulate their results
131 to a temporary file. No report should be generated if this is selected.
132 At the end, the report can be generated from the file.
133
134 rm -f FILE
135 falderal --record-to FILE tests1.markdown
136 falderal --record-to FILE tests2.markdown
137 falderal --report-from FILE
138
139 ### Support 'weak' testing
140
141 In which we only care about whether the command succeeded or failed.
142 In practice, this could be useful for testing the parser (just test
143 if these forms parse.) Or, if not this, then think of something that
144 would make just testing parsers more useful.
145
146 ### Split InterveningMarkdown blocks to make nice test descriptions
147
148 For example, if we have
149
150 ...test #1...
151 Some text.
152 Heading
153 -------
154 More text
155 ...test #2...
156
157 The description for test #2 should consist of "More text"; possibly also
158 the heading, but not "Some text". This can take place in a pre-processing
159 phase which simply splits every `InterveningMarkdown` block into multiple
160 blocks, at its headers. It should understand both underlined and atx-style
161 headers.
162
163 ### option to colourize test result output
164
165 Using one of the approaches listed here:
166
167 http://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python
168
169 ...py-falderal ought to provide an option (not default, of course, and not
170 if stdout is not a tty) to colorize the output with, of course, pass=green,
171 fail=red.
172
173 But, you'd often want to pipe the output to `less`, which by default makes
174 control characters visible, defeating colourization. But there is a flag
175 to less, `less -R`, which retains colourization. So use that.
176
177 ### Flag invalid sequences of lines as errors
178
179 2011-08-05
180
181 Currently, in `convertLinesToBlocks`, some invalid sequences of lines are
182 ignored. They should be flagged as errors in the test suite file.
183
184 (This was written against Test.Falderal but similar considerations could
185 be made for py-falderal.)
0 TODO
1 ====
2
3 (collected from the Falderal issue tracker on Bitbucket and the
4 py-falderal issue tracker on github)
5
6 Falderal Literate Test Format
7 -----------------------------
8
9 ### Policy for expecting both errors and output, success and failure
10
11 Policy should be this, by example:
12
13 | test
14 = foo
15
16 Means: I expect this to succeed and to produce `foo` on stdout, and
17 I don't care what's on stderr (or — stderr should be empty?)
18
19 | test
20 = foo
21 ? bar
22
23 Means: I expect this to succeed, to produce `foo` on stdout, and to
24 produce `bar` on stderr.
25
26 | test
27 ? foo
28
29 Means: I expect this to fail, and to produce `foo` on stderr.
30 And to not care about stdout (or expect it to be empty.)
31
32 | test
33 ? foo
34 = bar
35
36 Means: I expect this to fail, to produce `foo` on stderr, and to
37 produce `bar` on stdout.
38
39 In other words, an error expectation may follow an output expectation
40 and vice versa. Error expectations always match stderr, output expectations
41 always match stdout. Which one's first should dictate whether we expect
42 the command to succeed or fail.
43
44 What's after here hasn't been re-edited yet.
45
46 When you have a program that produces output on both stdout and stderr,
47 whether it fails or not, you might want to expect text on both stdout and
48 stderr.
49
50 Currently it expects the text on stdout if it is a `=` expectation, and on
51 stderr if it is a `?` expectation.
52
53 You can't work around this so well by tacking `2>&1` onto the end of the
54 command, because then stderr will always be empty.
55
56 We could, by default, tack `2>&1` on the end ourselves and look only at
57 stdout. This might be the simplest approach.
58
59 We might want to add options that avoid doing that, but if so, what should
60 they be? Should each test be able to configure this? Should a single test
61 be able to have both `=` and `?` expectations, each for each stream?
62
63 This is complicated by the presence of `%(output-file)`; currently, if that
64 is given, stdout is ignored in preference to it (but stderr is still
65 checked, if the command failed. There should probably be a corresponding
66 `%(error-file)` variable.)
67
68 I think the current behaviour could work, with the following policy:
69
70 If the command succeeds, your `=` expectation will be matched against
71 stdout only. If you wish to match against both stdout and stderr in these
72 cases, add `2>&1` to your shell command.
73
74 If the command fails, your `?` expectation will be matched against stderr
75 only. If you wish to match against both stdout and stderr in these cases,
76 add `1>&2` to your shell command.
77
78 Either way, it's still worth investigating whether it's worthwhile to have
79 both `=` and `?` expectations on a single test. (I can't convince myself
80 that stdout and stderr will always be combined deterministically, and
81 having both kinds of expectations would allow non-deterministic combinations
82 of the two to be matched.)
83
84 ### Allow expectations to be transformed during comparison
85
86 It would be nice to allow expectations to be transformed before they are
87 compared to the actual output. The main use case for this that I can think of
88 is to allow the expected output to be "pretty printed" (that is, nicely
89 formatted) in the Falderal file, while the functionality being tested just
90 produces a dump. The nicely formatted expected output should be "crunched"
91 into the same ugly format as the dump.
92
93 This doesn't work as well the other way; although one could compose the
94 functionality being tested with an actual pretty-printer, that would
95 prescribe a certain indentation scheme etc. that the expected output would
96 have to match exactly. It would be rather better if the writer of the tests
97 could format their expected output as they find most aesthetically pleasing
98 in their literate tests, and have that be transformed instead.
99
100 This might be somewhat tricky, however; if the transformation applied is
101 too powerful, it can distort or eliminate the meaning of the test, and erode
102 confidence.
103
104 ### Allow use of patterns in expected output
105
106 Likely by way of regexps. This would be particularly valuable in
107 exception-expecting tests, where we don't care about details such as the
108 line number of the Haskell file at which the exception occurred.
109
110 ### Allow equivalency tests to be defined and run.
111
112 To test functions of type `(Eq a) => String -> a`, you should be able to
113 give give multiple input strings in a set; if the function does not map
114 them all to the same value, that's a test failure.
115
116 Syntax for an equivalency test might look like this:
117
118 | 2+2
119 ==
120 | 3+1
121 ==
122 | 7-3
123
124
125 py-falderal
126 -----------
127
128 ### Test report accumulation
129
130 Multiple runs of `falderal` ought to be able to accumulate their results
131 to a temporary file. No report should be generated if this is selected.
132 At the end, the report can be generated from the file.
133
134 rm -f FILE
135 falderal --record-to FILE tests1.markdown
136 falderal --record-to FILE tests2.markdown
137 falderal --report-from FILE
138
139 ### Support 'weak' testing
140
141 In which we only care about whether the command succeeded or failed.
142 In practice, this could be useful for testing the parser (just test
143 if these forms parse.) Or, if not this, then think of something that
144 would make just testing parsers more useful.
145
146 ### Split InterveningMarkdown blocks to make nice test descriptions
147
148 For example, if we have
149
150 ...test #1...
151 Some text.
152 Heading
153 -------
154 More text
155 ...test #2...
156
157 The description for test #2 should consist of "More text"; possibly also
158 the heading, but not "Some text". This can take place in a pre-processing
159 phase which simply splits every `InterveningMarkdown` block into multiple
160 blocks, at its headers. It should understand both underlined and atx-style
161 headers.
162
163 ### option to colourize test result output
164
165 Using one of the approaches listed here:
166
167 http://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python
168
169 ...py-falderal ought to provide an option (not default, of course, and not
170 if stdout is not a tty) to colorize the output with, of course, pass=green,
171 fail=red.
172
173 But, you'd often want to pipe the output to `less`, which by default makes
174 control characters visible, defeating colourization. But there is a flag
175 to less, `less -R`, which retains colourization. So use that.
176
177 ### Flag invalid sequences of lines as errors
178
179 2011-08-05
180
181 Currently, in `convertLinesToBlocks`, some invalid sequences of lines are
182 ignored. They should be flagged as errors in the test suite file.
183
184 (This was written against Test.Falderal but similar considerations could
185 be made for py-falderal.)
+0
-340
doc/Falderal_Literate_Test_Format.markdown less more
0 Falderal Literate Test Format
1 =============================
2
3 This document describes the proposed Falderal Literate Test Format.
4
5 Status
6 ------
7
8 This document is a *draft*. It is nominally "version 0.12" because it
9 describes something that version 0.12 of `py-falderal` mostly implements.
10 We will deign to note which sections of this document the current released
11 version of `py-falderal` implements, and which it does not. However,
12 this document is a work in progress, subject to change, and subject to get
13 out of sync with `py-falderal`. It should not be considered to be
14 anything except a draft until it is described as "version 1.0".
15
16 Overview
17 --------
18
19 A Falderal Literate Test Suite is a plain text file where some of the lines
20 have special meaning to the Falderal Literate Test Format. Certain
21 groupings of the lines defined by this format are intended to denote tests,
22 while others denote pragmas which specify how those tests should be run.
23 Outside of these groupings, lines have no meaning to the Falderal Literate
24 Test Format other than that they are presumed to be mainly descriptive text.
25
26 The plain text file may also be formatted in some other format, such as
27 Markdown, although this is not required.
28
29 A tool which claims to understand the Falderal Literate Test Format may
30 choose to extract the tests from such a document, run them, and report which
31 and how many of them passed or failed. That would be a typical application
32 for this format. However, no tool is strictly required to do this with the
33 tests found in a document; a tool may, for example, simply reformat the
34 tests to produce an output file in a different format.
35
36 In the syntax templates below, words in `/slashes/` indicate a variable
37 rather than literal text.
38
39 Syntax
40 ------
41
42 Each grouping of lines which has special meaning to the Falderal Literate
43 Test Format always begins with an indent of four (4) spaces from the
44 leftmost column, preceded by non-indented text, and followed by either
45 non-indented text or the end of the file. Such a grouping of lines is
46 called a _block_.
47
48 There are two general formats to any block. In the first, "verbose" format,
49 each indent of 4 spaces is followed immediately on that line by distinguished
50 sequence of characters, called an _introducer_. The introducers which have
51 meaning to the Falderal Literate Test Format are as follows:
52
53 * `->` (hyphen, greater-than sign): pragma
54 * `| ` (vertical bar, space): test body text
55 * `+ ` (plus sign, space): test input text
56 * `= ` (equals sign, space): expected output text
57 * `? ` (question mark, space): expected error text
58
59 If the same introducer occurs on multiple adjacent lines, the text after
60 each introducer is concatenated to form one chunk of relevant text. This
61 allows, for example, multi-line text to be given as the body, the input,
62 or the expected output of a test.
63
64 There are some restrictions on the order in which introducers can sensibly
65 occur in a block:
66
67 * Test body text should occur at the start of a block.
68 * Test body test may be optionally followed by test input text.
69 * The first test input text must be immediately preceded by test body text.
70 * Subsequent test input texts need not be preceded by a test body text;
71 in this case, the previously-appearing test body text will be used again.
72 * Test body text must be followed by either test input text, expected
73 output text, or expected error text, with no intervening text.
74 * Either expected output or error text must follow either test body
75 text or test input text, with no intervening text.
76
77 See the sections for these introducers, below, for more details.
78
79 In the other, "freestyle" format, not all lines in a block require
80 introducers. A freestyle format block is indentified as such if one or
81 more of the final lines of the block begin with any of the following
82 introducers:
83
84 * `=> `: expected output text
85 * `==> `: expected output text
86 * `===> `: expected output text
87 * `?> `: expected error text
88 * `??> `: expected error text
89 * `???> `: expected error text
90
91 In addition, the following introducers may be used to mark a section
92 of test input text on the first of the final lines (but may not be
93 used to end a block):
94
95 * `<= `: test input text
96 * `<== `: test input text
97 * `<=== `: test input text
98
99 If a block is identified as a freestyle block, all lines preceding the
100 first final line appearing with one of these introducers, are interpreted
101 as having no introducer at all (even if they begin with `| ` or some other
102 sequence already mentioned) and are used as the test body block.
103
104 Lines without introducers are called _intervening text_.
105 Lines of intervening text are classified as either blank or non-blank. A
106 line is blank if it contains no characters, or if it contains only whitespace.
107 A group of non-blank lines is referred to as a _paragraph_.
108
109 Pragmas
110 -------
111
112 ### encoding ###
113
114 The encoding pragma allows a Falderal file to specify what encoding is
115 used for the characters in it. An implementation of Falderal is not
116 expected to be able to handle any coding other than UTF-8, however,
117 this pragma is included for the benefit of text editors and other tools,
118 to indicate that the document is in fact in UTF-8 encoding.
119
120 Example:
121
122 -> encoding: UTF-8
123
124 ### Functionality-definition ###
125
126 The Functionality-definition pragma allows a Falderal file to describe
127 ways in which a functionality being tested is implemented. It has the
128 following syntax:
129
130 -> Functionality /functionality-name/ is implemented by /functionality-type/ /functionality-specifier/
131
132 _functionality-type_ must at present be `shell command`. The format of the
133 _functionality-specifier_ differs according to the functionality-type.
134 The _functionality-name_ is arbitrary text enclosed within double quotes,
135 which may be referenced in a later Tests-for pragma.
136
137 Note that the Functionality-definitions given in a Falderal file should
138 not be considered exhaustive, or even requisite, by a tool. The tool may
139 accept additional definitions of a functionality, referencing it by its
140 name, from an external source such as the command line or a configuration
141 file, and may be instructed to ignore certain Functionality-definitions in
142 a Falderal file (if, for example, certain implementation are not currently
143 available or of interest to the user.) Indeed, the functionality referred
144 to by a _functionality-name_ in a Tests-for pragma need not be defined by any
145 Functionality-definition pragma in the same Falderal file, and this
146 situation requires the functionality to be specified to the tool in some
147 other manner.
148
149 #### Shell commands ####
150
151 For shell commands, the _functionality-specifier_ is in the format
152 `"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may
153 be used, so pipes, redirection, etc., are supported. Note that the double
154 quotation mark characters used to enclosed the command have meaning only to
155 the Falderal format — they are not part of the command, are not passed to the
156 shell, and do not require double quotation mark characters that are enclosed
157 by them to be escaped with a backslash.
158
159 Certain subsequences, called _variables_, if present in the command string,
160 will be expanded before execution, and will alter how the command reads the
161 text of the test and produces its output, to be compared with the expected
162 output.
163
164 When a variable is expanded, it is Falderal's responsibility to escape any
165 shell metacharacters that occur as part of the substitution text so that
166 it appears to the shell as a single token, and cannot otherwise affect the
167 meaning of the instantiated shell command. It is not necessary for a
168 variable to be enclosed in any kind of quotes in the command line template.
169 It is also not possible for the substitution text to e.g. expand to two
170 arguments, or to form a pipe or redirection specifier, etc.
171
172 The exact method by which the Falderal implementation escapes or quotes
173 the substitution text so that it appears unambiguously as a single argument
174 is left up to the implementation.
175
176 ##### `%(test-body-file)` #####
177
178 The variable `%(test-body-file)` will be replaced by the name of a file which
179 contains the text of the test body. This may be a temporary file created
180 solely for this purpose by the Falderal implementation.
181
182 ##### `%(test-body-text)` #####
183
184 The variable `%(test-body-text)` will be replaced by the actual text of the
185 test body.
186
187 ##### `%(test-input-file)` #####
188
189 The variable `%(test-input-file)` will be replaced by the name of a file which
190 contains the text of the test input. This may be a temporary file created
191 solely for this purpose by the Falderal implementation.
192
193 ##### `%(test-input-text)` #####
194
195 The variable `%(test-input-text)` will be replaced by the actual text of the
196 test input.
197
198 If neither of the variables `%(test-body-file)` nor `%(test-body-text)` appear
199 in the command string, the test body text will be provided on the standard
200 input of the shell command.
201
202 If neither of the variables `%(test-input-file)` nor `%(test-input-text)` appear
203 in the command string, the test input text will be provided on the standard
204 input of the shell command.
205
206 If both the test body text and the test input text are slated to appear on the
207 input of the shell command, then the behaviour is (presently) undefined.
208
209 ##### `%(output-file)` #####
210
211 The variable `%(output-file)` will be replaced by the name of a file
212 (temporary file) to which the test results will be written. If it does
213 not appear in the command string, the output text will be read from
214 the standard output of the command.
215
216 How shell commands support error output is not yet standardized.
217
218 For example:
219
220 -> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %(test-file) > %(output-file)"
221
222 ### Tests-for ###
223
224 The Tests-for pragma determines what functionality will be used to run all
225 following tests, until the next Tests-for pragma. It has the following
226 syntax:
227
228 -> Tests for /functionality-name/
229
230 The _functionality-name_ refers to a functionality, which may be specified
231 by a Functionality-definition pragma elsewhere in the Falderal file.
232
233 For example:
234
235 -> Tests for 'Reversing a string'
236
237 Test Body, Test Input and Expected Text
238 ---------------------------------------
239
240 Each section of test body text may or may not be followed by a section of
241 test input text; either way it must then be followed immediately by either
242 and expected output section or expected error section.
243
244 Valid examples in the "verbose" format:
245
246 | thing to test
247 = output to expect
248
249 | thing to test
250 ? error to expect
251
252 | thing to test
253 + input to give it
254 = output to expect
255
256 | thing to test
257 + input to give it
258 ? error to expect
259
260 + different input to give the immediately previously defined test body
261 ? different error to expect
262
263 Valid examples in the "freestyle" format:
264
265 thing to test
266 => output to expect
267
268 thing to test
269 ==> output to expect
270
271 thing to test
272 ===> output to expect
273
274 thing to test
275 ?> error to expect
276
277 thing to test
278 ??> error to expect
279
280 thing to test
281 ???> error to expect
282
283 thing to test
284 <=== input to give it
285 ===> output to expect
286
287 thing to test
288 <=== input to give it
289 ???> error to expect
290
291 Invalid examples:
292
293 | thing to test
294
295 ...needs an expectation.
296
297 + input to give it
298 = output to expect
299
300 ...test input must be preceded by a test body, if this is the first test.
301
302 <=== input to give it
303 ???> output to expect
304
305 ...test input must be preceded by a test body always, in freestyle format.
306
307 ? error to expect
308
309 ...expectation must be preceded by either test input or test body.
310
311 A test body section may also be preceded by a paragraph of text; the
312 intent of the Falderal Literate Test Format is that this text should
313 describe the test, or rather, the aspect of the behaviour of the system
314 that the test is meant to check. It is therefore reasonable that this
315 text should be displayed along with the contents (test body text and
316 expected output or error) of the test, in, for example, a test result
317 report.
318
319 Discussion
320 ==========
321
322 (This section is non-normative.)
323
324 Typically, a file in Falderal Literate Test Format will also be in
325 Markdown format.
326
327 The format of the lines which comprise the Falderal Literate Test Format
328 was chosen to not conflict with many other common text formats (including
329 but not limited to Bird-style Literate Haskell, and Markdown); thus
330 literate test suites may be embedded in a wide variety of other formats.
331 However, there are inevitably some conflicts with some textual formats;
332 for example, when embedded in C code and many other languages, Falderal
333 entries should be preceded by `/*` and followed by `*/`, to ensure that
334 they are regarded as comments. Also, reStructuredText uses the `|` line
335 prefix to denote preformatted plain text.
336
337 The format of pragmas was chosen such that they could be read literately,
338 and as such, a formatting tool could format them in the output document
339 with little if any change.
0 Falderal Literate Test Format
1 =============================
2
3 This document describes the proposed Falderal Literate Test Format.
4
5 Status
6 ------
7
8 This document is a *draft*. It is nominally "version 0.12" because it
9 describes something that version 0.12 of `py-falderal` mostly implements.
10 We will deign to note which sections of this document the current released
11 version of `py-falderal` implements, and which it does not. However,
12 this document is a work in progress, subject to change, and subject to get
13 out of sync with `py-falderal`. It should not be considered to be
14 anything except a draft until it is described as "version 1.0".
15
16 Overview
17 --------
18
19 A Falderal Literate Test Suite is a plain text file where some of the lines
20 have special meaning to the Falderal Literate Test Format. Certain
21 groupings of the lines defined by this format are intended to denote tests,
22 while others denote pragmas which specify how those tests should be run.
23 Outside of these groupings, lines have no meaning to the Falderal Literate
24 Test Format other than that they are presumed to be mainly descriptive text.
25
26 The plain text file may also be formatted in some other format, such as
27 Markdown, although this is not required.
28
29 A tool which claims to understand the Falderal Literate Test Format may
30 choose to extract the tests from such a document, run them, and report which
31 and how many of them passed or failed. That would be a typical application
32 for this format. However, no tool is strictly required to do this with the
33 tests found in a document; a tool may, for example, simply reformat the
34 tests to produce an output file in a different format.
35
36 In the syntax templates below, words in `/slashes/` indicate a variable
37 rather than literal text.
38
39 Syntax
40 ------
41
42 Each grouping of lines which has special meaning to the Falderal Literate
43 Test Format always begins with an indent of four (4) spaces from the
44 leftmost column, preceded by non-indented text, and followed by either
45 non-indented text or the end of the file. Such a grouping of lines is
46 called a _block_.
47
48 There are two general formats to any block. In the first, "verbose" format,
49 each indent of 4 spaces is followed immediately on that line by distinguished
50 sequence of characters, called an _introducer_. The introducers which have
51 meaning to the Falderal Literate Test Format are as follows:
52
53 * `->` (hyphen, greater-than sign): pragma
54 * `| ` (vertical bar, space): test body text
55 * `+ ` (plus sign, space): test input text
56 * `= ` (equals sign, space): expected output text
57 * `? ` (question mark, space): expected error text
58
59 If the same introducer occurs on multiple adjacent lines, the text after
60 each introducer is concatenated to form one chunk of relevant text. This
61 allows, for example, multi-line text to be given as the body, the input,
62 or the expected output of a test.
63
64 There are some restrictions on the order in which introducers can sensibly
65 occur in a block:
66
67 * Test body text should occur at the start of a block.
68 * Test body test may be optionally followed by test input text.
69 * The first test input text must be immediately preceded by test body text.
70 * Subsequent test input texts need not be preceded by a test body text;
71 in this case, the previously-appearing test body text will be used again.
72 * Test body text must be followed by either test input text, expected
73 output text, or expected error text, with no intervening text.
74 * Either expected output or error text must follow either test body
75 text or test input text, with no intervening text.
76
77 See the sections for these introducers, below, for more details.
78
79 In the other, "freestyle" format, not all lines in a block require
80 introducers. A freestyle format block is indentified as such if one or
81 more of the final lines of the block begin with any of the following
82 introducers:
83
84 * `=> `: expected output text
85 * `==> `: expected output text
86 * `===> `: expected output text
87 * `?> `: expected error text
88 * `??> `: expected error text
89 * `???> `: expected error text
90
91 In addition, the following introducers may be used to mark a section
92 of test input text on the first of the final lines (but may not be
93 used to end a block):
94
95 * `<= `: test input text
96 * `<== `: test input text
97 * `<=== `: test input text
98
99 If a block is identified as a freestyle block, all lines preceding the
100 first final line appearing with one of these introducers, are interpreted
101 as having no introducer at all (even if they begin with `| ` or some other
102 sequence already mentioned) and are used as the test body block.
103
104 Lines without introducers are called _intervening text_.
105 Lines of intervening text are classified as either blank or non-blank. A
106 line is blank if it contains no characters, or if it contains only whitespace.
107 A group of non-blank lines is referred to as a _paragraph_.
108
109 Pragmas
110 -------
111
112 ### encoding ###
113
114 The encoding pragma allows a Falderal file to specify what encoding is
115 used for the characters in it. An implementation of Falderal is not
116 expected to be able to handle any coding other than UTF-8, however,
117 this pragma is included for the benefit of text editors and other tools,
118 to indicate that the document is in fact in UTF-8 encoding.
119
120 Example:
121
122 -> encoding: UTF-8
123
124 ### Functionality-definition ###
125
126 The Functionality-definition pragma allows a Falderal file to describe
127 ways in which a functionality being tested is implemented. It has the
128 following syntax:
129
130 -> Functionality /functionality-name/ is implemented by /functionality-type/ /functionality-specifier/
131
132 _functionality-type_ must at present be `shell command`. The format of the
133 _functionality-specifier_ differs according to the functionality-type.
134 The _functionality-name_ is arbitrary text enclosed within double quotes,
135 which may be referenced in a later Tests-for pragma.
136
137 Note that the Functionality-definitions given in a Falderal file should
138 not be considered exhaustive, or even requisite, by a tool. The tool may
139 accept additional definitions of a functionality, referencing it by its
140 name, from an external source such as the command line or a configuration
141 file, and may be instructed to ignore certain Functionality-definitions in
142 a Falderal file (if, for example, certain implementation are not currently
143 available or of interest to the user.) Indeed, the functionality referred
144 to by a _functionality-name_ in a Tests-for pragma need not be defined by any
145 Functionality-definition pragma in the same Falderal file, and this
146 situation requires the functionality to be specified to the tool in some
147 other manner.
148
149 #### Shell commands ####
150
151 For shell commands, the _functionality-specifier_ is in the format
152 `"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may
153 be used, so pipes, redirection, etc., are supported. Note that the double
154 quotation mark characters used to enclosed the command have meaning only to
155 the Falderal format — they are not part of the command, are not passed to the
156 shell, and do not require double quotation mark characters that are enclosed
157 by them to be escaped with a backslash.
158
159 Certain subsequences, called _variables_, if present in the command string,
160 will be expanded before execution, and will alter how the command reads the
161 text of the test and produces its output, to be compared with the expected
162 output.
163
164 When a variable is expanded, it is Falderal's responsibility to escape any
165 shell metacharacters that occur as part of the substitution text so that
166 it appears to the shell as a single token, and cannot otherwise affect the
167 meaning of the instantiated shell command. It is not necessary for a
168 variable to be enclosed in any kind of quotes in the command line template.
169 It is also not possible for the substitution text to e.g. expand to two
170 arguments, or to form a pipe or redirection specifier, etc.
171
172 The exact method by which the Falderal implementation escapes or quotes
173 the substitution text so that it appears unambiguously as a single argument
174 is left up to the implementation.
175
176 ##### `%(test-body-file)` #####
177
178 The variable `%(test-body-file)` will be replaced by the name of a file which
179 contains the text of the test body. This may be a temporary file created
180 solely for this purpose by the Falderal implementation.
181
182 ##### `%(test-body-text)` #####
183
184 The variable `%(test-body-text)` will be replaced by the actual text of the
185 test body.
186
187 ##### `%(test-input-file)` #####
188
189 The variable `%(test-input-file)` will be replaced by the name of a file which
190 contains the text of the test input. This may be a temporary file created
191 solely for this purpose by the Falderal implementation.
192
193 ##### `%(test-input-text)` #####
194
195 The variable `%(test-input-text)` will be replaced by the actual text of the
196 test input.
197
198 If neither of the variables `%(test-body-file)` nor `%(test-body-text)` appear
199 in the command string, the test body text will be provided on the standard
200 input of the shell command.
201
202 If neither of the variables `%(test-input-file)` nor `%(test-input-text)` appear
203 in the command string, the test input text will be provided on the standard
204 input of the shell command.
205
206 If both the test body text and the test input text are slated to appear on the
207 input of the shell command, then the behaviour is (presently) undefined.
208
209 ##### `%(output-file)` #####
210
211 The variable `%(output-file)` will be replaced by the name of a file
212 (temporary file) to which the test results will be written. If it does
213 not appear in the command string, the output text will be read from
214 the standard output of the command.
215
216 How shell commands support error output is not yet standardized.
217
218 For example:
219
220 -> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %(test-file) > %(output-file)"
221
222 ### Tests-for ###
223
224 The Tests-for pragma determines what functionality will be used to run all
225 following tests, until the next Tests-for pragma. It has the following
226 syntax:
227
228 -> Tests for /functionality-name/
229
230 The _functionality-name_ refers to a functionality, which may be specified
231 by a Functionality-definition pragma elsewhere in the Falderal file.
232
233 For example:
234
235 -> Tests for 'Reversing a string'
236
237 Test Body, Test Input and Expected Text
238 ---------------------------------------
239
240 Each section of test body text may or may not be followed by a section of
241 test input text; either way it must then be followed immediately by either
242 and expected output section or expected error section.
243
244 Valid examples in the "verbose" format:
245
246 | thing to test
247 = output to expect
248
249 | thing to test
250 ? error to expect
251
252 | thing to test
253 + input to give it
254 = output to expect
255
256 | thing to test
257 + input to give it
258 ? error to expect
259
260 + different input to give the immediately previously defined test body
261 ? different error to expect
262
263 Valid examples in the "freestyle" format:
264
265 thing to test
266 => output to expect
267
268 thing to test
269 ==> output to expect
270
271 thing to test
272 ===> output to expect
273
274 thing to test
275 ?> error to expect
276
277 thing to test
278 ??> error to expect
279
280 thing to test
281 ???> error to expect
282
283 thing to test
284 <=== input to give it
285 ===> output to expect
286
287 thing to test
288 <=== input to give it
289 ???> error to expect
290
291 Invalid examples:
292
293 | thing to test
294
295 ...needs an expectation.
296
297 + input to give it
298 = output to expect
299
300 ...test input must be preceded by a test body, if this is the first test.
301
302 <=== input to give it
303 ???> output to expect
304
305 ...test input must be preceded by a test body always, in freestyle format.
306
307 ? error to expect
308
309 ...expectation must be preceded by either test input or test body.
310
311 A test body section may also be preceded by a paragraph of text; the
312 intent of the Falderal Literate Test Format is that this text should
313 describe the test, or rather, the aspect of the behaviour of the system
314 that the test is meant to check. It is therefore reasonable that this
315 text should be displayed along with the contents (test body text and
316 expected output or error) of the test, in, for example, a test result
317 report.
318
319 Discussion
320 ==========
321
322 (This section is non-normative.)
323
324 Typically, a file in Falderal Literate Test Format will also be in
325 Markdown format.
326
327 The format of the lines which comprise the Falderal Literate Test Format
328 was chosen to not conflict with many other common text formats (including
329 but not limited to Bird-style Literate Haskell, and Markdown); thus
330 literate test suites may be embedded in a wide variety of other formats.
331 However, there are inevitably some conflicts with some textual formats;
332 for example, when embedded in C code and many other languages, Falderal
333 entries should be preceded by `/*` and followed by `*/`, to ensure that
334 they are regarded as comments. Also, reStructuredText uses the `|` line
335 prefix to denote preformatted plain text.
336
337 The format of pragmas was chosen such that they could be read literately,
338 and as such, a formatting tool could format them in the output document
339 with little if any change.
+0
-88
doc/py-falderal.markdown less more
0 `py-falderal`
1 =============
2
3 `py-falderal` is an implementation of Falderal in Python 2.5.x.
4
5 Motivation
6 ----------
7
8 The original reference implementation of Falderal was written in Haskell.
9 There are a few reasons I had for re-implementing it in Python:
10
11 * The original Falderal implementation grew out of a Haskell-specific hack,
12 and it showed in how it was written.
13
14 * Fewer discrepancies between platforms. In particular, `ghc` for Windows
15 likes to operate in terms of MS-DOS end-of-lines (`CR`, `LF`), but I tend
16 to use it under Cygwin using Unix end-of-lines (`LF`).
17
18 * Smaller install burden: Python sometimes comes bundled with the operating
19 system; Haskell rarely does.
20
21 * Haskell, being lazy, makes it harder to deal with exceptions; unless the
22 Haskell expression is being evaluated both strictly and deeply, exceptions
23 can slip by. Haskell's facilities for forcing an expression to be
24 evaluated strictly and deeply are rather hacky (last I checked, the most
25 portable way to do so was to use `show`.) For Falderal's purposes, this
26 limitation seems artificial, at best.
27
28 * Relatedly, Python (or CPython, anyway) has better error behavior than
29 Haskell (or `ghc`, anyway); when it crashes, it dumps a backtrace (which I
30 can then analyze), instead of just saying something pithy like `Prelude:
31 empty list` (which I can't.)
32
33 Features
34 --------
35
36 `py-falderal` implements a slightly different subset of the
37 [Falderal Literate Test Format](Falderal_Literate_Test_Format.markdown) than
38 the Haskell implementation did.
39
40 In particular,
41
42 * It mainly implements `shell command` implementations. In practice, partly
43 due to the "strict & deep" evaluation problem mentioned above, that's how
44 I've been using Falderal anyway; also, its approach makes it somewhat more
45 useful for "end-to-end" testing of compilers and interpreters, than for
46 unit-testing individual text-processing functions inside a program.
47 (Technically, it implements `Haskell function` implementations too, but it's
48 something of a hack that uses `ghc -e`.)
49
50 * I plan for it to *only* understand indented Falderal blocks. The idea is
51 that the Falderal tests will almost certainly be embedded in a Markdown
52 document (possibly with Bird-style literate code also embedded therein,)
53 and no extra processing should be required to format something readable
54 from that Markdown. The four-space-indented Falderal blocks come out as
55 preformatted blocks, which is quite good enough. Relatedly,
56
57 * It does no formatting. There are a set of classes for parsing Falderal
58 files (`Document`s) into blocks (`Block`s), and methods for formatting
59 blocks into text. But aside from reporting the results of a test run,
60 they're not used by the utility to format Falderal files. If you want to
61 do extra processing on your Falderal/Markdown file so that Falderal
62 blocks are prettier, you certainly can do that, but you'll need to write
63 your own script which uses these classes, for it is outside the scope of
64 `py-falderal`.
65
66 * I'm not so sure about `-> Functionality "blah" is implemented by shell
67 command "flargh"` anymore. I mean, it should certainly be the case that
68 functionalities can have multiple implementations, but
69
70 * Perhaps implementations should not be specified in Falderal documents
71 at all — that approach is more abstract. But it requires them to be
72 specified on the tool's command line, which in practice requires there
73 to be a driver script to run the tests, for a particular implementation;
74 but this is not necessarily a bad thing. (The effective practice has
75 been, actually, to write out a bit of "configuration Markdown" with
76 that "Functionality is implemented by" snippet in it, and to load that
77 before loading the main tests. Rather than to configure the functionality
78 implementation(s) on the command line itself.)
79 * When `falderal` is run on more than one input file, what is the scope
80 of a functionality, and what is the scope of the implementations of a
81 functionality? Currently, in the Falderal semantics, that scope is
82 global, and maybe that is appropriate; but maybe finer-grained control
83 would be nice.
84
85 * `py-falderal` also does not try to optimize the test runs into a block of
86 test runs (which didn't work out so well in the Haskell implementation,
87 for `shell command`s, anyway.)
0 `py-falderal`
1 =============
2
3 `py-falderal` is an implementation of Falderal in Python 2.5.x.
4
5 Motivation
6 ----------
7
8 The original reference implementation of Falderal was written in Haskell.
9 There are a few reasons I had for re-implementing it in Python:
10
11 * The original Falderal implementation grew out of a Haskell-specific hack,
12 and it showed in how it was written.
13
14 * Fewer discrepancies between platforms. In particular, `ghc` for Windows
15 likes to operate in terms of MS-DOS end-of-lines (`CR`, `LF`), but I tend
16 to use it under Cygwin using Unix end-of-lines (`LF`).
17
18 * Smaller install burden: Python sometimes comes bundled with the operating
19 system; Haskell rarely does.
20
21 * Haskell, being lazy, makes it harder to deal with exceptions; unless the
22 Haskell expression is being evaluated both strictly and deeply, exceptions
23 can slip by. Haskell's facilities for forcing an expression to be
24 evaluated strictly and deeply are rather hacky (last I checked, the most
25 portable way to do so was to use `show`.) For Falderal's purposes, this
26 limitation seems artificial, at best.
27
28 * Relatedly, Python (or CPython, anyway) has better error behavior than
29 Haskell (or `ghc`, anyway); when it crashes, it dumps a backtrace (which I
30 can then analyze), instead of just saying something pithy like `Prelude:
31 empty list` (which I can't.)
32
33 Features
34 --------
35
36 `py-falderal` implements a slightly different subset of the
37 [Falderal Literate Test Format](Falderal_Literate_Test_Format.markdown) than
38 the Haskell implementation did.
39
40 In particular,
41
42 * It mainly implements `shell command` implementations. In practice, partly
43 due to the "strict & deep" evaluation problem mentioned above, that's how
44 I've been using Falderal anyway; also, its approach makes it somewhat more
45 useful for "end-to-end" testing of compilers and interpreters, than for
46 unit-testing individual text-processing functions inside a program.
47 (Technically, it implements `Haskell function` implementations too, but it's
48 something of a hack that uses `ghc -e`.)
49
50 * I plan for it to *only* understand indented Falderal blocks. The idea is
51 that the Falderal tests will almost certainly be embedded in a Markdown
52 document (possibly with Bird-style literate code also embedded therein,)
53 and no extra processing should be required to format something readable
54 from that Markdown. The four-space-indented Falderal blocks come out as
55 preformatted blocks, which is quite good enough. Relatedly,
56
57 * It does no formatting. There are a set of classes for parsing Falderal
58 files (`Document`s) into blocks (`Block`s), and methods for formatting
59 blocks into text. But aside from reporting the results of a test run,
60 they're not used by the utility to format Falderal files. If you want to
61 do extra processing on your Falderal/Markdown file so that Falderal
62 blocks are prettier, you certainly can do that, but you'll need to write
63 your own script which uses these classes, for it is outside the scope of
64 `py-falderal`.
65
66 * I'm not so sure about `-> Functionality "blah" is implemented by shell
67 command "flargh"` anymore. I mean, it should certainly be the case that
68 functionalities can have multiple implementations, but
69
70 * Perhaps implementations should not be specified in Falderal documents
71 at all — that approach is more abstract. But it requires them to be
72 specified on the tool's command line, which in practice requires there
73 to be a driver script to run the tests, for a particular implementation;
74 but this is not necessarily a bad thing. (The effective practice has
75 been, actually, to write out a bit of "configuration Markdown" with
76 that "Functionality is implemented by" snippet in it, and to load that
77 before loading the main tests. Rather than to configure the functionality
78 implementation(s) on the command line itself.)
79 * When `falderal` is run on more than one input file, what is the scope
80 of a functionality, and what is the scope of the implementations of a
81 functionality? Currently, in the Falderal semantics, that scope is
82 global, and maybe that is appropriate; but maybe finer-grained control
83 would be nice.
84
85 * `py-falderal` also does not try to optimize the test runs into a block of
86 test runs (which didn't work out so well in the Haskell implementation,
87 for `shell command`s, anyway.)
00 import setuptools
11
22
3 with open("README.markdown", "r") as fh:
3 with open("README.md", "r") as fh:
44 long_description = fh.read()
55
66 setuptools.setup(
77 name='Falderal',
8 version='0.13',
8 version='0.14',
99 description='Definition of, and tools for using, the Falderal literate testing format',
1010 long_description=long_description,
1111 long_description_content_type="text/markdown",