git @ Cat's Eye Technologies Falderal / 077d58f
Add documents, from wiki, plus a doc describing the mechanism. catseye 8 years ago
3 changed file(s) with 300 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
+198
-0
doc/Falderal_Literate_Test_Format.markdown less more
0 Falderal Literate Test Format
1 =============================
2
3 This document describes the proposed Falderal Literate Test Format.
4
5 Status
6 ======
7
8 This document is a *draft*. It is nominally "version 0.4" because it
9 describes something that version 0.4 of `Test.Falderal` mostly implements.
10 We will deign to note which sections of this document the current released
11 version of `Test.Falderal` implements, and which it does not. However,
12 this document is a work in progress, subject to change, and subject to get
13 out of sync with `Test.Falderal`. It should not be considered to be
14 anything except a draft until it is described as "version 1.0".
15
16 Overview
17 ========
18
19 A Falderal Literate Test Suite is a text file where some of the lines
20 have special meaning to the Falderal Literate Test Format. Certain
21 groupings of the lines defined by this format are intended to denote tests.
22 A tool which claims to understand this format may chose to extract these
23 lines, interpret them as tests, and execute these tests and report which
24 and how many of those tests passed or failed. It may also reformat these
25 lines to produce an output file which is no longer in the Falderal
26 Literate Test Format, but in some other format, generally to present the
27 tests and their descriptions in a nicely human-readable fashion.
28
29 Syntax
30 ======
31
32 Lines which have special meaning to the Falderal Literate Test Format
33 always begin (in the leftmost column) with a series of special characters,
34 called an //introducer//. The introducers which have meaning to the
35 Falderal Literate Test Format are as follows:
36
37 * `->` (hyphen, greater-than sign): pragma
38 * `| ` (vertical bar, space): input text
39 * `= ` (equals sign, space): expected output text
40 * `? ` (question mark, space): expected error text
41 * `> ` (greater-than sign, space): Bird-style embedded code
42
43 If the same introducer occurs on multiple adjacent lines, the text after
44 each introducer is concatenated to form one chunk of relevant text. This
45 allows, for example, multi-line text to be given as input or expected
46 output of a test.
47
48 There are some restrictions on the order in which introducers can sensibly
49 occur. Expected output or error text must follow input text, with no
50 intervening introducers or non-introducer lines. See the sections for
51 these introducers, below, for more details.
52
53 Bird-style embedded code is not considered part of a test by the Falderal
54 Literate Test Format, but may be recognized as such by a formatting tool,
55 for purposes of formatting.
56
57 Lines which do not begin with an introducer are classified as either blank
58 or non-blank. A line is blank if it contains no characters, or if it
59 contains only whitespace. A group of non-blank lines is referred to as a
60 _paragraph_.
61
62 Pragmas
63 =======
64
65 Functionality-definition
66 ------------------------
67
68 The Functionality-definition pragma allows a Falderal file to describe
69 ways in which a functionality being tested is implemented. It has the
70 following syntax:
71
72 -> Functionality /functionality-name/ is implemented by /functionality-type/ /functionality-specifier/
73
74 _functionality-type_ may be either `Haskell function` or `shell command`.
75 The format of the _functionality-specifier_ differs with each of these.
76 The _functionality-name_ is arbitrary text enclosed within double quotes,
77 which may be referenced in a later Tests-for pragma.
78
79 Note that the Functionality-definitions given in a Falderal file should
80 not be considered exhaustive, or even requisite, by a tool. The tool may
81 accept additional definitions of the name of a functionality, from an
82 external source such as the command line or a configuration file, and may
83 be instructed to ignore certain Functionality-definitions in a Falderal
84 file (if, for example, certain implementation are not currently available
85 or of interest to the user.) Indeed, the functionality referred to by a
86 _functionality-name_ in a Tests-for pragma need not be defined by any
87 Functionality-definition pragma in the same Falderal file, and this
88 situation requires the functionality to be specified to the tool in some
89 other manner.
90
91 Haskell functions
92 -----------------
93
94 For Haskell functions, the _functionality-specifier_ is in the format
95 `Module:functionName`. The function should have a signature of
96 `String -> String`. If the function raises an exception, the text of that
97 exception will be compared against expected error output, if any. For
98 example:
99
100 -> Functionality 'Reversing a string' is implemented by Haskell function Data.Backwards:reverseString
101
102 Shell commands
103 --------------
104
105 For shell commands, the _functionality-specifier_ is in the format
106 `"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may
107 be used, so pipes, redirection, etc., are supported. The sequence
108 `%input`, if it appears in the string, is replaced by the name of a
109 temporary file which is to be created to hold the input text. If it does
110 not appear, the input text will be provided on the standard input of the
111 shell command. The sequence `%output`, if it appears in the string,
112 is replaced by the name of a temporary file from which the output text
113 will be read. If it does not appear, the output text will be read from
114 the standard output. Shell commands do not (yet) support expected error
115 output.
116
117 For example:
118
119 -> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %input > %output"
120
121 `Test.Falderal` version 0.4 implements shell commands, but does not yet
122 support the `%input` and `%output` sequences.
123
124 Tests-for
125 ---------
126
127 The Tests-for pragma determines what functionality will be used to run all
128 following tests, until the next Tests-for pragma. It has the following
129 syntax:
130
131 -> Tests for _functionality-name_
132
133 The _functionality-name_ refers to a functionality, which may be specified
134 by a Functionality-definition pragma elsewhere in the Falderal file.
135
136 For example:
137
138 -> Tests for 'Reversing a string'
139
140 Alternatively, the following direct way of associating tests with an
141 implementation of a functionality, may be used. However, this direct way
142 of specifying a functionality is discouraged when there may be conceivably
143 be multiple implementation of the functionality.
144
145 -> Tests for _functionality-type_ _functionality-specifier_
146
147 _functionality-type_ and _functionality-specifier_ have the same meaning
148 as given in the description of the Functionality-definition pragma.
149
150 For example:
151
152 -> Tests for Haskell function Data.Backwards:reverseString
153
154 Input and Expected Text
155 =======================
156
157 Each section of input text should be followed immediately by either a
158 section of expected output or expected error output. It may also be
159 preceded by a paragraph of text; this paragraph should be displayed along
160 with the input text and expected text, in a test report.
161
162 Discussion
163 ==========
164
165 (This section is non-normative, and possibly out-of-date.)
166
167 The format of the lines which comprise the Falderal Literate Test Format
168 was chosen to not conflict with many other common text formats (including
169 but not limited to Bird-style Literate Haskell, and Markdown); thus
170 literate test suites may be embedded in a wide variety of other formats.
171 However, there are inevitably some conflicts with some textual formats;
172 for example, when embedded in C code and many other languages, Falderal
173 entries should be preceded by `/*` and followed by `*/`, to ensure that
174 they are regarded as comments. Also, reStructuredText uses the `|` line
175 prefix to denote preformatted plain text.
176
177 The format of pragmas was chosen such that they could be read literately,
178 and as such, a formatting tool could format them in the output document
179 with little if any change.
180
181 The Haskell function functionality type is essentially an optimization,
182 and a convenience to avoid writing Haskell code for a system program which
183 reads strings, passes them to a function, and outputs the result. However,
184 restricting oneself to Haskell functions provides a fairly good guarantee
185 that the tests are idempotent; shell commands may change the state of the
186 system. (It is of course recommended that when shell commands are used,
187 care is taken to leave the system in the same state as it was when testing
188 started.)
189
190 Here is why directly specifying the functionality implementation in a
191 Tests-for pragma is discouraged. Saying `-> Tests for Haskell function
192 Foo:blah` ties a set of tests to a particular function which is being
193 tested, but a large part of the point of Falderal is to let you write
194 tests for a *language*, and that language might have many implementations.
195 Decoupling them allows you to change what actual functions or programs
196 are being tested, and basically allow you to have multiple implementations
197 of a language and use the same tests for all of them.
+64
-0
doc/Quick_Start.markdown less more
0 Quick Start
1 ===========
2
3 First, install `Test.Falderal`.
4
5 % hg clone https://bitbucket.org/catseye/falderal/ -r rel_0_4
6 % cd falderal
7 % cabal install --prefix=$HOME --user
8
9 Define a programming language, or some other file format -- basically,
10 anything you can model as a function which takes strings to strings. In
11 Falderal terminology, this is a "functionality". Implement your
12 functionality in any programming language for which you can produce
13 executables for your system. (If you implement it in Haskell, you get
14 some side benefits, but it's not necessary.)
15
16 Often, depending on the syntax of your implementation language, you can
17 place your literate tests in the same file as your code. We'll use
18 Bird-style literate Haskell in this example.
19
20 #!lhs
21 module Gobvert
22
23 This is some kind of really trivial little language.
24
25 > gobvert "A" = "Z"
26 > gobvert "Z" = "A"
27
28 Then give your functionality a name, and write some tests for your
29 functionality. You use a Falderal pragma to identify which functionality
30 these tests are for.
31
32 #!lhs
33
34 -> Functionality "Gobvert a string" is implemented by Haskell function Gobvert:gobvert
35
36 -> Tests for functionality "Gobvert a string"
37
38 The gobversion of A is Z.
39
40 | A
41 = Z
42
43 The gobversion of Z is A.
44
45 | Z
46 = A
47
48 The gobversions of other letters are not defined.
49
50 | Q
51 ? Not matched
52
53 Then, use the {{{falderal}}} tool to run these tests:
54
55 % falderal test Gobvert.lhs
56
57 All failures will be listed in a nicely-formatted report, including the
58 literate description that appears before each failing test.
59
60 You can also use the `falderal` tool to format your literate Haskell
61 file, including embedded tests, to a document format such as Markdown:
62
63 % falderal format markdown Gobvert.lhs >Gobvert.markdown
+38
-0
doc/Theory_of_Operation.markdown less more
0 Theory of Operation
1 ===================
2
3 The `falderal` tool from the `Test.Falderal` implementation of the
4 Falderal Literate Test Format allows the user to format Falderal tests
5 to different formats, and to run those tests and generate a report.
6
7 This document briefly describes how it works internally.
8
9 When `falderal` is asked to run a set of tests, first it formats them
10 to a set of programs which run the functionalities being tested with the
11 input text of the tests. These programs are called *results generators*.
12 Since each test may have one or more implementations, multiple results
13 generators may be generated, one for each implementation language
14 (currently Haskell and Bourne shell).
15
16 Each results generator runs many functions in a batch, for efficiency.
17 The results of running the functions are written to standard output
18 (which is redirected to a temporary file by `falderal`) in an intermediate
19 format. `falderal` then reads these temporary files, parses the
20 intermediate format, checks which of the test results do not match the
21 expected output, and generates a test report based on that.
22
23 The intermediate format is a simple text-based format containing groups of
24 lines. A single group may look like the following.
25
26 output
27 4
28 2
29 Test with ID 4 generated
30 two lines of output.
31
32 The first line is either `output` or `exception`. The next line contains
33 the ID of the test that generated this result. The line following that
34 contains the number of lines of text that the test generated (call it _n_).
35 The next _n_ lines contain the actual text generated. (If _n_ = 0, there
36 will be no such lines.) Immediately following this group will be either
37 another group, or the end-of-file.