|
0 |
Falderal Literate Test Format
|
|
1 |
=============================
|
|
2 |
|
|
3 |
This document describes the proposed Falderal Literate Test Format.
|
|
4 |
|
|
5 |
Status
|
|
6 |
======
|
|
7 |
|
|
8 |
This document is a *draft*. It is nominally "version 0.4" because it
|
|
9 |
describes something that version 0.4 of `Test.Falderal` mostly implements.
|
|
10 |
We will deign to note which sections of this document the current released
|
|
11 |
version of `Test.Falderal` implements, and which it does not. However,
|
|
12 |
this document is a work in progress, subject to change, and subject to get
|
|
13 |
out of sync with `Test.Falderal`. It should not be considered to be
|
|
14 |
anything except a draft until it is described as "version 1.0".
|
|
15 |
|
|
16 |
Overview
|
|
17 |
========
|
|
18 |
|
|
19 |
A Falderal Literate Test Suite is a text file where some of the lines
|
|
20 |
have special meaning to the Falderal Literate Test Format. Certain
|
|
21 |
groupings of the lines defined by this format are intended to denote tests.
|
|
22 |
A tool which claims to understand this format may chose to extract these
|
|
23 |
lines, interpret them as tests, and execute these tests and report which
|
|
24 |
and how many of those tests passed or failed. It may also reformat these
|
|
25 |
lines to produce an output file which is no longer in the Falderal
|
|
26 |
Literate Test Format, but in some other format, generally to present the
|
|
27 |
tests and their descriptions in a nicely human-readable fashion.
|
|
28 |
|
|
29 |
Syntax
|
|
30 |
======
|
|
31 |
|
|
32 |
Lines which have special meaning to the Falderal Literate Test Format
|
|
33 |
always begin (in the leftmost column) with a series of special characters,
|
|
34 |
called an //introducer//. The introducers which have meaning to the
|
|
35 |
Falderal Literate Test Format are as follows:
|
|
36 |
|
|
37 |
* `->` (hyphen, greater-than sign): pragma
|
|
38 |
* `| ` (vertical bar, space): input text
|
|
39 |
* `= ` (equals sign, space): expected output text
|
|
40 |
* `? ` (question mark, space): expected error text
|
|
41 |
* `> ` (greater-than sign, space): Bird-style embedded code
|
|
42 |
|
|
43 |
If the same introducer occurs on multiple adjacent lines, the text after
|
|
44 |
each introducer is concatenated to form one chunk of relevant text. This
|
|
45 |
allows, for example, multi-line text to be given as input or expected
|
|
46 |
output of a test.
|
|
47 |
|
|
48 |
There are some restrictions on the order in which introducers can sensibly
|
|
49 |
occur. Expected output or error text must follow input text, with no
|
|
50 |
intervening introducers or non-introducer lines. See the sections for
|
|
51 |
these introducers, below, for more details.
|
|
52 |
|
|
53 |
Bird-style embedded code is not considered part of a test by the Falderal
|
|
54 |
Literate Test Format, but may be recognized as such by a formatting tool,
|
|
55 |
for purposes of formatting.
|
|
56 |
|
|
57 |
Lines which do not begin with an introducer are classified as either blank
|
|
58 |
or non-blank. A line is blank if it contains no characters, or if it
|
|
59 |
contains only whitespace. A group of non-blank lines is referred to as a
|
|
60 |
_paragraph_.
|
|
61 |
|
|
62 |
Pragmas
|
|
63 |
=======
|
|
64 |
|
|
65 |
Functionality-definition
|
|
66 |
------------------------
|
|
67 |
|
|
68 |
The Functionality-definition pragma allows a Falderal file to describe
|
|
69 |
ways in which a functionality being tested is implemented. It has the
|
|
70 |
following syntax:
|
|
71 |
|
|
72 |
-> Functionality /functionality-name/ is implemented by /functionality-type/ /functionality-specifier/
|
|
73 |
|
|
74 |
_functionality-type_ may be either `Haskell function` or `shell command`.
|
|
75 |
The format of the _functionality-specifier_ differs with each of these.
|
|
76 |
The _functionality-name_ is arbitrary text enclosed within double quotes,
|
|
77 |
which may be referenced in a later Tests-for pragma.
|
|
78 |
|
|
79 |
Note that the Functionality-definitions given in a Falderal file should
|
|
80 |
not be considered exhaustive, or even requisite, by a tool. The tool may
|
|
81 |
accept additional definitions of the name of a functionality, from an
|
|
82 |
external source such as the command line or a configuration file, and may
|
|
83 |
be instructed to ignore certain Functionality-definitions in a Falderal
|
|
84 |
file (if, for example, certain implementation are not currently available
|
|
85 |
or of interest to the user.) Indeed, the functionality referred to by a
|
|
86 |
_functionality-name_ in a Tests-for pragma need not be defined by any
|
|
87 |
Functionality-definition pragma in the same Falderal file, and this
|
|
88 |
situation requires the functionality to be specified to the tool in some
|
|
89 |
other manner.
|
|
90 |
|
|
91 |
Haskell functions
|
|
92 |
-----------------
|
|
93 |
|
|
94 |
For Haskell functions, the _functionality-specifier_ is in the format
|
|
95 |
`Module:functionName`. The function should have a signature of
|
|
96 |
`String -> String`. If the function raises an exception, the text of that
|
|
97 |
exception will be compared against expected error output, if any. For
|
|
98 |
example:
|
|
99 |
|
|
100 |
-> Functionality 'Reversing a string' is implemented by Haskell function Data.Backwards:reverseString
|
|
101 |
|
|
102 |
Shell commands
|
|
103 |
--------------
|
|
104 |
|
|
105 |
For shell commands, the _functionality-specifier_ is in the format
|
|
106 |
`"command arg1 arg2 ... argn"`. Any line of legal Bourne shell syntax may
|
|
107 |
be used, so pipes, redirection, etc., are supported. The sequence
|
|
108 |
`%input`, if it appears in the string, is replaced by the name of a
|
|
109 |
temporary file which is to be created to hold the input text. If it does
|
|
110 |
not appear, the input text will be provided on the standard input of the
|
|
111 |
shell command. The sequence `%output`, if it appears in the string,
|
|
112 |
is replaced by the name of a temporary file from which the output text
|
|
113 |
will be read. If it does not appear, the output text will be read from
|
|
114 |
the standard output. Shell commands do not (yet) support expected error
|
|
115 |
output.
|
|
116 |
|
|
117 |
For example:
|
|
118 |
|
|
119 |
-> Functionality 'Prepending foo.txt' is implemented by shell command "cat foo.txt %input > %output"
|
|
120 |
|
|
121 |
`Test.Falderal` version 0.4 implements shell commands, but does not yet
|
|
122 |
support the `%input` and `%output` sequences.
|
|
123 |
|
|
124 |
Tests-for
|
|
125 |
---------
|
|
126 |
|
|
127 |
The Tests-for pragma determines what functionality will be used to run all
|
|
128 |
following tests, until the next Tests-for pragma. It has the following
|
|
129 |
syntax:
|
|
130 |
|
|
131 |
-> Tests for _functionality-name_
|
|
132 |
|
|
133 |
The _functionality-name_ refers to a functionality, which may be specified
|
|
134 |
by a Functionality-definition pragma elsewhere in the Falderal file.
|
|
135 |
|
|
136 |
For example:
|
|
137 |
|
|
138 |
-> Tests for 'Reversing a string'
|
|
139 |
|
|
140 |
Alternatively, the following direct way of associating tests with an
|
|
141 |
implementation of a functionality, may be used. However, this direct way
|
|
142 |
of specifying a functionality is discouraged when there may be conceivably
|
|
143 |
be multiple implementation of the functionality.
|
|
144 |
|
|
145 |
-> Tests for _functionality-type_ _functionality-specifier_
|
|
146 |
|
|
147 |
_functionality-type_ and _functionality-specifier_ have the same meaning
|
|
148 |
as given in the description of the Functionality-definition pragma.
|
|
149 |
|
|
150 |
For example:
|
|
151 |
|
|
152 |
-> Tests for Haskell function Data.Backwards:reverseString
|
|
153 |
|
|
154 |
Input and Expected Text
|
|
155 |
=======================
|
|
156 |
|
|
157 |
Each section of input text should be followed immediately by either a
|
|
158 |
section of expected output or expected error output. It may also be
|
|
159 |
preceded by a paragraph of text; this paragraph should be displayed along
|
|
160 |
with the input text and expected text, in a test report.
|
|
161 |
|
|
162 |
Discussion
|
|
163 |
==========
|
|
164 |
|
|
165 |
(This section is non-normative, and possibly out-of-date.)
|
|
166 |
|
|
167 |
The format of the lines which comprise the Falderal Literate Test Format
|
|
168 |
was chosen to not conflict with many other common text formats (including
|
|
169 |
but not limited to Bird-style Literate Haskell, and Markdown); thus
|
|
170 |
literate test suites may be embedded in a wide variety of other formats.
|
|
171 |
However, there are inevitably some conflicts with some textual formats;
|
|
172 |
for example, when embedded in C code and many other languages, Falderal
|
|
173 |
entries should be preceded by `/*` and followed by `*/`, to ensure that
|
|
174 |
they are regarded as comments. Also, reStructuredText uses the `|` line
|
|
175 |
prefix to denote preformatted plain text.
|
|
176 |
|
|
177 |
The format of pragmas was chosen such that they could be read literately,
|
|
178 |
and as such, a formatting tool could format them in the output document
|
|
179 |
with little if any change.
|
|
180 |
|
|
181 |
The Haskell function functionality type is essentially an optimization,
|
|
182 |
and a convenience to avoid writing Haskell code for a system program which
|
|
183 |
reads strings, passes them to a function, and outputs the result. However,
|
|
184 |
restricting oneself to Haskell functions provides a fairly good guarantee
|
|
185 |
that the tests are idempotent; shell commands may change the state of the
|
|
186 |
system. (It is of course recommended that when shell commands are used,
|
|
187 |
care is taken to leave the system in the same state as it was when testing
|
|
188 |
started.)
|
|
189 |
|
|
190 |
Here is why directly specifying the functionality implementation in a
|
|
191 |
Tests-for pragma is discouraged. Saying `-> Tests for Haskell function
|
|
192 |
Foo:blah` ties a set of tests to a particular function which is being
|
|
193 |
tested, but a large part of the point of Falderal is to let you write
|
|
194 |
tests for a *language*, and that language might have many implementations.
|
|
195 |
Decoupling them allows you to change what actual functions or programs
|
|
196 |
are being tested, and basically allow you to have multiple implementations
|
|
197 |
of a language and use the same tests for all of them.
|