git @ Cat's Eye Technologies Falderal / e1f6f20
Show impl in failure, prevent results.txt race condition. catseye 10 years ago
5 changed file(s) with 59 addition(s) and 6 deletion(s). Raw diff Collapse all Expand all
9191 be styled independently from, and thus distinguished from, any plain
9292 Markdown indented code blocks which may appear in the literate portion
9393 of the source code.
94 * In failure reports, the implementation of the functionality of the test
95 that failed is now reported in each failure.
96 * A race condition(?) that could occur when testing multiple implementations
97 of a functionality, of different kinds (Haskell and shell), has been
98 prevented. Both tests were writing to `results.txt` and immediately
99 deleting it, and this would sometimes confuse `falderal` into thinking
100 one had produced no results (perhaps a result of some creative
101 scheduling by `ghc`, although really, I haven't a clue.) Results are
102 now written to different temporary files with different, generated
103 names.
94104
95105 TODO:
96106
97 * Figure out why one of the tests in wc.falderal doesn't run.
107 * Skip blank lines at end of each block in results. Test for missing eol.
98108 * Command-line option to clear functionalities.
99109 * Command-line option to skip tests for given functionalities.
100110
3535 data Functionality = HaskellTest String String -- module name, function name
3636 | ShellTest String -- command
3737 | NamedFunctionality String
38 deriving (Show, Eq, Ord)
38 deriving (Eq, Ord)
39
40 instance Show Functionality where
41 show (HaskellTest m f) = "Haskell function " ++ m ++ ":" ++ f
42 show (ShellTest c) = "Shell command \"" ++ c ++ "\""
43 show (NamedFunctionality f) = "functionality \"" ++ f ++ "\""
3944
4045 --
4146 -- ...and after.
1919 reportEachTest (Test id fns literalText testText expected (Just actual):rest) = do
2020 reportText 8 "FAILED" (stripLeading '\n' (stripTrailing '\n' literalText))
2121 putStrLn ""
22 reportText 8 "Impl" (show fn)
2223 reportText 8 "Input" testText
2324 reportText 8 "Expected" (show expected)
2425 reportText 8 "Actual" (show actual)
2526 putStrLn ""
2627 reportEachTest rest
28 where [fn] = fns
2729 reportEachTest (Test id fns literalText testText expected Nothing:rest) = do
2830 reportText 8 "NOT RUN" (stripLeading '\n' (stripTrailing '\n' literalText))
31 putStrLn ""
32 reportText 8 "Impl" (show fn)
2933 reportText 8 "Input" testText
3034 reportText 8 "Expected" (show expected)
3135 putStrLn ""
3236 reportEachTest rest
37 where [fn] = fns
3338
3439 reportText width fieldName text =
3540 if
3030 text <- return $ format formatName [] blocks
3131 hPutStr outputFileHandle text
3232 hClose outputFileHandle
33 exitCode <- system (command ++ " >results.txt")
34 contents <- readFile "results.txt"
33 (resultsFilename, h) <- openTempFile "." "results.txt"
34 hClose h
35 exitCode <- system (command ++ " >" ++ resultsFilename)
36 contents <- readFile resultsFilename
3537 results <- return $ collectResults $ lines $ contents
3638 cleanRun (not messy) ("rm -f " ++ filename)
37 cleanRun (not messy) ("rm -f results.txt")
39 cleanRun (not messy) ("rm -f " ++ resultsFilename)
3840 return $ decorateTestsWithResults blocks results
3941
4042 collectResults [] =
2222 This is an intentionally failing test, to demonstrate how Falderal will
2323 present it.
2424
25 Impl : Haskell function LiterateHaskellDemo:everySecond
2526 Input : Something
2627 Expected: Output "Anything"
2728 Actual : Output "oehn"
3031 Another intentionally failing test to demonstrate how Falderal will
3132 present expecting an exception and not getting one.
3233
34 Impl : Haskell function LiterateHaskellDemo:everySecond
3335 Input : ridiculous
3436 Expected: Exception "Prelude.head: empty list"
3537 Actual : Output "iiuos"
3840 An intentionally failing test to demonstrate that it is important
3941 to get the formatting of the output right, when testing with show.
4042
43 Impl : Haskell function LiterateHaskellDemo:showParseBits
4144 Input : 01
4245 Expected: Output "[False, True]"
4346 Actual : Output "[False,True]"
4649 An intentionally failing test to demonstrate show what a failure
4750 looks like on multi-line input.
4851
52 Impl : Haskell function LiterateHaskellDemo:showParseBits
4953 Input:
5054 01
5155 10
6064 annotated with the number of the test in the set that failed. The
6165 intentionally-failing third test below demonstrates this.
6266
67 Impl : Haskell function LiterateHaskellDemo:showParseBits
6368 Input : 0000
6469 Expected: Output "[False,False,False,Flse]"
6570 Actual : Output "[False,False,False,False]"
7277 E2=$?
7378 rm -f expected.txt actual.txt
7479
80 echo 'Testing wc.falderal (multiple impls, var exp...)'
81
82 cat >expected.txt <<EOF
83 --------------------------------
84 Total tests: 7, failures: 1
85 --------------------------------
86
87 FAILED : An intentionally failing test.
88
89 Impl : Shell command "wc -w"
90 Input : Not four words!
91 Expected: Output "4"
92 Actual : Output "3"
93
94 EOF
95 cd eg
96 falderal test wc.falderal >../actual.txt
97 cd ..
98 diff -u expected.txt actual.txt
99 EWC=$?
100 rm -f expected.txt actual.txt
101
75102 echo 'Testing Erroneous.falderal...'
76103
77104 cat >expected.txt <<EOF
80107 --------------------------------
81108
82109 NOT RUN : (#2)
110
111 Impl : Haskell function Erroneous:countLines
83112 Input:
84113 These are eight words
85114 that span two lines.
86115 Expected: Output "2"
87116
88117 NOT RUN : (#3)
118
119 Impl : Haskell function Erroneous:countLines
89120 Input:
90121 These are eight words
91122 that span
124155 E5=$?
125156 rm -f expected.txt actual.txt
126157
127 if [ $E1 != 0 -o $E2 != 0 -o $E3 != 0 -o $E4 != 0 -o $E5 != 0 ]
158 if [ $E1 != 0 -o $E2 != 0 -o $E3 != 0 -o $E4 != 0 -o $E5 != 0 -o $EWC != 0 ]
128159 then
129160 echo "Internal tests failed!"
130161 exit 1