tdelibs/kate/README.testing

 Testing Kate
==============

Author: Leo Savernik

Kate contains regression tests to ensure that fixed bugs do not reappear in
newer versions. To facilitate regression testing, a dedicated application
testkateregression will execute the regression tests and compare them to the
expecting results, indicating passed as well as failed testcases.


1. Using testkateregression
  --------------------------

We tried to make regression testing for Kate as easy as possible such that you
can run it before each commit and find out regressions caused by your changes
before they are shipped as part of a release.

Running all regression tests works by simply invoking

	> make check

in your kate build directory. While executing, testkateregression prints a line
for each executed testcase, prefixed with "PASS" if it passed, and "FAIL" if it
failed. Furthermore, testkateregression stores a comprehensive output log under
<katetests-directory>/output/index.html. The output log is invaluable for
determining why a certain testcase failed.

If you invoke testkateregression the first time, it will print instructions on
how to fetch the testsuite and pointing testkateregression to it. This setup
has only to be done once per branch.


2. Discriminating your regressions against existing regressions
  --------------------------------------------------------------

In an ideal universe, all testcases always pass. In this universe, however,
some testcases fail, be it because of anticipating future features not
implemented yet, be it because of nasty bugs which cannot be repaired easily.

This means if you've hacked on kate for quite some while and then fire up
"make check", you are likely to see many failed tests pass by, most of them
*not* caused by your very changes, as they failed already before.

To discriminate the failed tests caused by your changes against the unaffected
failures, testkateregression provides the option --save-failures=<name>, which
runs the regression tests and stores all failures under a failure snapshot
identified by <name>.

The next time you run "make check", testkateregression automatically picks up
the most recently stored failure snapshot and compares the failures and passes
with the one stored in the snapshot. Each failure not listed in the failure
snapshot will be prefixed with "FAIL (new)", indicating that this is a new
failure. Testcases which failed in the snapshot but do pass now are prefixed
with "PASS (new)", indicating that this testcase seems to be fixed now.


3. Using testkateregression efficiently
  --------------------------------------

Therefore, to get the most out of regression testing, we suggest the following
development approach:

   1. Before you change Kate, update and run testkateregression in the part-
      subdirectory.

	> make testkateregression && ./testkateregression --save-failures=last

      This will produce a failure snapshot called "last".
   
   2. Hack on Kate.

   3. Before you commit, run

	> make check

      It will automatically pick up the failure snapshot "last" (provided you
      didn't generate a newer one in the meantime) and compare all results with
      the previously stored ones.
      
      If you inspect <katetests-directory>/output/index.html, the new failures
      are marked red. Those are of interest to you, because they have been
      caused by your changes.
      
      New passes are marked green. These were former failures which started
      working due to your changes.
      
      Goto 2 while there are any new failures.
      
   4. Commit.


4. Invoking testkateregression directly
  --------------------------------------
  
While make check is handy and simple enough for the common case, you might
sometimes need more control over regression testing.

testkateregression features a broad range of options, enabling you to run
dedicated testcases only, specifying an alternate output directory for the
logs, etc.

	> ./testregression --help

will provide you with a complete list of options.


5. Structure of the regression test suite
  ----------------------------------------

Kate's regression testsuite is located in the KDE repository under

	trunk/tests/katetests/regression

and consists of two subdirectories

	baseline
	tests

The latter, tests, contains a directory hierarchy for all testcases to be run
by testkateregression. The former, baseline, contains results as they are
expected by correct operation. Mismatch between the output of a test and its
baseline is considered to be a failure.

Each directory under tests may optionally contain one of the following files.

	.kateconfig
	.kateconfig-commands
	ignore
	KNOWN_FAILURES

.kateconfig: This file works exactly like .kateconfig as supported by the kate
and kwrite editors. It may contain any kate line variable necessary to set up
the testcases proper. Note that .kateconfig files from parent directories are
not merged with .kateconfig files from child directories.

.kateconfig-commands: This file may contain all commands that can be entered by
kate's command line (F7). Each line will be interpreted as one command. To the
contrary of .kateconfig, .kateconfig-commands files are merged with
.kateconfig-commands files from parent directories. Nearer ancestors' commands
take precedence over farther ancestors'.

ignore: This file specifies on each line a file to be ignored in the directory
the ignore-file is located. This enables you to mark any helper files which
otherwise would be interpreted as testcases. Note that hidden files (.*) are
ignored by default, and cannot be "unignored".

KNOWN_FAILURES: This file specifies on each line a file name of a testcase
which is known to fail. Such known failures are counted towards the total count
of failures but they don't cause testkateregression to return a failure code.


6. Structure of a testcase
  -------------------------

A testcase is comprised of a simple plain text file <testcase>.txt which may
be located in any subdirectory under tests. This file contains the *initial*
content the testcase operates on.

Each <testcase>.txt must be accompanied with a <testcase>.txt-script which
contains the actual tests to be performed on the testcase. It consists of
simple JavaScript-statements for direct interfacing with Kate.

Last but not least, a <testcase>.txt-result exists under the baseline
subdirectory, which contains a mirrored directory hierarchy of tests. This very
file contains the expected *result* of the performed tests.


7. Writing a simple testcase
  ---------------------------

Writing your own testcases is easy once you know how to get started. Let's
test how Kate's C-Style indenter fares with indenting after opening braces.

First, we create the new initial content under tests/indent/csmart/openbrc.txt
and fill it with (the dashed lines are not part of the content)
---------------------------

int main() {

---------------------------

Now, we need to write a script performing some actions. We therefore create
a file tests/indent/csmart/openbrc.txt-script and fill it with
---------------------------
v.setCursorPosition(1,12);
v.enter();
v.type("good");
---------------------------

Here, we set the initial cursor position to line 2 (the coordinates are zero-
based) and column 13 which happens to be just after the opening brace. Then
v.enter() simulates pressing the return key in the editor, thus inserting a
new line. v.type simulates typing of the word "good" at the current position
of the cursor.

The options under .kateconfig specify the C-Style indenter to be applied to the
testcases and an indent width of two. With this information, we know what we
expect as a result.

What we are still missing is the expected result itself which we create under
baseline/indent/csmart/openbrc.txt-result and fill it with
---------------------------

int main() {
  good

---------------------------

You can see that "good" is indented by two spaces, even though we didn't
specify those with v.type. We expect from the indenter to provide them for us.

Last but not least we test the testcase by invoking in kate's part directory

	> ./testkateregression indent/csmart/openbrc.txt

and checking whether it works the way we intended it.


7. The JavaScript-interface to the testcases
  -------------------------------------------
  
testkateregression provides you with the following global objects for each
testcase:

	v - object of view
	d - object of document

Each object provides the same methods and fields as the respective JavaScript-
interfaces built in to Kate, like v.setCursorPosition.

Additionally, v provides the following methods unique to testkateregression.

type(<string>)
	Inserts <string> into the current cursor position as if <string> had
	been typed by the keyboard. Contrary to insert(<string>), it will
	trigger indentation and other checks.
enter(), returnKey()
	Inserts a new line as if the return key had been pressed. This will
	trigger special indentation rules.