zebrajs update, testing and testcase generation

What do you think of this post?
  • Awesome (0.0%)
  • Interesting (0.0%)
  • Useful (0.0%)
  • Boring (0.0%)
  • Sucks (0.0%)

ZebraJS Update

The last few days I have been rewriting the variables lib, this updates includes:

  • Added examples (exemples/logic/add.js), from my previews post,
  • Better memory management with version system (commit, revert, remove) instead the old dumb stack (save and load) system,
  • simplified the code to achieve better performance,
  • Testing and debugging libs:
    • lib/variables_test.js: Monitors all lib operations and throws a exception if it finds a inconsistency on the lib,
    • lib/variables_test_msg.js: Monitors all lib operations and stops the program with a error message,
    • lib/variables_testcase.js: Monitors all lib operations and stops the program with a error message and generates a testcase for mocha.

The sudoku puzzle takes now, almost 3 seconds to solve and check for all possibilities:
real 0m2.837s
user 0m2.496s
sys 0m0.038s

I think its good since we have to take into account the time nodejs takes to start up the program.

Testing and testcase generation

When we rewrite part of a program bugs will always appear especially on big rewrites, that’s why having a test framework is useful, we run all the test and fix everything that doesn’t pass the tests, then we run our examples and they all fail and burn lol…
So I think its good practice to find bugs and write some more test, this will prevent future fail, and will also confirm your bug suspicions…The problem is when you write a bunch of testcases but all of them passes, meaning that the bug is more tricky than you would expect.

So after I got tired to write tests, I tried to find a good tool to help me, but I found none, the problem was:

  • Debugger: Node-inspector doesn’t work very well for me and besides debugging requires a little work and inspection to find bugs,
  • Profiler: I cant understand the output of profilers and cant get the information that I need,
  • Symbolic Execution: I have been reading about this, seems great but didn’t find anything for nodejs,
  • Other options: There is probably other options, I just made a quick search didn’t bother to look how to personalize the tools to make what I expected…shame on me 😛 (any suggestion or comments are welcome)

So I decided to take on other approach, I made a proxy class, that has this features:

  • Monitor all calls of a object,
  • For every call made it constructs a call trace with the provided arguments,
  • Extensible with the functions:
    • before: execute before call,
    • after: execute after call,
    • error: execute if a exception occurs
  • Run the trace again and generate a minimal test case.
  • Setting up proxy trace should be only a matter of changing the require libs to proxy libs, no other changes in the code should be made.

And this is how its working for zebrajs lib:

  • I created a proxy to Variables and VariablesFactory,
  • On after and before calls I check my variables integrity with the should.js lib, if something wrong occurs I throw an exception,
  • On my error handling function I call proxy.sandbox to generate a small test case where the error occurs,
  • I go to one of examples, and change the ‘require(“lib/variables”)’ with my proxy ‘require(“lib/variables_testcase”)’, and then run the example like this
    ‘nodejs my_example.js > log.txt’. If a error occurs it will present me with a testcase on the bottom of the log.txt.
  • I copy the testcase to my mocha test/ folder, normally I change the testcase adding more tests, check the expected values and change them to what they should be, ect, etc…
  • I then correct the bugs, check if everything is right and keep the testcase for future development 😉

Until now I think this proxy object is pretty useful to test “real” programs/examples, of course not everything is great, proxy lib may have bugs (it probably has, since the code is a mess), the check functions may have bugs, the performance is decreased a lot and some testcases may take some time to generate.

Here is a testcase that was generated (This error occur as normal exception of js, the test case generated was really big, I then added some checks to catch the bug early…):

 it("should f_0 commit", function() {
                var f_0 = new VariableFactory();
                var v_71 = f_0.v({domain:[1,2,3,4,5,6,7,8,9], });
                var v_80 = f_0.v({domain:[1,2,3,4,5,6,7,8,9], });
                        invalid version number < 0
                        f_0 commit
                should(f_0.commit()).eql(0); // I changed this from -1 to 0 since it was the correct expected value,

This is a little trick bug, since it didn’t occur if I comment one of the lines with notUnify, It was cool to see that the test case generated was
exactly the smallest to the error occur, it made my day lol...
And that’s it 😉 ...

Testcase generator algorithm

The testcase generator is a pretty dumb piece of code, when a error occurs I do the following on proxy.sanbox:

  • It grabs only the calls on the root of the trace, its not interested on internal calls, and do a list,
  • After call list is created it will setup a "clean" and "controlled" environment to execute the calls,
  • Then it runs the call list, ignoring calls from the bottom of the list, if the error still occurs than it can remove the ignored call,
    it continues to do this until it reaches the beginning of the list,
  • It then creates a string, as a mocha testcase, with the remaining calls in the list, since it has all information like returned values it can
    create assertions of the expected values, of course some of them may be wrong but that’s must be checked by the programmer, and a wrong assertion is
    definitely a bug.

Well that’s it, Happy codding.

What do you think of this post?
  • Awesome (0.0%)
  • Interesting (0.0%)
  • Useful (0.0%)
  • Boring (0.0%)
  • Sucks (0.0%)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>