Better Reporting and Errors in web-component-tester

The sad path with web-component-tester is a little worrisome. So far, web-component-tester (WCT) has handled pretty much everything I had to throw at it with aplomb. It seems every bit a well thought out and implemented test framework for Polymer. But I have mostly worked with successful tests—the happy path. On occasion, something unexpected went wrong and I had trouble figuring out where. That bothered me enough to note, but at the time, I focused on the task at hand.

Now, I look closer at failures—the sad path.

I start by breaking a test intentionally so that I can take a closer look at what WCT reports:

Running "wct-test:local" (wct-test) task
Starting Selenium server for local browsers
Selenium server running on port 33554
Web server running on port 36678 and serving from /home/chris/repos/polymer-book/play/web-component-tester
firefox 37               Beginning tests via http://localhost:36678/js/test/index.html?cli_browser_id=0
firefox 37               ✖ index.html » <x-pizza> » adding a whole topping » updates the pizza state accordingly

  expected '{"firstHalfToppings":[],"secondHalfToppings":[],"wholeToppings":["green peppers"]}' to equal '{"firstHalfToppings":[],"secondHalfToppings":[],"wholeToppings":["red peppers"]}'

firefox 37               Tests failed: 1 failed tests
Test run ended in failure: 1 failed tests

firefox 37 (4/0/1)                    
1 failed tests

I have 5 tests scattered across 4 test files. Which one broke? I know the answer in this case because I intentionally broke it, but had I been merrily coding a separate feature when greeted by this there is no indication of where the failure occurred. My only option would be to hunt for the test description. That would be incredibly tedious if I have multiple tests with the same description (e.g. in different contexts).

I note that using Chrome instead of Firefox will report the bad line number:

chrome 41                ✖ index.html » <x-pizza> » adding a whole topping » updates the pizza state accordingly

  expected '{"firstHalfToppings":[],"secondHalfToppings":[],"wholeToppings":["green peppers"]}' to equal '{"firstHalfToppings":[],"secondHalfToppings":[],"wholeToppings":["red peppers"]}'
    <unknown> at   Context.<anonymous> at /js/test/page_objects.js:79:0

That can be easy to miss—especially if I am running more than one browser at a time. So I hope to get richer information from WCT with all browser.

WCT's --verbose option is not the answer. That is a swamp of output that, if anything, makes it harder to find the error. Instead, I try the --expanded option by adding it to my project's wct.cong.js file:

module.exports = {
  expanded: true,
  plugins: {
    local: {
      browsers: ['firefox']
    }
  }
};

Not surprisingly, this results in expanded output:

Running "wct-test:local" (wct-test) task
Starting Selenium server for local browsers
Selenium server running on port 57440
Web server running on port 53285 and serving from /home/chris/repos/polymer-book/play/web-component-tester
firefox 37               Beginning tests via http://localhost:53285/js/test/index.html?cli_browser_id=0
firefox 37               ✓ index.html » <x-pizza> » can add toppings to the whole pizza
firefox 37               ✓ index.html » <x-pizza> » defaults » has no toppings anywhere
firefox 37               ✖ index.html » <x-pizza> » adding a whole topping » updates the pizza state accordingly

  expected '{"firstHalfToppings":[],"secondHalfToppings":[],"wholeToppings":["green peppers"]}' to equal '{"firstHalfToppings":[],"secondHalfToppings":[],"wholeToppings":["blue peppers"]}'

firefox 37               ✓ simple_tests_01.html » <x-pizza> (1) » starts with no toppings
firefox 37               ✓ simple_tests_02.html » <x-pizza> (2) » starts with no toppings
firefox 37               Tests failed: 1 failed tests
Test run ended in failure: 1 failed tests

firefox 37 (4/0/1)                    
1 failed tests

That is of some use, but is not entirely satisfactory. I had not realized earlier that index.html in the output referred to the HTML context of the output. That is the entry point for all of my tests, so I assumed that any failure would be reported from there. In fact, alternate HTML contexts, such as simple_tests_01.html are reported as distinct from the JS tests.

All of my tests are loaded from separate files in index.html:

<!doctype html>
<html>
<head>
  <meta charset="utf-8">
  <script src="../bower_components/webcomponentsjs/webcomponents.min.js"></script>
  <script src="../../web-component-tester/browser.js"></script>
  <link rel="import" href="../elements/x-pizza.html">
</head>
<body>
  <x-pizza id="fixture"></x-pizza>
  <script>
    WCT.loadSuites([
      'simple_tests_01.html',
      'adding_toppings.js',
      'page_objects.js',
      'simple_tests_02.html',
    ]);
  </script>
</body>
</html>

The JS tests are all reported as coming from index.html while the HTML tests are reported as coming from their respective HTML files. With that understanding, I think I can live with this behavior.

Between knowing that I can find the actual line number failure in the output from Chrome and that all browsers will report the correct HTML context, I ought to be fine. For the most part, I imagine that I will just run Chrome tests for development (to keep the runs shorter) and run multiple browsers in continuous integration.

Furthermore, I think it very unlikely that I will ever use multiple JS test suites for a single HTML context like I have done here. I prefer my JS in a separate file, so it is likely that for each simple_tests_01.html and simple_tests_02.html, each will load a corresponding simple_tests_01.js and simple_tests_02.js. So if a test reports failure from a given HTML context, I will know where to start looking. I doubt that I wound ever use two JS contexts as I have done here. As I found last night, doing this makes it easy to confuse your test environment.

So one JS suite for each HTML context will likely resolve my objections. Another potential objection to WCT gone. One way or another, I am definitely sticking with the expanded option. The resulting output may not be as nice as Mocha's spec reporter, but it is close enough for my tastes:

firefox 37               ✓ index.html » <x-pizza> » can add toppings to the whole pizza
firefox 37               ✓ index.html » <x-pizza> » defaults » has no toppings anywhere
firefox 37               ✓ index.html » <x-pizza> » adding a whole topping » updates the pizza state accordingly
firefox 37               ✓ simple_tests_01.html » <x-pizza> (1) » starts with no toppings
firefox 37               ✓ simple_tests_02.html » <x-pizza> (2) » starts with no toppings
firefox 37               Tests passed
Test run ended with great success

It is looking more and more like I need to update a few chapters in Patterns in Polymer.


Day #19

‹prev | My Chain | next›