Sunday, December 28, 2014

A Solution for Karma Testing Polymer on IE (10 & 11)


I coded in WordPad yesterday. 15 years of successfully repressing that sensation and now I fear I may never recover.

Still, I successfully got Karma & Jasmine to test Polymer code with Internet Explorer yesterday. So it was worth it. Kinda.

Regardless, I am happy to have gotten testing of Polymer on IE working, but I still have a few outstanding questions that I'd like answered before moving on to less traumatizing subjects. My IE Polymer tests work with karma-ie-launcher and IE10. I would like to be able to run karma-webdriver-launcher and use it on both IE10 and IE11.

Switching to karam-webdriver-launcher would mean no more WordPad coding (since the code resides on the host machine, not the guest VM), so I will start there. I already have Karma configuration in place that should work:
module.exports = function(config) {
  config.set({
    // ...
    // Use IP so Windows guest VM can connect to host Karma server
    hostname: '192.168.1.129',
    customLaunchers: {
      'IE10': {
        base: 'WebDriver',
        config: {
          hostname: 'localhost',
          port: 4410
        },
        browserName: 'internet explorer',
        name: 'Karma'
      }
    },
    browsers: ['IE10']
  });
};
I was able to use this to connect to webdriver on Windows the other day, I just could not get the tests to pass. Without changing the tests, I still get very unhelpful failures:
$ karma start --single-run 
INFO [karma]: Karma v0.12.28 server started at http://192.168.1.129:9876/
INFO [launcher]: Starting browser internet explorer via Remote WebDriver
INFO [IE 10.0.0 (Windows 8)]: Connected on socket WUsG_gAEAHJFhm1lLF0G with id 61543069
IE 10.0.0 (Windows 8) ERROR
  Object doesn't support property or method 'indexOf'
  at /home/chris/repos/polymer-book/play/plain_forms/js/node_modules/karma-jasmine/lib/jasmine.js:1759
IE 10.0.0 (Windows 8): Executed 1 of 3 ERROR (0.645 secs / 0.547 secs)
This indexOf failure seems to come from calling a “contains” matcher in my tests on undefined values. As I found yesterday, that is more of a symptom of larger Karma issues than individual test failures.

So I try yesterday's wait-a-browser-event-loop / set-timeout-zero solution in my tests:
  describe('properties', function(){
    it('updates value when internal state changes', function(done) {
      el.model.firstHalfToppings.push('pepperoni');
      el.async(function(){
        setTimeout(function(){
          expect(el.value).toContain('pepperoni');
          done();
        }, 0);
      });
    });
  });
The el here is my <x-pizza> pizza building Polymer element. The asyc() method from Polymer accepts a callback that will be invoked after Polymer has updated the UI and all bound variables. That works on its own in Chrome and Firefox, but, as I found yesterday, Polymer's IE implementation seems to have a bug that requires an additional browser event loop before everything is ready.

And, with that set-timeout-zero, I have my IE WebDriver tests passing:
$ karma start --single-run 
INFO [karma]: Karma v0.12.28 server started at http://192.168.1.129:9876/
INFO [launcher]: Starting browser internet explorer via Remote WebDriver
INFO [IE 10.0.0 (Windows 8)]: Connected on socket oggMD2c6Q8c4-xq-Lk_- with id 68843724
IE 10.0.0 (Windows 8): Executed 3 of 3 SUCCESS (0.54 secs / 0.543 secs)
So what about IE11? Things were even worse in that VM because I have yet to get the console in the Web Developer Tools to start successfully, making troubleshooting next to impossible. Perhaps the set-timeout-zero fix works there as well?

In the Windows VM, I fire up a good old command prompt (I can't believe that it hasn't changed in 10+ years). In there, I start WebDriver via the webdriver node.js package:
C:\Users\IEUser>webdriver-manager start --seleniumPort 4411
I run the IE10 WebDriver on port 4410 and the IE11 WebDriver on 4411. So I update my karma.conf.js accordingly:
module.exports = function(config) {
  config.set({
    // ...
    // Use IP so Windows guest VM can connect to host Karma server
    hostname: '192.168.1.129',

    customLaunchers: {
      'IE10': { /* ... */ },
      'IE11': {
        base: 'WebDriver',
        config: {
          hostname: 'localhost',
          port: 4411
        },
        browserName: 'internet explorer',
        name: 'Karma'
      }
    },

    browsers: ['IE11']
  });
};
Unfortunately, when I try to run the same Karma tests against IE11, I find:
$ karma start --single-run 
INFO [karma]: Karma v0.12.28 server started at http://192.168.1.129:9876/
INFO [launcher]: Starting browser Internet Explorer 11 via Remote WebDriver
INFO [IE 11.0.0 (Windows 7)]: Connected on socket xu5uT_EaFT6kfCVrTOpE with id 86153250
IE 11.0.0 (Windows 7) <x-pizza> properties updates value when internal state changes FAILED
        TypeError: Unable to get property 'firstHalfToppings' of undefined or null reference
           at Anonymous function (/home/chris/repos/polymer-book/play/plain_forms/js/test/XPizzaSpec.js:29:7)
IE 11.0.0 (Windows 7) <x-pizza> syncing <input> values updates the input FAILED
...
IE 11.0.0 (Windows 7): Executed 3 of 3 (2 FAILED) (0.4 secs / 0.396 secs)
INFO [WebDriver]: Killed Karma test.
For whatever reason, it seems that my Polymer element is taking more than the usual single event loop to register in IE11. A single event loop is all that is required in Chrome, Firefox, and IE10. But IE11 requires an additional 10 milliseconds before it is ready:
describe('<x-pizza>', function(){
  var el, container;

  beforeEach(function(done){
    container = document.createElement("div");
    container.innerHTML = '<x-pizza></x-pizza>';
    document.body.appendChild(container);
    el = document.querySelector('x-pizza');

    setTimeout(done, 10); // Delay for elements to register in Polymer
  });

  describe('properties', function(){
    it('updates value when internal state changes', function(done) {
      // Same async + set-timeout-zero test here...
    });
  });
});
With that, I have my tests passing on IE11:
$ karma start --single-run 
INFO [karma]: Karma v0.12.28 server started at http://192.168.1.129:9876/
INFO [launcher]: Starting browser Internet Explorer 11 via Remote WebDriver
INFO [IE 11.0.0 (Windows 7)]: Connected on socket sQqPBHAEvsL1ujLtULSD with id 7115351
IE 11.0.0 (Windows 7): Executed 3 of 3 SUCCESS (0.634 secs / 0.566 secs)
And this seems to work reliably. I run through 10 single-run rounds of tests and all pass each time. If I drop down to 2 milliseconds, then tests start failing on occasion. If I really, really needed IE testing, I might bump these timeout delays for 50 or 100 milliseconds. For now, I'm just happy with a reliable local solution.

So it is ugly. It is rife with callbacks, set-timeout-zeros and other more arbitrary set-timeouts. But I have a working solution for testing Polymer on Internet Explorer. And I didn't even need to fire up WordPad.


Day #38

Saturday, December 27, 2014

Finally, Testing Polymer in Internet Explorer


I am not fond of Internet Explorer. The reasons are myriad, but not terribly helpful. Despite my dislike, I must admit that it tends to fail in ways that make sense. More often than not, when troubleshooting an “IE bug” I find myself wondering how code works in other browsers. Maybe that will wind up being the case with my Polymer testing woes.

I am running tests for my <x-pizza> Polymer element with Karma. The tests are written with the Jasmine testing library and look like:
describe('<x-pizza>', function(){
  var el, container;
  beforeEach(function(done){
    // Create <x-pizza> el and other setup...
  });
  // ...
  describe('properties', function(){
    it('updates value when internal state changes', function(done) {
      el.model.firstHalfToppings.push('pepperoni');
      el.async(function(){
        expect(el.value).toContain('pepperoni');
        done();
      });
    });
  });
});
The error for the past two nights (with karma-webdriver-launcher and karma-ie-launcher) occurs when I try to access the firstHalfToppings property of model. In the test, it is undefined:
C:\Users\IEUser\plain_old_forms>node node_modules\karma\bin\karma start --single-run --browsers IE
INFO [karma]: Karma v0.12.28 server started at http://localhost:9876/
INFO [launcher]: Starting browser IE
INFO [IE 11.0.0 (Windows 7)]: Connected on socket qNf8JVIEqjDNilL7AQFH with id 9528228
IE 11.0.0 (Windows 7) <x-pizza> properties updates value when internal state changes FAILED
        TypeError: Unable to get property 'firstHalfToppings' of undefined or null reference
           at Anonymous function (C:/Users/IEUser/plain_old_forms/test/XPizzaSpec.js:29:7)
But if I fire this element up in a web page, that property is very definitely defined:



After much futzing, I eventually trace this not to my code, but to Polymer's async() method, which comes on the line after the failure message. Things got so wonky that failures in previous tests would break other tests in non-helpful ways. Only after running single tests — with Karma/Jasmine's ddescribe() / iit() — was I able to identify the culprit as async(), which my test uses:
    it('updates value when internal state changes', function(done) {
      el.model.firstHalfToppings.push('pepperoni');
      el.async(function(){
        expect(el.value).toContain('pepperoni');
        done();
      });
    });
Polymer elements expose this method as a callback mechanism. The supplied callback (my test's assertion in this case), is only invoked once Polymer has updated the UI and all bound variables. This is ideal for testing because the test can be assured that all of the Polymer element's properties and visualizations have been updated—at least in Chrome and Firefox.

In Internet Explorer, it seems that nothing is updated until one more browser event loop. In other words, I have to wait for a setTimeout-0 in addition to the async() callback:
it('updates value when internal state changes', function(done) {
      el.model.firstHalfToppings.push('pepperoni');
      el.async(function(){
        setTimeout(function(){
          expect(el.value).toContain('pepperoni');
          done();
        }, 0);
      });
    });
This quite definitely fixes the test. I have other tests that magically pass with the addition of a setTimeout-0, but reliably fail without.

And once again, after some investigation of an “IE Bug,” I am left with a code error that is not IE's fault. This is almost certainly a Polymer bug. The entire point of async() is that the element is updated when the supplied callback is invoked. That I have to wait for another event loop on top of this seems quite wrong.


Day #37

Friday, December 26, 2014

Polymer and Karma-Ie-Launcher


Testing in Internet Explorer is a low priority for me. It always seems to be more effort than it is worth, given its dwindling market share. Still, I recognize that there is value and that some folks need it. So, even if I will not run automated tests for the code in Patterns in Polymer, I would like to be able to tell people that it does work.

Except that I was unable to get karma-webdriver-launcher to launch IE tests against Polymer last night. I suspect that the problems lie with WebDriver, so tonight, I try karma-ie-launcher instead.

First, I copy the code and tests onto my Windows VM. In there I add karma-ie-launcher to the list of NPM package.json dependencies:
{
  "name": "plain_old_forms",
  "devDependencies": {
    "grunt": "~0.4.0",
    "grunt-contrib-watch": "~0.5.0",
    "karma-jasmine": "~0.2.0",
    "karma-ie-launcher": ">0.0"
  }
}
After an npm install, I try to start Karma:
C:\Users\IEUser\plain_old_forms>karma start --single-run --browsers IE
'karma' is not recognized as an internal or external command,
operable program or batch file.
OK, so I install it globally:
C:\Users\IEUser\plain_old_forms>npm install -g karma
npm WARN optional dep failed, continuing fsevents@0.3.1

> ws@0.4.32 install C:\Users\IEUser\AppData\Roaming\npm\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\ws
> (node-gyp rebuild 2> builderror.log) || (exit 0)

C:\Users\IEUser\AppData\Roaming\npm\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\ws>node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild
karma@0.12.28 C:\Users\IEUser\AppData\Roaming\npm\node_modules\karma
├── ...
With that, when I try Karma now, I get:
C:\Users\IEUser\plain_old_forms>karma start --single-run --browsers IE
'karma' is not recognized as an internal or external command,
operable program or batch file.
Sigh. It looks like I need the full path instead:
C:\Users\IEUser\plain_old_forms>node node_modules\karma\bin\karma start --single-run --browsers IE
INFO [karma]: Karma v0.12.28 server started at http://localhost:9876/
INFO [launcher]: Starting browser IE
WARN [watcher]: Pattern "C:/Users/IEUser/plain_old_forms/bower_components/webcomponentsjs/webcomponents.js" does not match any file.
WARN [watcher]: Pattern "C:/Users/IEUser/plain_old_forms/bower_components/**" does not match any file.
INFO [IE 11.0.0 (Windows 7)]: Connected on socket _0g9fTtXw2PJOCjb5b2y with id 98303942
IE 11.0.0 (Windows 7) <x-pizza> element content has a shadow DOM FAILED
        ReferenceError: 'Polymer' is undefined
           at waitForPolymer (C:/Users/IEUser/plain_old_forms/test/PolymerSetup.js:19:5)
           at Anonymous function (C:/Users/IEUser/plain_old_forms/test/PolymerSetup.js:29:3)
...
IE 11.0.0 (Windows 7): Executed 3 of 3 (3 FAILED) ERROR (0.047 secs / 0.038 secs)
Oops! I neglected to bower install my client-side dependencies (this time I globally install from the start):
C:\Users\IEUser\plain_old_forms>npm install -g bower
C:\Users\IEUser\AppData\Roaming\npm\bower -> C:\Users\IEUser\AppData\Roaming\npm\node_modules\bower\bin\bower
bower@1.3.12 C:\Users\IEUser\AppData\Roaming\npm\node_modules\bower
└── ...
That does get installed in a useful location and it works… to a point:
C:\Users\IEUser\plain_old_forms>bower install
bower polymer#*                 ENOGIT git is not installed or not in the PATH
So it seems that I need Git installed to work with Polymer on Windows. This is already more trouble than it is worth to me, but I must see it through. I install Git the usual way and accept all of the default options but one:



This will allow bower to access Git when running from the command prompt:
C:\Users\IEUser\plain_old_forms>bower install
...
a-form-input#0.0.1 bower_components\a-form-input
└── polymer#0.5.2
polymer#0.5.2 bower_components\polymer
├── core-component-page#0.5.2
└── webcomponentsjs#0.5.2
core-component-page#0.5.2 bower_components\core-component-page
├── polymer#0.5.2
└── webcomponentsjs#0.5.2
webcomponentsjs#0.5.2 bower_components\webcomponentsjs
Yay!

Now, when I karma, I find:
C:\Users\IEUser\plain_old_forms>node node_modules\karma\bin\karma start --single-run --browsers IE
INFO [karma]: Karma v0.12.28 server started at http://localhost:9876/
INFO [launcher]: Starting browser IE
INFO [IE 11.0.0 (Windows 7)]: Connected on socket qNf8JVIEqjDNilL7AQFH with id 9528228
IE 11.0.0 (Windows 7) <x-pizza> properties updates value when internal state changes FAILED
        TypeError: Unable to get property 'firstHalfToppings' of undefined or null reference
           at Anonymous function (C:/Users/IEUser/plain_old_forms/test/XPizzaSpec.js:29:7)
IE 11.0.0 (Windows 7) <x-pizza> syncing <input> values updates the input FAILED
        TypeError: Unable to get property 'firstHalfToppings' of undefined or null reference
           at Anonymous function (C:/Users/IEUser/plain_old_forms/test/XPizzaSpec.js:42:7)
        TypeError: Unable to get property 'value' of undefined or null reference
           at Anonymous function (C:/Users/IEUser/plain_old_forms/test/XPizzaSpec.js:47:7)
IE 11.0.0 (Windows 7): Executed 3 of 3 (2 FAILED) (0.453 secs / 0.444 secs)
Dang it! Those are the same errors that I saw yesterday with WebDriver. So it seems that this is a problem with IE and Polymer rather than WebDriver. The failure is occurring when the test adds a topping to the first half of the pizza being built by the <x-pizza> Polymer element:
  describe('properties', function(){
    it('updates value when internal state changes', function(done) {
      el.model.firstHalfToppings.push('pepperoni');
      el.async(function(){
        expect(el.value).toContain('pepperoni');
        done();
      });
    });
  });
This works in Chrome and Firefox, but it seems that I need a different approach in Internet Explorer. Unfortunately, the web developer tools are still broken in my install so this is going to be tricky to troubleshoot.



Day #36

Thursday, December 25, 2014

Karma IE Testing of Polymer Elements with WebDriver


I think I am resigned to Karma as the best solution for Polymer—especially for the testing chapters in Patterns in Polymer. My experience with WebDriver, and Protractor in particular, made this a tougher decision than I expected. In the end, the combination of WebDriver's lack of shadow DOM support (kinda important with Polymer) and conceptual overload lead me to set Protractor aside. For now.

There are two things that I will miss from Protactor: wait-for-element baked in to element finders and easy Internet Explorer testing. There is not much to be done with async testing in Karma—that is more or less up to the testing library, most of which rely on done() callbacks. Those work, but are not as nice as Protractor's promise-based finders. I can live without them—especially in the book. But crazy as it might seem, I would like to be able to test on Internet Explorer.

In Protractor, I could establish a WebDriver instance in a Windows VM that my Protractor tests could drive from my Linux box. In Karma, I think I am stuck with karma-ie-launcher. There is not a ton of documentation on that project, but I assume that I would have to install Node.js, Karma, and my code on the Windows VM in order to make that work. I much prefer the code residing entirely on my machine with only a WebDriver instance running on the Windows VM.

Enter karma-webdriver-launcher. I add it to the list of NPM package dependencies:
{
  "name": "plain_old_forms",
  "devDependencies": {
    // ....
    "karma-webdriver-launcher": "~ 1.0.1"
  }
}
And then install:
$ npm install

karma-webdriver-launcher@1.0.1 node_modules/karma-webdriver-launcher
└── wd@0.2.8 (vargs@0.1.0, async@0.2.10, q@0.9.7, underscore.string@2.3.3, archiver@0.4.10, lodash@1.3.1, request@2.21.0)
To use in my Karma configuration, I add a custom launcher for IE11:
module.exports = function(config) {
  var webdriverConfig = {
    hostname: 'localhost',
    port: 4411
  };

  config.set({
    // ...
    customLaunchers: {
      'IE11': {
        base: 'WebDriver',
        config: webdriverConfig,
        browserName: 'internet explorer',
        name: 'Karma'
      }
    },

    browsers: ['IE11']
  });
};
I already have WebDriver installed on my Windows VM via the webdriver Node.js package. I start it with:
C:\Users\IEUser>webdriver-manager start --seleniumPort=4411
(I also have port forwarding to 4411 in place)

Then I run the tests from my Linux box, or I try. Instead of successful or failing tests, I see an error:
$ karma start --single-run
INFO [karma]: Karma v0.12.28 server started at http://localhost:9876/
...
INFO [launcher]: Trying to start internet explorer via Remote WebDriver again (2/2).
WARN [launcher]: internet explorer via Remote WebDriver have not captured in 60000 ms, killing.
INFO [WebDriver]: Killed Karma test.
ERROR [launcher]: internet explorer via Remote WebDriver failed 2 times (timeout). Giving up.
On the Windows side, I see that I am unable to connect to the Karma server on port 9876:



I was unaware of this, but it is possible to set the hostname for the Karma web server in the configuration file. So I add the IP address of my Linux box:
  config.set({
    // ...
    hostname: '192.168.1.129',

    customLaunchers: {
      'IE11': {
        base: 'WebDriver',
        config: webdriverConfig,
        browserName: 'internet explorer',
        name: 'Karma'
      }
    },
    browsers: ['IE11']
  });
And that actually works:
$ karma start --single-run
INFO [karma]: Karma v0.12.28 server started at http://192.168.1.129:9876/
INFO [launcher]: Starting browser internet explorer via Remote WebDriver
INFO [IE 11.0.0 (Windows 7)]: Connected on socket JiWBPa2Dzoq-fiGXsSvi with id 96899745
IE 11.0.0 (Windows 7) <x-pizza> properties updates value when internal state changes FAILED
        TypeError: Unable to get property 'firstHalfToppings' of undefined or null reference
           at Anonymous function (/home/chris/repos/polymer-book/play/plain_forms/js/test/XPizzaSpec.js:29:7)
IE 11.0.0 (Windows 7) <x-pizza> syncing <input> values updates the input FAILED
        TypeError: Unable to get property 'firstHalfToppings' of undefined or null reference
           at Anonymous function (/home/chris/repos/polymer-book/play/plain_forms/js/test/XPizzaSpec.js:42:7)
        TypeError: Unable to get property 'value' of undefined or null reference
           at Anonymous function (/home/chris/repos/polymer-book/play/plain_forms/js/test/XPizzaSpec.js:47:7)
IE 11.0.0 (Windows 7): Executed 3 of 3 (2 FAILED) (0.415 secs / 0.409 secs)
INFO [WebDriver]: Killed Karma test.
Well, it kind of works.

The connection is made and one of the test even passes. The two failing tests seem an awful lot like WebDriver and Polymer issues that I have seen over the past two weeks. Satisfied that I can connect, I will call it a night here. I may investigate the failures tomorrow.


Day #35

Wednesday, December 24, 2014

TDDing New Polymer Element Features with Protractor


Even after a week or so of working with Protractor as testing solution for Polymer, I still do not know if I have a good feel for recommending it as a solution. I begin to understand its strengths and weaknesses as a Polymer testing tool—its async support works brilliantly with Polymer while its complete lack of shadow DOM support is nearly a show stopper.

Nearly a show stopper, but when I can write tests like this:
describe('<x-pizza>', function(){
  beforeEach(function(){
    browser.get('http://localhost:8000');
  });
  it('updates value when internal state changes', function() {
    new XPizzaComponent().
      addFirstHalfTopping('pepperoni');

    expect($('x-pizza').getAttribute('value')).
      toMatch('pepperoni');
  });
});
How can I dismiss it? To be sure, the XPizzaComponent Page Object helps to mask some ugliness—but the same test in Karma has just as much shadow DOM ugliness, plus async ugliness.

I have already tried TDDing a bug fix with Protractor, which went reasonably well. So tonight I wonder what it is like TDDing a new feature? I have been playing with an early version of the <x-pizza> pizza building Polymer element as I explore these questions:



This version of the element has the exploration advantage of including some old-timey HTML form elements with which to play. The advantage here is that I made old-timey mistakes in the backing code. The bug that I TDD'd the other day resulted from adding unknown toppings to the pizza. I eliminated the bug, but it is still possible to click the “Add First Half Topping” button without choosing a topping. I would like to change that. So I write a Protractor test:
describe('<x-pizza>', function(){
  beforeEach(function(){
    browser.get('http://localhost:8000');
  });
  // ...
  it('does nothing when adding a topping w/o choosing first', function(){
    new XPizzaComponent().
      clickAddFirstHalfToppingButton();

    expect($('x-pizza').getAttribute('value')).
      toEqual('\n\n');
  });
});
I need to write the clickAddFirstHalfToppingButton() method for my page object. This is already done as part of the addFirstHalfTopping() method, but now I need it separate:
function XPizzaComponent() {
  this.selector = 'x-pizza';
}

XPizzaComponent.prototype = {
  // ...
  clickAddFirstHalfToppingButton: function() {
    browser.executeScript(function(selector){
      var el = document.querySelector(selector),
          button = el.$.firstHalf.querySelector("button");
      button.click();
    }, this.selector);
  }
};
The executeScript() method is hackery to work around Protractor's lack of shadow DOM support. It is not too horrible, but I am extremely tempted to factor the el assignment out into its own method, especially since both addFirstHalfTopping() and clickAddFirstHalfToppingButton() both assign it in the exact same way:
XPizzaComponent.prototype = {
  addFirstHalfTopping: function(topping) {
    browser.
      executeScript(function(selector, v){
        var el = document.querySelector(selector),
            select = el.$.firstHalf.querySelector("select");
      }, this.selector, topping);
  },

  clickAddFirstHalfToppingButton: function() {
    browser.
      executeScript(function(selector){
        var el = document.querySelector(selector),
            button = el.$.firstHalf.querySelector("button");
        // ...
      }, this.selector);
  }
};
The problem with trying to factor this out is that I am obtaining the element inside an anonymous function that is called by browser.executeScript() and the reason that I am doing this is because regular Protractor locators cannot find Polymer elements. The best that I can come up with is:
XPizzaComponent.prototype = {
  el: function(){
    return browser.executeScript(function(selector){
      return document.querySelector(selector);
    }, this.selector);
  },
  // ...
  clickAddFirstHalfToppingButton: function() {
    browser.
      executeScript(function(el){
        var button = el.$.firstHalf.querySelector("button");
        button.click();
      }, this.el());
  }
});
This is not too horrible, I suppose. But I am writing as much test code (and working as hard to write it) as I am with the actual code.

Regardless, I have a failing test:
$ protractor --verbose
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
<x-pizza>
  has a shadow DOM - pass
  updates value when internal state changes - pass
  syncs <input> values - pass
  does nothing when adding a topping w/o choosing first - fail

Failures:

  1) <x-pizza> does nothing when adding a topping w/o choosing first
   Message:
     Expected 'unknown

' to equal '

'.
   Stacktrace:
     Error: Failed expectation
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:30:7)

Finished in 2.032 seconds
4 tests, 4 assertions, 1 failure
I can make that pass easily enough. My Polymer element's addFirstHalf() method simply needs a guard clause:
Polymer('x-pizza', {
  // ..
  addFirstHalf: function() {
    if (this.currentFirstHalf == '') return;
    this.model.firstHalfToppings.push(this.currentFirstHalf);
  },
  // ...
});
With that, I have my passing test:
$ protractor --verbose
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
<x-pizza>
  has a shadow DOM - pass
  updates value when internal state changes - pass
  syncs <input> values - pass
  does nothing when adding a topping w/o choosing first - pass

Finished in 2.012 seconds
4 tests, 4 assertions, 0 failures
And a nice new feature.

I normally do not care for refactoring tests, so the difficulty experienced tonight really should not bother me. But it does. The lack of shadow DOM support makes me think that I will need to write many helpers—if not an entire library—to make Protractor and Polymer really play nicely. If I struggle with the simple stuff like getting the Polymer element itself, this does not bode well for more in-depth testing adventures. I may play with this some more over the next few days, but I am leaning toward not using Protractor at this point.


Day #34

Tuesday, December 23, 2014

Can't Protractor Polymer on Internet Explorer


All right, let's see if I can figure out what went wrong while running my Polymer + Protractor tests in Internet Explorer. On the face of it, Polymer and Protractor are not an obvious combination since Polymer is a web component (not an application) and Protractor is designed to test single-page applications, specifically AngularJS. Also, WebDriver (on which Protractor is based) does not support shadow DOM. That said, there are some definite reasons to like the Polymer + Protractor combination. Plus, tests should work on all browsers, including Internet Explorer.

Emphasis on should.

I am unsure where I went wrong the other night, but trying to run this stuff on IE11 instead of 10 seemed a next logical step. So I retrace my steps from that post (and from the source post). On the windows virtual machine, I install node.js. I install Java. I open a command prompt and install protractor globally (npm install -g protractor).

Side note, I omitted installing the IE WebDriver in the previous post. When I try to start WebDriver without it, I see:
C:\Users\IEUser>webdriver-manager start --seleniumPort=4411
Selenium Standalone is not present. Install with webdriver-manager update --standalone
The fix is to tell the webdriver manager to install the IE driver:
C:\Users\IEUser>webdriver-manager update --ie
With that, I can start my WebDriver server:
C:\Users\IEUser>webdriver-manager start --seleniumPort=4411
20:27:18.602 INFO - Launching a standalone server
Setting system property webdriver.ie.driver to C:\Users\IEUser\AppData\Roaming\npm\node_modules\protractor\selenium\IEDriverServer.exe
20:27:18.894 INFO - Java: Oracle Corporation 25.25-b02
20:27:18.908 INFO - OS: Windows 7 6.1 x86
20:27:18.922 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
20:27:19.145 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4411/wd/hub
...
20:27:19.353 INFO - Started SocketListener on 0.0.0.0:4411
Back on my Linux machine, I update the protractor.conf.js to point to this IE11 instance of WebDriver:
exports.config = {
  seleniumAddress: 'http://localhost:4411/wd/hub',
  capabilities: {
    'browserName': 'internet explorer',
    'platform': 'ANY',
    'version': '11'
  },
  // seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['tests/XPizzaSpec.js'],
  onPrepare: function() {
    require('./tests/polymerBrowser.js');
    require('./tests/xPizzaComponent.js');
  }
};
And, I also need to point the specs to my IP address instead of localhost so that the IE VM can connect to it:
describe('', function(){
  beforeEach(function(){
    browser.get('http://192.168.1.129:8000');
    // browser.get('http://localhost:8000');
  });
  // Tests here...
});
Last, I add a port forwarding rule on the VM so that my Protractor tests on the Linux box can connect to that WebDriver address on post 4411 of localhost.

With that… I see the exact same behavior that I saw the other night with IE10:
$ protractor --verbose
Using the selenium server at http://localhost:4411/wd/hub
[launcher] Running 1 instances of WebDriver
<x-pizza>
  has a shadow DOM - pass
A Jasmine spec timed out. Resetting the WebDriver Control Flow.
The last active task was:
WebDriver.findElements(By.cssSelector("x-pizza"))
    at [object Object].webdriver.WebDriver.schedule (/home/chris/local/node-v0.10.20/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:345:15)
    at [object Object].webdriver.WebDriver.findElements (/home/chris/local/node-v0.10.20/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:934:17)
  ...
  updates value when internal state changes - fail
Sigh.

As I saw with IE, the first test, which asks the Polymer element to send back its shadow DOM innerHTML via a Protractor executeScript(), works. The second test, which adds pepperoni toppings to the <x-pizza> element then asserts on the value attribute, fails. What is weird about the failure is that I can see the pepperoni toppings added in WebDriver IE:



So the problem is not interacting with the Polymer element, it is querying the value property. Interacting with the element involved a bit of hackery since Protractor does not support the shadow DOM, but the hackery would seem to work. What does not work is the thing that should work without any hackery at all—querying element attributes.

Of course, the hackery could be partially to blame. This works in Chrome and Firefox, but that does not necessarily work in all browsers. To see if that is the case, I remove the page object that updates the pizza toppings:
describe('<x-pizza>', function(){
  beforeEach(function(){
    browser.get('http://192.168.1.129:8000');
    // browser.get('http://localhost:8000');
  });

  it('updates value when internal state changes', function() {
    // new XPizzaComponent().
    //   addFirstHalfTopping('pepperoni');

    expect($('x-pizza').getAttribute('value')).
      toMatch('pepperoni');
  });
});
The test should fail, but hopefully it will be an assertion failure, not a timeout.

Unfortunately, it has no effect.

I am at a loss at what to try next. The developer tools on my IE11 installation are useless (I mean more than normal—I cannot type in the console). I would like to be able to test against IE, but it is not that important to me. I am inclined to set this aside until inspiration (or a driver update) arrive. Until then, I will likely move onto to other topics.


Day #33

Monday, December 22, 2014

Fixing Polymer Bugs with Protractor


Today, I hope to get a feel for fixing bugs in Polymer elements by driving the fixes with Protractor.

A while back, I noticed a problem in my <x-pizza> pizza building element:



The problem occurs when adding non-standard toppings, which results in a JavaScript error. I actually noticed the problem while writing Protractor tests, so if I can fix it with another Protractor test, it would make for nice symmetry.

My Protractor test for adding valid toppings looks like:
describe('<x-pizza>', function(){
  beforeEach(function(){
    browser.get('http://localhost:8000');
  });
  it('updates value when internal state changes', function() {
    new XPizzaComponent().
      addFirstHalfTopping('pepperoni');

    expect($('x-pizza').getAttribute('value')).
      toMatch('pepperoni');
  });
});
That is nice and compact thanks to the Page Objects pattern—the XPizzaComponent in this case.

The page object makes it easy to write a new test that breaks things:
  it('adds "unknown" topping when invalid item selected', function() {
    new XPizzaComponent().
      addFirstHalfTopping('');

    expect($('x-pizza').getAttribute('value')).
      toMatch('unknown');
  });
Instead of choosing an option from the list, this should select the first item on the drop down list, which is blank. When I run the Protractor test, I get a failure, but not an error:
  1) <x-pizza> adds "unknown" topping when invalid item selected
   Message:
     Expected '

' to match 'unknown'.
   Stacktrace:
     Error: Failed expectation
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:31:7)
If I add a browser.pause():
  it('adds "unknown" topping when invalid item selected', function() {
    new XPizzaComponent().
      addFirstHalfTopping('');

    browser.pause();

    expect($('x-pizza').getAttribute('value')).
      toMatch('unknown');
  });
Then I can see the error in the WebDriver console:



I find that I can also use this very verbose code to print out the console:
  it('adds "unknown" topping when invalid item selected', function() {
    new XPizzaComponent().
      addFirstHalfTopping('');

    browser.manage().logs().get('browser').then(function(browserLog) {
      console.log('log: ' + require('util').inspect(browserLog));
    });

    expect($('x-pizza').getAttribute('value')).
      toMatch('unknown');
  });
Regardless of how I find the error, it is fairly easy to fix it. The console in both cases reports the error to be at:
ReferenceError: _svgUnknown is not defined
    at x-pizza.Polymer._toppingMaker (http://localhost:8000/elements/x_pizza.js:94:12)
Looking at the _toppingMaker() method, I see right away that, unlike the SVG makers for know elements, I am trying to call _svgUnknown() as a function instead of a method:
  _toppingMaker: function(topping) {
    if (topping == 'pepperoni') return this._svgPepperoni;
    if (topping == 'sausage') return this._svgSausage;
    if (topping == 'green peppers') return this._svgGreenPepper;
    return _svgUnknown;
  },
And, indeed, converting that to a method resolves the bug:
  _toppingMaker: function(topping) {
    if (topping == 'pepperoni') return this._svgPepperoni;
    if (topping == 'sausage') return this._svgSausage;
    if (topping == 'green peppers') return this._svgGreenPepper;
    return this._svgUnknown;
  },
Just as importantly, I know have an e2e test that exercises this code, ensuring that I do not make a dumb mistake again:
<x-pizza>
  has a shadow DOM - pass
  updates value when internal state changes - pass
  syncs <input> values - pass
  adds "unknown" topping when invalid item selected - pass


Finished in 1.938 seconds
4 tests, 4 assertions, 0 failures
This might seem like a small thing for all that test code, but if I made that mistake once, I can make it again. Unless I have a test.


Day #32


Sunday, December 21, 2014

Protractor, Polymer, and Other Browsers


I am quickly warming to the idea of using Protractor for testing my Polymer elements. As of last night, I have some very nice tests.

They work well in Chrome:
$ protractor --verbose --browser=chrome
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
<x-pizza>
  has a shadow DOM - pass
  updates value when internal state changes - pass
  syncs <input> values - pass


Finished in 1.397 seconds
3 tests, 3 assertions, 0 failures

[launcher] 0 instance(s) of WebDriver still running
[launcher] chrome #1 passed
They even run well in Firefox:
$ protractor --verbose --browser=firefox
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
<x-pizza>
  has a shadow DOM - pass
  updates value when internal state changes - pass
  syncs <input> values - pass


Finished in 2.183 seconds
3 tests, 3 assertions, 0 failures

[launcher] 0 instance(s) of WebDriver still running
[launcher] firefox #1 passed
I will probably regret this, but what about Internet Explorer?

There is a nicely done post on getting Protractor working on Windows, but there are a crazy number of steps in there. I opt for the post's alternative #1 and install via Node.js. I do this in my VirtualBox VM for IE10.

I install Node the usual way then run npm install -g protractor to install it and the webdriver-manager globally:



When I try to run webdriver-manager start --seleniumPort=4410 however, I find that I do not have Java installed:



So it seems that I cannot skip all of the steps from that post. No matter, I install Java the usual way. I accept the defaults and enable any networking for which I am prompted:



With that, I have WebDriver running in my Windows VM.

Back in my Linux system that runs the actual tests, I change the configuration to point to the Windows webdriver instance:
exports.config = {
  //seleniumAddress: 'http://localhost:4444/wd/hub',
  seleniumAddress: 'http://localhost:4410/wd/hub',
  capabilities: {
    'browserName': 'internet explorer',
    'platform': 'ANY',
    'version': '10'
  },
  specs: ['tests/XPizzaSpec.js'],
  onPrepare: function() {
    require('./tests/polymerBrowser.js');
    require('./tests/xPizzaComponent.js');
  }
}
Ah, but that will not work without port forward. In the network setting of my VM, I click the port forwarding button:



In the resulting dialog, I tell VirtualBox to forward requests to my Linux box's port 4410 to the same TCP port in my Windows VM:



Even that does not quite work as my Windows VM cannot access the server that hosts my Polymer element. Bleh.

I work around this by rewriting my test to request the hostname of my Linux machine (instead of localhost):
describe('<x-pizza>', function(){
  beforeEach(function(){
    // browser.get('http://localhost:8000');
    browser.get('http://serenity.local:8000');
  });
  // Tests here...
});
That gets Protractor on my Linux box talking successfully to WebDriver on the Windows VM. I can even see it interacting with my <x-pizza> Polymer element:



But try as I might, I cannot get it to query the document for values against which to run my assertions:
$ protractor --verbose
Using the selenium server at http://localhost:4410/wd/hub
[launcher] Running 1 instances of WebDriver
A Jasmine spec timed out. Resetting the WebDriver Control Flow.
The last active task was:
WebDriver.findElements(By.cssSelector("input[type=hidden]"))
    at [object Object].webdriver.WebDriver.schedule (/home/chris/local/node-v0.10.20/lib/node_modules/protractor/node_modules/selenium-webdriver/lib/w
ebdriver/webdriver.js:345:15)
    at [object Object].webdriver.WebDriver.findElements (/home/chris/local/node-v0.10.20/lib/node_modules/protractor/node_modules/selenium-webdriver/l
ib/webdriver/webdriver.js:934:17)
...
<x-pizza>
  syncs <input> values - fail


Failures:

  1) <x-pizza> syncs <input> values
   Message:
     timeout: timed out after 30000 msec waiting for spec to complete
   Stacktrace:
     undefined

Finished in 32.849 seconds
1 test, 1 assertion, 1 failure
I cannot figure out why this is timing out. I can run assertions against some text values, but cannot get properties of elements on the page—Polymer or regular elements. While the test is timing out, I can even pop open the Developer Tools' console on IE and verify that it can query the value for which it is testing:



But, for some as yet unknown reason, I cannot get my local Protractor to successfully run this same query.

I really hate debugging code on IE so I may just punt on this. Or I could try a simpler test case to attempt to isolate the problem. Or maybe, if I am feeling industrious tomorrow, I will install all of this on a VM with IE11 instead of 10. But I'd have to be feeling pretty industrious to go through all of this again.


Day #31

Saturday, December 20, 2014

Inflicting Bad Things on Protractor for Great Polymer Good


I never seriously considered Protractor as a possible Polymer testing solution. For the last few days, I have been messing around with it in an effort to understand Protractor and what I might want in a real end-to-end testing framework for Polymer. But at no point did I think that Protractor might be anything close to a real solution. Until last night.

Using Page Objects, I wrote the following test:
describe('<x-pizza>', function(){
  beforeEach(function(){
    // (1) Get the page containing my Polymer element (served
    //     by Python simple HTTP server)
    browser.get('http://localhost:8000');
    expect($('[unresolved]').waitAbsent()).toBeTruthy();
  });
  it('updates value when internal state changes', function() {
    // (2) Tell the element on the page to add pepperoni
    new XPizzaComponent().
      addFirstHalfTopping('pepperoni');

    // (3) Verify pepperoni was added
    expect($('x-pizza').getAttribute('value')).
      toMatch('pepperoni');
  });
});
From a readability (and hence maintainability) standpoint, it does not get much better than that. Load the page, add something to the component, assert that the component changed as expected.

To be sure, this hides a fair bit of complexity, including some ugly hackery that works around Protractor/WebDriver's lack of shadow DOM support. That lack of support was the main reason that I never thought Protractor could be a legitimate functional Polymer testing tool—the shadow DOM being a fundamental piece of Polymer elements and all. And yet, that test code is beautiful.

The page object test code even compares favorably to the Karma test code that I wrote for an earlier version of the same <x-pizza> element:
describe('<x-pizza>', function(){
  var container, xPizza;

  beforeEach(function(done){
    container = document.createElement("div");
    var el = document.createElement("x-pizza");
    container.appendChild(el);
    document.body.appendChild(container);

    xPizza = new XPizzaComponent(el);
    xPizza.flush(done);
  });

  describe('adding a whole topping', function(){
    beforeEach(function(done){
      xPizza.addWholeTopping('green peppers', done);
    });

    it('updates the pizza state accordingly', function(){
      expect(xPizza.currentPizzaStateDisplay()).
        toMatch('green peppers');
    });
  });
});
I have been perfectly content with this Karma solution. There is a little work to add the element to the page, but that is not a big deal. What is something of a big deal is the need to pass those done callbacks into the xPizza page object. Last night's <x-pizza> page object for Protractor was a direct copy of the page object for Karma—except that I removed the callback code.

Protractor's built-in promises eliminate the worry about Polymer's asynchronous nature—and it is a complex worry. I am almost tempted to rewrite the page objects chapter in Patterns in Polymer for Protractor. Almost.

The problem is a tradeoff of conceptual complexity. If I stick with Karma, all of the testing framework complexity is described in the unit test chapter—and I will have already discussed dealing with the async nature of Polymer in that chapter. In other words, a Karma page objects chapter is just about pages objects. If I opt for Protractor, I have to (1) introduce Protractor, (2) detail how its asynchronous support helps with Polymer, (3) explain that, because of Protractor's lack of shadow DOM support, readers can never use built-in WebElements, and (4) describe how to workaround Protractor's lack of shadow DOM support. Oh yeah, and I have to explain what Page Objects are.

I still might do just that, but I need more experience with this before I make that call. I think I would like to start by improving the setup code:
  beforeEach(function(){
    browser.get('http://localhost:8000');
    expect($('[unresolved]').waitAbsent()).toBeTruthy();
  });
The expect() is a bit awkward. Even experienced Polymer… uh ers? Sure, why not. Even experienced Polymerers would find that hard to read—even though they know that the unresolved attribute is removed by Polymer when it initializes to deal with FOUC.

I would much rather just write:
  beforeEach(function(){
    browser.get('http://localhost:8000');
  });
And I can can do just that by mucking with the browser object in the Protractor configuration's onPrepare block:
exports.config = {
  // ...
  onPrepare: function() {
    browser.ignoreSynchronization = true;

    browser._originalGet = browser.get;

    browser.get = function(url) {
      browser._originalGet('http://localhost:8000');
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
    };
    // ...
  }
}
Is that horrible? I am rewriting the very important browser.get() to do something very different. I think I am OK with that. The first onPrepare line has already set this test run on a very different, non- AngularJS path than it would otherwise be on. If I am in for a penny, I might as well embrace the pound.

Especially if the pound gives me tests like:
describe('<x-pizza>', function(){
  beforeEach(function(){
    browser.get('http://localhost:8000');
  });
  it('updates value when internal state changes', function() {
    new XPizzaComponent().
      addFirstHalfTopping('pepperoni');

    expect($('x-pizza').getAttribute('value')).
      toMatch('pepperoni');
  });
});
That is one pretty pound.


Day #30

Friday, December 19, 2014

On Polymer, Protractor, and Page Objects


Tonight, I continue to abuse Protractor, forcing it to perform end-to-end (e2e) tests against Polymer instead of its preferred AngularJS. Protractor provides some facilities for testing non-Angular applications, but is hampered in testing Polymer elements since the underlying WebDriver does not support the Shadow DOM on which Polymer relies heavily. Even so, it has already provided useful insights into my Polymer element—and made for some pretty tests.

The Polymer element that is being tested remains an early version of the <x-pizza> pizza builder used extensively in Patterns in Polymer:



Last night's test veered a bit away from pretty tests, mostly thanks to a call to Protractor's browser.executeScript(), which I use to workaround WebDriver's lack of shadow DOM support. Said script comes in the form of a brutal string:
describe('<x-pizza>', function(){
  beforeEach(function(){
    browser.get('http://localhost:8000');
    expect($('[unresolved]').waitAbsent()).toBeTruthy();
  });
  it('updates value when internal state changes', function() {
    var selector = 'x-pizza';
    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'var options = el.shadowRoot.querySelectorAll("#firstHalf option"); ' +
      'options[1].selected = true; ' +
      '' +
      'var e = document.createEvent("Event"); ' +
      'e.initEvent("change", true, true); ' +
      'options[1].dispatchEvent(e); ' +
      '' +
      'button = el.shadowRoot.querySelector("#firstHalf button"); ' +
      'button.click(); '
    );
    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
  // ...
});
The code in there is ugly enough to begin with, but why-oh-why does Protractor force me to build strings like that?!

Er… it doesn't.

It turns out that I am as guilty as anyone else in finding a solution when scrambling to meet a deadline. And I am just as guilty here in never bothering to check the documentation which is perfectly clear. At any rate, I can clean that code up a bit with:
  it('updates value when internal state changes', function() {
    var selector = 'x-pizza';

    browser.executeScript(function(selector){
      var el = document.querySelector(selector);
      var options = el.shadowRoot.querySelectorAll("#firstHalf option");
      options[1].selected = true;

      var e = document.createEvent("Event");
      e.initEvent("change", true, true);
      options[1].dispatchEvent(e);

      button = el.shadowRoot.querySelector("#firstHalf button");
      button.click();
    }, [selector]);

    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
That is still not pretty, but the ugly is down by an order of magnitude. It is a little annoying supplying the "x-pizza" selector in the list of arguments at the end of the executeScript(), but the real ugly remains in the form of that createEvent() inside the executeScript() that triggers a Polymer-required event.

It sure would be nice if I could click() the <select>, then <click> the correct <option>. Alas, this is WebDriver shadow DOM limitation in action. If I grab the <select> from executeScript(), then try to click it:
  it('updates value when internal state changes', function() {
    var selector = 'x-pizza';
    var toppings = browser.executeScript(function(selector){
      var el = document.querySelector(selector);
      return el.shadowRoot.querySelector('#firstHalf select');
    }, [selector]);

    toppings.then(function(toppings){
      toppings.click();
    });
    // ...
  });
I get ye olde stale element error:
1) <x-pizza> updates value when internal state changes
   Message:
     StaleElementReferenceError: stale element reference: element is not attached to the page document
  (Session info: chrome=39.0.2171.95)
  (Driver info: chromedriver=2.12.301324 (de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux 3.13.0-37-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 10 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/stale_element_reference.html
Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:02:37'
System info: host: 'serenity', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-37-generic', java.version: '1.7.0_65'
Session ID: c91503b6a80a9931478f6908f80f16d4
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities [{platform=LINUX, acceptSslCerts=true, javascriptEnabled=true, browserName=chrome, chrome={userDataDir=/tmp/.com.google.Chrome.Dfu9DJ}, rotatable=false, locationContextEnabled=true, mobileEmulationEnabled=false, version=39.0.2171.95, takesHeapSnapshot=true, cssSelectorsEnabled=true, databaseEnabled=false, handlesAlerts=true, browserConnectionEnabled=false, webStorageEnabled=true, nativeEvents=true, applicationCacheEnabled=false, takesScreenshot=true}]
The usual state element fixes have no effect since this element is still attached—but attached in the unreachable (for WebDriver) shadow DOM.

So I am stuck with the ugly. Or am I?

Page Objects to the rescue:
  it('updates value when internal state changes', function() {
    new XPizzaComponent().
      addFirstHalfTopping('pepperoni');

    expect($('x-pizza').getAttribute('value')).
      toMatch('pepperoni');
  });
To make that work, I need to define the object in the Protractor onPrepare(), which resides in the configuration file. First, I need to add the object to Protractor's global namespace:
exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['tests/XPizzaSpec.js'],
  onPrepare: function() {
    browser.ignoreSynchronization = true;

    global.XPizzaComponent = XPizzaComponent;
    function XPizzaComponent() {
      this.selector = 'x-pizza';
    }
  }
};
Then I only need to define the addFirstHalfTopping() method, which mostly comes from the "ugly" test code:
    XPizzaComponent.prototype = {
      addFirstHalfTopping: function(topping) {
        browser.executeScript(function(selector, v){
          var el = document.querySelector(selector);

          var select = el.$.firstHalf.querySelector("select"),
              button = el.$.firstHalf.querySelector("button");

          var index = -1;
          for (var i=0; i<select.length; i++) {
            if (select.options[i].value == v) index = i;
          }
          select.selectedIndex = index;

          var event = document.createEvent('Event');
          event.initEvent('change', true, true);
          select.dispatchEvent(event);

          button.click();
        }, this.selector, topping);
      }
    };
Since I "hide" this ugly in a page object, I have less compunction over using Polymer properties or methods. In this case, I use the dollar sign property to get the container element with the ID of firstHalf.

In the end, this is very similar to the page object work that I did in Karma. Only now I can rely on Protractor's promises to wait for the element's value property to be present instead of the async() callback craziness that was required for Karma. That is an exciting win.

Now the ugly, at least what is left of it, all resides in the Page Object defined in the configuration. I can definitely live with that. More than that—I may be starting to like Protractor as a Polymer testing solution.


Day #29

Thursday, December 18, 2014

A First Attempt at E2E Testing of Polymer with Protractor


I continue my experiments using Protractor to test Polymer elements. I do not expect to get this working completely. Instead I am trying to understand what end-to-end testing of a Polymer element might look like. I also hope to understand where the gaps still exist between current WebDriver tools and real end-to-end testing.

Mostly, I am just having fun.

Tonight's fun takes the form of trying to convert last night's unit test into a real e2e Polymer test. The test that I wound up with is almost pretty:
  it('updates value when internal state changes', function() {
    var selector = 'x-pizza';

    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'el.model.firstHalfToppings.push("pepperoni"); '
    );

    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
A beforeEach() block loads the page containing the element to be tested: the <x-pizza> pizza building Polymer element. This test then adds pepperoni to the first half and then expects that the element's value property is updated to include pepperoni in it.

This is not a real unit test, in that it does not test a single method of the Polymer element. It is also not a true e2e test because of the executeScript() call that manipulates the innards of the Polymer element. This is something in between. It is a functional test (it tests functionality across concerns), but it still lies closer to unit test than e2e, which is what Protractor wants to do.

The version of the <x-pizza> element being tested here looks like:



(I switched to a Material Design later)

If those drop downs were not part of <x-pizza>, I might tell protractor to interact with them by replacing the executeScript() work-under-the-covers with something like:


    // browser.executeScript(
    //   'var el = document.querySelector("' + selector + '"); ' +
    //   'el.model.firstHalfToppings.push("pepperoni"); '
    // );

    $('#firstHalf options').
      then(function(options){
        options[1].click();
      });

    $('#firstHalf button').
      then(function(button){
        button.click();
      });
But, since WebDriver (and hence Protractor) does not support the shadow DOM, I get an error like:
  1) <x-pizza> updates value when internal state changes
   Message:
     NoSuchElementError: No element found using locator: By.cssSelector("#firstHalf options")
So what is a fellow to do? Well, hopefully recreate this with executeScript:
    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'var options = el.shadowRoot.querySelectorAll("#firstHalf option"); ' +
      'options[1].selected = true; ' +
      'button = el.shadowRoot.querySelector("#firstHalf button"); ' +
      'button.click(); '
    );
Ick. This code still finds the <x-pizza> element, but now is selects the second option (pepperoni) and clicks the “Add Topping” button.

I suppose that is not too horrible, but I sure hate all that coding inside JavaScript strings. Especially since it does not even work:
  1) <x-pizza> updates value when internal state changes
   Message:
     Expected '

' to match 'pepperoni'.
If I add a browser.wait(30*1000) (waiting 30 seconds is my preferred debugging methodology in Protractor), I can see that this code would seem to do what I expect. The pepperoni option is selected. The first half topping button is clicked. But I still get a failure. More importantly, I get a code failure in the JavaScript console of the waiting browser.

The problem occurs in the code that adds unknown toppings. I will fix that another day (my tests found a bug!). For now, I need to figure out why this is trying to add an unknown topping when I clearly added pepperoni. Thankfully, I have already solved this. Polymer needs a change event to be triggered before it will update its bound values:
    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'var options = el.shadowRoot.querySelectorAll("#firstHalf option"); ' +
      'options[1].selected = true; ' +
      '' +
      'var e = document.createEvent("Event"); ' +
      'e.initEvent("change", true, true); ' +
      'options[1].dispatchEvent(e); ' +
      '' +
      'button = el.shadowRoot.querySelector("#firstHalf button"); ' +
      'button.click(); '
    );
That actually gets my test working:
$ protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
...

Finished in 1.363 seconds
3 tests, 6 assertions, 0 failures
That is a good stopping point for tonight. I have learned that it is possible to write actual end-to-end tests for Polymer. I also learned that it is pretty ugly doing so inside Protractor browser.executeScript() blocks. Not only is it ugly, but there is definite gap between Protractor/WebDriver and a Polymer e2e testing tool. The lack of access to the shadow DOM is one problem, but now I see that Polymer events are also tricky.

Those problems aside, I also see definite advantages to e2e testing of Polymer elements—the most obvious being the bug that I found tonight by interacting with my Polymer element like a user would. And yes, I am still having fun.


Day #28

Wednesday, December 17, 2014

Testing Polymer Async with Protractor


I have a fair Protractor test for a Polymer element. It reads fairly well. It actually tests the Polymer element (i.e. it fails if the element is changed). It makes me happy. But it is limited in scope and I worry that expanding that scope will cause things to get messy.

The current test verifies that there is a shadow DOM for my <x-pizza> pizza building element and that the element has a title:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      expect($('x-pizza').getShadowRootHtml()).
        toMatch("<h2>Build Your Pizza</h2>");
    });
  });
});
Pretty, right?

There are three problems with it, however. First, the test is simple—it asserts against the initial HTML and I have no idea how well this approach will work with the asynchronous nature of Polymer. Second, this involves no user interaction, which is really the point of Protractor. Last, it lacks the ability to probe the internal state of the Polymer element because of WebDriver's lack of shadow DOM support.

For tonight, I would like to see if Protractor built-in asynchronous support, in the form of promises, will help when working with Polymer elements. The current Karma Jasmine test for <x-pizza> makes assertions on the value after updating the internal state:
describe('properties', function(){
    it('updates value when internal state changes', function(done) {
      el.model.firstHalfToppings.push('pepperoni');
      el.async(function(){
        expect(el.value).toContain('pepperoni');
        done();
      });
    });
  });
To ensure that Karma does not check the assertion until Polymer has had a chance to update the element, I had to set the assertion inside Polymer's async() callback. The supplied callback will be invoked once the most recent Polymer operation has flushed all changes to the screen and DOM. I would definitely prefer to avoid that in my tests since it adds framework complexity to my tests. Also, accessing async() in Protractor would difficult, if not impossible.

I copy that into Protactor-ese, which should look something like:
  it('updates value when internal state changes', function(done) {
    var selector = 'x-pizza';

    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'el.model.firstHalfToppings.push("pepperoni"); '
    );

    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
As I did last night, I am forced to interact with the element via browser.executeScript() since Protractor / WebDriver does not allow access.

Unfortunately, I get:
A Jasmine spec timed out. Resetting the WebDriver Control Flow.
The last active task was:
unknown
F

Failures:

  1) <x-pizza> properties updates value when internal state changes
   Message:
     timeout: timed out after 30000 msec waiting for spec to complete
   Stacktrace:
     undefined

Finished in 31.427 seconds
2 tests, 5 assertions, 1 failure
There is no expectation error in there, just an "unknown" failure. It takes me longer than I care to admit to realize that I had copied the done callback from the original Karma / Jasmine test into this Protractor version. I simply remove the done argument to the test's function:
  it('updates value when internal state changes', function() {
    var selector = 'x-pizza';

    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'el.model.firstHalfToppings.push("pepperoni"); '
    );

    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
Which actually works.

And really, aside from the ugliness of the code in browser.executeScript(), this is a nice test. I update the internal state and expect the attribute to change as a result. Brilliant.

Of course, the ugliness of the browser.executeScript() is trying to tell me that I am doing something wrong. In this case, I am manipulating a page through code rather than user interaction. That brings me to the next item on my list of worries, which I will investigate tomorrow.

Day #27

Tuesday, December 16, 2014

Improving Protractor Testing of Polymer Elements


I can barely test Polymer elements with Protractor. And yet, I still rather like it.

I may be misplacing my extreme gratitude over getting a working Polymer test. I had previously tried and failed to get the Intern, another WebDriver-based testing framework, to test Polymer elements. Still, it seems that at least one intrepid soul was able to make it work, so my gratitude may be personal rather than based on legitimate criteria. That said, my Protractor test is rather nice:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();

      var shadowRootHtml = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.innerHTML'
      );

      expect(shadowRootHtml).toMatch("<h2>Build Your Pizza</h2>");
    });
  });
});
The beforeEach() setup block disables Protractor's AngularJS-specific synchronization code and requests my test content (being served separately by a Python simple HTTP server):
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
The actual test involves a little hackery due to WebDriver's lack of shadow DOM support. Still, it remains fairly readable:
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();

      var shadowRootHtml = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.innerHTML'
      );

      expect(shadowRootHtml).toMatch("<h2>Build Your Pizza</h2>");
    });
The expectation on the [unresolved] attribute (square brackets being the CSS attribute matcher) waits for the Polymer library to handle FOUC, thus proving that it is operating properly on the page. Additional hackery grabs the string representation of my element's shadow root for actually assertion.

Despite Protactor's claims to the contrary, it will not resolve standard HTML elements inside a shadow DOM to WebDriver WebElement objects. This assertion on an <h2> element inside my element's shadow DOM fails:
      // Fails with Error: Expected undefined to match '<h2>Build Your Pizza</h2>'.
      var shadowRoot = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.children[0]'
      );
      expect(shadowRoot.outerHTML).toMatch("<h2>Build Your Pizza</h2>");
But if I return the outerHTML string from executeScript(), then it passes:
      // This passes...
      var shadowRoot = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.children[0].outerHTML'
      );
      expect(shadowRoot).toMatch("<h2>Build Your Pizza</h2>");
So it seems that I am restricted to working with strings from executeScript(). That is a big pile of meh.

Instead, I would like to write that test as something along the lines of:
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      expect($('x-pizza').getShadowRootHtml()).
        toMatch("<h2>Build Your Pizza</h2>");
    });
I can get that with the following in my Protractor conf.js:
exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['tests/XPizzaSpec.js'],
  onPrepare: function() {
    var ElementFinder = $('').constructor;

    ElementFinder.prototype.getShadowRootHtml = function() {
      var locator = this.locator();
      if (locator.using != 'css selector') return '';

      var selector = locator.value;
      return browser.executeScript(
        'return document.querySelector("' + selector + '").shadowRoot.innerHTML'
      );
    };

    require('./tests/waitAbsent.js');
  }
};
Some of this is based on the waitAbsent.js code required at the bottom of the onPrepare() block. I borrowed this code yesterday, which is how I know how to access the ElementFinder prototype. The rest of this code assumes that a CSS selector is being used. If that is the case, then I work through the same executeScript() hackery.

The difference here is that the hackery is now encapsulated outside of my test, which reads better and, more importantly, continues to pass:
$ protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
.

Finished in 0.581 seconds
1 test, 2 assertions, 0 failures
That makes me happy. At least happier. I may continue this effort tomorrow to see if true testing happiness with Polymer and Protactor is possible.



Day #26

Monday, December 15, 2014

Protractor Can Test Polymer (Sort of)


I was unable to test Polymer elements with Protractor last night. But, thanks to some helpful suggestions this morning, I think it just might be possible after all.

My barrier last night was the seeming hard coupling between Protractor and AngularJS, which resulted in testing errors that indicated that Angular was not loaded:
  1) <x-pizza> element content has a shadow DOM
   Message:
     Error: Angular could not be found on the page http://localhost:8000/ : retries looking for angular exceeded
The solution suggested by various people is the undocumented ignoreSynchronization property on the Protractor browser object.

This property is meant to be set in the Protractor configuration or a beforeEach() block. I opt for the latter here:
describe('<x-pizza>', function(){
  var el;

  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      // Tests here...
    });
  });
Unfortunately, that does not solve all of my problems. I am still using last night's simple test—that the <x-pizza> Polymer element, being served by a Python simple HTTP server on port 8000, has a shadow root:
    it('has a shadow DOM', function(){
      el = $('x-pizza');
      expect(el.shadowRoot).toBeDefined();
    });
This simple test still fails:
Failures:

  1) <x-pizza> element content has a shadow DOM
   Message:
     Expected undefined to be defined.
The problem here is twofold. First, I have to wait for Polymer to have upgraded the <x-pizza> element on the page being served. Second, that el reference is not a regular DOM element—it is a WebDriver WebElement wrapper. Neither problem seems to be easily solved, though I am able to come up with a solution for each.

To wait for Polymer to be ready, I need to pull in an external library. Protractor has great support for waiting for Angular operations, but no support for waiting for other operations. Fortunately for me, others have had similar problems. I swipe one of those solutions, waitAbsent.js, from a Gist and save it in tests/waitAbsent.js. I then use it in my current setup by adding to the onPrepare section of my Protractor configuration:
exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['tests/XPizzaSpec.js'],
  onPrepare: function() {
    require('./tests/waitAbsent.js');
  }
};
There is also a waitReady.js, but I specifically want to use a wait-until-absent operation for Polymer so that I can wait until Polymer has loaded and removed the unresolved attribute from the document:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      el = $('x-pizza');
      expect(el.shadowRoot).toBeDefined();
  });
});
That will block the remainder of the test from being evaluated until the unresolved attribute is removed from the document. Polymer does this to deal with FOUC, so once that attribute is removed I know that Polymer is active on the page (this does assume that the author has added unresolved somewhere in the page).

Even waiting for Polymer to be active is not sufficient, however, because the el in that test is a WebElement wrapper. And to my intense consternation, there is no way to access the underlying element from the wrapper. This would be fine… if WebDriver had some way to access the shadow DOM for testing, but it does not. All of this leaves me resigned to hackery.

The hackery in this case comes in the form of executeScript(), which I use to execute a simple query of the element's shadow DOM, returning its the innerHTML:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });

    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      var shadowRoot = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.innerHTML'
      );
      expect(shadowRoot).toMatch("<h2>Build Your Pizza</h2>");
  });
});
And that actually works:
$ protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
.

Finished in 0.621 seconds
1 test, 2 assertions, 0 failures

[launcher] 0 instance(s) of WebDriver still running
[launcher] chrome #1 passed
I know that is a legitimate test because I can make it fail by changing the expectation:
      expect(shadowRoot).toMatch("<h2>Build a Pizza</h2>");
Which results in:
Failures:

  1) <x-pizza> element content has a shadow DOM
   Message:
     Expected '
    <h2>Build Your Pizza</h2>
    <! ... -->
  ' to match '<h2>Build a Pizza</h2>'.
   Stacktrace:
     Error: Failed expectation
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:27:26)

Finished in 0.608 seconds
1 test, 2 assertions, 1 failure
I should probably convert the working executeScript into another onPrepare addition to Protractor, but I will leave that to another day. For now, I suffer with functional hackery.

One note about that functional hackery is that I have to be extremely careful with the return value from executeScript() and the matcher used. I had originally tried to return a typeof() for it:
      var shadowRoot = browser.executeScript(
        'return typeof(document.querySelector("x-pizza").shadowRoot);'
      );
The result of that is "object", which seems OK, until there is a problem. If there is no shadow root for any reason, the shadowRoot local variable is assigned to a promise that does not resolve to unresolved. In other words, I might not have a shadowRoot, but the following will still pass:
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      var shadowRoot = browser.executeScript(
        'return typeof(document.querySelector("h1").shadowRoot);'
      );
      expect(shadowRoot).toBeDefined();
    });
That already bit me during my investigation, so it is sure to cause problems down the line—all the more reason to want shadow DOM support of some kind in WebDriver. But, at least for one night, I was able to write a successful Protractor test for a Polymer element.


Day #25