Send to Kindle

Thursday, December 18, 2014

A First Attempt at E2E Testing of Polymer with Protractor


I continue my experiments using Protractor to test Polymer elements. I do not expect to get this working completely. Instead I am trying to understand what end-to-end testing of a Polymer element might look like. I also hope to understand where the gaps still exist between current WebDriver tools and real end-to-end testing.

Mostly, I am just having fun.

Tonight's fun takes the form of trying to convert last night's unit test into a real e2e Polymer test. The test that I wound up with is almost pretty:
  it('updates value when internal state changes', function() {
    var selector = 'x-pizza';

    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'el.model.firstHalfToppings.push("pepperoni"); '
    );

    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
A beforeEach() block loads the page containing the element to be tested: the <x-pizza> pizza building Polymer element. This test then adds pepperoni to the first half and then expects that the element's value property is updated to include pepperoni in it.

This is not a real unit test, in that it does not test a single method of the Polymer element. It is also not a true e2e test because of the executeScript() call that manipulates the innards of the Polymer element. This is something in between. It is a functional test (it tests functionality across concerns), but it still lies closer to unit test than e2e, which is what Protractor wants to do.

The version of the <x-pizza> element being tested here looks like:



(I switched to a Material Design later)

If those drop downs were not part of <x-pizza>, I might tell protractor to interact with them by replacing the executeScript() work-under-the-covers with something like:


    // browser.executeScript(
    //   'var el = document.querySelector("' + selector + '"); ' +
    //   'el.model.firstHalfToppings.push("pepperoni"); '
    // );

    $('#firstHalf options').
      then(function(options){
        options[1].click();
      });

    $('#firstHalf button').
      then(function(button){
        button.click();
      });
But, since WebDriver (and hence Protractor) does not support the shadow DOM, I get an error like:
  1) <x-pizza> updates value when internal state changes
   Message:
     NoSuchElementError: No element found using locator: By.cssSelector("#firstHalf options")
So what is a fellow to do? Well, hopefully recreate this with executeScript:
    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'var options = el.shadowRoot.querySelectorAll("#firstHalf option"); ' +
      'options[1].selected = true; ' +
      'button = el.shadowRoot.querySelector("#firstHalf button"); ' +
      'button.click(); '
    );
Ick. This code still finds the <x-pizza> element, but now is selects the second option (pepperoni) and clicks the “Add Topping” button.

I suppose that is not too horrible, but I sure hate all that coding inside JavaScript strings. Especially since it does not even work:
  1) <x-pizza> updates value when internal state changes
   Message:
     Expected '

' to match 'pepperoni'.
If I add a browser.wait(30*1000) (waiting 30 seconds is my preferred debugging methodology in Protractor), I can see that this code would seem to do what I expect. The pepperoni option is selected. The first half topping button is clicked. But I still get a failure. More importantly, I get a code failure in the JavaScript console of the waiting browser.

The problem occurs in the code that adds unknown toppings. I will fix that another day (my tests found a bug!). For now, I need to figure out why this is trying to add an unknown topping when I clearly added pepperoni. Thankfully, I have already solved this. Polymer needs a change event to be triggered before it will update its bound values:
    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'var options = el.shadowRoot.querySelectorAll("#firstHalf option"); ' +
      'options[1].selected = true; ' +
      '' +
      'var e = document.createEvent("Event"); ' +
      'e.initEvent("change", true, true); ' +
      'options[1].dispatchEvent(e); ' +
      '' +
      'button = el.shadowRoot.querySelector("#firstHalf button"); ' +
      'button.click(); '
    );
That actually gets my test working:
$ protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
...

Finished in 1.363 seconds
3 tests, 6 assertions, 0 failures
That is a good stopping point for tonight. I have learned that it is possible to write actual end-to-end tests for Polymer. I also learned that it is pretty ugly doing so inside Protractor browser.executeScript() blocks. Not only is it ugly, but there is definite gap between Protractor/WebDriver and a Polymer e2e testing tool. The lack of access to the shadow DOM is one problem, but now I see that Polymer events are also tricky.

Those problems aside, I also see definite advantages to e2e testing of Polymer elements—the most obvious being the bug that I found tonight by interacting with my Polymer element like a user would. And yes, I am still having fun.


Day #28

Wednesday, December 17, 2014

Testing Polymer Async with Protractor


I have a fair Protractor test for a Polymer element. It reads fairly well. It actually tests the Polymer element (i.e. it fails if the element is changed). It makes me happy. But it is limited in scope and I worry that expanding that scope will cause things to get messy.

The current test verifies that there is a shadow DOM for my <x-pizza> pizza building element and that the element has a title:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      expect($('x-pizza').getShadowRootHtml()).
        toMatch("<h2>Build Your Pizza</h2>");
    });
  });
});
Pretty, right?

There are three problems with it, however. First, the test is simple—it asserts against the initial HTML and I have no idea how well this approach will work with the asynchronous nature of Polymer. Second, this involves no user interaction, which is really the point of Protractor. Last, it lacks the ability to probe the internal state of the Polymer element because of WebDriver's lack of shadow DOM support.

For tonight, I would like to see if Protractor built-in asynchronous support, in the form of promises, will help when working with Polymer elements. The current Karma Jasmine test for <x-pizza> makes assertions on the value after updating the internal state:
describe('properties', function(){
    it('updates value when internal state changes', function(done) {
      el.model.firstHalfToppings.push('pepperoni');
      el.async(function(){
        expect(el.value).toContain('pepperoni');
        done();
      });
    });
  });
To ensure that Karma does not check the assertion until Polymer has had a chance to update the element, I had to set the assertion inside Polymer's async() callback. The supplied callback will be invoked once the most recent Polymer operation has flushed all changes to the screen and DOM. I would definitely prefer to avoid that in my tests since it adds framework complexity to my tests. Also, accessing async() in Protractor would difficult, if not impossible.

I copy that into Protactor-ese, which should look something like:
  it('updates value when internal state changes', function(done) {
    var selector = 'x-pizza';

    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'el.model.firstHalfToppings.push("pepperoni"); '
    );

    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
As I did last night, I am forced to interact with the element via browser.executeScript() since Protractor / WebDriver does not allow access.

Unfortunately, I get:
A Jasmine spec timed out. Resetting the WebDriver Control Flow.
The last active task was:
unknown
F

Failures:

  1) <x-pizza> properties updates value when internal state changes
   Message:
     timeout: timed out after 30000 msec waiting for spec to complete
   Stacktrace:
     undefined

Finished in 31.427 seconds
2 tests, 5 assertions, 1 failure
There is no expectation error in there, just an "unknown" failure. It takes me longer than I care to admit to realize that I had copied the done callback from the original Karma / Jasmine test into this Protractor version. I simply remove the done argument to the test's function:
  it('updates value when internal state changes', function() {
    var selector = 'x-pizza';

    browser.executeScript(
      'var el = document.querySelector("' + selector + '"); ' +
      'el.model.firstHalfToppings.push("pepperoni"); '
    );

    expect($(selector).getAttribute('value')).
      toMatch('pepperoni');
  });
Which actually works.

And really, aside from the ugliness of the code in browser.executeScript(), this is a nice test. I update the internal state and expect the attribute to change as a result. Brilliant.

Of course, the ugliness of the browser.executeScript() is trying to tell me that I am doing something wrong. In this case, I am manipulating a page through code rather than user interaction. That brings me to the next item on my list of worries, which I will investigate tomorrow.

Day #27

Tuesday, December 16, 2014

Improving Protractor Testing of Polymer Elements


I can barely test Polymer elements with Protractor. And yet, I still rather like it.

I may be misplacing my extreme gratitude over getting a working Polymer test. I had previously tried and failed to get the Intern, another WebDriver-based testing framework, to test Polymer elements. Still, it seems that at least one intrepid soul was able to make it work, so my gratitude may be personal rather than based on legitimate criteria. That said, my Protractor test is rather nice:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();

      var shadowRootHtml = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.innerHTML'
      );

      expect(shadowRootHtml).toMatch("<h2>Build Your Pizza</h2>");
    });
  });
});
The beforeEach() setup block disables Protractor's AngularJS-specific synchronization code and requests my test content (being served separately by a Python simple HTTP server):
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
The actual test involves a little hackery due to WebDriver's lack of shadow DOM support. Still, it remains fairly readable:
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();

      var shadowRootHtml = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.innerHTML'
      );

      expect(shadowRootHtml).toMatch("<h2>Build Your Pizza</h2>");
    });
The expectation on the [unresolved] attribute (square brackets being the CSS attribute matcher) waits for the Polymer library to handle FOUC, thus proving that it is operating properly on the page. Additional hackery grabs the string representation of my element's shadow root for actually assertion.

Despite Protactor's claims to the contrary, it will not resolve standard HTML elements inside a shadow DOM to WebDriver WebElement objects. This assertion on an <h2> element inside my element's shadow DOM fails:
      // Fails with Error: Expected undefined to match '<h2>Build Your Pizza</h2>'.
      var shadowRoot = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.children[0]'
      );
      expect(shadowRoot.outerHTML).toMatch("<h2>Build Your Pizza</h2>");
But if I return the outerHTML string from executeScript(), then it passes:
      // This passes...
      var shadowRoot = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.children[0].outerHTML'
      );
      expect(shadowRoot).toMatch("<h2>Build Your Pizza</h2>");
So it seems that I am restricted to working with strings from executeScript(). That is a big pile of meh.

Instead, I would like to write that test as something along the lines of:
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      expect($('x-pizza').getShadowRootHtml()).
        toMatch("<h2>Build Your Pizza</h2>");
    });
I can get that with the following in my Protractor conf.js:
exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['tests/XPizzaSpec.js'],
  onPrepare: function() {
    var ElementFinder = $('').constructor;

    ElementFinder.prototype.getShadowRootHtml = function() {
      var locator = this.locator();
      if (locator.using != 'css selector') return '';

      var selector = locator.value;
      return browser.executeScript(
        'return document.querySelector("' + selector + '").shadowRoot.innerHTML'
      );
    };

    require('./tests/waitAbsent.js');
  }
};
Some of this is based on the waitAbsent.js code required at the bottom of the onPrepare() block. I borrowed this code yesterday, which is how I know how to access the ElementFinder prototype. The rest of this code assumes that a CSS selector is being used. If that is the case, then I work through the same executeScript() hackery.

The difference here is that the hackery is now encapsulated outside of my test, which reads better and, more importantly, continues to pass:
$ protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
.

Finished in 0.581 seconds
1 test, 2 assertions, 0 failures
That makes me happy. At least happier. I may continue this effort tomorrow to see if true testing happiness with Polymer and Protactor is possible.



Day #26

Monday, December 15, 2014

Protractor Can Test Polymer (Sort of)


I was unable to test Polymer elements with Protractor last night. But, thanks to some helpful suggestions this morning, I think it just might be possible after all.

My barrier last night was the seeming hard coupling between Protractor and AngularJS, which resulted in testing errors that indicated that Angular was not loaded:
  1) <x-pizza> element content has a shadow DOM
   Message:
     Error: Angular could not be found on the page http://localhost:8000/ : retries looking for angular exceeded
The solution suggested by various people is the undocumented ignoreSynchronization property on the Protractor browser object.

This property is meant to be set in the Protractor configuration or a beforeEach() block. I opt for the latter here:
describe('<x-pizza>', function(){
  var el;

  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      // Tests here...
    });
  });
Unfortunately, that does not solve all of my problems. I am still using last night's simple test—that the <x-pizza> Polymer element, being served by a Python simple HTTP server on port 8000, has a shadow root:
    it('has a shadow DOM', function(){
      el = $('x-pizza');
      expect(el.shadowRoot).toBeDefined();
    });
This simple test still fails:
Failures:

  1) <x-pizza> element content has a shadow DOM
   Message:
     Expected undefined to be defined.
The problem here is twofold. First, I have to wait for Polymer to have upgraded the <x-pizza> element on the page being served. Second, that el reference is not a regular DOM element—it is a WebDriver WebElement wrapper. Neither problem seems to be easily solved, though I am able to come up with a solution for each.

To wait for Polymer to be ready, I need to pull in an external library. Protractor has great support for waiting for Angular operations, but no support for waiting for other operations. Fortunately for me, others have had similar problems. I swipe one of those solutions, waitAbsent.js, from a Gist and save it in tests/waitAbsent.js. I then use it in my current setup by adding to the onPrepare section of my Protractor configuration:
exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: ['tests/XPizzaSpec.js'],
  onPrepare: function() {
    require('./tests/waitAbsent.js');
  }
};
There is also a waitReady.js, but I specifically want to use a wait-until-absent operation for Polymer so that I can wait until Polymer has loaded and removed the unresolved attribute from the document:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      el = $('x-pizza');
      expect(el.shadowRoot).toBeDefined();
  });
});
That will block the remainder of the test from being evaluated until the unresolved attribute is removed from the document. Polymer does this to deal with FOUC, so once that attribute is removed I know that Polymer is active on the page (this does assume that the author has added unresolved somewhere in the page).

Even waiting for Polymer to be active is not sufficient, however, because the el in that test is a WebElement wrapper. And to my intense consternation, there is no way to access the underlying element from the wrapper. This would be fine… if WebDriver had some way to access the shadow DOM for testing, but it does not. All of this leaves me resigned to hackery.

The hackery in this case comes in the form of executeScript(), which I use to execute a simple query of the element's shadow DOM, returning its the innerHTML:
describe('<x-pizza>', function(){
  describe('element content', function(){
    beforeEach(function(){
      browser.ignoreSynchronization = true;
      browser.get('http://localhost:8000');
    });

    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      var shadowRoot = browser.executeScript(
        'return document.querySelector("x-pizza").shadowRoot.innerHTML'
      );
      expect(shadowRoot).toMatch("<h2>Build Your Pizza</h2>");
  });
});
And that actually works:
$ protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
.

Finished in 0.621 seconds
1 test, 2 assertions, 0 failures

[launcher] 0 instance(s) of WebDriver still running
[launcher] chrome #1 passed
I know that is a legitimate test because I can make it fail by changing the expectation:
      expect(shadowRoot).toMatch("<h2>Build a Pizza</h2>");
Which results in:
Failures:

  1) <x-pizza> element content has a shadow DOM
   Message:
     Expected '
    <h2>Build Your Pizza</h2>
    <! ... -->
  ' to match '<h2>Build a Pizza</h2>'.
   Stacktrace:
     Error: Failed expectation
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:27:26)

Finished in 0.608 seconds
1 test, 2 assertions, 1 failure
I should probably convert the working executeScript into another onPrepare addition to Protractor, but I will leave that to another day. For now, I suffer with functional hackery.

One note about that functional hackery is that I have to be extremely careful with the return value from executeScript() and the matcher used. I had originally tried to return a typeof() for it:
      var shadowRoot = browser.executeScript(
        'return typeof(document.querySelector("x-pizza").shadowRoot);'
      );
The result of that is "object", which seems OK, until there is a problem. If there is no shadow root for any reason, the shadowRoot local variable is assigned to a promise that does not resolve to unresolved. In other words, I might not have a shadowRoot, but the following will still pass:
    it('has a shadow DOM', function(){
      expect($('[unresolved]').waitAbsent()).toBeTruthy();
      var shadowRoot = browser.executeScript(
        'return typeof(document.querySelector("h1").shadowRoot);'
      );
      expect(shadowRoot).toBeDefined();
    });
That already bit me during my investigation, so it is sure to cause problems down the line—all the more reason to want shadow DOM support of some kind in WebDriver. But, at least for one night, I was able to write a successful Protractor test for a Polymer element.


Day #25

Sunday, December 14, 2014

Trying Protractor to Test Polymer


Well, if I've gone to all of the trouble of getting Selenium installed...

Despite not being able to get The Intern to properly test Polymer elements, the effort was not a complete loss. There are some nice features in Intern that I wish existed elsewhere in JavaScript testing space. My favorite is probably the built-in web server. I know it's stupid, but I hate having to run my own web server in tests—there are just so many other moving parts that adding another is a source of some consternation. I also picked up a useful little trick from the Intern's configuration. They suggest an NPM post-install script to Bower install client side packages:
{
  "name": "plain_old_forms",
  // ...
  "scripts": {
    "postinstall": "bower install",
    "test": "intern-runner config=tests/intern"
  }
}
That seems perfectly reasonable, to the point of a best practice kind of thing, so I will likely adopt it going forward.

That said, my experience with the Intern was ultimately a failure since it was not able to reference Polymer elements, which puts a damper on testing them. But since I went to all the trouble of getting all things Selenium and Web Driver installed yesterday, I think that I will give Protractor a shot with testing Polymer elements. Sure, Protractor is really meant for testing AngularJS code and it requires a separate web server to test code, but it installs a bunch of the WebDriver stuff for me and, who knows, maybe it'll actually be able to reference Polymer elements.

I have fiddled with Protractor before, but it has been a while:
$ protractor --version
Version 0.22.0
So I upgrade:
$ npm install -g protractor
/home/chris/local/node-v0.10.20/bin/protractor -> /home/chris/local/node-v0.10.20/lib/node_modules/protractor/bin/protractor
/home/chris/local/node-v0.10.20/bin/webdriver-manager -> /home/chris/local/node-v0.10.20/lib/node_modules/protractor/bin/webdriver-manager
protractor@1.5.0 /home/chris/local/node-v0.10.20/lib/node_modules/protractor
...
➜  protractor git:(master) ✗ protractor --version     
Version 1.5.0
I already have the latest Selenium installed, but Protractor prefers Chrome as the default browser testing environment, which I do not have. So I let Protractor install it:
$ webdriver-manager update
Updating selenium standalone
...
Updating chromedriver
...

I have to admit that, even though it is a thin wrapper around the normal Java Selenium server startup command, I find Protractor's thin wrapper much more appealing:
webdriver-manager start
seleniumProcess.pid: 26247
23:40:45.524 INFO - Launching a standalone server
Setting system property webdriver.chrome.driver to /home/chris/local/node-v0.10.20/lib/node_modules/protractor/selenium/chromedriver
23:40:45.570 INFO - Java: Oracle Corporation 24.65-b04
23:40:45.571 INFO - OS: Linux 3.13.0-37-generic amd64
23:40:45.585 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
explorer, version=}] does not match with current platform: LINUX
23:40:45.734 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
...
23:40:45.766 INFO - Started HttpContext[/wd,/wd]
23:40:45.770 INFO - Started SocketListener on 0.0.0.0:4444
23:40:45.770 INFO - Started org.openqa.jetty.jetty.Server

I already have a smoke test page for the <x-pizza> pizza building element that I would like to test, so I fire up a Python simple server against which my tests can run:
$ python -m SimpleHTTPServer 8000
Serving HTTP on 0.0.0.0 port 8000 ...

This is a fresh copy of my <x-pizza> element. It has a bower.json for all of the necessary dependencies as well as an npm package.json for helper packages like Grunt. I add the post-install "trick" from the Intern to my package.json:
{
  "name": "plain_old_forms",
  "devDependencies": {
    "grunt": "~0.4.0",
    "grunt-contrib-watch": "~0.5.0"
  },
  "scripts": {
    "postinstall": "bower install"
  }
}
After an npm install, I am ready to test:
describe('<x-pizza>', function(){
  var el;

  describe('element content', function(){
    beforeEach(function(){
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      el = $('x-pizza');
      expect(el.shadowRoot).not.toBe(null);
    });
  });
});
This is not a "real" test. It just verifies that the <x-pizza> element is a real Polymer element—that it has a shadow root. This is the test that I was never able to get running on the Intern.

Unfortunately, this does not work any better in Protractor than in Intern. When I run the test, I get:
protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
F

Failures:

  1) <x-pizza> element content has a shadow DOM
   Message:
     Error: Angular could not be found on the page http://localhost:8000/ : retries looking for angular exceeded
   Stacktrace:
     Error: Angular could not be found on the page http://localhost:8000/ : retries looking for angular exceeded
==== async task ====
Protractor.get(http://localhost:8000/) - test for angular
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:6:15)
==== async task ====
Asynchronous test function: beforeEach()
Error
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:5:5)
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:4:3)
    at Object.<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:1:63)

Finished in 10.667 seconds
1 test, 2 assertions, 1 failure
So it seems that Protractor is more than a little coupled to Angular. Bummer.

I am somewhat surprised that Protractor is coupled to Angular to the point that it does not work with anything else. I had just assumed that it worked better with Angular than with other things. That is good to know, I suppose, but not terribly helpful for my current efforts. I may give the basic selenium-webdriver NPM package a try tomorrow, but that may be veering too close to my dislike of raw Selenium.


Day #24

Saturday, December 13, 2014

Can't Even Polymer with Intern and Local Selenium Server


I hate a problem unsolved. I really ought to move onto more useful ground, but I cannot for the life of me understand why I cannot get The Intern tester running with Polymer elements. So tonight, I try to remove one potential problem point from the equation: Sauce Labs.

Really, there is no reason that this should work, but if nothing else, running locally will give me more real time feedback on what might possibly be going on here. But I'll be honest here, I hate Selenium locally (which is one of the reasons I really appreciate Sauce Labs).

Running it is easy enough. I download the latest standalone server from the download page. I can then run the server immediately with:
$ java -jar Downloads/selenium-server-standalone-2.44.0.jar
00:07:06.282 INFO - Launching a standalone server
00:07:06.366 INFO - Java: Oracle Corporation 24.65-b04
00:07:06.366 INFO - OS: Linux 3.13.0-37-generic amd64
00:07:06.388 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
...
00:07:06.606 INFO - Started HttpContext[/wd,/wd]
00:07:06.611 INFO - Started SocketListener on 0.0.0.0:4444
But for the life of me, I cannot figure out how to install the Firefox driver. How stupid is that? This must be so obvious to just about everyone else, but I've got nothing.

Eventually, I read that the extension is contained in the standalone server. I would think that would mean that I could install it from the running server, but if that is case, I am unable to make it happen. So I unzip the JAR file:
$ unzip ../selenium-server-standalone-2.44.0.jar
...
   creating: org/openqa/grid/selenium/
   creating: org/openqa/grid/selenium/proxy/
   creating: org/openqa/grid/selenium/utils/
  inflating: org/openqa/grid/selenium/GridLauncher.class
  inflating: org/openqa/grid/selenium/proxy/DefaultRemoteProxy$1.class
  inflating: org/openqa/grid/selenium/proxy/DefaultRemoteProxy.class
  inflating: org/openqa/grid/selenium/utils/WebProxyHtmlRenderer.class
That does contain the XPI:
$ find | grep xpi$
./org/openqa/selenium/firefox/webdriver.xpi
Armed with that, I install it by directly pointing Firefox to the location on disk:



After allowing this to occur:



I am ready to make some changes to the Intern tests/intern.js configuration code. I switch from a specific browser version on Sauce Labs to my local Firefox:
  environments: [
  // { browserName: 'chrome', version: '34', platform: [ 'Linux' ] }
  { browserName: 'firefox' }
  ]
I switch from the Sauce Labs tunnel to no tunnel:
  // Name of the tunnel class to use for WebDriver tests
  //tunnel: 'SauceLabsTunnel',
  tunnel: 'NullTunnel',
  useSauceConnect: false,
Finally, I add webdriver configuration (this may not be necessary):
  webdriver: {
    host: 'localhost',
    port: 4444
  },
With that in place, I am ready to try again… which results in:
$ ./node_modules/.bin/intern-runner config=tests/intern
Listening on 0.0.0.0:9000
Starting tunnel...
Initialised firefox 34.0 on LINUX
Test main - index - <x-pizza> FAILED on firefox 34.0 on LINUX:
CancelError: Timeout reached on main - index - <x-pizza>
Error: Timeout reached on main - index - <x-pizza>
  at Error  <anonymous>
  at new ErrorCtor  <node_modules/intern/node_modules/dojo/errors/create.js:13:21>
  at null._onTimeout  <node_modules/intern/lib/Test.js:180:38>
  at Timer.listOnTimeout [as ontimeout]  <timers.js:110:15>
Unfortunately, this seems even worse than yesterday. The timeout is occurring while waiting for the Polymer element's shadow root property to come into existence:
    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .then(pollUntil('return document.querySelector("x-pizza").shadowRoot;', 1000))
        .findByTagName('x-pizza')
        // .execute('return document.querySelector("x-pizza");')
        .then(function (el) {
           console.log(el);
           console.log(el.$);
           console.log(el.constructor);
           console.log(el.shadowRoot);
           console.log(el[0]);
           assert.isDefined(el.shadowRoot);
         });
    }
That at least happened with Sauce Labs, but never returns locally. This is baffling because I can see the <x-pizza> element just starting to be resolved after Polymer has initialized and removed the unresolved attribute from the <body>:



Unfortunately, I cannot think of anything else to try tonight that I had not already tried last night with Sauce Labs. If nothing else, I can see very definitely tonight that the <x-pizza> element is present and should have a shadowRoot (which is where the SVG pizza resides). But for whatever reason, the Intern is unable to access this element for testing.

I think I am ready to give up on Intern for real now, but I may give Protractor a whirl with this tomorrow. Even if this particular WebDriver tool cannot yet test Polymer, perhaps another tool can.


Day #23


Friday, December 12, 2014

Delaying Intern for Polymer Tests


I started evaluating The Intern last night as an alternative to Karma for testing Polymer elements. Although I was not able to get it actually working, I can already see that there is much to like about Intern.

The biggest win so far is the ability to run acceptance tests against the same page on which I run my manual smoke tests. Configuration and setup does not seem much different than Karma. I also have yet to try my hand with Protractor for testing Polymer elements (another selenium driver based framework). So I cannot say that Intern is far and away the best, but it was relatively quick and easy to get my first test running.

Hopefully tonight I can actually get it passing.

I am still trying to test the <x-pizza> pizza building Polymer element. I am reasonably sure that last night's setup and configuration is solid. The problem lies in the test itself:
    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .findByTagName('x-pizza')
        .getProperty('shadowRoot')
        .then(function (doc) {
           console.log(doc);
           assert.isDefined(doc);
         });
    }
If I watch the test in the Sauce Labs viewer, everything looks good through finding the <body> tag without an unresolved attribute. From there, things are wonky.

The unresolved attribute is a Polymer convention for dealing with FOUC. Before Polymer loads and initializes the <body> content is not displayed. Once Polymer is ready, the unresolved attribute is removed from <body>, which starts a CSS transition to display contents.

And all of that works—even in the Sauce Labs viewer. The problem is finding the <x-pizza> element in the next step. It is in the original page:
  <body unresolved>
    <div class="container">
      <form action="test" method="post">
        <h1>Plain-old Form</h1>
        <x-pizza name="pizza_toppings"></x-pizza>
        <button>Order my pizza!</button>
      </form>
    </div>
  </body>
But before Polymer does its thing, it is an unresolved HTML element. After it is upgraded to a web component by Polymer, it should have a shadow root as the test attempts to assert (along with all kinds of pizza building behavior). But my test never even gets a chance to try that. It fails with a stale element error:
StaleElementReference: ... stale element reference: element is not attached to the page document
My guess here is that, even though Polymer is ready, it has not replaced the original, unresolved <x-pizza> element with the Polymer web component. That is, I believe that I need to wait a few milliseconds (possibly as little as a single event loop) before my test finds that element.

Unfortunately for me, this seems to be impossibly hard in Intern. Unless you read the documentation. I don't know how I missed that last night.

So I add pollUntil to my list of dependencies and use it:
define([
  'intern!object',
  'intern/chai!assert',
  'intern/node_modules/dojo/node!leadfoot/helpers/pollUntil',
  'require'
], function (registerSuite, assert, pollUntil, require) {
  registerSuite({
    name: 'index',

    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .then(pollUntil('return document.querySelector("x-pizza").shadowRoot;', 1000))
        .findByTagName('x-pizza')
        .getProperty('shadowRoot')
        .then(function (doc) {
           console.log(doc);
           assert.isDefined(doc);
         });
    }
  });
});
Only that has no effect whatsoever. I still get a stale element error.

In fact, if I try to get the element directly and run assertions against it, I find no shadow root even though I have just polled until it exists:
    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .then(pollUntil('return document.querySelector("x-pizza").shadowRoot;', 1000))
        .findByTagName('x-pizza')
        // .execute('return document.querySelector("x-pizza");')
        .then(function (el) {
           console.log(el);
           console.log(el.constructor);
           console.log(el.$);
           console.log(el.shadowRoot);
           assert.isDefined(el.shadowRoot);
         });
    }
The test always fails with shadowRoot being undefined. The properties logged are also undefined, except for the constructor, which is Element instead of Polymer.

At this point, I have to confess that I am at a loss for what to do next. No matter what I try, I get the original, unresolved <x-pizza> element. Even though the Polymer element is definitely in the document, I cannot gain access to it through Intern. Unless inspiration strikes, I am left to believe that Intern does not work with Polymer yet.


Day #22