Sunday, December 14, 2014

Trying Protractor to Test Polymer


Well, if I've gone to all of the trouble of getting Selenium installed...

Despite not being able to get The Intern to properly test Polymer elements, the effort was not a complete loss. There are some nice features in Intern that I wish existed elsewhere in JavaScript testing space. My favorite is probably the built-in web server. I know it's stupid, but I hate having to run my own web server in tests—there are just so many other moving parts that adding another is a source of some consternation. I also picked up a useful little trick from the Intern's configuration. They suggest an NPM post-install script to Bower install client side packages:
{
  "name": "plain_old_forms",
  // ...
  "scripts": {
    "postinstall": "bower install",
    "test": "intern-runner config=tests/intern"
  }
}
That seems perfectly reasonable, to the point of a best practice kind of thing, so I will likely adopt it going forward.

That said, my experience with the Intern was ultimately a failure since it was not able to reference Polymer elements, which puts a damper on testing them. But since I went to all the trouble of getting all things Selenium and Web Driver installed yesterday, I think that I will give Protractor a shot with testing Polymer elements. Sure, Protractor is really meant for testing AngularJS code and it requires a separate web server to test code, but it installs a bunch of the WebDriver stuff for me and, who knows, maybe it'll actually be able to reference Polymer elements.

I have fiddled with Protractor before, but it has been a while:
$ protractor --version
Version 0.22.0
So I upgrade:
$ npm install -g protractor
/home/chris/local/node-v0.10.20/bin/protractor -> /home/chris/local/node-v0.10.20/lib/node_modules/protractor/bin/protractor
/home/chris/local/node-v0.10.20/bin/webdriver-manager -> /home/chris/local/node-v0.10.20/lib/node_modules/protractor/bin/webdriver-manager
protractor@1.5.0 /home/chris/local/node-v0.10.20/lib/node_modules/protractor
...
➜  protractor git:(master) ✗ protractor --version     
Version 1.5.0
I already have the latest Selenium installed, but Protractor prefers Chrome as the default browser testing environment, which I do not have. So I let Protractor install it:
$ webdriver-manager update
Updating selenium standalone
...
Updating chromedriver
...

I have to admit that, even though it is a thin wrapper around the normal Java Selenium server startup command, I find Protractor's thin wrapper much more appealing:
webdriver-manager start
seleniumProcess.pid: 26247
23:40:45.524 INFO - Launching a standalone server
Setting system property webdriver.chrome.driver to /home/chris/local/node-v0.10.20/lib/node_modules/protractor/selenium/chromedriver
23:40:45.570 INFO - Java: Oracle Corporation 24.65-b04
23:40:45.571 INFO - OS: Linux 3.13.0-37-generic amd64
23:40:45.585 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
explorer, version=}] does not match with current platform: LINUX
23:40:45.734 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
...
23:40:45.766 INFO - Started HttpContext[/wd,/wd]
23:40:45.770 INFO - Started SocketListener on 0.0.0.0:4444
23:40:45.770 INFO - Started org.openqa.jetty.jetty.Server

I already have a smoke test page for the <x-pizza> pizza building element that I would like to test, so I fire up a Python simple server against which my tests can run:
$ python -m SimpleHTTPServer 8000
Serving HTTP on 0.0.0.0 port 8000 ...

This is a fresh copy of my <x-pizza> element. It has a bower.json for all of the necessary dependencies as well as an npm package.json for helper packages like Grunt. I add the post-install "trick" from the Intern to my package.json:
{
  "name": "plain_old_forms",
  "devDependencies": {
    "grunt": "~0.4.0",
    "grunt-contrib-watch": "~0.5.0"
  },
  "scripts": {
    "postinstall": "bower install"
  }
}
After an npm install, I am ready to test:
describe('<x-pizza>', function(){
  var el;

  describe('element content', function(){
    beforeEach(function(){
      browser.get('http://localhost:8000');
    });
    it('has a shadow DOM', function(){
      el = $('x-pizza');
      expect(el.shadowRoot).not.toBe(null);
    });
  });
});
This is not a "real" test. It just verifies that the <x-pizza> element is a real Polymer element—that it has a shadow root. This is the test that I was never able to get running on the Intern.

Unfortunately, this does not work any better in Protractor than in Intern. When I run the test, I get:
protractor conf.js
Using the selenium server at http://localhost:4444/wd/hub
[launcher] Running 1 instances of WebDriver
F

Failures:

  1) <x-pizza> element content has a shadow DOM
   Message:
     Error: Angular could not be found on the page http://localhost:8000/ : retries looking for angular exceeded
   Stacktrace:
     Error: Angular could not be found on the page http://localhost:8000/ : retries looking for angular exceeded
==== async task ====
Protractor.get(http://localhost:8000/) - test for angular
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:6:15)
==== async task ====
Asynchronous test function: beforeEach()
Error
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:5:5)
    at [object Object].<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:4:3)
    at Object.<anonymous> (/home/chris/repos/polymer-book/play/protractor/tests/XPizzaSpec.js:1:63)

Finished in 10.667 seconds
1 test, 2 assertions, 1 failure
So it seems that Protractor is more than a little coupled to Angular. Bummer.

I am somewhat surprised that Protractor is coupled to Angular to the point that it does not work with anything else. I had just assumed that it worked better with Angular than with other things. That is good to know, I suppose, but not terribly helpful for my current efforts. I may give the basic selenium-webdriver NPM package a try tomorrow, but that may be veering too close to my dislike of raw Selenium.


Day #24

Saturday, December 13, 2014

Can't Even Polymer with Intern and Local Selenium Server


I hate a problem unsolved. I really ought to move onto more useful ground, but I cannot for the life of me understand why I cannot get The Intern tester running with Polymer elements. So tonight, I try to remove one potential problem point from the equation: Sauce Labs.

Really, there is no reason that this should work, but if nothing else, running locally will give me more real time feedback on what might possibly be going on here. But I'll be honest here, I hate Selenium locally (which is one of the reasons I really appreciate Sauce Labs).

Running it is easy enough. I download the latest standalone server from the download page. I can then run the server immediately with:
$ java -jar Downloads/selenium-server-standalone-2.44.0.jar
00:07:06.282 INFO - Launching a standalone server
00:07:06.366 INFO - Java: Oracle Corporation 24.65-b04
00:07:06.366 INFO - OS: Linux 3.13.0-37-generic amd64
00:07:06.388 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
...
00:07:06.606 INFO - Started HttpContext[/wd,/wd]
00:07:06.611 INFO - Started SocketListener on 0.0.0.0:4444
But for the life of me, I cannot figure out how to install the Firefox driver. How stupid is that? This must be so obvious to just about everyone else, but I've got nothing.

Eventually, I read that the extension is contained in the standalone server. I would think that would mean that I could install it from the running server, but if that is case, I am unable to make it happen. So I unzip the JAR file:
$ unzip ../selenium-server-standalone-2.44.0.jar
...
   creating: org/openqa/grid/selenium/
   creating: org/openqa/grid/selenium/proxy/
   creating: org/openqa/grid/selenium/utils/
  inflating: org/openqa/grid/selenium/GridLauncher.class
  inflating: org/openqa/grid/selenium/proxy/DefaultRemoteProxy$1.class
  inflating: org/openqa/grid/selenium/proxy/DefaultRemoteProxy.class
  inflating: org/openqa/grid/selenium/utils/WebProxyHtmlRenderer.class
That does contain the XPI:
$ find | grep xpi$
./org/openqa/selenium/firefox/webdriver.xpi
Armed with that, I install it by directly pointing Firefox to the location on disk:



After allowing this to occur:



I am ready to make some changes to the Intern tests/intern.js configuration code. I switch from a specific browser version on Sauce Labs to my local Firefox:
  environments: [
  // { browserName: 'chrome', version: '34', platform: [ 'Linux' ] }
  { browserName: 'firefox' }
  ]
I switch from the Sauce Labs tunnel to no tunnel:
  // Name of the tunnel class to use for WebDriver tests
  //tunnel: 'SauceLabsTunnel',
  tunnel: 'NullTunnel',
  useSauceConnect: false,
Finally, I add webdriver configuration (this may not be necessary):
  webdriver: {
    host: 'localhost',
    port: 4444
  },
With that in place, I am ready to try again… which results in:
$ ./node_modules/.bin/intern-runner config=tests/intern
Listening on 0.0.0.0:9000
Starting tunnel...
Initialised firefox 34.0 on LINUX
Test main - index - <x-pizza> FAILED on firefox 34.0 on LINUX:
CancelError: Timeout reached on main - index - <x-pizza>
Error: Timeout reached on main - index - <x-pizza>
  at Error  <anonymous>
  at new ErrorCtor  <node_modules/intern/node_modules/dojo/errors/create.js:13:21>
  at null._onTimeout  <node_modules/intern/lib/Test.js:180:38>
  at Timer.listOnTimeout [as ontimeout]  <timers.js:110:15>
Unfortunately, this seems even worse than yesterday. The timeout is occurring while waiting for the Polymer element's shadow root property to come into existence:
    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .then(pollUntil('return document.querySelector("x-pizza").shadowRoot;', 1000))
        .findByTagName('x-pizza')
        // .execute('return document.querySelector("x-pizza");')
        .then(function (el) {
           console.log(el);
           console.log(el.$);
           console.log(el.constructor);
           console.log(el.shadowRoot);
           console.log(el[0]);
           assert.isDefined(el.shadowRoot);
         });
    }
That at least happened with Sauce Labs, but never returns locally. This is baffling because I can see the <x-pizza> element just starting to be resolved after Polymer has initialized and removed the unresolved attribute from the <body>:



Unfortunately, I cannot think of anything else to try tonight that I had not already tried last night with Sauce Labs. If nothing else, I can see very definitely tonight that the <x-pizza> element is present and should have a shadowRoot (which is where the SVG pizza resides). But for whatever reason, the Intern is unable to access this element for testing.

I think I am ready to give up on Intern for real now, but I may give Protractor a whirl with this tomorrow. Even if this particular WebDriver tool cannot yet test Polymer, perhaps another tool can.


Day #23


Friday, December 12, 2014

Delaying Intern for Polymer Tests


I started evaluating The Intern last night as an alternative to Karma for testing Polymer elements. Although I was not able to get it actually working, I can already see that there is much to like about Intern.

The biggest win so far is the ability to run acceptance tests against the same page on which I run my manual smoke tests. Configuration and setup does not seem much different than Karma. I also have yet to try my hand with Protractor for testing Polymer elements (another selenium driver based framework). So I cannot say that Intern is far and away the best, but it was relatively quick and easy to get my first test running.

Hopefully tonight I can actually get it passing.

I am still trying to test the <x-pizza> pizza building Polymer element. I am reasonably sure that last night's setup and configuration is solid. The problem lies in the test itself:
    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .findByTagName('x-pizza')
        .getProperty('shadowRoot')
        .then(function (doc) {
           console.log(doc);
           assert.isDefined(doc);
         });
    }
If I watch the test in the Sauce Labs viewer, everything looks good through finding the <body> tag without an unresolved attribute. From there, things are wonky.

The unresolved attribute is a Polymer convention for dealing with FOUC. Before Polymer loads and initializes the <body> content is not displayed. Once Polymer is ready, the unresolved attribute is removed from <body>, which starts a CSS transition to display contents.

And all of that works—even in the Sauce Labs viewer. The problem is finding the <x-pizza> element in the next step. It is in the original page:
  <body unresolved>
    <div class="container">
      <form action="test" method="post">
        <h1>Plain-old Form</h1>
        <x-pizza name="pizza_toppings"></x-pizza>
        <button>Order my pizza!</button>
      </form>
    </div>
  </body>
But before Polymer does its thing, it is an unresolved HTML element. After it is upgraded to a web component by Polymer, it should have a shadow root as the test attempts to assert (along with all kinds of pizza building behavior). But my test never even gets a chance to try that. It fails with a stale element error:
StaleElementReference: ... stale element reference: element is not attached to the page document
My guess here is that, even though Polymer is ready, it has not replaced the original, unresolved <x-pizza> element with the Polymer web component. That is, I believe that I need to wait a few milliseconds (possibly as little as a single event loop) before my test finds that element.

Unfortunately for me, this seems to be impossibly hard in Intern. Unless you read the documentation. I don't know how I missed that last night.

So I add pollUntil to my list of dependencies and use it:
define([
  'intern!object',
  'intern/chai!assert',
  'intern/node_modules/dojo/node!leadfoot/helpers/pollUntil',
  'require'
], function (registerSuite, assert, pollUntil, require) {
  registerSuite({
    name: 'index',

    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .then(pollUntil('return document.querySelector("x-pizza").shadowRoot;', 1000))
        .findByTagName('x-pizza')
        .getProperty('shadowRoot')
        .then(function (doc) {
           console.log(doc);
           assert.isDefined(doc);
         });
    }
  });
});
Only that has no effect whatsoever. I still get a stale element error.

In fact, if I try to get the element directly and run assertions against it, I find no shadow root even though I have just polled until it exists:
    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .then(pollUntil('return document.querySelector("x-pizza").shadowRoot;', 1000))
        .findByTagName('x-pizza')
        // .execute('return document.querySelector("x-pizza");')
        .then(function (el) {
           console.log(el);
           console.log(el.constructor);
           console.log(el.$);
           console.log(el.shadowRoot);
           assert.isDefined(el.shadowRoot);
         });
    }
The test always fails with shadowRoot being undefined. The properties logged are also undefined, except for the constructor, which is Element instead of Polymer.

At this point, I have to confess that I am at a loss for what to do next. No matter what I try, I get the original, unresolved <x-pizza> element. Even though the Polymer element is definitely in the document, I cannot gain access to it through Intern. Unless inspiration strikes, I am left to believe that Intern does not work with Polymer yet.


Day #22

Thursday, December 11, 2014

Kicking the Tires on Polymer and The Intern


One of the things that I love about JavaScript is the dynamic ecosystem that exists just about everywhere. Of course, that's also the thing that I hate most about it.

It is thrilling that so much cool, new stuff pops up daily. It is especially fun for someone who needs to learn something every day lest the gods smite him (or her). That said, it is frustrating when new best practices pop up with alarming frequency—especially for things that ought to be solved problems like testing.

Along those lines, I was recently accused (in the nice, playful way) of being stuck in 2012 by testing my Polymer JavaScript with Karma. I will say this for Karma: it works and is easy to get running under CI services. So there is something to be said for 2012. That said, it is almost 2015 so perhaps there are better options out there, like The Intern.

I start by copying a Karma tested Polymer project and removing Karma configuration and dependency references. Next I install Intern:
npm install intern --save-dev
...
intern@2.2.2 node_modules/intern
...
Next, I update the NPM package.json to include an Intern test runner:
{
  "name": "plain_old_forms",
  "devDependencies": {
    "grunt": "~0.4.0",
    "grunt-contrib-watch": "~0.5.0",
    "intern": "^2.2.2"
  },
  "scripts": {
    "postinstall": "bower install",
    "test": "intern-runner config=tests/intern"
  }
}
I copy the default test config:
$ mkdir tests
$ cp node_modules/intern/tests/example.intern.js tests/intern.js
Normally in that configuration, I would need to list AMD compatible modules to be loaded into each test:
  // ...
 loader: {
  // Packages that should be registered with the loader in each testing environment

  packages: [
   { name: 'todo', location: 'js' },
   { name: 'jquery', location: 'bower_components/jquery' },
   { name: 'underscore', location: 'bower_components/underscore' },
   { name: 'backbone', location: 'bower_components/backbone' },
   { name: 'backboneStorage', location: 'bower_components/backbone.localStorage' }
  ]

 },
 // ...
As far as I know there is nothing AMD compatible in Polymer, so I leave this blank instead:
 // ...
 loader: { },
 // ...
I have yet to write a test, but I start Intern anyway to see where I am at:
$ SAUCE_USERNAME=eee-c SAUCE_ACCESS_KEY=<my-key> ./node_modules/.bin/intern-runner config=tests/intern 
Listening on 0.0.0.0:9000
Starting tunnel...
...
Starting secure remote tunnel VM...
...
Secure remote tunnel VM is now: booting
Secure remote tunnel VM is now: running
...
Starting Selenium listener...
Establishing secure TLS connection to tunnel...
Selenium listener started on port 4444.
Connection established.
Ready
Error: Failed to load module tests/functional from /home/chris/repos/polymer-book/play/theintern/tests/functional.js (parent: *34)
  at <node_modules/intern/node_modules/dojo/dojo.js:757:12>
  at <fs.js:207:20>
  at Object.oncomplete  <fs.js:107:15>
Where I am at is that I need to write a test, which is good news. Connectivity to Sauce Labs seems to just work—always nice not to have to start my own Selenium Server (ick).

So I actually add a first pass at testing my <x-pizza> Polymer element. I am pretty sure this is going to need to be a functional test, I save it as tests/functional.js:
define([
  'intern!object',
  'intern/chai!assert',
  'require'
], function (registerSuite, assert, require) {
  registerSuite({
    name: 'index',

    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByTagName('x-pizza')
        .getProperty('shadowRoot')
        .then(function (doc) {
           assert.isDefined(doc);
         });
        }
    });
});
The test in this case simply tried to assert that my <x-pizza> Polymer element has a shadow DOM.

After adding this test to the intern.js configuration:
  // ...
  functionalSuites: [ 'tests/functional' ],
  // ...
I am ready to try my test again.

But that does not seem to work. The test process just hangs:
$ SAUCE_USERNAME=eee-c SAUCE_ACCESS_KEY=<my-key> ./node_modules/.bin/intern-runner config=tests/intern
Listening on 0.0.0.0:9000
...
Secure remote tunnel VM is now: booting
Secure remote tunnel VM is now: running
...
Starting Selenium listener...
Establishing secure TLS connection to tunnel...
Selenium listener started on port 4444.
Connection established.
Ready
This may have been a lack of patience on my part (though I did wait for more than 2 minutes). Eventually I was able to get this working by fiddling with the environments value in intern.js I had copied in a local value when I should have been using a Sauce Labs compatible version:
 environments: [
  { browserName: 'chrome', version: '34', platform: [ 'Linux' ] }
            ]
With that, I am able to get the test to run, though it still fails:
Test main - index - <x-pizza> FAILED on chrome 34.0.1847.116 on Linux:
StaleElementReference: [POST http://(redacted)@localhost:4444/wd/hub/session/239770a379f045a2ad8d3b098e9e05df/execute / {"script":"return arguments[0]
[arguments[1]];","args":[{"ELEMENT":"0.7332835071720183-1"},"shadowRoot"]}] stale element reference: element is not attached to the page document
  (Session info: chrome=34.0.1847.116)
...
=============================== Coverage summary ===============================
Statements   : 19% ( 19/100 )
Branches     : 0% ( 0/20 )
Functions    : 32% ( 8/25 )
Lines        : 20.43% ( 19/93 )
================================================================================
chrome 34.0.1847.116 on Linux: 1/1 tests failed

-----------------|-----------|-----------|-----------|-----------|
File             |   % Stmts |% Branches |   % Funcs |   % Lines |
-----------------|-----------|-----------|-----------|-----------|
   elements/     |        19 |         0 |        32 |     20.43 |
      x_pizza.js |        19 |         0 |        32 |     20.43 |
-----------------|-----------|-----------|-----------|-----------|
All files        |        19 |         0 |        32 |     20.43 |
-----------------|-----------|-----------|-----------|-----------|

TOTAL: tested 1 platforms, 1/1 tests failed
Unfortunately, my attempts to get this fixed do not go well. I believe that the stale element reference is due to gaining a reference to the <x-pizza> element before Polymer upgrades it (Sauce Lab replays would seem to confirm this). I try to wait for the <body> element to lose its unresolved attribute, which Polymer removes when it is loaded:
    '<x-pizza>': function () {
      return this.remote
        .get(require.toUrl('../index.html'))
        .setFindTimeout(5000)
        .findByXpath('//body[not(@unresolved)]')
        .findByTagName('x-pizza')
        .getProperty('shadowRoot')
        .then(function (doc) {
           console.log(doc);
           assert.isDefined(doc);
         });
    }
Again, Sauce Lab replays would seem to confirm that this has the desired effect—immediately after that step passes, the display starts to fade-in. But I still get the stale element error.

My best guess is that, even though Polymer is loaded, it needs just a moment longer before it replaces the original <x-pizza> with the version that I am trying to test. Unfortunately, if brief delays are easy in Intern, I cannot find them. So I call it a night here and will pick back up with the effort tomorrow.

Day #21

Wednesday, December 10, 2014

Polymer (JS) Testing on Travis CI


Last night, I got a Drone.io build for a Polymer.dart element working in relatively short order. Tonight, I hope to repeat that success for a JavaScript Polymer element.

To be sure, JavaScript testing is far more tedious than the equivalent Dart code. Still, there are services out there, like the excellent Travis CI, that help to ease some of the pain. What I appreciate about Travis CI is that it is one of those tools that makes it easy to work 80% of the use cases and still manages to stay out of the way when working the other 20%. Hopefully that will prove to be the case here since testing Polymer elements almost certainly falls in the latter 20%.

I am working with the a-form-input Polymer class, which enables other Polymer elements to behave like native HTML form inputs. I already have passing Karma / Jasmine tests for this project, so all I need to accomplish tonight is to get them passing on Travis CI.

I start in the Travis CI interface to enable my project from among my GitHub projects:



The Karma project includes Travis CI documentation, which indicates that Firefox testing is likely the easiest path for me. Unfortunately, I have not been testing with Firefox in this project:
$ karma start --single-run --browsers Firefox
INFO [karma]: Karma v0.12.28 server started at http://localhost:9876/
WARN [launcher]: Can not load "Firefox", it is not registered!
  Perhaps you are missing some plugin?
To address this, I need to add the Firefox launcher to my npm package.json's list of dependencies:
{
  "name": "a-form-input",
  "desciption": "Polymer element that mimics a native <form> input",
  "repository": "https://github.com/eee-c/a-form-input",
  "devDependencies": {
    "grunt": "^0.4.5",
    "grunt-contrib-watch": "^0.6.1",
    "karma": "^0.12.24",
    "karma-chrome-launcher": "^0.1.5",
    "karma-firefox-launcher": "^0.1.3",
    "karma-jasmine": "^0.2.2"
  }
}
After a quick install:
$ npm install
karma-firefox-launcher@0.1.3 node_modules/karma-firefox-launcher
I have my Karma setup working with Firefox:
$ karma start --single-run --browsers Firefox
INFO [karma]: Karma v0.12.28 server started at http://localhost:9876/
INFO [launcher]: Starting browser Firefox
INFO [Firefox 34.0.0 (Ubuntu)]: Connected on socket Yd6CO1SBzegq4s_R-7CC with id 53464891
.....
Firefox 34.0.0 (Ubuntu): Executed 5 of 5 SUCCESS (0.336 secs / 0.333 secs)
For that to work on Travis, I need to supply that same command-line as the test script in the npm package.json file:
{
  "name": "a-form-input",
  // ...
  "scripts": {
    "test": "./node_modules/karma/bin/karma start --single-run --browsers Firefox"
  }
}
By default, Travis runs npm test, which looks for this entry.

At this point, I am ready for Travis CI (I hope). I add a .travis.yml to the repository and push it to Github:
language: node_js
node_js:
  - 0.10
before_script:
  - export DISPLAY=:99.0
  - sh -e /etc/init.d/xvfb start
The before_script section comes straight from the Karma Travis CI documentation for running Firefox, which requires an X Window system (virtual in this case).

Just like that, Travis CI springs into action and... I have my first build failure:
WARN [watcher]: Pattern "/home/travis/build/eee-c/a-form-input/bower_components/webcomponentsjs/webcomponents.js" does not match any file.
WARN [watcher]: Pattern "/home/travis/build/eee-c/a-form-input/bower_components/!(a-form-input)/**" does not match any file.
INFO [Firefox 31.0.0 (Linux)]: Connected on socket qvBVzvavVfpYhsSXB3og with id 12698000
Firefox 31.0.0 (Linux) <a-form-input> element form input element is created in the parent light DOM FAILED
 ReferenceError: Polymer is not defined in /home/travis/build/eee-c/a-form-input/test/PolymerSetup.js (line 18)
 waitForPolymer@/home/travis/build/eee-c/a-form-input/test/PolymerSetup.js:18:1
 @/home/travis/build/eee-c/a-form-input/test/PolymerSetup.js:28:3
 env.executeFiltered@/home/travis/build/eee-c/a-form-input/node_modules/karma-jasmine/lib/boot.js:126:7
 createStartFn/<@/home/travis/build/eee-c/a-form-input/node_modules/karma-jasmine/lib/adapter.js:171:5
 [2]</Karma/this.loaded@http://localhost:9876/karma.js:185:7
 @http://localhost:9876/context.html:48:5
 
 Expected null not to be null.
 @/home/travis/build/eee-c/a-form-input/test/AFormInputSpec.js:20:7
Yikes! That is only a portion of the failure even that portion is awfully noisy. Fortunately, I have seen enough Polymer Karma failures to know that I need to look to the top of the error messages before troubleshooting the actual tests. At the top of the build output, I see:
WARN [watcher]: Pattern "/home/travis/build/eee-c/a-form-input/bower_components/webcomponentsjs/webcomponents.js" does not match any file.
WARN [watcher]: Pattern "/home/travis/build/eee-c/a-form-input/bower_components/!(a-form-input)/**" does not match any file.
Wonderful. It seems that my Bower packages are not installed. So in addition to the nice and easy npm install that Travis CI boasts, I am going to have to convince it to also install the Bower npm package and to run the Bower install command.

Installing Bower is the easy part. I add it to the list of the npm package.json development dependencies:
{
  "name": "a-form-input",
  "devDependencies": {
    // ...
    "bower", "^1.3.12"
  },
  "scripts": {
    "test": "./node_modules/karma/bin/karma start --single-run --browsers Firefox"
  }
}
To run a bower install, I add it to the list of before_script actions in my .travis.yml file:
language: node_js
node_js:
  - 0.10
before_script:
  - ./node_modules/bower/bin/bower install
  - export DISPLAY=:99.0
  - sh -e /etc/init.d/xvfb start
And, amazingly, that actually works. All my bower packages are installed. All my npm packages are installed. And the build passes:



In the end, it was not quite as easy getting JavaScript Polymer tests running on Travis CI as it was running Dart Polymer tests on Drone.io. But it was close—and certainly easy enough once I pushed through the Bower nonsense. It might be nice if Travis CI supported Bower out of the box, but let's face it, this is more a win for Dart packaging than anything else. Travis CI's minimal (non) approach to Bower is likely the only rational way to deal with two different package managers for the same language.

Well, that or just use Dart.

At any rate, I have my CI build working for a JavaScript Polymer element thanks to Travis CI.

Day #20

Tuesday, December 9, 2014

Continuous Integration Testing Polymer (Dart) Elements


Tonight, I place a Polymer.dart project under continuous integration. I have done similar work with some of the tests in the Patterns in Polymer book, but that is running on a custom install Jenkins server. This time, I am going to install on a standard CI service—Drone.io.

Last night I got a command-line build script ready for the AFormInput class/mixin. I hope that getting that test running on the service will be similar to "regular" Dart projects, but there is only one way to be sure. The only difference between these tests and regular Dart CI that I can foresee causing trouble is the need to fire up a Dart Pub server during the tests. The test script from last night should take care of this—it is more a question of if the CI service will allow it.

I start by publishing the project to GitHub at: https://github.com/eee-c/a-form-input-dart. With that, I am ready to create a new project in Drone, which is done by selecting a project from the list of GitHub projects associated with my account:



The choice of project template is easy:



And the initial test setup looks great (I created test/run.sh last night):



With that, I think I'm ready to test:



Or not. The first build fails with:
$ test/run.sh
Analyzing [lib/a_form_input.dart]...
No issues found
Looks good!

Loading source assets...
Loading polymer transformers...
Serving a-form-input test on http://localhost:8080
Build completed successfully
test/run.sh: line 20: content_shell: command not found
That's a solved problem, so I solve it again by adding a step in test/run.sh to install content_shell:
#!/bin/bash

# Static type analysis...

# Install content_shell if not already present
which content_shell
if [[ $? -ne 0 ]]; then
  $DART_SDK/../chromium/download_contentshell.sh
  unzip content_shell-linux-x64-release.zip

  cs_path=$(ls -d drt-*)
  PATH=$cs_path:$PATH
fi

# Run tests...
With that... the build still fails. I was not sure it would be necessary, but indeed, I do need to start xvfb to provide the X windowing environment for content_shell:
sudo start xvfb
pub install
test/test_runner.sh
And with that, I have a passing build—pub server and all:
Loading source assets...
Loading polymer transformers...
Serving a-form-input test on http://localhost:8080
Build completed successfully
....
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> has a shadowRoot
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - value property is updated when internal state changes
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - value attribute is updated when internal state changes
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - form value attribute is updated when internal state changes
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - containing form includes input with supplied name attribute
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - setting the name property updates the name attribute
CONSOLE MESSAGE: 
CONSOLE MESSAGE: All 6 tests passed.
With the build passing:



All that is left is to copy the badge code:



Into the project README:



And that is all there is to it! So my worries were unfounded—Drone.io makes it just as easy to test a Polymer.dart project as it does to test any other Dart project. That is great news. Up tomorrow, I will push my luck by attempting to do the same with a JavaScript Polymer project.


Day #19

Monday, December 8, 2014

Testing Polymer (Dart) Mixins


I need to improve my continuous integration solutions for Patterns in Polymer. I do not know that I necessarily need to include a chapter on it in the book, but I ought to at least be able to point to some solutions in my research notes. Especially since I have been asked about it multiple times this week alone.

With that in mind, I first need to test my AFormInputMixin class from last night, which serves as mixin or baseclass for other Polymer.dart elements so that they can behave like native HTML form input elements. This turns out to be a little trickier than I had expected, but I'll get to that in a bit.

I start with the usual Polymer.dart test/index.html entry point in this package's pubspec.yaml file:
name: a-form-input
author: "Chris Strom <github@eeecomputes.com>"
description: "Polymer element that mimics a native <form> input."
dependencies:
  polymer: any
dev_dependencies:
  unittest: any
transformers:
- polymer:
    entry_points:
      - test/index.html
The test/index.html page loads a concrete element that uses the mixin, then pulls in the necessary test code:
<!doctype html>
<html>
<head>
  <!-- Load component(s) -->
  <link rel="import" href="x-double.html">

  <!-- The actual tests -->
  <script type="application/dart" src="test.dart"></script>
  <script src="packages/unittest/test_controller.js"></script>
</head>
<body>
</body>
</html>
The test/test.dart file is pretty typical Polymer testing code. It pulls in the necessary libraries, initializes Polymer (there can be only one main() so the test entry point has to start Polymer), creates DOM containers for the test elements and establishes tear downs for the same:
library plain_old_forms_test;

import 'package:unittest/unittest.dart';
import 'dart:html';
import 'dart:async';
import 'package:polymer/polymer.dart';

main() {
  initPolymer();

  var _el, _form, _container;
  group("AFormInputMixin mixed into <x-double>", (){
    setUp((){
      _container = createElement('<div></div>');
      _form = createElement('<form></form>');
      _el = createElement('<x-double></x-double>');

      _form.append(_el);
      _container.append(_form);
      document.body.append(_container);
    });

    tearDown((){
      _container.remove();
    });

    group('acts like <input> -', (){
      setUp((){
        _el.name = 'my_field_name';
        _el.start = '21';

        var completer = new Completer();
        _el.async(completer.complete);
        return completer.future;
      });

      test('value property is updated when internal state changes', (){
        expect(_el.value, '42'); // Doubled by <x-double>
      });
      // ...
    });

  });
}
Aside from the funkiness of the async() callback in setUp(), the tests themselves are straight-forward. In the test above, I ensure that the <x-double> example element places the doubled start value on the value property. This all works in the browser, so now I need a script that can be used on a continuous integration server.

I start by running some static type analysis with test/run.sh:
#!/bin/bash

# Static type analysis
results=$(dartanalyzer lib/a_form_input.dart 2>&1)
echo "$results"
if [[ "$results" != *"No issues found"* ]]
then
    exit 1
fi
echo "Looks good!"
echo
That works fine, so now it is time for the actual tests. I start those with the standard content_shell:
#!/bin/bash

# Static type analysis....

# Run a set of Dart Unit tests
results=$(content_shell --dump-render-tree test/index.html)
echo -e "$results"

# check to see if DumpRenderTree tests
# fails, since it always returns 0
if [[ "$results" == *"Some tests failed"* ]]
then
    exit 1
fi

if [[ "$results" == *"Exception: "* ]]
then
    exit 1
fi
And here is where I run into problems. All of the tests pass when run from pub serve, but with content_shell, I get:
CONSOLE MESSAGE: warning: polymer.html not found.
CONSOLE WARNING: line 213: Error:

www.polymer-project.org.

CONSOLE WARNING: line 213: FAIL
CONSOLE ERROR: Exception: Bad state: polymer.js must be loaded before polymer.dart, please add <link rel="import" href="packages/polymer/polymer.html"> to your <head> before any Dart scripts. Alternatively you can get a different version of polymer.js by following the instructions at http://www.polymer-project.org

Since I am mixing the solution in this package into another class, I seem to be running afoul of resource loading limitations and the lack of transformers being run on local resources. So I have to borrow my solution for testing ajax resources with a pub server in the test:
#!/bin/bash

# Static type analysis....

pub serve &
pub_pid=$!

# Wait for server to build elements and spin up...
sleep 15

# Run a set of Dart Unit tests
results=$(content_shell --dump-render-tree http://localhost:8080)
echo -e "$results"

kill $pub_pid

# check to see if DumpRenderTree tests fail...
That is less than ideal, but it does the trick:
$ /test/run.sh
Analyzing [lib/a_form_input.dart]...
No issues found
Looks good!

Loading source assets...
Loading polymer transformers...
Serving a-form-input test on http://localhost:8080
Build completed successfully
...
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> has a shadowRoot
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - value property is updated when internal state changes
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - value attribute is updated when internal state changes
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - form value attribute is updated when internal state changes
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - containing form includes input with supplied name attribute
CONSOLE MESSAGE: PASS: AFormInputMixin mixed into <x-double> acts like <input> - setting the name property updates the name attribute
CONSOLE MESSAGE:
CONSOLE MESSAGE: All 6 tests passed.
CONSOLE MESSAGE: unittest-suite-success
CONSOLE WARNING: line 213: PASS
It might be nice for this to work without the server, so I may take some time tomorrow to explore that a bit. Regardless, I have a solution for testing Polymer.dart base and mixin classes.


Day #18