google-nomulus/javatests/google/registry/webdriver

Webdriver Tests

This is the home of the Nomulus webdriver tests. These tests run our user-facing HTML/JavaScript code in the context of a set of common browsers. See theWebdriver documentation for more information on using webdriver.

My Screenshot Tests are Failing!

1. Make sure Docker is installed

• Docker is used to provision the browser and the Webdriver service, so please make sure Docker is installed and the docker command is availbe in the shell.

2. Missing golden images

• If you added a new test using screenshot comparison, you have to generate the golden image for that test in advance and copy it to goldens/ folder. There is an auxiliary Gradle build task to help with this, and here are some examples:

  # Generate golden images for all tests. All generated images are stored in
  # ${projectRoot}/gradle/core/build/new_golden_images
  $ ./gradlew :core:generateGoldenImages
  
  # Generate the golden image for a certain test
  $ ./gradlew :core:generateGoldenImages \
    --tests "google.registry.webdriver.OteSetupConsoleScreenshotTest.get_owner_fails"
  ```-webkit-transition
  

3. Screenshot differs by X pixels

• If you made any change to the existing test and expected to affect the web page, the screenshot should be different from the previously generated golden image. In this case, you can just use the screenshot as the new golden image.
• If you didn't expect to have any change, you have to use your own justification to mitigate it. For example, if the number of different pixels is very small, like 1 piexel, it may be because the test is flaky and a retry can probably solve it; also, if there is any change to the version of browser or Webdriver library, it could affect the screenshot as well.

My Screenshot Tests are Timing Out!

The screenshot tests display your HTML and generated Javascript in an actual browser. There can be multiple HTTP queries sent to the browser and the tests need to know when everything is finished loading before taking a screenshot. This is normally done by waiting for the browser page to contain a specific HTML element that isn't present until the end of the load and render. When the element shows up, it's safe to take a screenshot.

If that element doesn't show up, the tests will eventually time out. This could be due to a difference that you've introduced but more likely it is due to a bug in your javascript. Unfortunately, Javascript errors aren't visible from the screenshot tests. This can make these problems very difficult to debug.

Compounding this problem is the fact that we do have unit tests, but they run against the uncompiled javascript. The screenshot tests run against the closure compiled javascript, which means that they are susceptible to at least one new class of errors due to the name-mangling that happens during a compile. One thing to consider is that name-mangling breaks the use of object attributes accessed via strings: e.g. "foo['bar']" is a different attribute from "foo.bar" after name mangling - the second case gets mangled, the first does not.

In any case, the following tools can be helpful in debugging these kinds of problems:

• The "webm" files. Along with the pngs in the output files, there are a set of .webm files. These are movies taking of the browser during the test, and will show all intermediate steps up to the timeout. Useful information can be gleaned from them.
• Alerts. Javascript alerts (generated by the plain old "alert()" function) show up right in the logfile, which is directly available from the shell. You can instrument your code with alerts, but keep in mind that the driver doesn't respond to alerts, so the first one you hit is the only one you'll see.