mirror of
https://github.com/google/nomulus.git
synced 2025-05-17 01:47:14 +02:00
a7d099d2a3
This change actually enabled the screenshot comparison in the visual regression tests. We used Docker to provision Chrome and ChromeDriver to eliminate the discrepancy of environment between local development and Travis CI. ------------- Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=237811918
70 lines
3.9 KiB
Markdown
70 lines
3.9 KiB
Markdown
# Webdriver Tests
|
|
|
|
This is the home of the Nomulus webdriver tests. These tests run our user-facing
|
|
HTML/JavaScript code in the context of a set of common browsers. See the[Webdriver documentation](https://www.seleniumhq.org/docs/03_webdriver.jsp)
|
|
for more information on using webdriver.
|
|
|
|
## My Screenshot Tests are Failing!
|
|
|
|
1. Make sure [Docker](https://www.docker.com/) is installed
|
|
* Docker is used to provision the browser and the Webdriver service, so please
|
|
make sure Docker is installed and the `docker` command is availbe in the shell.
|
|
2. Missing golden images
|
|
* If you added a new test using screenshot comparison, you have to generate
|
|
the golden image for that test in advance and copy it to
|
|
[goldens/](https://github.com/google/nomulus/tree/master/javatests/google/registry/webdriver/goldens)
|
|
folder. There
|
|
is an auxiliary Gradle build task to help with this, and here are some examples:
|
|
```shell
|
|
# Generate golden images for all tests. All generated images are stored in
|
|
# ${projectRoot}/gradle/core/build/new_golden_images
|
|
$ ./gradlew :core:generateGoldenImages
|
|
|
|
# Generate the golden image for a certain test
|
|
$ ./gradlew :core:generateGoldenImages \
|
|
--tests "google.registry.webdriver.OteSetupConsoleScreenshotTest.get_owner_fails"
|
|
```-webkit-transition
|
|
3. Screenshot differs by X pixels
|
|
* If you made any change to the existing test and expected to affect the web page,
|
|
the screenshot should be different from the previously generated golden image.
|
|
In this case, you can just use the screenshot as the new golden image.
|
|
* If you didn't expect to have any change, you have to use your own justification
|
|
to mitigate it. For example, if the number of different pixels is very small,
|
|
like 1 piexel, it may be because the test is flaky and a retry can probably solve
|
|
it; also, if there is any change to the version of browser or Webdriver library,
|
|
it could affect the screenshot as well.
|
|
|
|
## My Screenshot Tests are Timing Out!
|
|
|
|
The screenshot tests display your HTML and generated Javascript in an actual
|
|
browser. There can be multiple HTTP queries sent to the browser and the tests
|
|
need to know when everything is finished loading before taking a screenshot.
|
|
This is normally done by waiting for the browser page to contain a specific HTML
|
|
element that isn't present until the end of the load and render. When the
|
|
element shows up, it's safe to take a screenshot.
|
|
|
|
If that element doesn't show up, the tests will eventually time out. This could
|
|
be due to a difference that you've introduced but more likely it is due to a bug
|
|
in your javascript. Unfortunately, Javascript errors aren't visible from the
|
|
screenshot tests. This can make these problems very difficult to debug.
|
|
|
|
Compounding this problem is the fact that we do have unit tests, but they run
|
|
against the uncompiled javascript. The screenshot tests run against the closure
|
|
compiled javascript, which means that they are susceptible to at least one new
|
|
class of errors due to the name-mangling that happens during a compile. One
|
|
thing to consider is that name-mangling breaks the use of object attributes
|
|
accessed via strings: e.g. "foo['bar']" is a different attribute from "foo.bar"
|
|
after name mangling - the second case gets mangled, the first does not.
|
|
|
|
In any case, the following tools can be helpful in debugging these kinds of
|
|
problems:
|
|
|
|
- The "webm" files. Along with the pngs in the output files, there are a set
|
|
of .webm files. These are movies taking of the browser during the test, and
|
|
will show all intermediate steps up to the timeout. Useful information can
|
|
be gleaned from them.
|
|
- Alerts. Javascript alerts (generated by the plain old "`alert()`" function)
|
|
show up right in the logfile, which is directly available from the shell.
|
|
You can instrument your code with alerts, but keep in mind that the driver
|
|
doesn't respond to alerts, so the first one you hit is the only one you'll
|
|
see.
|