Pseudorandom Deterministic Cypress Tests

This post is a guide for creating pseudorandom yet deterministic tests in Cypress. Cypress can “test anything that runs in a browser”. As such it is mostly agnostic about the implementation details of your application, such as a front-end framework, or what language and framework the server is running. Therefore, orchestrating your database cleaning and seeding for tests is left entirely up to you.

If you’re familiar with a language-specific unit-testing framework, then you may also be familiar with the concept of a “seed” which may control the test order or “random” features of the language such as random numbers and array sampling. Out of the box, Cypress itself provides nothing of the sort, and this post will show how to build it. I’ll be using Ruby on Rails for the server examples, as that is what I am most comfortable with, but the idea should apply regardless of what framework you are using.

A note on test determinism and randomness

By “pseudorandom”, I mean that test inputs, data, and possibly even test behavior are not hardcoded, but rather determined by an initial seed value. If a test is started with a randomly chosen seed, then certain properties of the test will be random as well. The reason for using a strategy like this is that randomizing certain values will increase test coverage. If the underlying bits of numeric and text data are always slightly different, we can find bugs due to edge cases that we may have failed to consider if we had hardcoded those values and used the same ones for every test run.

Since this may turn up bugs, it is very important that we be able to replay a failing build to understand and fix the bug. This is what is meant by “deterministic”. Although the test runs have randomized properties, we can always use the seed of a particular test run to replay it. Tests should always be deterministic, regardless of whether or not one is using this strategy of randomizing test data. Without it, tests would sometimes pass and sometimes fail, and it would be difficult if not impossible to know why.

Obviously tests cannot (and should not) be fully randomized. Some elements of a test must be explicitly given, as they correspond to the constant properties of the application. Visit a page, see a header, click a button — these are all explicit, non-random instructions. But if a particular part of an application is complex, possibly a form that changes in response to certain values being selected, it can be useful to randomize the test a bit to encompass all the interactions between behaviors that cause dynamic changes to the page. It is up to you to decide where to introduce randomness, and how much to use.

Guide

There are several different ways of preparing the test database with data for tests. Here are the key points of the method shown below:

– Cypress picks a seed at the start of the suite run

– At the beginning of each test, Cypress sends the seed to the server. The server resets the database and seeds a minimum amount of data, such as an account or user.

– Individual tests have the option to seed additional data as needed, backed by Faker and FactoryBot on the server.

Here is the Rails server route to reset the database:

# config/routes.rb
Rails.application.routes.draw do
  # ...
  if Rails.env.test?
    post 'test/reset_db', to: 'test#reset_db'
  end
end

# app/controllers/test_controller.rb
class TestController < ApplicationController
  before_action -> { fail unless Rails.env.test? }

  def reset_db
    # TODO: Wipe database, clear cache, etc.
    seed = params[:seed]
    # Set Ruby's PRNG to make its core functions deterministic *on this seed*
    Kernel.srand(seed)
    # Ensure Faker is deterministic on the same seed
    Faker::Config.random = Random.new(seed)
    Faker::UniqueGenerator.clear

    # TODO: Seed data necessary for all test runs, e.g. account, user, etc.
    render plain: "Database reset", status: :ok
  end
end

Now that we have created the route, we need to configure Cypress to use it.

// cypress/plugins/index.js
// Use the seed specified in the environment for replayability. Otherwise generate a random seed.
// TODO: implement `getRandomSeed`. JavaScript does not have a native function to generate random integers.
const seed = process.env.SEED ? parseInt(process.env.SEED) : getRandomSeed();

on("task", {
  async reset_db() {
    const params = { seed: seed };
    const { data } = await axios.post(`${testServerUrl}/test/reset_db`, params);
    return data;
  },
  getSeed() {
    return seed;
  }
});

// cypress/support/index.js
// At the beginning of the test suite, log the seed to the console. If we need to replay a test run,
// we can put this seed in the env, and it will be picked up when `seed` is initialized.

before(() => {
  cy.task("getSeed").then(seed => {
    console.log(`[seed ${seed}]`);
  });
});

// Before each test, reset the database and set the seed.

beforeEach(() => {
  cy.task("reset_db");
});

Our test suite is now set up such that any randomness introduced in our tests (specifically, data created by the server) will be replayable.

This works well for the initial database reset and seeding. Given this user factory:

FactoryBot.define do
  factory :user do
    sequence(:username) { |n| "user_#{n}" }
    email { "#{username}@test.invalid" }
    first_name { Faker::Name.first_name }
    last_name { Faker::Name.last_name }
    password { 'password' }
  end
end

If we add FactoryBot.create(:user) to our reset_db route, then each test will be seeded with a user with username user_1. Each test, the user will have a different name. If that name causes issues in the UI (e.g. the name overflows its container and can’t be fully read, failing the Cypress test), then we can replay the test using the seed for that run, and find out exactly what to fix.

But what about when we need to seed different data for individual tests? For that we’ll create a custom Cypress command to hit a server route to create records on the fly.

# config/routes.rb
Rails.application.routes.draw do
  # ...
  if Rails.env.test?
    # ...
    post 'test/seed_record', to: 'test#seed_record'
  end
end

# app/controllers/test_controller.rb
class TestController < ApplicationController
# ...

  def seed_record
    attrs = params[:attributes]&.to_unsafe_h || {}
    record = FactoryBot.create(params[:factory], attrs)
    render json: { id: record.id, record: record }, status: :ok
  end
end

// cypress/support/commands.js

Cypress.Commands.add("seedRecord", (factory, attributes = {}) => {
  let body = { factory, attributes };
  return cy.request({
    method: "POST",
    url: `${testServerUrl}/test/seed_record`,
    body: body,
    form: true
  });
});

Now we have the ability to create pseudorandom data per test, or for every test in a block:

describe("Address edit page", () => {
  beforeEach(() => cy.seedRecord("address", { user_id: 1 }));
  // ...
});

Here is another take on that same pattern that simply returns the record attributes without creating the record, in case the test itself should create the record.

# config/routes.rb

Rails.application.routes.draw do
  # ...
  if Rails.env.test?
    # ...
    post 'test/get_attributes', to: 'test#get_attributes'
  end
end

# app/controllers/test_controller.rb
class TestController < ApplicationController
  # ...

  def get_attributes
    attrs = params[:attributes]&.to_unsafe_h || {}
    attributes = FactoryBot.attributes_for(params[:factory], attrs)
    render json: { attributes: attributes }, status: :ok
  end
end

// cypress/support/commands.js
Cypress.Commands.add("getAttributes", (factory, attributes = {}) => {
  let body = { factory, attributes };
  return cy.request({
    method: "POST",
    url: `${testServerUrl}/test/get_attributes`,
    body: body,
    form: true
  });
});

An example test using the getAttributes command to construct pseudorandom payment details that we can use to create a payment through the UI during the test:

describe("Payments page", () => {
  it("Accepts payment", () => {
    cy.getAttributes("payment").then(resp => {
      const payment = resp.body.attributes;
      cy.visit("/billing");
      cy.contains('a', 'Make a payment');
      cy.contains('a', payment.payment_type).click();
      cy.get("#amount").type(payment.amount_cents);
      cy.click('Submit');
      cy.contains(".toast", "Payment successful")
    });
  });
});

To reduce the amount of randomness in this test case, specify attributes for FactoryBot. Aside from testing particular paths from the get-go, this is useful for creating regression tests when your randomized tests reveal a bug. Since the conditions for the bug are not always present, hardcode the conditions into another test case:

describe("Payments page", () => {
  it("Does not allow negative payments", () => {
    cy.getAttributes("payment", {amount_cents: -1}).then(resp => {
      const payment = resp.body.attributes;
      cy.visit("/billing");
      cy.contains('a', 'Make a payment').click();
      cy.contains('a', payment.payment_type).click();
      cy.get("#amount").type(payment.amount_cents);
      cy.click('Submit');
      cy.contains(".toast", "Payment amount cannot be negative")
    });
  });
});

I’ll close with a description of one of the bugs I have seen surfaced by this setup. The application uses Rails on the back-end, and Vue.js on the front-end. For handling and formatting money, it uses the money Ruby gem on the back-end, and dinero.js on the front. We use dinero.js mainly for formatting money in Vue, but also in Cypress to calculate sums used in test expectations. A test failed because the expectation was one cent off from the value it was checking.

The source of the issue was, at the time, entirely unexpected. The calculation on the back-end was using normal rounding rules (“half up”) while the front-end was using banker’s rounding (“half even”). This resulted in a discrepancy that only happened 25% of the time when rounding exactly half a cent. Money calculations were isolated to the back-end, so the bug did no damage, but I am still glad we found it. If we weren’t running pseudorandom deterministic tests, it would have likely taken much longer to find.