ENV variables are a major cause of test failures on ci

This is part of the Semicolon&Sons Code Diary - consisting of lessons learned on the job. You're in the testing category.

Last Updated: 2024-03-02

All my JavaScript tests passed locally, but on my CI server, the cookie notice test failed, as if cookies had been accepted when in fact they hadn't been.

Here was my code.

export function CookieNotice() {
  // railsEnv() is defined below
  const showCookieNotice = railsEnv() == "test"

  if (showCookieNotice) {
    attachCookieNotice()
  } else {
    simulateAccepting()
  }
}

export function railsEnv() {
  return process.env.RAILS_ENV
}

Looking at it this way makes the problem easy to diagnose, but the reality was that I did not expect RAILS_ENV to be set for during my jest tests (since I never explicitly set any ENV variables related to jest on my local machine). However this variable was set due to a general configuration on my CircleCI and production server, leading to divergences.

The moral of the story, probably, is that switches based on ENV variables are a common source of confusion, and ought to be used with care. Perhaps the dependency could have been documented or logged to highlighted during test runs.