Dealing with intermittent failures on CI

DCE uses a Continous Integration (CI) service to run specs when pushing to GitHub. CI is a unique environment that may have slight differences from production environments or development machines. You may encounter intermittent failures when specs run under CI that you don’t run into on your local machine.

If you have a spec that is passing locally, but sometimes failing on Travis you can use a strategy that Chris Colvard came up with on the Avalon project. The original Avalon commit for this functionality is here.

You add this module to spec/support:

module OptionalExample
  RSpec.configure do |config|
    config.after do |example|
      if example.metadata[:optional] && (RSpec::Core::Pending::PendingExampleFixedError === example.display_exception)
        ex = example.display_exception
        example.display_exception = nil
        example.execution_result.pending_exception = ex
      end
    end
  end

  def optional(message)
    RSpec.current_example.metadata[:optional] = true
    pending(message)
  end
end

Then include that module in the rails_helper:

config.include OptionalExample

This allows you to make a test as optional for a specific enviroment. It is similar to pending, but will not fail if the test passes. Here’s how to use it in a spec:

it 'should delete existing tokens' do
 optional "Sometimes fails on travis" if ENV['TRAVIS']
 expect(false).to be_falsey
end