Recent RSpec Configuration Warnings and Errors

Myron Marston

Nov 4, 2011

RSpec 2.6 introduced a deprecation warning when using RSpec.configure { ... } after defining an example group. In RSpec 2.7, this warning was removed, and now an error is raised when particular configuration settings (expect_with and mock_with) are set after defining an example group.

Recently, there have been comments and complaints on these changes from several different people on twitter.

I’m the one who made those changes to RSpec, and people are rightfully annoyed with these changes…but there’s a lot more to the story. I’m not sure what the right solution is to the problem I was trying to solve by making those changes, but I’m hoping that by blogging about it, we can get some good ideas from the community.

RSpec 2.0

One of the primary goals of RSpec 2 was to decouple the spec runner (rspec-core) from the mocking framework (rspec-mocks) and the expectation framework (rspec-expectations). Besides the fact that decoupling is A Good Thing™, this separation opened up new possibilities for people to pick and choose which parts of RSpec they want to use.

In particular, it allows people to use rspec-core to define and run their tests using RSpec’s example definition DSL (describe, it, before, let, etc) while sticking with the assert_foo assertion methods from Test::Unit or minitest, rather than using RSpec’s object.should whatever syntax. For better or worse, some people really dislike rspec-expectations but love the runner.

RSpec 2 allows you to configure which you want to use:

# spec_helper.rb
RSpec.configure do |config|
  config.expect_with :stdlib
end

# or

RSpec.configure do |config|
  # not strictly necessary; this is the default config anyway
  config.expect_with :rspec
end

Both rspec-expectations and the standard library assertions are available as modules–RSpec::Matchers and Test::Unit::Assertions, respectively. RSpec 2.0 to 2.5 included the appropriate module into RSpec::Core::ExampleGroup just prior to running the examples (that is, after all of them have been defined) to allow people to configure this whenever they want.

Unfortunately, this triggered an unfortunate bug in ruby 1.9…which I’ll get to below.

Infinite Recursion Issues

Shortly after RSpec 2 was released, we began to get some intermittent reports that users were occasionally getting a SystemStackError from RSpec, indicating infinite recursion was occurring. I myself saw this error when working on the rspec-core specs at one point.

The recursion always happened in rspec-expectations’ method_missing hook. In particular, the call to super triggered infinite recursion.

I found this very, very puzzling, and spent many hours troubleshooting trying to figure out why super would infinitely recurse on itself.

It’s a Bug in Ruby 1.9

I eventually managed to boil the problem down to a simple rspec-less example:

# example.rb
module MyModule
  def some_method; super; end
end

class MyBaseClass; end

class MySubClass < MyBaseClass;
  include MyModule
end

# To trigger this bug, we must include the module in the base class after
# the module has already been included in the subclass.  If we move this line
# above the subclass declaration, this bug will not occur.
MyBaseClass.send(:include, MyModule)

MySubClass.new.some_method

If you run this on ruby 1.8, you will (correctly) get a NoMethodError. On ruby 1.9, you get infinite recursion and a SystemStackError. Here’s my short explanation of the conditions that trigger this bug:

Note that this error does not occur if the module is included in the superclass before it is included in the subclass.

How this Bug Manifested Itself in RSpec

Here’s how this bug manifested itself in RSpec:

Fixing the Issue…by Introducing Other Problems :(

To prevent users from getting infinite recursion, we need to prevent RSpec::Matchers (and really, any other module that may use super) from being included in an example group before it is included in RSpec::Core::ExampleGroup. At one point, I considered adding a hook to either RSpec::Core::ExampleGroup or RSpec::Matchers to detect when a user is including it, and somehow prevent or warn them. However, I quickly realized that the extreme flexibility of Ruby’s module system makes this very complicated. A user may not be including RSpec::Matchers directly; they may be including a module from some library or plugin, that itself includes RSpec::Matchers, or includes a module that includes RSpec::Matchers. I realized this wasn’t going to be a simple solution to get right.

Instead, after talking with David Chelimsky and some other members of the RSpec core team, we decided it was best to change when RSpec::Matchers gets included. By including RSpec::Matchers in RSpec::Core::ExampleGroup before it has been subclassed the issue goes away entirely.

We still wanted to let people configure expect_with :stdlib, though. the best solution we could come up with is to delay the inclusion of the expectation module until the moment when RSpec::Core::ExampleGroup is being subclassed for the first time. This gives the user a chance to configure expect_with :stdlib as long as they do so before defining any example groups. Once an example group has been defined, we automatically default to expect_with :rspec–which means that any future expect_with configuration would be effectively ignored.

Here’s the commit, if you’re interested. You’ll notice that it also changes when the mocking framework adapter module gets included. Since we don’t want to deal with this issue again if/when one of the mocking framework adapter modules uses super, I thought it best to change it as well.

This whole issue suggested to me that it’s problematic to allow users to configure RSpec after defining examples. In my mind, since the configuration affects how RSpec works, it’s best to set it before you start to use RSpec (i.e. by defining examples). I made a change to cause RSpec to print a deprecation warning when you use RSpec.configure after defining an example group.

In retrospect, I should have realized that this would affect a lot of users. At the time, it didn’t occur to me that it would. I do all of my RSpec configuration before defining any example groups and I assumed that’s what everyone else did, too. The deprecation warning was mostly just put there on the off chance that someone might configure RSpec after defining an example.

RSpec 2.6

RSpec 2.6 was released with these changes and we immediately began getting questions and complaints about this warning. People like Gary Bernhardt and Corey Haines have a technique of speeding up their tests by loading as little as possible–and this usually involves not loading spec_helper from each spec file. This can trigger the deprecation warning when one spec file (say, spec/lib/my_class_spec.rb) does not require spec_helper, but another file in the same suite (say spec/models/user_spec.rb) does. If spec/lib/my_class_spec.rb is loaded before spec/models/user_spec.rb (which usually happens–they tend to get loaded in alphabetical order), it will trigger the warning since examples are defined in spec/lib/my_class_spec.rb before the configuration happens in spec_helper.rb.

I’m a big fan of the “don’t load spec_helper” approach now, but at the time I made the changes, I had never heard or thought of doing it that way.

We had a conversation about this on an rspec-rails issue. The best suggestion to come out of that (and one that no one argued against) was to remove the warning, and instead raise an error on the specific, problematic configs (expect_with and mock_with), if they get set after defining an example group.

I made this change and it was released in RSpec 2.7.

RSpec 2.7

After RSpec 2.7 came out, there were yet more complaints about this change. Now some users were unable to run their specs because of the error. Effectively, this had made the problem worse–the previous warning could be ignored, but not the error.

I committed a change a few days ago that should improve things here: instead of raising an error anytime expect_with or mock_with are called after an example group is defined, the error is only raised if the method call is changing the setting. No error will be raised if you’re simply explicitly setting the default (i.e. mock_with :rspec) or re-setting the existing config value.

This is certainly an important, needed change that I simply didn’t consider when I made the previous changes. I apologize.

Note that this does not remove the error entirely: if you are configuring RSpec to use a different expectation/assertion framework or mocking framework, this must still be done before an example group is defined so that RSpec can include the appropriate module in RSpec::Core::ExampleGroup before it has been subclassed.

Avoiding these warnings/errors

If you’re on RSpec 2.6 or 2.7 and have gotten these warnings or errors, there are some very simple changes you can make to avoid them.

First, make sure all of your spec_helper requires are simply require 'spec_helper'. If you use a path relative to __FILE__, as people often do, spec_helper.rb can be loaded multiple times (since ruby will happily re-require a file if it is specified as a different file path). RSpec puts the spec directoy on the load path for you, so you can (and generally should) just require spec_helper.

Second, if you follow the “don’t load spec_helper” approach, and you need to configure either expect_with or mock_with, you’ll need to create a secondary helper file for your isolated, fast specs.

Here’s one way to do it:

# spec/fast_spec_helper.rb
require 'rspec'

RSpec.configure do |c|
  c.mock_with :mocha
end

# spec/spec_helper.rb
require 'fast_spec_helper'

# load rails or whatever to get your full app environment booted

RSpec.configure do |c|
  # other RSpec configuration
end

# spec/lib/my_class_spec.rb
require 'fast_spec_helper'

describe MyClass do
end

# spec/controllers/my_controller_spec.rb
require 'spec_helper'

describe MyController do
end

I know this goes against the “don’t load spec_helper” approach a bit, but the important thing is that the rails/sinatra/whatever environment is not fully loaded in the isolated specs. We’re using fast_spec_helper.rb to only configure the bare minimum–the specific expect_with or mock_with settings that must be set before an example group is defined. The one extra require isn’t going to make a noticeable difference in the speed of your isolated tests.

Of course, if you are just setting mock_with or expect_with to the default (:rspec) then you should just remove that configuration entirely and the error should go away–no need for a separate fast_spec_helper.rb file.

Alternatively, you could use ruby’s -r flag to force a helper file to be loaded before any isolated specs, rather than having to require a helper file from each.

Is There a Better Way to Work Around this Bug?

So there you have it…the full story behind the recent configuration warnings and errors in RSpec. I apologize if this has caused upgrade pain for you. I solved the issues in the best way I could figure out.

If you think I totally screwed up, or if you can think of a better way to deal with the ruby 1.9 bug, please let me know in the comments!

Also, I filed a bug on the ruby issue tracker. Please comment there if you would like to see it fixed.