Old Testing Interviews
Back in January 2009, I did a bunch of interviews with prominent Rubyists about their test practices. The interviews vanished when I moved the site to WordPress, but I still get hits from a link to the interviews, and I thought it would be useful to get them all in one place.
Remember, this was 2009, and I’m sure everybody’s habits have changed since then. Other than putting them all together in one post, I haven’t edited these at all.
Noel Rappin
How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?
I started with automated testing very shortly after reading Kent Beck’s original XP book. The XP book came out at a time when I was very receptive to the ideas—I had just come off a project that had suffered from a lot of regressions and all the other kinds of pain that XP promised.
It didn’t take long for me to see how much better my code was when I did a lot of testing… what was more surprising was how much more fun writing code test-first turned out to be. The quick feedback and the ability to clean up code with confidence turned out to be really satisfying.
What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?
When I’m adding a new feature, I tend to start with a skeleton controller test that validates that the controller sets the right variables. I don’t put much code in the controller, so then I move to testing the model for the new functionality. If the view logic seems to require it, then I’ll add view tests after I get the view in place. I go back and forth between test and code pretty quickly, but sometimes the code will get ahead of the tests, especially when doing view layer stuff. I don’t use integration tests very much at the moment.
I seem to have moved back into the core Rails test features recently, although I still use Shoulda for contexts and for the additional assertions. My most recent project used core Rails plus Matchy. I’ve also been using the various factory replacements for fixtures, which I like quite a bit.
What’s the most interesting thing you’ve discovered about testing recently?
The biggest change I’ve made recently is using factories for generating test data, which makes the tests much more readable and stable by keeping the setup closer to the actual test.
Is there a tool you wish you had for testing that you don’t think currently exists?
I wish I had a really good way of validating view logic, none of the ones I’ve tried have been completely satisfying. It’d also be nice to have more sophisticated coverage reports. Of course, these things might actually be impossible…
What advice would you give somebody looking to write more effective tests?
Automated testing is much easier and more valuable if you keep a tight feedback loop between your tests and your code.
Geoffrey Grosenbach
First up on the Testing Practices Interview series is Geoffrey Grosenbach, Senior Visionary of Topfunky, and also the person behind PeepCode. Geoffrey blogs at http://nubyonrails.com/, and is responsible for the gruff graph generator and, most recently, a task tracker based on David Seah’s Online CEO. He was also kind enough to write his responses in Textile, which I heartily endorse. Take it away…
How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?
I remember watching tests run during installation back when I was using Perl (wouldn’t that be nice if RubyGems optionally ran their test suite during installation?). But it seemed like an advanced topic that only some programmers did.
When I started using Ruby and met the testing fanatics at Seattle.rb, I started to understand what it was about and why I might want to write a test.
Three things helped me get started with test-driven development:
- Watching other people do it.
- Writing a graphics library (gruff). I needed to generate a bunch of samples with various data inputs and Test::Unit was a great way to do it.
- Finally, I started out by writing a single test for an existing Rails app whenever I encountered a bug. It gave me peace of mind.
Since then, I’ve appreciated the process of thinking that I get into when I code test-first.
What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?
I’ve tried several libraries and tools.
I started with Test::Unit and still use it on some existing projects.
For a while I used Jay Fields’ Unit-Record style of separating out unit and functional tests for both models and controllers (see also). It also provided a nice test method that took a string and a block of assertions (similar to what’s in Rails now).
I’ve used RSpec’s ability to add should syntax into Test::Unit.
I’m currently happy with straightforward RSpec. I also have some Test::Unit integration tests in my Rails apps but have also used RSpec User Stories and their replacement, Cucumber. Given the fact that I’m both the designer and coder of most of my apps, I don’t get much benefit from Cucumber. But I can see how it would be useful to people working for a semi-technical client.
Honestly, it’s a bit overwhelming with all the options out there!
When I’m actually coding, I’ll use Eric Hodel’s ubiquitous
autotest or rstakeout to run my test suite.
What’s the most interesting thing you’ve discovered about testing recently?
I’ve been working on a small command-line Objective-C app and am experimenting with using Ruby to run a suite against the command-line app to check the inputs and outputs. Ruby is useful that way and works better for me than trying to use Objective-C for the same purpose.
Is there a tool you wish you had for testing, but which you don’t think currently exists?
I can’t think of one. Usually I end up writing a tool if I need it and I can’t find it anywhere, such as test_benchmark to show individual test runtimes for Test::Unit. Someone recently imported it to GitHub as well.
What advice would you give somebody looking to write more effective tests?
Find a mentor. Work with someone else who is doing it. Concentrate on testing the effects of your code, not the way they are implemented. It’s easy to write tests that really don’t do anything and won’t reveal meaningful changes in the code if it stops behaving properly.
Gregg Pollack
Next up in the Testing Practices interview series is Gregg Pollack. Gregg is one of the proprietors of Rails Envy, and is one of the co-hosts of their Rails Envy Podcast, which mentioned Rails Prescriptions today—thanks Gregg!
Gregg is also one of the founding members of the Rails Activists. He’s also done a lot of video production, including some excellent ruby and rails screencasts, and his series condensing various Ruby and Rails conferences. Put it all that way, and he seems kind of busy, actually…
Take it away, Gregg…
How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?
I have to admit, my first 3 Rails projects didn’t contain any tests. I got into testing when RSpec started on the scene. RSpec just made more sense and the documentation on the RSpec website was really useful. Once I figured out how to use RSpec with autotest and growl, testing became much more fun.
I also ended up doing a talk at my local users group about how I came to love testing. It’s a little dated, but still quite relevant:
http://www.railsenvy.com/2007/10/4/how-i-learned-to-love-testing-presentation
What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?
I used to write isolated tests at the model and controller level using RSpec, properly mocking and stubbing at the controller level. Often times integration tests would get ignored.
Currently I’m working on a project using Shoulda for Model tests and Integration tests with Webrat. Yes, we’re not doing controller or view tests. So far I’m really liking the combination, and it seems to cover things pretty well without having to deal with much mocking and stubbing of controllers/view tests which have very little logic anyways.
What’s the most interesting thing you’ve discovered about testing recently?
Webrat rocks for integration tests.
Is there a tool you wish you had for testing, but which you don’t think currently exists?
Hmmm.. James Golick recently blogged about one line tests using a library called Zebra. He also argues that test names are essentially comments, and well.. too many comments are a code smell.
So what’s the alternative?
Creating a DSL that allows us to write tests in a way that doesn’t need names/comments. Shoulda gets us pretty close with all of it’s one line helpers like:
should\_belong\_to :user
should\_have\_many :tags, :through =\> :taggings
should\_require\_unique\_attributes :title
should\_require\_attributes
should\_only\_allow\_numeric\_values\_for :user\_id
None of these tests need comments to figure out what they’re testing! I’d love to figure out a way to do more of this, and the Zebra library James put out is certainly a step in the right direction.
What advice would you give somebody looking to write more effective tests?
Integration tests are probably more important then anything else, and using a library like Webrat makes them very easy to do.
If you’re working with a team of people and you want to ensure you’re building a solid test library then a Continuous Integration server is imperative. Set it to run all your tests every time something is checked in and if it fails to email everyone. Make a rule that whoever checks in code that causes tests to fail has to buy everyone a round of beer, or has to perform some humiliating task. Checked in code that fails should be fixed immediately.
Ryan Bates
Next up on the testing interviews is Ryan Bates. Ryan runs Railscasts, a weekly screencast on a new Rails topic that is simply one of the best ongoing sources for tutorials about Rails. Seriously, if you aren’t familiar with it, drop everything and prepare to spend some time watching his videos. Ryan has also done two screencast series for Pragmatic, Everyday Active Record and Mastering Rails Forms, both of which are available at pragmatic.tv.
On a related topic, I didn’t introduce myself to Ryan at last year’s RailsConf, even though I walked past him a few times. This is because every time I walked by him, people were coming up to him and thanking for all the great Railscasts. So, thanks Ryan.
How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?
About 6 years ago I read the book Extreme Programming by Kent Beck. This sparked my interest in testing (specifically TDD), but I could not find many practical examples of the practice. I spent some time researching the topic but honestly did not “get” it until Rails came along. Rails provided practical testing patterns which were fairly easy to follow.
I find the most difficult part of testing is coming up with a pattern which works well for a specific situation. Once that is done, adding similar tests becomes very easy, and that is where it starts to really pay off.
What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?
I primarily use RSpec for testing, however my tests are quite different than what they recommend. For one thing, I use controller tests like functional and integration tests. That is, they execute the entire stack (including models and views). I don’t test views exclusively beyond this because I find that requires too many mocks and leads to brittle tests.
I test models and helpers exclusively (like unit tests). That is, I test each method on its own to ensure it functions properly. My theory is, the more deeper a piece of code is, then the more it is used by various parts of the application, and therefore should have better test coverage.
Fixtures are kind of interesting. I don’t use them at all in unit tests, instead I prefer factories (factory_girl) for this. However I do use fixtures in controller tests as filler data to help catch errors. Each fixture usually does not have more than two records, and I often leave them with their default generated content.
What’s the most interesting thing you’ve discovered about testing recently?
I just recently discovered the RR mocking framework and I like its syntax more than Mocha. However I haven’t moved many projects over to it yet, so I can’t say how well it works in real world use.
Is there a tool you wish you had for testing, but which you don’t think currently exists?
Perhaps a tool for testing private methods and accessing instance variables in a clean way. I know in theory tests shouldn’t need to do this, but I would still find it useful. There may already be one out there, I haven’t looked much.
Beyond this I would love to see more documentation, examples and experiments done with regard to testing. I still feel it is a fairly unfamiliar territory in the programming world, and everyone seems to have their own way of doing things.
What advice would you give somebody looking to write more effective tests?
Be careful with mocks and stubs. They are often an easy, immediate solution but can lead to brittle or deceiving tests. Only use them if you can find no other cleaner way to test something.
Overall, don’t give up on testing if you don’t grasp it right away. Try a technique for a while, if it doesn’t work with your flow, try something else. Don’t feel bad if you find testing is hard – it really is. But it is so worth it.
James Golick
On to the interivew. James Golick is the founder of GiraffeSoft, a boutique Rails consulting firm out of Montreal. He also maintains the James on Software blog. Within the Rails community, he’s the author of resource_controller, a common parent for RESTful controllers, and active_presenter, which allows you to create presenter objects out of aggregations of ActiveRecord models.
Most recently, James has created zebra, a test library for the quick creation of single line tests.
How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?
I worked at a few really cheaply run companies, where I was the one man technology department. I didn’t sleep much in those days.
In those days, I was always desperately looking for ways to get better at what I was doing, if only to get the occasional extra hour of sleep. At one point, I came accross an article about automated testing and TDD, and it all just made so much sense to me. When you’re that strapped for time, automation is a necessity. Once I was introduced to the idea that tests could be automated, too, I jumped all over it.
What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?
At giraffesoft, we’re very serious about TDD. So, naturally, everything we do is test-first.
Currently, we write extensive unit tests and functional tests. We’re slowly adding cucumber to the mix. I’m definitely sold on the benefits of acceptance test driven development. So, that’s where we’re going.
We use Shoulda, largely because of its macros. Duplication in tests becomes incredibly tedious. So, expressing certain kinds of repetitive tests as one-liners is a huge win.
However, as I mentioned in the release announcement for zebra, we’ve started to feel the burn with Shoulda’s macros in certain situations. Often, test failures result in completely useless backtraces and the string programming catches up to you after a while.
So, we’re currently moving towards a context & zebra stack to replace Shoulda.
What’s the most interesting thing you’ve discovered about testing recently?
Using Jay Fields’ expectations testing framework completely changed the way that I approach unit tests. Not having to describe each test in english is incredibly liberating.
If your code is readable, you shouldn’t need to document it at a micro level, except in special cases. Your code probably isn’t littered with comments describing every couple of lines. So, why should your tests be?
Is there a tool you wish you had for testing, but which you don’t think currently exists?
If you’d asked me 2 weeks ago, I’d have replied, something that would allow me to write more expectations-like tests in my every day hacking. That’s what zebra is. So, now I’m feeling pretty happy about my toolset and where things are headed.
What advice would you give somebody looking to write more effective tests?
Be pragmatic. I get a lot of questions about the “proper” way to mock or stub something, for example. Stop worrying about getting things “right” and try to make judgements that get you the best possible test, while striking a balance between productivity and the longevity of the test.
Additionally, I’d really encourage people to use expectations for a small project – a rails plugin or something. For various reasons, you probably won’t want to use it for every day stuff, but you might learn a lot from giving it a serious look.
Chad Fowler
I’m very excited to have Chad Fowler as the latest participant in the Testing Practice interview series.
Chad Fowler is an internationally known software developer, trainer, manager, speaker, and musician. Over the past decade he has worked with some of the world’s largest companies and most admired software developers. He loves to program computers and, as part of his role as CTO of InfoEther, Inc., spends much of his time solving hard problems for customers in the Ruby language. He is co-organizer of RubyConf, and RailsConf and author or co-author of a number of popular software books.
And so, let’s hear from Chad…
How did you get into writing tests regularly? Did you have a specific moment when you realized automated testing was valuable?
For me it was when I was exposed to TDD for the first time. I had been practicing the typical “guru reads the output” style of testing, primarily in Java at the time. That means every class typically had a main() method at the bottom which would exercise my code and print a bunch of hardly decipherable junk to the screen. As I was developing, I knew what that junk meant. Nobody else ever did. Two days later, neither did I. But the main() methods remained. Because, hey, those were the tests.
I think the way most people worked in an environment like that is that if they needed to change anything in the production code, they would hack the existing “tests” in the main() method such that they could understand what they meant (since this stuff was embedded in production code, you didn’t typically venture out of main() for fear of polluting the namespace). So it was more of a scratchpad than a test.
At the same time, I was doing a lot of mentoring of junior developers on how to do object oriented programming. Specifically, I was trying to help a large group of developers stop generating unmaintainable spaghetti Java. I started using a technique that I call Error Driven Development. It was very much like TDD, which I had not yet heard of. You start in the area of the application you’re trying to implement. Maybe it’s a controller in an MVC setup, for example. And you type in the code you wish already existed as an API. You express your intent as clearly and succinctly as possible within the scope of what would be technically possible in this magical but not yet existing API (paying attention to which data needs to be passed around as parameters, which imaginary objects would best play the role of owning which function, etc.). Then you try to compile and/or run your code and follow the error stream until everything works.
With this kind of development, you get to always work at the level of abstraction you’re interested in while you’re developing. When I’m in a controller, I don’t care about SQL. When I’m in a view, I don’t care how business logic is implemented. To make everything run, of course, I have to walk down through the layers of abstraction and repeat this process—-imagining the perfect API and just using it. Eventually I’m done. It’s motivating and makes better code.
So, the question was about testing. This Error Driven Development technique was just a lame version of how good TDD works. TDD gives you the extra advantage of a nice framework with assertions and reports. So the first time I saw TDD, I was hooked. It was a better version of what I’d been working toward in my quest to write and help others write better code. It just happened to also create tests as a side effect.
What is your Rails testing process? What kinds of tests do you write, and what tools do you use to write them?
I use whatever testing framework and tools the team I’m working with uses. I’ve done everything from out-of-the-box test/unit to RSpec. Given a choice these days, I’d probably depend on Shoulda and Mocha. RR is really interesting as well but I haven’t fully switched to it. I suspect I might replace Mocha with it at some point in the near future.
I typically start by writing model tests. They’re called unit tests in Rails, but they’re rarely isolated unit tests. Ideally, as much logic as possible lives in the models, so I spend a lot of time here. By far more than anywhere else. I do everything test-driven unless I’m hacking something together quickly. Even then, I usually reach a point where I wish I was doing things test-driven and switch.
Ideally, your controllers are going to be tiny. They’re also likely to be composed of calls to objects and methods from your model layer. So, though I do functional (controller) tests, I try to minimize the need for them. If you’re already testing a method in a model, you don’t need to duplicate that test in a controller if the controller is simply calling the model. You do need to make sure the controller is doing its job well. That’s where things like mocks come in. I use mocks not so much to avoid calling code in the dependencies an object has (such as the classic credit card processor example) but more to allow me to specify a process in terms of what the code I’m testing is supposed to do vs. how it does it.
Ultimately, it’s a good idea for someone to actually write real automated tests. These aren’t the ones I do unless I’m playing the role of tester. For example, Selenium is an excellent tool for really testing a Rails app. It’s great to have automated tests at that level that run with a continuous build and so on. I don’t usually actually do that so it’s not part of my process per se, but I’d always advocate that it be part of a team’s process. I don’t get much value out of trying to do selenium-first development, though, for example.
What’s the most interesting thing you’ve discovered about testing recently?
I wouldn’t call it a discovery, but I’m starting to change a long held opinion about the value of tests. I used to think of them as executable requirements specifications. I know a lot of people do. I’m from the school of TDD that blossomed into what’s now called BDD. BDD people even tend to use the word “specify” when they talk about tests they write up front.
That always sounded like a great idea. Requirements docs you can execute for validity. In my experience it rarely works out that way, though. Tests don’t end up being readable like english. RSpec and especially Cucumber are a step in that direction, but I’m starting to believe that ultimately developer tests should read like good code. Maybe a little different from normal good code, but they are code after all. And they’re not for validating the production code you write. They’re for motivating the production code you write.
So maybe the goals of executable requirements documentation and motivational specs are at odds with each other. And if the tests do the job of driving you to make well designed code but don’t really read like requirements documentation afterward, that’s nothing to feel bad about.
Maybe it’s even OK to throw away the tests eventually in the same way we used to throw away the code in our main()-method scratch pads.
I’m pretty sure I’m exaggerating, but that’s the general idea.
Is there a tool you wish you had for testing, but which you don’t think currently exists?
Not really. Marcel Molina and I used to talk about how we longed for something that would sit in the middle of our objects and their method invocations, such that you could set up how you expect an object to behave and then verify, without overriding the method’s behaviors, that the object did the things you asked. Like a partial mock which doesn’t change the method implementations it’s setting expectations for.
RR now does this with its proxy implementation. Now that it exists, though I like the feature, it doesn’t feel like it filled a big hole. Go figure.
What advice would you give somebody looking to write more effective tests?
Assume that writing more effective tests means writing better code. Developer testing (which is what I do) isn’t primarily about verifying code. It’s about making great code. If you can’t test something, it might be your testing skills failing you but it’s probably your code code’s design. Testable code is almost always better code. Code written as a result of a series of failing tests is very likely (by definition) testable. (An example of a design choice in Ruby that might result from this sort of approach is to use more mixins and less inheritance.)
That being said, it’s easy to fall into a trap when you start coding this way. I often come across heavily TDD’d code with a huge focus on some trivial but easily tested feature. For example, you might start writing tests for the Ruby comparison method (<=>) or to_s on an object, simply because you know how to write tests for it. I’ve seen programmers spend a disproportionate amount of time testing and implementing ancillary features because they get the TDD (or BDD) bug and find a comfort zone to hang out in.
Instead, always focus on testing the core of your domain. Other tests are nice to have, but when you focus on the core of your domain (the features that define the product you’re implementing), you drive that domain model forward and avoid spinning your wheels on testing for the sake of testing. Kent Beck used to say “test everything that could possibly break”. That’s good advice, but I’d add “and actually matters”.