December 18, 2015

Tom CopelandHTTP verb and response code mismatches

2015/12/18 05:00 AM

Recently I ran across a web app bug that I caused by breaking a route. I had changed a match to a get in my config/routes.rb and I should have changed it to a post instead. So we fixed the bug and backfilled a routing test, etc.

The interesting thing was that we were aided in finding it by checking the logs and noticing a mismatch in the HTTP request verbs and the response codes. In this case, we found requests using the POST verb that were resulting in a 404 response code. That got us looking in the right direction.

If a client is doing a POST to a resource and getting a 404 in return, that means that the client somehow got hold of a resource identifier that is no longer there. Of course, if a client is URL hacking, that's going to happen. But in a normal day to day flow I wouldn't expect to see that. If I do, that might indicate there's either some bug in the page or perhaps a missing abstraction in an API.

The "bug in the page" case might be straightforward. We might have a list of notes on a page, and when a customer deletes one we're not removing it from the page. Then the customer attempts to update the note and gets an error. That might be a PUT or PATCH verb, but same idea.

Another more interesting scenario might be a nested parent resource that can be removed. Suppose we have projects that can have many notes and the note resource is nested under it, so creating a new note involves a POST to a URL like /projects/42/notes. If a project gets deleted while a customer is looking at a page, you could imagine a POST getting a 404 in response. That would be unpleasant, but if it only happens once in a blue moon it might not be a real problem. At the very least it would be a low-priority bug on the list to fix, though.

The API scenario is a little more open-ended. A POST to a URL with a 404 response could indicate a bug in an API client where they're just using the wrong ids. Or it could indicate a client that's just not paying attention to response codes and is iterating through a list. Depending on how much contact you have with your API consumers it might be worthy of a conversation though.

So we've established that a POST that gets a 404 should be an indicator that something's wrong. There are some other combinations that also seem like potential issues:

  • A POST with a 301 response: generally Rails responds with a 302; I checked a somewhat busy Rails app for this combination and found no hits.
  • A POST or a PUT with a 401 response: why is the client not passing the appropriate authentication credentials?
  • A POST or a PUT with a 403 response: why is the client attempting to modify a resource that it doesn't have access to? Is this a hack attempt or just a confused client?
  • Any non-GET with a 404 response seems like an avoidable issue; it's a client error, but is there anything we can do to help the client avoid stumbling into that error?

Can you think of any other combinations of verb / response code mismatches? Let me know!

October 27, 2015

Ryan DavisHappy Birfday to me!

2015/10/27 07:00 PM

I actually missed my Ruby Birfday, but this is my Actual Birfday, so that counts for something, right?

The 15th of October was my Fifteen Year Anniversary with Ruby! Yay!

959 gem releases (up by 136), 10409 commits (up by 604), and bazillions of test runs later, and I’m still going strong. Rawr!

October 20, 2015

Tom CopelandPicking the right data structure

2015/10/20 04:00 AM

Suppose you came across some code like this:

def foo
  teams = ["Yankees", "Cubs"]
  puts "There are #{teams.size} teams"
  puts "Mariners are included? #{teams.include?('Mariners')}"

Probably you would think that it was odd that it was using an Array for teams; order doesn't seem to be important. All the operations are supported by Set which is much more efficient if it does what's needed. So this could be replaced with:

def foo
  teams =["Yankees", "Cubs"])
  puts "There are #{teams.size} teams"
  puts "Mariners are included? #{teams.include?('Mariners')}"

Now we're using constant rather than linear time operations. This doesn't matter here, but if we had a million items in the collection it would make a big difference.

Here's another example of using an Array when another type would work better:

def foo
  Team =, :city)
  teams = []
  teams <<"Yankees", "New York")
  teams <<"Mets", "New York")
  teams <<"Cubs", "Chicago")
  teams <<"White Sox", "Chicago") {|t| == "New York" }

In this case order is important because clearly the Yankees are the best team of the bunch, ever, in the world. So a Set wouldn't work. But we can avoid scanning the array with select by using a Hash with the city name as the key:

def foo
  Team =, :city)
  teams = {
    "New York" => ["Mets", "New York"),"Yankees", "New York")],
    "Chicago" => ["Cubs", "Chicago"),"Cubs", "White Sox")],
  teams["New York"]

We've replaced another linear time operation with a constant time operation.

This is more than a performance enhancement, though. It also communicates something to the next developer who looks at this code. If we use a Set, we're communicating that order doesn't matter, and that we'll never expect to be able to index into this collection. If we use a Hash, we're communicating that we expect lookups to happen a particular way.

So these are worthwhile changes. But in a largish codebase - or even a small one - how do we find these refactoring / performance opportunities? In the case of "replace Array with Set", for a given Array, we need to ensure that we're not using any methods that aren't also available on Set. So if we have an Array that's using any of these methods, we can't swap them out:

irb(main):004:0> -
=> [:*, :[], :[]=, :assoc, :at, :bsearch, :combination, :compact, :compact!, :concat,
:delete_at, :each_index, :fetch, :fill, :index, :insert, :join, :last, :pack, :permutation,
 :pop, :product, :push, :rassoc, :repeated_combination, :repeated_permutation, :reverse,
 :reverse!, :rindex, :rotate, :rotate!, :sample, :shift, :shuffle, :shuffle!, :slice,
 :slice!, :sort!, :sort_by!, :to_ary, :transpose, :uniq, :uniq!, :unshift, :values_at]

It's not quite that simple though. Consider this code:

def foo
  # teams best to worst
  teams = ["Yankees", "Cubs", "Red Sox"]
  teams.each {|t| puts t }

That usage of an Array has an implicit order - best to worst - but we can't tell that by looking at the methods invoked on the object. There aren't any giveaway usages of shuffle or at or whatever.

For a real-world example of this I poked around the sass gem (v3.4.13) that I'm using in a project. I looked at a bunch of different array usages, but for most of them it seemed like they were relying on the array being an ordered sequence, not just a collection. The one instance that seemed like it might be a candidate was the watched_paths array, since only unique directories are desirable there (per the remove_redundant_directories method) and order doesn't appear to be important. But even if so, what are we talking about here - maybe a dozen or so directories? So not a big savings.

When I first started thinking about this I considered writing a runtime utility to watch usage of Array instances and suggest replacing with other types. It'd be something like pippi except focused on type usage, not method call sequences. I'm still noodling on it, but as things stand I think it would result in too many false positives.

Maybe it would be more useful on gems than Rails apps? I feel like there's not a lot of upside there, though. When I think about the gems I use day to day - aasm, declarative_authorization, nokogiri, etc - most of the time I'm using small collections where iterating over a few dozen items just isn't a big deal. Or I'm using some API wrapper client where 99% of the time is spent waiting for data on a socket. The code might be more clear if a specific type were used, but for the most part, no big deal.

In conclusion, I don't think there's a useful tool to be written here. But comments welcome on the twitters.

October 19, 2015

Tom CopelandA tribute to my Dad

2015/10/19 04:00 AM

My Dad, Lee Copeland, died last Tuesday. Folks have been very kind in expressing their sympathies; I really appreciate it. We're going to miss him a lot. Here are some things that I mentioned at the funeral and have been thinking about.

Dad invested in people. There was a big crowd at his funeral, many of whom Dad met 20-30 years ago when they were attending Longwood or Hampden-Sydney college and his church, Faith Bible Fellowship. As I was going through his effects, I was tracking down upcoming appointments, and I had to call students that he was scheduled to meet with that week. He poured many many hours into people's lives, listening to them and offering wisdom that you only get from a lifetime of counseling.

Dad was a big Yankees fan. He and I would talk about the latest injuries, trades, slumps, and rookies once every few weeks. He had a bunch of DVDs of Yankees World Series victories and such. The night before he died he was at a good friend's house watching the Cubs-Cards game. But he never let his enjoyment of the game dominate him. If the Yankees lost he would groan, but then would move on rather than getting depressed or getting fussy with us kids about it. He kept his enjoyment of baseball in its proper place.

Dad was a man of God. He pastored at a small church in a small town for 40 years, through many people moving through the church, through a variety of buildings, etc. He was a scholar; he read Hebrew and Greek, had read through the Bible many, many times, and had a stack of concordances - all of which he had read! - 10 feet high. One of my fondest memories is waking up in the morning and walking into the living room and seeing him sitting on the couch with a cup of coffee perched atop a stack of books and reading his Bible.

Marvin Fisher, the pastor who gave the benediction at the funeral, said that my Dad would sit in the Farmville Ministerial association meetings and not say a whole lot as the various discussions went back and forth. But he said that a day or two after the meeting "you'd get a phone call, and Lee would say 'hey, why don't we grab coffee', and then he would have some helpful things to say". I think my Dad was smart enough to understand not just how to talk people but when to talk to them.

My wife and I recently told my parents that we were expecting another child. The next day a nice bouquet of flowers showed up at our house. On the card was not "we're happy for you" or "congrats!" or "we're so proud" or anything like that, which would have been great. Instead, my Dad's message was to my wife, and it read "you make us very happy". It was a wonderful thing to say and it showed what a kind and thoughtful guy he was.

Dad and I had many gallows humor-laden discussions about death. He had many phrases that he'd employ: "just put me in a pine box", "don't say 'there's Dad in the coffin', that's not me, that's just a shell", "no matter what you've done, you never know when a beer truck will hit you". All of those discussions reflected his understanding of death being part of a fallen world. He had what he called a "death box" under his bed, which was a cardboard box containing insurance policies and whatnot. On the cover of it he had written "Oh death where is thy sting." So, all that to say that for him death was not something to be ignored or denied but instead was a deliverance. I think his was a good way to go - to be in good health and mentally sharp, to watch a ballgame with a friend, to come home, and to die in his sleep was a great blessing.

I'll miss my Dad terribly, and my Mom and sisters will also. But we rejoice in the thought of him being welcomed home as a good and faithful servant.

August 12, 2015

Evan PhoenixWhat do 100 strangers think? (A science experiment on ethnic perception)

evanphx @ 2015/08/12 04:59 PM

Yesterday in the car, Abby and I happened upon a conversation about how other people will perceive our daughters’ ethnicity. They’re both half White / half Chinese, something that in Los Angeles is hardly rare, and we were split on how people would see them as they grow up (Zoe is 4 years old and mostly the focus of the discussion).

Rather than leave it as a hypothetical discussion, something we’d wait years to see how it played out mostly through what Zoe chooses to tell us, I decided to do something more scientific. I had Abby pick a picture of Zoe and we wrote up a Mechanical Turk job to have people answer what ethnicity they thought the person in the picture was.

Mechanical Turk, for those who haven’t heard of it, is an Amazon service that allows you to ask a set of humans to perform some work on their computer and report back on the results. You pay them a small reward (we paid them $0.05 each) and pick how many answers you’d like to get (we picked 100). You formulate the work, in our case in the form of a picture with a multiple choice question, and Amazon gives it out to people that make money performing the work.

To my delight, it only took about an hour to get 100 replies, making this science experiment quite fulfilling. The results were also really interesting.

So without further ado, the picture:11270640_10153835836635744_2082581986329784894_o

And the results:


I gave people the ability to pick multiple, that’s why the numbers don’t add up to 100. Of the people that picked multiples, no one picked the correct one either.

Ethnicity identification is very much a regional effect. It’s not possible to know where the answers were from, but if we assume they were mostly American (given the time that I was using Mechanical Turk), this provides an interesting insight into how people will see Zoe and Kira in their lives.

It was also a fun mini-experiment.

August 06, 2015

Tom CopelandRevisiting rails:update

2015/08/06 04:00 AM

Lately I've been revisiting some Rails apps which have been around for a while. These are Rails apps that started life somewhere in the 2.x era and are now on the latest (4.2.3 as of this writing), and they were upgraded in a series of steps over the past couple of years. So all that to say that these apps have been through a lot of changes.

When I'm doing Rails upgrades I tend to do the minimum necessary to get the app to the next version. Then I make some notes on things to revisit after the dust has settled. Sometimes those things happen, sometimes not. One thing I hadn't done before is run rake rails:update on those apps. I had tried it a couple of times, but I would see a huge diff and shelve it for later.

But I think I was approaching it in the wrong way. Here's what I've been doing lately.

bundle exec rake rails:update # and hit 'a' to accept all changes
git diff > ../rails_update.diff
git add . && git reset --hard # throw changes away
git checkout -b newify

Now in a terminal window I more ../rails_update.diff and just kind of page through it, looking for useful bits and fixing them as I go. Stuff like:

  • Changing Foo::Application to Rails.application in Rakefile and whatnot.
  • Removing unnecessary MIME type registrations like Mime::Type.register("text/plain", :txt).
  • Removing any explicit requiring of rubygems in config/boot.rb.
  • Simplifying the bundler required invocation in application.rb from Bundler.require(:default, Rails.env) if defined?(Bundler) down to Bundler.require(*Rails.groups).
  • Seeing if there are any redundant configuration item settings in application.rb. For example, I might be explicitly setting ActiveRecord::Base.include_root_in_json to false, but that's the default value in 4.2.3.
  • Similarly, seeing if I'm adding any now-non-existent (due to code cleanups or whatever) directories to the autoload path.
  • Seeing if I'm adding any autoload paths that are unnecessary since modern Rails autoloads all directories under app/. I've found apps that were adding app/mailers to the autoload path, for example.

There are usually some other things I spot when poking through these files - old constant settings, old environment variable checks, etc. Usually a git blame will help me figure out if this is something that was added 5 years ago and never deleted or will at least let me know who I can talk to about it.

None of these changes are big wins, it's just a few milliseconds here and there. But they all add up and help to make an old and crufty application more approachable. Plus, red commits always feel good!

Thanks to Jeremy Stuckey for reviewing this post.

July 10, 2015

Tom CopelandRemoving blocknsurf adware infection

2015/07/10 04:00 AM

Recently I was doing some family Windows tech support. We had been given a laptop and I wanted to square it away so my wife could use it and pass on her current laptop to one of the kids. So it was the usual routine - uninstall lots of stuff, run Windows Update, install Open Office, generally make it safe for humanity.

Chrome and other browsers kept behaving weirdly though. A page would load, but then a couple of addresses would be displayed in the status bar and the page would get all fouled up - ads would appear, various words would get munged so that they had an ad-loading overlay when clicked, links would be similarly munged, various Javascript window slideins would interfere with the page, etc. All of this stuff had a "blocknsurf" logo at the bottom. Googled that, obvious adware infection. Checked the browser extensions - nothing. Checked the installed programs - nothing. Set up an ad blocker - no effect. Ran a adware checker utility, it found some misc files and deleted them and rebooted - no effect. Reset Chrome to defaults - nothing.

What puzzled me was that I didn't see another process in the task manager; there wasn't a proxy service or something like that running. Then it hit me. Blocknsurf must have modified Chrome. It must have wrapped the Chrome launcher, or one of the executables or DLLs so that when launched it would include the blocknsurf code in-process. Perhaps it automatically loaded an extension and hid it from the extensions list; that would have been consistent with the observed behavior.

The fix was to simply uninstall Chrome and then download and reinstall it. With the new binary in place, all was well; no more blocknsurf ads!

May 11, 2015

Tom CopelandRails 4.2 class loading

2015/05/11 04:00 AM

Supposing you have a Rails 4.2 application with a Book model, and there's a record of who has read that book, so it has_many :readings. If you start a new console, reopen the class, and try to use that association, you might be surprised by the result:

irb(main):001:0> class Book < ActiveRecord::Base ; end
=> nil
irb(main):002:0> Book.first.readings
  Book Load (0.7ms)  SELECT  "books".* FROM "books"  ORDER BY "books"."id" ASC LIMIT 1
NoMethodError: undefined method `readings' for #<Book:0x007fef076616d0>
	from gems/activemodel-4.2.1/lib/active_model/attribute_methods.rb:433:in `method_missing'
	from (irb):2

What? Where did my associations go?

This puzzled me for a while until I realized that the first line of code in that irb session is not reopening the class. Instead, it's defining it for the first time, and once Rails has a class definition in place it won't attempt to load the definition that's in app/models/.

This bit me recently on a Rails 4.1 to 4.2 upgrade. I was running the test suite and getting errors as if fixtures were not working - i.e., errors like ActiveRecord::StatementInvalid: Mysql2::Error: Unknown column 'readings' in 'field list':. But it was because I was reopening model classes in test_helper to add some methods, and those reopenings didn't have the full class definitions with all the associations and module inclusions and whatnot. Yup, I know, bad practice, hey, reasons. So I "fixed" it by adding Book (thanks to Michael Cohen for noting that Book.first did an unnecessary query) just before my class Book ActiveRecord::Base in test_helper; with that in place the app/models/book.rb class definition was loaded first and then my reopening code was actually reopening as I intended.

Joe Rafaniello had an even better idea. Rather than using autoloading by referencing the constant, just do a plain old require 'book'; that explicitly shows that we're loading the class rather than trying to fool autoloading.

More generally, Michael Cohen also pointed to Justin Weiss's article on monkeypatching and suggested putting these test-related monkeypatches in a module rather than just slapping them into test_helper.rb.

April 27, 2015

Tom CopelandWorking with the Google Directory API Ruby client

2015/04/27 04:00 AM

Recently I had to transition a Rails app from Google's old provisioning API to the new-ish Directory API. There were a few bits that I had trouble with, and some of the Stack Overflow threads reflected similar confusion, so here's a writeup in hopes of helping others.

The Directory API is a pretty standard REST setup, so you could just hit it with httparty or faraday or some such HTTP client. But I think most people will want to use the google-api-client gem since that's what Google provides. This gem uses an interesting technique; it doesn't define a bunch of methods like Group#update and whatnot. Instead it downloads an API definition file and at runtime uses that to generate classes and methods. This means that Google doesn't have to release a new gem version every time they add an API endpoint, which is pretty cool. But it does mean that you can't just bundle open the gem and see what the method names are.

You don't want to download that API file every time you hit the API, so you'll want to do something to cache it locally. I created a GoogleAdminDirectoryWrapper class in lib/ for easier testing and command line usage; here's the part of that class that caches the API definition file:

  def dump_directory_to_file, "w") do |f|
      f.syswrite(JSON.dump(client.discovery_document("admin", "directory_v1")))
  def discovered_directory_api_file

Then you can jump in a console, run, and you've got the file locally. In order to use the API definition you need to register it with an instance of Google::APIClient, so you'll want something like:

  def directory_api_from_file
    return @directory_api if @directory_api
    contents =
    client.register_discovery_document("admin", "directory_v1", contents)
    @directory_api = client.discovered_api("admin", "directory_v1")

The client method is another helper I wrote that returns a properly initialized Google::APIClient, i.e., one that's been authorized with OAuth using an instance of Signet::OAuth2::Client. Getting a service (i.e., a web app) to authenticate is a whole discussion in itself. The one hint (and the documentation calls this out, but still) that I'd give is that when authorizing you need to use a :person entry in the hash where the value is the email of the administrative account that you're "impersonating", that is, the account with which you created the application that's making these calls.

So, back to API usage. When I was writing the code that calls the endpoints it wasn't always clear to me how to actually compose those method invocations. The invocations vary based on the HTTP method and the content of the request; they depend on what's in the URL itself and what's in the request body. It's a sort of mapping technique; for example, here's more or less my function for deleting a group:

  def group_delete(email)
    client.execute(directory_api_from_file.groups.delete, :groupKey => email)

Seems pretty straightforward. But adding a member to a group looks a little different since it's a HTTP POST that requires information in the request body. You have to supply an api_method parameter, a parameters containing the URL parameters, and a body_object containing the POST request body:

  def group_add_member(group_email, user_email)
    p = {:groupKey => group_email}
    b = {:email => user_email, :role => "MEMBER"}
    m = directory_api_from_file.members.insert
    client.execute(:api_method => m, :parameters => p, :body_object => b)

Here's another example. To fetch all members of a group you're doing a GET and so everything is contained in the URL and in the parameters. So we need api_method and parameters but no body_object. The method below also takes care of unpacking the relevant (for my purposes) bits from the data structures that the gem returns:

  def group_retrieve_all_members(group_email)
    p = {:groupKey => group_email}
    m = directory_api_from_file.members.list
    res = client.execute(:api_method => m, :parameters => p).data {|m|'@').first.downcase }.sort

After doing a couple of these a pattern becomes clear and you can just bang them out. I found it really helpful to sit in a console and do load 'lib/google_admin_directory_wrapper.rb' while I experimented with the method definitions and only after I nailed it down would I write client code within my application.

I thought some of the error handling was a little surprising. For example, the API responds with a permissions error if the group does not exist, so you can do something like this to check for a group's existence:

  def group_exists?(group_email)
    p = {:groupKey => group_email}
    m = directory_api_from_file.groups.get
    !client.execute(:api_method => m, :parameters => p).error?

For testing, I used mocha to mock out method invocations on GoogleAdminDirectoryWrapper and supply the expected results. I feel a little bad about not mocking things at a lower level - i.e., using vcr. I didn't actually change anything, but I do feel bad, so that's got to count for something.

I think the lessons learned are pretty much the usual ones. Get started on API migrations early (which I didn't). Do yourself a favor and put together a rapid development cycle. Get simple things working and build from there. RTFM.

April 15, 2015

Tom CopelandClass methods and singleton methods

2015/04/15 04:00 AM

Class methods are methods on a object's singleton class. Everyone knows this (1). I think I sort of knew it also, but recently I was working on a thing and this was brought home to me.

I was working on integration tests for filter_decrufter, so I wanted to define a sort of stubbed out ActionController::Base (2) with a class method that simulated defining a before action:

module ActionController
  class Base
    def self.before_action(*args)

And I had a subclass that attempted to call that before_action class method:

class AnotherFakeController < ApplicationController
  before_action :foo, :only => [:bar]
  def foo
  def self.my_action_methods

Then filter_decrufter could define a singleton method that would check the before_action arguments and flag any options for missing actions:

# in a loop where filter_sym is before_action, after_action, around_filter, etc
ActionController::Base.define_singleton_method(filter_sym) do |*args, &blk|
  # ... gather some data about *args ...
  super(*args, &blk)

What I was seeing, though, was that AnotherFakeController would raise an exception when I loaded it and it attempted to call the parent class method as part of the class definition:

  1) Error:
NoMethodError: super: no superclass method `before_action' for AnotherFakeController:Class
    /lib/filter_decrufter/checker.rb:100:in `block in patch_method'
    /test/integration/another_fake_controller.rb:3:in `<class:AnotherFakeController>'

But why? The before_action method is declared right there in ActionController::Base!

The problem was that the before_action method that ActionController::Base defined was living on ActionController::Base's singleton class. No need to take my word for it though; you can verify this by defining a class method and checking the singleton methods:

irb(main):001:0> class Foo ; def ; puts "Foo#bar" ; end ; end
=> :bar
irb(main):002:0> Foo.singleton_methods
=> [:bar]

So when I defined a singleton method on ActionController::Base I was not intercepting the method call like I intended. Instead, I was redefining the existing method. And my new method definition called super, but since I'd redefined the only method with that name in this class's ancestor chain, there was no superclass method by that name available, and so bam, exception.

As a side note, singleton_methods looks up the inheritance chain, so it's not quite reliable for saying "this method is defined right here":

irb(main):001:0> class Foo ; def ; end ; end
=> :bar
irb(main):002:0> class Buz < Foo ; end ; class Biz < Buz ; end
=> nil
irb(main):003:0> Biz.singleton_methods
=> [:bar]

Back to the original problem - how to solve it? By defining the method not on the singleton class but instead further up the ancestor chain. And how to do that? By defining the method in a module and extend'ing that module:

# Define a method with our method in it
irb(main):001:0> module Buz ; def bar ; puts "Buz#bar" ; end ; end
=> :bar
# extend that module so that it's a class method
# rather than include'ing which would make it an instance method
irb(main):002:0> class Foo ; extend Buz ; end
=> Foo
# now define a singleton method that will intercept invocations of that method
irb(main):003:0> Foo.define_singleton_method(:bar) { puts "Foo#bar" ; super() }
=> :bar
# and demonstrate the that interceptor gets called first and then calls the superclass

I think the lessons learned are the usual ones. Unexpected exceptions are an opportunity for learning something. Don't confuse Java static methods with Ruby's class methods. Verify expected behavior in irb or in a small program. And read books written by people who have poured a lot of time and energy into the topic that's currently giving you trouble!

(1) Because everyone's read Paolo Perrotta's excellent Metaprogramming Ruby 2nd Ed.

(2) You could argue that I should just declare a dependency on actionpack and use it. That probably would be better; I might do that.

April 14, 2015

Ryan DavisGreat Expectations

2015/04/14 07:00 PM

minitest/spec has existed since v1.3.0 (6.5 years). It started as a simple proof of concept to show that there was a 1:1 mapping between “spec-style” and “unit-style” tests (not to be confused with BDD vs TDD, which is a development process). It showed (in 67 lines!), that every describe was just a Test subclass, and every it was just a test method (with strange inheritance).

minitest/spec has only grown to 152 lines since then but it really hasn’t changed all that much. Until yesterday.

The older spec system relied on a thread-local variable in order to understand what the current test method was. It was a limitation of how you called the expectation methods. So, when you wrote a test like this:

describe Something do
  it "should blah-blah" do                                      
    (1+1).must_equal 2

we call must_equal on the result of 1+1. This implies that the method is on Object in order to be called on the value 2. In order to map must_equal back to Minitest::Test’s assert_equal, minitest needed to know what test the expectation was in and that was done via Minitest::Spec.current, which just accesses a thread-local variable. That works great 99.9% of the time, until, you know, you use threads in your tests. (Why you would use threads inside your tests is irrelevant.)

So, if your test changed to this:

describe Something do
  it "should blah-blah threaded" do                                      
    my_threaded_test_thingy do
      (1+1).must_equal 2

then it would blow up, because you’d be wrapping must_equal in your own thread and that would make the thread-local variable unavailable (they’re not scoped to parent threads, for better or worse).

Yesterday’s release of 5.6 brings new expect system from tenderlove that adds a struct and one new method _. The method wraps the resultant value in a Expectation value monad. All in all, it only adds 11 lines of code to minitest. All of the usual expectation methods are defined, but this time around, they know what test they’re in because it was stored off when the monad was created.

So, one character change later, and the following works:

describe Something do
  it "should blah-blah threaded" do                                      
    my_threaded_test_thingy do
      _(1+1).must_equal 2

The method _ is also aliased to value if you prefer verbier tests. It is also aliased to expect, to make transitioning easier, but <bikeshed>1I really dislike that name</bikeshed> because it is non-descriptive. value is much better, and imo, _ is even better because it gets out of your way and lets you focus on the actual meat of the test.

At some point (pre-6.0) the old expectations are going to be deprecated in favor of the new system. Until then, they’re fine to use and won’t make any noise until the deprecation is officially planned. That will remove the monkeypatches on Object for all of the expectations that minitest supports. Hopefully that’ll reclaim the 11 lines we added.

Apologies are owed to Dickens. I couldn’t help myself, even tho I hated that book.

  1. <bikeshed> means: don’t argue with me about it. I don’t care to argue about it.

Ryan DavisDebride gets better rails support

2015/04/14 07:00 AM

I just released debride 1.3.0. Debride statically analyzes your ruby for potentially uncalled / dead methods. It should help you strip out your dead weight from your large (or small) projects.

Thanks to contributions from Pete and Ian, it now has an --exclude flag as well as better rails DSL support.

Pete’s also released debride-haml 1.0.0 as debride’s second plugin.

If you have any problems with it or have suggestions on how to make it work better for you, please file an issue.

March 26, 2015

Ryan DavisDebride gets Plugins!

2015/03/26 07:00 PM

I just released debride 1.2.0 and extended it with a flexible plugin system that’ll allow it to process multiple types of files.

I also released debride-erb 1.0.0 as debride’s first plugin.

Between the whitelisting and ERB support, I’ve been able to run this across’s website and actually found some stuff we could remove!

If you have any problems with it or have suggestions on how to make it work better for you, please file an issue.

March 18, 2015

Ryan DavisDebride gets Whitelisting

2015/03/18 09:00 AM

I just released debride 1.1.0 and extended it with whitelisting. So previously when you ran debride on your code you might see something like this:

% debride lib

These methods MIGHT not be called:

  good_method                         lib/some/file.rb:16
  bad_method                          lib/some/file.rb:20

But if you know that good_method is exempt (perhaps because it is public API), then you can whitelist it:

% echo good_method > whitelist.txt
% debride --whitelist whitelist.txt lib

These methods MIGHT not be called:

  bad_method                          lib/some/file.rb:20

You can also use regexps in your whitelist by delimiting them with slashes (eg. /^good_/).

If you have any problems with it or have suggestions on how to make it work better for you, please file an issue.

March 12, 2015

Ryan DavisMeet Debride

2015/03/12 07:00 PM

I released a new tool named debride while on the road for Ruby on Ales and Mountain West Ruby Conf. It’s a fairly simple tool that can analyze code for potentially uncalled / dead methods. If you suspect that you have dead code but don’t know how to go about identifying it, you might want to check out debride.

% debride lib

These methods MIGHT not be called:

  method!                            lib/some/file.rb:16

What’s dead code? To debride, it is any method defined that doesn’t appear to have a call anywhere else. So this tool isn’t as good for public API unless you also run your tests against it (but that is prone to keeping well-tested dead code alive).


There are obvious problems to using a static analyzer for a highly dynamic language like ruby. The first is send. I’m toying with the idea of providing a send wrapper that’ll log all method names to a file that you can then review and whitelist for subsequent runs on debride, but I’m totally open to suggestions & PRs.

March 09, 2015

Ryan DavisStanding on the Shoulders of Giants

2015/03/09 06:00 PM

I just posted my slides for my talk at both Ruby on Ales and MountainWest Ruby Conf 2015.

Enjoy! Let me know if you have any questions or comments!

February 10, 2015

Tom CopelandString method use and misuse

2015/02/10 05:00 AM

Consider this little code snippet:

[1,2,3].select {|x| x > 1 }.first

This should be replaced by [1,2,3].detect {|x| x > 1 }; that's what pippi is for. Pippi also finds a bunch of variants - select followed by any?, etc.

But after writing a couple rules I ran out of ideas on method sequences to detect. There's an infinite number of method sequence possibilities, of course, but most of them are nonsense, and pippi is slow enough as it is without filling it with checks that probably won't find anything.

So I wanted to figure out which sequences of method calls were happening. This isn't looking for just any method sequence, though, it's focusing on method invocations on objects where that invocation returns that same type. So Array#select is interesting, because it returns another Array, but Array#first is not because the array could be holding a variety of types. After a bit of flailing around I came up with a gizmo to find such sequences and ran it (e.g., USE_PIPPI=true PIPPI_CHECKSET=research bundle exec rake test:units | grep occurred) on some code that was lying around, producing:

select followed by each occurred 34 times
select followed by map occurred 20 times
select followed by size occurred 13 times
select followed by empty? occurred 10 times
select followed by any? occurred 7 times
select followed by first occurred 7 times
select followed by sort_by occurred 4 times
select followed by reject occurred 3 times
select followed by select occurred 3 times
select followed by count occurred 3 times
select followed by last occurred 3 times
select followed by detect occurred 2 times
select followed by length occurred 2 times
select followed by group_by occurred 2 times
select followed by none? occurred 2 times
select followed by to_a occurred 1 times
select followed by each_with_index occurred 1 times
select followed by slice occurred 1 times
select followed by inject occurred 1 times
select followed by all? occurred 1 times
select followed by collect occurred 1 times
select followed by map! occurred 1 times
select followed by join occurred 1 times

The grep is there because the utility prints out the location of each hit so you can go take a look at it, and I wanted to filter those out here. I'm printing out a frequency map, but that's not to say that low numbers on the histogram aren't be interesting; if a Pippi rule only finds one hit in a 10KLOC codebase it's still a good check if it actually found something to fix. But while we're accumulating data we may as well get a feel for frequency.

There's some good stuff there - for example, that select followed by all? should probably be stamped out - but we've already got a bunch of checks around Array#select. How about String#strip?

strip followed by empty? occurred 15 times
strip followed by downcase occurred 10 times
strip followed by split occurred 7 times
strip followed by strip occurred 4 times
strip followed by gsub occurred 4 times
strip followed by encode occurred 3 times
strip followed by length occurred 3 times
strip followed by encoding occurred 3 times
strip followed by force_encoding occurred 3 times
strip followed by index occurred 3 times
strip followed by count occurred 2 times
strip followed by size occurred 2 times
strip followed by ascii_only? occurred 2 times
strip followed by unpack occurred 1 times
strip followed by strip! occurred 1 times
strip followed by gsub! occurred 1 times
strip followed by match occurred 1 times
strip followed by tr occurred 1 times
strip followed by dump occurred 1 times
strip followed by b occurred 1 times
strip followed by delete occurred 1 times
strip followed by encode! occurred 1 times

Definitely some good fodder for rules there. Why would you ever call strip on a String that you got as a result of calling String#strip? Maybe strip followed by empty? could be replaced with a regex check like if foo.match(/^\s*$/). And you can imagine some other possibilities when seeing that list.

To repeat the warning in the pippi README, not all of these issues are fixable. A method might call strip on a String and return it, and the caller might then invoke strip on that same object because a different code path would bring an un-strip'd String to that same location. I don't think there's a good way around that short of a refinement or some such that would tell pippi to ignore that hit, and I'm not sure that'd be worth the trouble.

Anyhow, should be good check potentials here. We'll see!

January 15, 2015

Ryan DavisYou can go home now, I solved testing

2015/01/15 08:00 PM

I need to work on a real blog post about this topic, but all I could come up with yesterday was this snarky snippet of code:

def test_some_method
  assert_something_output do
    assert_nothing_raised do
      assert_some_side_effect do
        assert_some_response do
          object.some_method # testing SOLVED

This is, of course, me responding to more resistance to minitest not having useless assertions like assert_nothing_raised. I’ve talked about it before. I’ve blogged about it before. Pretty sure it’ll never die.

January 07, 2015

Ryan DavisEmacs Automodes

2015/01/07 08:00 PM

I have a system that I’ve already detailed called “autohooks”, but I also hinted about something I use called “automodes”. automodes? What is that and how does it differ from autohooks?

Hooks are intended to be run every time you load a file and go into a mode. When I load a ruby file, my the enh-ruby-mode-hook gets triggered. That’s what my autohook system is for. Automodes tho, is intended to load all the stuff that needs to be loaded before the hook triggers. An easy example is to set up mapping file patterns (eg “*.rb” files) to their modes (enh-ruby-mode) or to set up variables needed for the mode to be set on load (eg enh-ruby-program).

How simple? It’s just 4 lines long:

(defun rwd-load-modes ()
  (dolist (path (directory-files (concat user-init-dir "modes") t ".*el$"))
    (load path)))

This is triggered right after loading emacs via my after-init autohook file:

;; after-init.el:
(when (and running-osx (string= "/" default-directory))
  (cd "~"))

(global-whitespace-mode 1)


As an example, here is my modes/enh-ruby-mode.el:

(autoload 'enh-ruby-mode "enh-ruby-mode" "Major mode for ruby files" t)

(dolist (spec '(("\\.mab$"   . enh-ruby-mode)
                ("\\.rb$"    . enh-ruby-mode)
                ("Rakefile"  . enh-ruby-mode)
                ("Gemfile"   . enh-ruby-mode)
                ("\\.rake$"  . enh-ruby-mode)))
  (add-to-list 'auto-mode-alist spec))

(add-to-list 'interpreter-mode-alist '("ruby" . enh-ruby-mode))

(add-to-list 'load-path (expand-file-name "~/Work/git/enhanced-ruby-mode"))
(setq enh-ruby-program (expand-file-name "/usr/bin/ruby"))

It is a really simple system and really only exists so that I can break up a monolithic setup into many small single-topic files. This has made things a lot more maintainable over time.

December 19, 2014

Ryan DavisUpdated iTunes Bedtime script

2014/12/19 08:00 AM

Minor changes this time. This works in iTunes v12, which is the latest release for Yosemite. The only real fixes centered around verbage changes to menu items. I wish it had applescript support that didn’t require UI scripting to do basic things like listen to a smart playlist or a particular band.

This script starts up itunes and airfoil, sets the volumes of both, picks a specified playlist, and plays it. I fire this one up just before going to bed. I have another copy that sets up my programming playlist and another for cooking.

They’re always available via the script menu by putting them in "~/Library/Scripts/".

I hope this helps.


property myPlaylist : "Mellow"
property myVolume : 50
property airfoilVolume : 0.5

tell application "iTunes"
    stop application
    my setShuffle()
    set sound volume to myVolume
    my setSource()
    my minimizeITunes()
end tell

my setSpeakers()

on setShuffle()
    tell application "System Events"
        tell process "iTunes"
            tell menu 1 of menu bar item "Controls" of menu bar 1
                tell menu 1 of menu item "Shuffle"
                    if "Turn On Shuffle" is in (title of menu items) then
                        click menu item "Turn On Shuffle"
                    end if
                    click menu item "Songs"
                end tell
                tell menu 1 of menu item "Repeat"
                    click menu item "All"
                end tell
            end tell
        end tell
    end tell
end setShuffle

on minimizeITunes()
    tell application "iTunes" to close every window
    tell application "System Events" to tell process "iTunes"
            click menu item "Switch to MiniPlayer" of menu 1 of menu bar item "Window" of menu bar 1
        end try
    end tell
end minimizeITunes

on setSource()
    tell application "iTunes"
        tell playlist myPlaylist
        end tell
        next track
    end tell
end setSource

on setSpeakers()
    set volume output volume 100
    set volume input volume 100
    set volume alert volume 100
    set volume without output muted
    tell application "Airfoil"
        close every window
        set current audio source to application source "iTunes"
        set linked volume to false
        disconnect from every speaker
        set (volume of every speaker) to airfoilVolume
        connect to speaker "airport2"
        connect to speaker "Computer"
    end tell
end setSpeakers
Last updated at 2016/02/09 05:00 AM | Presented by Aredridel and Christoffer Sawicki | Hosted at The Internet Company