November 10, 2014

Evan PhoenixOn Portable Computation…

evanphx @ 2014/11/10 07:42 AM

Since the first time two computers communicated directly over a network, programmers have mused about the programs running on those computers. Back in those early days, there was careful external synchronization of programs “Bob, will you load number 3 while I’ll do the same over here?” We like to think that we’re well past that, but in reality, it’s basically still the same situation today. We’ve invented versioning schemes to make that external synchronization easier to swallow, but we’re still just manually putting 2 programs on different computers and hoping they can talk to each other.

And so the industry has slowly tried to evolve solutions to this problem. Java, for instance, strove to allow a server to push down the client version of a program to the browser to run, allowing the server to decide at load time which to push (you remember applet’s don’t you?). Modern web programmers effectively worked their butts off to avoid the problem and instead push a very heavy standards drivin’ approach (hi there W3C!). And because web programming has dominated so much of the industry for the past decade, there hasn’t been a big push to work on portable computing.

Disclaimer: I’m sure people will comment and say “OMG Evan! This person and that person have been working on it for 20+ years! I’m sure thats true, but almost nothing has gone mainstream, such that it has no effect on your everyday programmers.

There are 3 really interesting attempts at bringing portable computing to the everyday programmer I’ve been thinking about this weekend that I thought I’d do a quick post about.

  1. NaCL – Google’s approach to applets, basically. With a heavy emphasis on C++ as the target language, the aim is squarely at game developers. But I think if you look past immediate browser usage, you can see a separate idea: The ability to generate managed machine code that only runs inside a special harness and can be easily transported. I’m sure the irony of that last sentence NOT being applied to Java will be lost on no one. And in that irony, a nugget of truth: this approach isn’t new. The exact same thing could be applied to Java or really any VM. The place that NaCL deviates is that because it’s the generation of machine code and that the toolkit is a normal C/C++ compiler, it’s possible to compile a VM itself with NaCL to deploy it. Thus NaCL can be seen as a safe way to distribute binaries, which makes it pretty interesting. I’ll call this approach to portable computing the specialized environment approach.
  2. Docker – I’m talking about docker rather than generically containers or other similar technologies because it’s Dockers pairing of containers with images that are delivered over a network. Certainly a service to download a binary and run it would be trivial, thusly it’s usage of containers that Docker into a proper portable computing environment. The interesting aspect to Docker is that it’s approach is squarely on the computing “backend”. NaCL and it’s spiritual father Java applet focus getting new computation units to and end user directly, where as Docker is about the management of backend services. It’s just that the code for those backend services is delivered remotely on demand, making it a portable computing environment. This is the constrained environment type of portable computing.
  3. Urbit – Urbit is, well, it feels like the implementation of an idea from a scifi novel. Part of that is on purpose, it’s founder is known to be quite esoteric. Cutting through all the weirdness, I find a kernel of an idea for portable computing. They’ve defined a VM that can run a very simple bytecode and a whole ecosystem around it. The service I found most interesting is the fact that all programs are in their bytecode and all programs are stored in their global filesystem service. This means that it’s trivial for programs written and stored by another computer to be run by an other computer. Add in versioning, public identities, and encryption and you’ve got yourself a really nice little portable computing environment. The fact that the VM so effectively stateless and the services promote storing immutable data, it’s easy to see how a distributed system would work. The problem? It’s all in a really esoteric environment that it going to be incredibly difficult for even the most motivated programmers to get into. This is the isolate environment type.

So, that’s the point of cataloging these here? Well, I started looking at Urbit and thinking how difficult it is for it to get uptick. But the kernel of the idea is super interesting. So could that kernel be transplanted to something like NaCL or Docker to provide a meaningful environment for distributed programming? Urbit’s immutable global file system seems perfect to apply CRDT to, allowing multiple nodes to share a meaningful view of some shared data. Wire in a discovery mechanism that will tie environments together and I think there could be something big there.


October 22, 2014

Tom CopelandFinding suboptimal Ruby class API usage

2014/10/22 04:00 AM

Consider this little array:

[1,2,3]

Now suppose we want to find the first element in that array that's greater than one. We can use Array#select and then use Array#first on the result:

[1,2,3].select {|x| x > 1 }.first

Of course that's terribly inefficient. Since we only need one element we don't need to select all elements that match the predicate. We should use Array#detect instead:

[1,2,3].detect {|x| x > 1}

This is an example of refactoring to use the Ruby standard library. These are small optimizations, but they can add up. More importantly, they communicate the intent of the programmer; the use of Array#detect makes it clear that we're just looking for the first item to match the predicate.

This sort of thing can be be found during a code review, or maybe when you're just poking around the code. But why not have a tool find it instead? Thus, pippi. Pippi observes code while it's running - by hooking into your test suite execution - and reports misuse of class-level APIs. Currently pippi checks for "select followed by first", "select followed by size", "reverse followed by each", and "map followed by flatten". No doubt folks will think of other checks that can be added.

Here's an important caveat: pippi is not, and more importantly cannot, be free of false positives. That's because of the halting problem. Pippi finds suboptimal API usage based on data flows as driven by a project's test suite. There may be alternate data flows where this API usage is correct. For example, in the code below, if rand 0.5 is true, then the Array will be mutated and the program cannot correctly be simplified by replacing "select followed by first" with "detect":

x = [1,2,3].select {|y| y > 1 }
x.reject! {|y| y > 2} if rand < 0.5
x.first

At one point I considered pulling the plug on this utility because of this fundamental issue. But after thinking it over and seeing what pippi found in my code, I decided to stay with it because there were various techniques that eliminated most of these false positives. For example, after flagging an issue, pippi watches subsequent method invocations and if those indicate the initial problem report was in error it'll remove the problem from the report.

There are many nifty Ruby static analysis tools - flay, reek, flog, etc. This is not like those. It doesn't parse source code; it doesn't examine an abstract syntax tree or even sequences of MRI instructions. So it cannot find the types of issues that those tools can find. Instead, it's focused on runtime analysis; that is, method calls and method call sequences.

To see what pippi finds in your Rails project, follow the instructions in the pippi_demo example Rails app. TLDR: Add gem 'pippi' to your Gemfile, add a snippet to test_helper.rb, and run a test with USE_PIPPI=true to see what it finds.

Note that pippi is entirely dependent on the test suite to execute code in order to find problems. If a project's test code coverage is small, pippi probably won't find much.

Here's how pippi stacks up using the Aaron Quint Ruby Performance Character Profiles system:

  • Specificity - very specific, finds actual detailed usages of bad code
  • Impact - very impactful, slows things down lots
  • Difficulty of Operator Use - easy to install, just a new gemfile entry
  • Readability - results are easy to read
  • Realtimedness - finds stuff right away
  • Special Abilities - ?

Finally, why "pippi"? Because Pippi Longstocking was a Thing-Finder, and pippi finds things.

October 21, 2014

Ruby InsideRaptor: A Forthcoming Ruby Web Server for Faster App Deployment

Peter Cooper @ 2014/10/21 01:20 PM

Raptor bills itself as a new Ruby "app server” and it claims to blow everything else out of the water performance-wise (by between 2-4x!) whether that’s Unicorn, Puma, Passenger, or even TorqueBox on JRuby. The bad news for now is there’s no source or repo yet and only a handful of people (including me) have been given a sneak peek, although a public beta is promised on November 25th.

The history of Ruby webapp deployment

The deployment of Ruby (and therefore Rails) webapps was a painful mess for years, a state I lamented 7 years ago in No True ‘mod_ruby’ is Damaging Ruby’s Viability on the Web. Thankfully, shortly thereafter a number of projects came out to make life easier, the most famous being Phusion Passenger (then known as mod_rails) in April 2008.

Things have continued to improve gradually over the years, with Passenger getting consistently better, and new approaches such as those offered by Unicorn and Puma, using JRuby, as well as proxying through Nginx, coming into the picture.

Enter Raptor

Raptor, a new entry to the burgeoning world of Ruby Web servers, boasts some compelling features. "Visibility" is cited as a key feature so that you can look ‘into’ your app and analyze its performance as easily as possible using a JSON API (so building your own tools around the API should be simple). Raptor also uses the HTTP parser from Node which itself was derived from Nginx’s HTTP parser; both are renowned for their speed and stability. Raptor boasts a zero-copy, concurrent, evented architecture which makes it efficient memory and IO-wise - so even if you have slow clients or a slow network, these won’t bring your app server to a stuttering standstill.

Another feature that jumped out at me is integrated caching. Raptor doesn’t rely on an external services like memcached or Redis at all, but is truly internal and optimized specifically for Web workloads. If you’ve never set up caching before, this could provide a big boost as with Raptor it’ll be available “out of the box”.

The initial results seem promising. Fabio Akita has already shared some early benchmark results which broadly mirror my own experience (disclaimer: as someone with rather little experience and authority in benchmarking, my benchmarks are oriented around Raptor’s own benchmarking suite) but, as always, YMMV and such benchmarks are often criticized.

The waiting game..

The team behind Raptor promise they’ll be releasing some interesting blog posts soon about the technology behind it, including how the cache is implemented and has been optimized, how the zero-copy system works and how it’ll benefit your code, and similar things. So keep an eye on rubyraptor.org, especially around November 25th.

October 18, 2014

Tom CopelandGood technical videos

2014/10/18 04:00 AM

I rarely get a chance to go to technical conferences, but I do watch lot of videos. I tweet about the good ones; here's a collection for posterity:

October 16, 2014

Ruby InsideRuby’s Unary Operators and How to Redefine Their Functionality

Peter Cooper @ 2014/10/16 09:34 AM

In math, a unary operation is an operation with a single input. In Ruby, a unary operator is an operator which only takes a single 'argument' in the form of a receiver. For example, the - on -5 or ! on !true.

In contrast, a binary operator, such as in 2 + 3, deals with two arguments. Here, 2 and 3 (which become one receiver and one argument in a method call to +).

Ruby only has a handful of unary operators, and while it's common to redefine binary operators like + or [] to give your objects some added syntactic sugar, unary operators are less commonly redefined. In my experience, many Rubyists aren't aware that unary operators can be redefined and.. technically you can't "redefine an operator" but Ruby's operators frequently use specially named methods behind the scenes, and as you'll know.. redefining a method is easy in Ruby!

A Quick Example with -@

Let's ease into things with the - unary operator. The - unary operator is not the same thing as the - binary operator (where a binary operator has two operants). By default, the - unary operator is used as notation for a negative number, as in -25, whereas the - binary operator performs subtraction, as in 50 - 25. While they look similar, these are different concepts, different operators, and resolve to different methods in Ruby.

Using the - unary operator on a string in irb:

> -"this is a test"
NoMethodError: undefined method `-@' for "this is a test":String

The String class doesn't have unary - defined but irb gives us a clue on where to go. Due to the conflict between the unary and binary versions of -, the unary version's method has a suffix of @. This helps us come up with a solution:

str = "This is my STRING!"

def str.-@
  downcase
end

p str     # => "This is my STRING!"
p -str    # => "this is my string!"

We've defined the unary - operator by defining its associated -@ method to translate its receiving object to lower case.

Some Other Operators: +@, ~, ! (and not)

Let's try a larger example where we subclass String and add our own versions of several other easily overridden unary operators:

class MagicString  String
  def +@
    upcase
  end

  def -@
    downcase
  end

  def !
    swapcase
  end

  def ~
    # Do a ROT13 transformation - http://en.wikipedia.org/wiki/ROT13
    tr 'A-Za-z', 'N-ZA-Mn-za-m'
  end
end

str = MagicString.new("This is my string!")
p +str         # => "THIS IS MY STRING!"
p !str         # => "tHIS IS MY STRING!"
p (not str)    # => "tHIS IS MY STRING!"
p ~str         # => "Guvf vf zl fgevat!"
p +~str        # => "GUVF VF ZL FGEVAT!"
p !(~str)      # => "gUVF VF ZL FGEVAT!"

This time we've not only redefined -/-@, but the + unary operator (using the +@ method), ! and not (using the ! method), and ~.

I'm not going to explain the example in full because it's as simple as I could get it while still being more illustrative than reams of text. Note what operation each unary operator is performing and see how that relates to what is called and what results in the output.

Special Cases: & and *

& and * are also unary operators in Ruby, but they're special cases, bordering on 'mysterious syntax magic.' What do they do?

& and to_proc

Reg Braithwaite's The unary ampersand in Ruby post gives a great explanation of &, but in short & can turn objects into procs/blocks by calling the to_proc method upon the object. For example:

p ['hello', 'world'].map(&:reverse)  # => ["olleh", "dlrow"]

Enumerable#map usually takes a block instead of an argument, but & calls Symbol#to_proc and generates a special proc object for the reverse method. This proc becomes the block for the map and thereby reverses the strings in the array.

You could, therefore, 'override' the & unary operator (not to be confused by the equivalent binary operator!) by defining to_proc on an object, with the only restriction being that you must return a Proc object for things to behave. You'll see an example of this later on.

* and splatting

There's a lot of magic to splatting but in short, * can be considered to be a unary operator that will 'explode' an array or an object that implements to_a and returns an array.

To override the unary * (and not the binary * - as in 20 * 32), then, you can define a to_a method and return an array. The array you return, however, will face further consequences thanks to *'s typical behavior!

A Full Example

We've reached the end of our quick tour through Ruby's unary operators, so I wanted to provide an example that shows how to override (or partially override) them that should stand as its own documentation:

class MagicString < String
  def +@
    upcase
  end

  def -@
    downcase
  end

  def ~
    # Do a ROT13 transformation - http://en.wikipedia.org/wiki/ROT13
    tr 'A-Za-z', 'N-ZA-Mn-za-m'
  end

  def to_proc
    Proc.new { self }
  end

  def to_a
    [self.reverse]
  end

 def !
   swapcase
 end
end

str = MagicString.new("This is my string!")
p +str                   # => "THIS IS MY STRING!"
p ~str                   # => "Guvf vf zl fgevat!"
p +~str                  # => "GUVF VF ZL FGEVAT!"
p %w{a b}.map &str       # => ["This is my string!", "This is my string!"]
p *str                   # => "!gnirts ym si sihT"

p !str                   # => "tHIS IS MY STRING!"
p (not str)              # => "tHIS IS MY STRING!"
p !(~str)                # => "gUVF VF ZL FGEVAT!"

It's almost a cheat sheet of unary operators :-)

A Further Example: The TestRocket

TestRocket is a tiny testing library I built for fun a few years ago. It leans heavily on unary operators. For example, you can write tests like this:

+-> { Die.new(2) }
--> { raise }
+-> { 2 + 2 == 4 }

# These two tests will deliberately fail
+-> { raise }
--> { true }

# A 'pending' test
~-> { "this is a pending test" }

# A description
!-> { "use this for descriptive output and to separate your test parts" }

The -> { } sections are just Ruby 1.9+ style 'stabby lambdas' but, with assistance from Christoph Grabo, I added unary methods to them so that you can prefix +, -, ~, or ! to get different behaviors.

Hopefully you can come up with some more useful application for unary methods on your own objects ;-)

Ryan DavisHappy Birfday to me!

2014/10/16 01:13 AM

Today is my Fourteenth Year Anniversary with Ruby! Yay!

823 gem releases (up by 93), 9805 commits (up by 823), and bazillions of test runs later, and I’m still going strong. Rawr!

October 07, 2014

Tom CopelandCleaning up a Rails routes file

2014/10/07 04:00 AM

Large Rails apps have a way of gathering cruft. One popular cruft hideout is config/routes.rb. It doesn't change very often, and when it does change it's usually a person getting in there, adding a new resource or a new action, and getting out. So from time to time it's worth checking the routes file and tidying things up.

Probably the most obvious cleanup is removing unused routes. A route which has no corresponding action won't produce a warning, so you have to ferret these out yourself. Take a look at the routes file and see if anything obvious jumps out at you; if you know that a bunch of admin code was just extracted into another app you can look around for routes in an admin namespace. Do a git whatchanged -p app/controllers/ config/routes.rb (or as Brian Ewins suggests, git log -p --no-merges app/controllers/ config/routes.rb because whatchanged is deprecated) and see if recent controller or action removals included removal of orphaned routes. Grab the traceroute gem and see what it finds. This will lead to a smaller file, faster load time, and generally less confusion.

There are some minor cleanups that are worth a look. Sometimes I'll see resources with empty blocks:

resources :products, :only => [:index, :show] do
end

This is usually leftover from an earlier cleanup; just delete those empty blocks. Same goes for empty member or collection blocks. This is trivial, but it makes the file easier to read.

The default resource declaration - e.g., resources :products - results in routes to seven actions: new, create, edit, update, index, show, and destroy. Occasionally you'll see a resource declaration with all seven actions declared explicitly in an only constraint; in that case you can remove the entire constraint. So this:

resources :products, :only => [:create, :new, :edit, :update, :index, :show, :destroy]

can be boiled down to:

resources :products

The same applies to resource declarations, of course, but with six actions instead of seven.

A variant on this is to replace only blocks with except blocks when those would be shorter. That is, you can replace this:

resources :products, :only => [:create, :new, :edit, :update, :index, :show]

with this:

resources :products, :except => [:destroy]

Some people prefer wrapping single actions in an array when declaring resource restrictions; that is, they prefer resources :products, :only => [:create] to resources :products, :only => :create. I tend to use the bare symbol, but I can see where those folks are coming from - makes it easier to add other actions - so whatever works for your team.

This one is not like the others in that it makes your routes file bigger. But it reduces overall complexity, so I think it belongs here anyway. If you have an action that is serving only to redirect requests to another URL, you can replace that action with a route. So you can replace:

class ProductsController < ApplicationController
  def show
    redirect_to root_path
  end
end

With this route:

match '/products/:id' => redirect('/')

This trades five lines of code and a file for a single route, which seems like a win to me. There are times when this isn't quite as clearcut, like when the redirect is to a more complicated route, but generally this is a just a more effective use of the Rails API. Shades of 'refactor to the standard library'!

Another change which increases the routes file size but reduces the number of routes generated is to add only blocks where appropriate. I've found that many resources don't need a destroy action since that's handled in an admin app. So add only blocks and reduce your rake routes output.

There are other cleanups which could be made around shared constraints, unnecessary controller specifications, and redundant path_names usage. But I feel like usage of these features is rare enough that if you're using them you're probably reading the documentation.

When making changes like this, one option is to do a clean sweep and tidy up everything at once. I can see why people do this; it's one code review, one deploy, etc. But I usually do these things piecemeal - one or two changes at a time. I do this because there's lower risk; if I'm only changing a few things it's easy to locate the culprit if something goes wrong. It also lets me do the work in small chunks; instead of devoting a full day to routes file cleanup I can do 10 minutes of work here and there whenever I have time. Also, small changes are more helpful for co-workers to see what's going on. If someone sees a commit with a small and straightforward route cleanup, that person might be more likely to do a similar cleanup next time they see a similar issue. And if I'm headed down a wrong road with a cleanup strategy, someone can catch it earlier on in the process before I do too much damage.

October 01, 2014

Tom CopelandPreserving technical knowledge

2014/10/01 04:00 AM

If you've worked for a single company in a tech job for a while you know that there's a fair bit of turnover in our world. Folks move on to other jobs, new folks come on, people move between technical teams and over into management, etc. Why, chances are you might even have switched jobs yourself a few times!

It's nice to be able to preserve some technical knowledge during these transitions. If the person is switching roles within the company this is not such a big deal as that person will still be available in a crisis. But even in that case, there are a couple of ways to approach this.

One way is to have the outgoing person do a bunch of diagrams explaining stuff. To me this doesn't help a whole lot; these diagrams all end up looking the same: "this box connects to this cloud where things happen, then there are these other boxes". But it can't hurt. My advice here is to keep the diagrams simple. Also, do them in a tool that the next system owner knows how to use, otherwise they bitrot. I suspect there are folks who really learn a lot from these diagrams, so if you are one of those, have at it. But they don't seem to work for me. I wonder if a way to ensure these pictures have meaning is for the outgoing person to point to a connecting line and say "what would happen if this got cut" and the incoming person has to take a crack at it. That's probably a decent way to learn about things in general - drop a raise "BOOM" in a random place and think/talk through what effect that would have.

Another way is to do face to face interviews where the outgoing person explains the system. I think this is a bit more effective, however, the incoming person has to know something about what's going on otherwise none of the information makes sense. "We've had a longstanding problem with frobnicators not cromulating" doesn't mean much unless you know why a frobnicator should cromulate. However, this interview method can result in some frank and earnest sharing about system weaknesses that the outgoing person may be embarrassed to write down. Meetings like this have to have some structure, though, otherwise we just end up shooting the breeze for an hour and the knowledge passing doesn't actually happen.

My preferred way is to work out a shared handover. So Sally has been running the payroll system, she's leaving, and Fred is getting it. Sally should start redirecting all emails to Fred. If a report fails once or twice a day, Sally can make sure that there's some kind of alert set up, but Fred has to be the one fixing it. This is a fine opportunity for Sally to say something like "yeah I could never figure out how to get these gizmos to be 100%, but maybe you'll be able to sort it out". Humility goes a long way when you're outgoing; got to set Fred up for success. If there's some functionality move in progress - the tax calculations are getting extracted to a third party system - then Fred goes to the next status meeting and Sally doesn't. He'll be clueless, but then he'll know he's clueless, rather than sitting there with glazed eyes while Sally talks shop. Once the meeting is over he can race back to Sally's desk with a list of questions and now the answers will mean something to him.

The last chapter of Dave Copeland's excellent book The Senior Software Engineer talks about an outgoing dev's responsibility to leave a project in capable hands and covers some of this same material: one on one interviews, reviewing current or potential issues, etc. That's assuming that it's an amicable departure, but so is everything in this blog post.

Finally, I think it's helpful if the outgoing person can give the incoming developer a feel for the importance and impact of the job. "You're the key person for the payroll team", "you'll be able to save 200 accountants hours each day", etc. A little vision goes a long way towards motivating a developer to make a system better.

September 22, 2014

Tom CopelandVarious ways to create structs

2014/09/22 04:00 AM

The first chapter of Understanding Computation does a whirlwind tour of Ruby programming - syntax, control structures, etc. One of the constructs the author calls out is the Struct class. I've always used it like this:

Team = Struct.new(:players, :coach)
class Team
  def slogan
    "A stitch in time saves nine"
  end
end
some_team = Team.new(some_players, some_coach)

That is, I'd use the Struct constructor as a factory and assign the result to a constant. Then, if I needed to, I'd reopen the class that had just been created in order to add methods.

The example in the book does it a different way:

class Team < Struct.new(:players, :coach)
  def slogan
    "It takes one to know one"
  end
end

This creates an anonymous Struct, uses it as a superclass, assigns it to a constant, and reopens the new class all in one fell swoop. I poked around various gems and this seems like a pretty common approach - for example, Unicorn::App::Inetd::CatBody does this, and so does Arel::Attributes::Attribute.

The latter approach does result in more noise in the ancestor chain:

irb(main):001:0> Foo = Struct.new(:x) ; Foo.ancestors
=> [Foo, Struct, Enumerable, Object, Kernel, BasicObject]
irb(main):002:0> class Bar < Struct.new(:x) ; end ; Bar.ancestors
=> [Bar, #<Class:0x007ff4abb8b508>, Struct, Enumerable, Object, Kernel, BasicObject]

That's because Struct.new returns an anonymous subclass of Struct, and that subclass hasn't been assigned to a constant, so Ruby has to synthesize a to_s value from the type and object id. You could get around that using the two-arg constructor, but that creates the new class as a constant in the class Struct, which is a little weird:

irb(main):001:0> Struct.new("Bar", :x).ancestors
=> [Struct::Bar, Struct, Enumerable, Object, Kernel, BasicObject]

Sometimes I see code create Struct subclasses using the class keyword without adding methods; for example from WebSocket::Driver:

class OpenEvent < Struct.new(nil) ; end

Is there a reason to do the above rather than OpenEvent = Struct.new(nil)?

Edit: Bruno Michel noted that the Struct constructor also accepts a block:

Foo = Struct.new(:x) do
  def hello
    "hi"
  end
end

This is a nice technique because it results in a clean ancestor chain.

Edit 2: Tom Stuart (the author of 'Understanding Computation') pointed out this great writeup on Structs by James Edward Grey II where he explains why he's using Structs in the way he does. Definitely worth a read, especially with Ara Howard weighing in with a comment about his Map implementation.

Thanks to Thomas Olausson for reviewing this post!

September 12, 2014

Ryan DavisOne Reservoir Nerd Left

2014/09/12 07:00 PM

reservoir_nerd.png

And then there was one. (2,060 days later, a bit slower than the movie).

Unlike last time, this time it’s just me. 23 years experience, 14 (next month) in ruby, I’m available and looking.

I would love to wind up doing more open source work, or helping to optimize your people/processes/codes. I’ll be speaking at gogaruco next week. If you’re nearby, you should come talk to me.

June 04, 2014

RubyLoveIntro to my possible Ruby Book

2014/06/04 07:00 AM

My own history

I have been coding since I was thirteen, when my Father came home with old IBM clone that he traded for during the 80’s barter craze. Being nearly forty years old now, I have stubbed my toe and left it bleeding in more than a dozen languages now. If I were a construction worker instead of a knowledge worker, I would probably be missing fingers, toes, and possibly a hand or leg.

When I look at my whole career in code, I see a common theme. From BASIC, to C to C++, to VB, to Javascript to C# to Ruby I have been writing imperative, stateful code for my whole career. I have lived with loops, state mutation, and C syntax, and was blissfully unaware of anything else as I had skipped the CS degree and opted to go directly into the workforce at eighteen.

I worked through the 90s as the Internet was in it’s very early years. I had a Geocities page for my Warcraft tips and strategies. It probably didn’t blink, but it must have been fugly. I worked for companies that turned email into snail mail, companies who turned snail mail into email, and companies who made adware for Windows machines and a little bit of everything in between.

When I coded in C, we hated those ‘weird’ C++ guys. We would tease them about using a language before the improvement, since it wasn’t ++C. Then I joined that group to write DCOM and COM+ for Intel. We made fun of the VB programmers. Then I became a VB programmer when ASP exploded, since it was essentially VBScript, and I don’t like context switches. Then it was semi-colons that were stupid.

By the time I got to Ruby I hated working with the Microsoft stack. The guy that introduced me to Ruby in 2006 was working in C# with me, and constantly opined at the lack of jobs in Ruby and LISP. We hacked a bit during lunches, but I never had time to get serious about Ruby, as I worked at a startup. Like 80 hours a week worked at a startup.

He never made fun of anything in languages. He didn’t point out differences in the languages in a snide way. Every tool had uses, and every language had interesting points and beautiful aspects. It was almost his attitude alone that made me want to work with Ruby. The language seemed very much like this programmer. Friendly, smart, concise, and there wasn’t a lot of, no you cannot do this, you shouldn’t do that.

I was introduced to the concepts in LISP and some functional programming by him as well. He also was a versed smalltalker. He really opened my eyes. Then I got a Ruby job and moved to Amsterdam for it. I can’t even remember this guy’s name beyond Courtney, but I will forever be grateful to what he shared with me.

Debugging

I have spent more hours in my life with debug statements, than probably anything else, except possibly sleeping or growing older. I used to think that this was just the state of things. You wrote some code, then you ran it, and it either, A) ran and acted as expected or B) politely suggested I write some debug statements to what was really going on.

Debugging is basically detective work. When the code you interact with is convoluted, and obfuscated through constant laziness and ignorance, you constantly have new mysteries to solve. The mystery you really are solving here is not syntax, it is not structure, it is almost always state, and the way that state is formed and managed.

Why state is hard to reason and debug

When I speak of state what I am really speaking of, are memory addresses. In the early days of computers, memory was very pricey and in limited supply due to space. This meant programmers were forced to write efficient code, and efficiency meant small size and fast speed. So developers wouldn’t think twice about reusing a memory address instead of allocating more, it was in fact preferred given the space you needed was equal or less to what was already allocated.

Still today, programmers will reassign a variable over and over, changing it’s state. Memory is now very cheap and machines have lots of capacity, and yet variables are rewritten habitually. And not just setting something else to that memory address, but also changing what was there originally.

So imagine we have a Person class. The class has several pieces of internal state, we will just focus on ‘name’ and ‘favorite_color’, for our example, I will use my Uncle Bob, because he has taught me so much.

So I create a person class instance, Bob. Then I save him off to a database, the how doesn’t really concern us, so I will just use a descriptive stand-in double.

Then later on, I find that I need to fill in the last name as well and then re-save it:

But then later on again, I alter it again, but adding some other field and overwrite it again. And so on and so forth.

Now trying to rationalize about the state of that object is difficult because it changes and changes and there is no audit trail of what it used to be like. It is like a video stream that cannot be rewound.

Immutability

I will now do the same thing as above, but this time, I will create new copies of Bob before I alter his state:

I now need to change his name to his full name, so I create a clone, alter that clone, then save.

Now I change it again, this time because we find out my Uncle’s favorite color is, in fact, green.

But you might say, what about in the database we are ignoring? What about primary keys? It isn’t the concern here, but you could use last updated at and update a join table with a static guid, and the current id, but again, not the point here.

Now I want you to think of this process not as a video stream that cannot be rewound, but a set of many still snapshots. If you were combine the stills, you could recreate the stream again. However, if you want to, you can lay these versions of my Uncle together out in front of you on a desktop and see all states of Uncle Bob before you at once, and not be stuck debugging the current state of him.

In a way, this style of state management gives you an almost superhuman power of all seeing, where you can look at a person and instead of seeing them as they are you can see all versions of them as any given state of time. Pretty cool, next you might be able to dodge bullets!

The death of a generation of programmers

For the entirety of computing up until 2012-2013, memory and speed were doubling every year as noted by “Moore’s law”. During that time, the amount of abstraction, or layer on layer of code to make dealing with the complexities of voltages and gates and registers easier on up to windowing tools and the GUI.

These two have gone at about the same pace, the latter being limited by the former. However, seeing “ahead” of time by measuring the slow of growth per cycle, the CPU industry changed gears and instead of making the single core much faster, they bundled cores together to create a multi-core architecture.

Programmers who do not adapt to this new age of multi-core will cease to be hireable and their families will starve like in a Dickens’ novel. Do not let this happen to you!

The multi-core, memory address problem

Now that processors are no longer growing exponentially, and multi-core architecture has taken over, we have threads that operate against different cores. One core can accessed at the same time as other cores, which is where they get their speed improvement over the single core.

Yet there is only one memory register. Well, this is not entirely true, but there is likely only one memory register your application has access to. That means you could have something like the following:

So I create Bob again, but this time, pretend this is going to happen on core 1:

Bob’s favorite color is about to change again, but it will be done by core 2:

Then core 1 tries to access their copy again, but wait…. it is not in an unexpected state. It was modified by another core, and this can create race conditions that include memory corruption.

That is a highly over-simplified explanation because I am not a hardware engineer nor are the absolute internals of the CPU our domain. What you should gather here is that a developer can no longer ignore threads and multiple cores or how they might interact with each other. We need to write software that naturally works in this brave new world.

To do so, we dig into the braver old world, where smart people have already figured out the problem and the solution before it even existed. In this book I will help guide you on a path from the imperative to the explicit, and the object oriented to the functional. These concepts are not polar, they are orthogonal, that is, you can plug them into each other for a hybrid experience.

Let’s embark into the magical world of value objects, immutability, hybrid functional programming, threads, and homoiconicity, but do so in the comfortable and familiar, friendly world of Ruby!

June 01, 2014

RubyLoveRubyLoveLive Sunday Sessions

2014/06/01 07:00 AM

This week we started to outside-in, TDD some code towards a static website generator. Here is the video!

May 31, 2014

RubyLoveRefactoring conditionals out of your code

2014/05/31 07:00 AM

Today I will be doing something a little different.

I have prepared some Ruby code to show how conditionals can breed like rabbits, and how to spay and neuter those bunnies before it gets out of control.

Along with the code, there is a screencast or as I will call it, a RubyLovecast. In the video, I will walk you through several revisions of the code, first showing how conditionals grow, then how to take a weedwacker to them.

I am looking for feedback, so tweet that shit at @thatrubylove and hit me hard. My plan is to take a weeks worth of feedback and then ‘reshoot’ the screencast, taking your input into account.

So until next week’s rubylovecast…. love and hacking

May 26, 2014

RubyLoveI am now a till time Ruby/Rails coach

2014/05/26 07:00 AM
  • Please note, I am not accepting students…. unless I suppose you have an exceptional mind.

I have to apologize for being away so long. I ended a relationship that had drug on for far too long and going nowhere, and at the same time I decided that I will no longer accept clients that refuse to place the proper amount of importance on quality and training.

The two of those catalysts essentially derailed my life and then gave me a hovercraft instead! I started mentoring @ codementor.io and then was convinced by a friend of mine to also become a full time mentor @ bloc.io.

Needless to say, I have been very, very busy.

Along with now mentoring about 12 people from various sources (we do 2 hour sessions monday through thursday and students chose from the morning or evening session) I have been speaking with awesome developers who have been my heroes.

I have been gleening their thoughts about my thoughts and about their own and others thoughts… honestly, it is a lot like Primer of the mind for developers. It has been a blast. For instance, here is myself and one of my heroes, J.B. Rainsberger discussing my Ruby Manifesto.

I promise I will post another in depth very soon. My question for my readers…. What do you want to learn? Also, would you like to join me every Sunday 12pm to 2pm CST for live sessions?

Let me know on twitter please. @thatrubylove

May 21, 2014

Charles Oliver NutterJRubyConf.eu 2014!

Charles Nutter (noreply@blogger.com) @ 2014/05/21 11:44 AM
I'm thrilled to announce that we'll have another edition of JRubyConf.eu this year!

Last year's event was a great success. We had a two-day conference in Berlin immediately before Eurucamp 2013, with two speakers from the core team (myself and Tom Enebo) and a whopping fifteen non-core speakers. A great event was had by all.

This year, we've decided to pull the event back to its roots, as part of Eurucamp 2014. We'll return to the single-track, single-day event co-located with and immediately preceding Eurucamp on 1st August. We really wanted to bring JRuby back to Rubyists, and we're looking forward to hanging out at Eurucamp the whole weekend!

Why not visit Eurucamp early and spend a day learning about JRuby with the best JRubyists in Europe?

If you're interested in attending, tickets are available for only €99 at the Eurucamp ticket site now!

We're also looking for speakers from the JRuby community. You can submit to the CFP (which ends Sunday 28 May) using the Eurucamp CFP app.

Looking forward to seeing you at JRubyConf and Eurucamp this summer!

May 02, 2014

Ryan DavisCatching Up

2014/05/02 07:00 PM

I realize that is has been a hellishly long time since I last updated this thing.

When last we met I brain-dumped most of my emacs setup and then promptly disappeared. I know that at least a few of you have used that to good effect and that’s been encouraging.

But since then, I’ve had some fun.

Gems:

I’ve done 63 gem releases in the last 6ish months, mostly thanks to a bunch of generous contributors. In all it includes 6 major, 109 minor, and 65 bug fixes. Fan-fucking-tastic! This brings a bunch of my more stable projects up to date for 2.0 & 2.1: autotest-rails, flog, graph, image_science, minitest-bacon, minitest-focus, minitest_tu_shim, omnifocus, omnifocus-github, rubygems-sandbox, RubyInline, vlad, ZenTest, and ZenWeb.

I spent a lot of time focusing on 2 of my favorite gems: hoe & minitest.

Hoe & Minitest

Hoe makes it a breeze to manage all my gems and really is the reason why I can do what I do with so many projects. It is incredibly important to me that my projects don’t have any duplication across them. By using Hoe, any changes that happen across my projects happen in one place only. DRY at its finest.

Minitest went through a complete rewrite for version 5 early last year. Thanks to a lot of people for contributing to minitest and helping make it what it is today. The architecture is vastly improved over minitest 4 and it shows. Ever since the v5 rewrite, I absolutely love this project and thank you all for it.

Language Love

ruby_parser has seen some serious love in that time, as well as some love for its supporting projects.

For the supporting projects, sexp_processor got some nice refactorings from other projects, including adding MethodBasedSexpProcessor from Flog. ruby2ruby was extended to 2.0 and 2.1 support. The nice thing about these guys was how cleanly designed they were to begin with so they didn’t see too much over the last year.

But… For ruby_parser… which saw 17(!!) releases in 2012 but only 5 in 2013… I got a bit burnt out on the project, but I’ve had a bit of a comeback. I’ve done 5 releases already this year. I’ve added support for 2.0 and initial support for 2.11. But most importantly, I rewrote the hand-written lexer from scratch using a brand new project: oedipus_lex.

Oedipus Lex

Oedipus Lex is my independent lexer fork of Rexical. Rexical was in turn a fork of Rex. While Oedipus Lex is a cleanroom re-implementation of rexical, but it is also massively extended including proper ruby regexp support, predicate method prerequisites, grouping for extra performance, shortcuts on state changes, and action methods. I love this project for what it has done to the maintainability of ruby_parser, but more importantly, Oedipus Lex makes ruby_parser roughly 60% faster than my highly-optimized hand-written lexer.

Talks:

I had a lot of fun on the conference circuit in the last year.

Last year, I gave Let’s Write an Interpreter! at both MWRC and GoGaRuCo. I spoke about Minitest & Rails in Chicago. And I spoke about craftsmenship and tools at cascadia.

So far this year I’ve given one talk at MWRC, Nerd Party v3.1, which was my take on Seattle.rb’s format and how it can work for you.

  1. Please! if you want to help out with ruby_parser (or flog/flay/etc), I could use a PR for the new rational & imaginary lexemes!

October 24, 2013

Ryan Davis9000 Commits

2013/10/24 07:00 PM

I just committed the 9000th commit to my perforce repo. Woot!

% p4 counter change
9000

Time between 1000s. You can see some burnout/slowdown between 2009-06 to 2012-02, but it’s been maintaining >1000 per year since then.

% p4 changes | grep "Change .000" | cut -f4 -d\   | ruby -rdate -e 'puts $stdin.read.lines.map { |s| Date.parse s }.each_cons(2).map { |a, b| [a, (a - b).to_i].join " " }'
2013-10-24 335
2012-11-23 294
2012-02-03 436
2010-11-24 540
2009-06-02 358
2008-06-09 489
2007-02-06 655
2005-04-22 1065

Granted, # of commits or commits / day is a stupid metric to follow if you’re trying to measure productivity… but since I’m not trying to game my own system, this ain’t bad.

October 18, 2013

Ryan DavisTime Machine Repair

2013/10/18 07:00 PM

Turn off time machine on all involved parties, then:

% sudo chflags -R nouchg greed.sparsebundle
% sudo emacs -q -nw greed.sparsebundle/com.apple.TimeMachine.MachineID.plist

  # Setting VerificationState to 0 and removing RecoveryBackupDeclinedDate

% sudo hdiutil attach -nomount -noverify -noautofsck -readwrite greed.sparsebundle
% tail -f /var/log/fsck_hfs.log

  # wait for regular check to finish... if it passes, go for it, otherwise:

% sudo fsck_hfs -drfy /dev/diskXs2

  # once you're done:

% hdiutil detach  /dev/diskXs2

For me, everything fixed itself so I was able to skip the last step.

From: http://www.garth.org/archives/2010,07,16,124,fixing-time-machine-sparsebundle-network-backup-errors.html

October 15, 2013

Ryan DavisHappy Birfday to me!

2013/10/15 10:04 PM

Today is my Thirteen Year Anniversary with Ruby! Yay!

780 gem releases (up by 150), 8982 commits (up by 1146), and bazillions of test runs later, and I’m still going strong. Rawr!

July 30, 2013

Ryan DavisTriage Applescript for OmniFocus

2013/07/30 07:00 PM

I have a bunch of omnifocus tasks that tell me to go review my projects’ issue trackers. Each one has a note with a url in it that is hyperlinked. Unfortunately, omnifocus doesn’t let you cmd-click the url to open in the background. It’s incredibly aggravating that you have to flip back and forth between safari and omnifocus over and over. It is even worse when you do it across spaces (I always have omnifocus fullscreened).

I wrote this applescript (blech) because I couldn’t take it anymore and omnigroup has ignored my bug report for 2+ years now. All it does is go through each task in the current window and open a new Safari tab for each note that starts with “http”.

I hope it helps bring sanity to someone else.

tell application "Safari"
	activate
	
	tell application "OmniFocus"
		set urls to note of value of leaves of content of window 1
	end tell
	
	make new document
	
	repeat with aUrl in urls
		if aUrl starts with "http" then
			make new tab at end of window 1 with properties {URL:aUrl}
		end if
	end repeat
	
	delete tab 1 of window 1
	
end tell

I have this in the script menu titled “Triage”. Enjoy

Last updated at 2014/11/23 02:18 AM | Presented by Aredridel and Christoffer Sawicki | Hosted at The Internet Company