Category Archives: Programming

Fixing the Gemfile not found (Bundler::GemfileNotFound) error

I was working on an app (bundler, unicorn, Rails3) that had a strange deploy issue. Five deploys (using capistrano) after our unicorn processes had started unicorn would fail to restart, this is the capistrano output:

* executing `deploy:restart'
* executing `unicorn:restart'
* executing "cd /u/apps/dash/current && unicornctl restart"
servers: ["stats-01"]
[stats-01] executing command
** [out :: stats-01] Restarting pid 15160...
** [out :: stats-01] PID 15160 has not changed, so the deploy may have failed. Check the unicorn log for issues.

I checked the unicorn log for details:

I, [2011-08-02T15:59:32.498371 #11790] INFO -- : executing ["/u/apps/dash/shared/bundle/ruby/1.9.1/bin/unicorn", "/u/apps/dash/current/config.ru", "-Dc", "/u/apps/dash/current/config/unicorn.conf.rb", "-E", "production"] (in /u/apps/dash/releases/20110802155921)
/opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler/definition.rb:14:in `build': /u/apps/dash/releases/20110802152815/Gemfile not found (Bundler::GemfileNotFound)
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:136:in `definition'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:124:in `load'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:107:in `setup'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler/setup.rb:17:in `'
from :29:in `require'
from :29:in `require'
E, [2011-08-02T15:59:32.682270 #11225] ERROR -- : reaped # exec()-ed

Sure enough there’s the exception, “Gemfile not found (Bundler::GemfileNotFound)“, and the file referenced (/u/apps/dash/releases/20110802152815/Gemfile) didn’t exist. The directory that was being looked in (20110802152815) was from a previous deploy and had been rotated off the filesystem. We keep five historical deploys so that explained why the problem only happened five deploys after a full unicorn restart.

I suspected an environment variable was getting set somewhere, and never updated, so I added some debugging to our unicorn.conf.rb file:

ENV.each{ |k, v| puts "#{k}:tt#{v}" }

I then restarted the unicorns fully and tailed the unicorn log file while deploying the app. Sure enough one of the environment variables stuck out:

BUNDLE_GEMFILE: /u/apps/dash/releases/20110802165726/Gemfile

I deployed again, it remained the same, still pointing to /u/apps/dash/releases/20110802165726/Gemfile. I continued to deploy until release 20110802165726 was rotated off the filesystem and up pops the error again. This looks like the problem.

I committed a change to our unicorn.conf.rb that set the BUNDLE_GEMFILE variable explicitly in the before_exec block:

before_exec do |server|
ENV['BUNDLE_GEMFILE'] = "/u/apps/dash/current/Gemfile"

end

Over 5 deploys later and the env var is still set to /u/apps/dash/current/Gemfile and there are no more errors. Let me know if you found this useful!

Notes

  • There may be other issues that cause errors of this type, this was just the solution that worked for us, YMMV.
  • There may be better places to set the environment variable other than the unicorn.conf.rb, I’m open to suggestions (we’re using bluepill, I may be able to set it there for intance).
Update: I’ve changed this on our systems so the environment variable is set in bluepill, it works the same.
Tagged , ,

Rails 2 -> 3 undefined method `html_safe’ for nil:NilClass error

I am converting the Recycling Group Finder site from Rails 2 to Rails 3 and though it has mostly gone to plan I was temporarily held up by this error which I was getting on some pages:

The error was hard to track down as the error message wasn’t very descriptive, but in the end it turned out to be caused by a comment. I am using content_for blocks to generate sections of page content and for a long section I had added a comment to the end of the block to help be know which block was closing:

It turns out that this ‘# some_section’ comment was the problem, possibly because of the change to erubis in Rails 3. Removing the comment caused the page to start working again:

I hope this page helps short-cut the debugging for anyone else that is bitten by this issue.

Tagged , ,

Running Unicorn under supervise/daemontools

I’m running a Rails3 app under daemontools, this is how I did it.

First, install then start daemontools and Unicorn. This is left as an exercise for the reader, it was easy under CentOS 5. You should end up with a /service directory on your server.

Run “mkdir /service/my_service_name && cd /service/my_service_name && ls”. If you have the right processes running you should have a “supervise” subdirectory automatically created for you under your new directory.

You should be in the “/service/my_service_name” directory. In order to get anything to run you need to create a “run” script. Here’s mine, you can copy and modify it for your own use. If you’re running Unicorn and you improve the script I’d appreciate feedback in the comments:

#!/bin/sh
cd /home/www/foo/current && unicorn_rails -E staging -c /home/www/foo/shared/config/unicorn.conf

Make sure the script is executable. If you have done everything right you should have a Unicorn process running. You should see something similar to this in the output of “ps auxww -H”:

If that’s not the sort of thing you see, check the logs. Lastly I added this cap snippet to kill the process on deploy and let supervise handle re-spawning it:

Done!

Why daemontools?

There is some propaganda on the daemontools site, but I used it because it’s quick to get going and in my experience reliable. I use monit a lot at Engine Yard, but there are no recent monit packages easily available for CentOS that I have found. One drawback is that you don’t get any of the resource monitoring that monit provides, such as http checks, memory checks etc. You’d have to implement this yourself, but in this instance that’s the tradeoff you make for simplicity.

CentOS 5 update:

If you get an error that looks like this:

Follow these instructions to fix the issue:

Simple http ping program in Ruby

Just a little http ping program I wrote to check request latency during my testing for zero-downtime rails deploys with Unicorn.

Example:

pinky:~ will$ ruby httping.rb willj.net
200 :   0.000000   0.000000   0.000000 (  0.761449)
200 :   0.010000   0.000000   0.010000 (  0.535888)
200 :   0.000000   0.010000   0.010000 (  0.800904)
200 :   0.000000   0.000000   0.000000 (  0.530763)
200 :   0.000000   0.000000   0.000000 (  0.811362)
200 :   0.000000   0.000000   0.000000 (  0.557995)
200 :   0.000000   0.010000   0.010000 (  0.774484)

Tagged

Unicorn and bundler issue

I am using unicorn with a rails3 app that uses bundler and I hit a weird error when trying to start a server:

`rescue in load_spec_files': git://github.com/odorcicd/authlogic.git (at rails3) is not checked out. Please run `bundle install` (Bundler::GitError)

The background to this is that I am starting unicorn_rails as root then dropping privs in the after_fork block, this is the relevant section from my unicorn.conf file:

after_fork do |server, worker|
  # …
  uid, gid = Process.euid, Process.egid
  user, group = 'will', 'will'
  target_uid = Etc.getpwnam(user).uid
  target_gid = Etc.getgrnam(group).gid
  worker.tmp.chown(target_uid, target_gid)

  if uid != target_uid || gid != target_gid
    Process.initgroups(user, target_gid)
    Process::GID.change_privilege(target_gid)
    Process::UID.change_privilege(target_uid)
  end
end

This is basically the same as the GitHub unicorn config.

Back to the problem. It works fine for all the standard rubygems.org gems, but was failing on the dependencies I had specified as ‘:git’ deps, for instance authlogic:

gem 'authlogic', :git => 'git://github.com/odorcicd/authlogic.git', :branch => 'rails3'

I checked where these gem/git dependencies were being stored and saw the problem:

[will@server current]$ bundle show activerecord
/usr/local/lib/ruby/gems/1.9.1/gems/activerecord-3.0.0

[will@server current]$ bundle show authlogic
/home/will/.bundler/ruby/1.9.1/authlogic-a087ad0

The :git dependencies were being bundled to a subdirectory of my home dir (the same user that does the deploy, and therefore the initial bundle install. The unicorn_rails process starts as root and has already run Bundler.setup before it forks, so Bundler.setup gets run as root, and the root user has no idea where the :git dependencies are, hence the error.

I solved this by adding a call to Bundler.setup from within the unicorn.conf after_fork block:

after_fork do |server, worker|
  Bundler.setup
  # …
  uid, gid = Process.euid, Process.egid
  user, group = 'will', 'will'
  target_uid = Etc.getpwnam(user).uid
  target_gid = Etc.getgrnam(group).gid
  worker.tmp.chown(target_uid, target_gid)

  if uid != target_uid || gid != target_gid
    Process.initgroups(user, target_gid)
    Process::GID.change_privilege(target_gid)
    Process::UID.change_privilege(target_uid)
  end
end

It seems to work fine, better solutions welcome!

Tagged , ,

Setting custom request headers with objective resource

For the last few evenings I have been working on a native Mac app for Staging that talks to a Rails restful API. Rather than write the code access the resource URLs myself I thought it would be pretty cool to use Objective Resource, the Objective-c library for accessing Rails-style restful APIs, from the site:

ObjectiveResource is an Objective-C port of Ruby on Rails’ ActiveResource. It provides a way to serialize objects to and from Rails’ standard RESTful web-services (via XML or JSON) and handles much of the complexity involved with invoking web-services of any language from the iPhone.

There are a couple of problems with this approach however. First ObjectiveResource is designed to work on iOS (iPhones, iPod touches and iPads) only, and second it expects access to be controlled by a username and password. The API for Staging uses an X-API-Token HTTP header sent with each request.

Getting ObjectiveResource to work on native Mac

This was pretty simple, just replace all instances of

#import <UIKit/UIKit.h>

with:

#import <AppKit/AppKit.h>

So far nothing has broken.

Getting ObjectiveResource to send a custom header

This was a little trickier and required some more invasive code changes. In Connection.h declare two new methods:

And define them in Connection.m:

Next you need to modify the request that gets called for every GET request:

The addition here is basically a for loop to loop over every header that we have configured and add them to the request object. Finally somewhere before you make any requests in your code:

Done! Now every request that GETs made will have the custom header sent along with it. You can obviously set any number of arbitrary headers using this method too, you aren’t limited to my X-API-Token. It is left as an exercise to the reader to implement the same for POST requests.

It would be nice to see these changes rolled into the ObjectiveResource framework in some way.

Tagged , ,

Basic zeroMQ Ruby example

Update: As Jake pointed out in the comments you obviously need zero MQ installed for this example to work. Just ‘brew install zeromq’ or ‘port install zmq’ on OS X, or use your Linux package manager.

I couldn’t find may examples of zeroMQ usage in Ruby so here is a basic sender/consumer I made to test it. First install the ‘zmq’ gem:

gem install zmq --no-ri --no-rdoc

Now start a worker, you can start as many as you want:

Now stick some messages on the queue:

You should get messages distributed to all the worker processes you started up. Pretty simple!

Tagged ,

map-reduce using mongoid

It took me a while to work out how to use map-reduce in Ruby using mongoid so I thought I’d share it here in-case it helps anyone else get there quicker. I start out with a model that includes Mongoid::Document:

To map-reduce across the collection I need to define a map and reduce function in javascript then run the on the collection:

I can roll this into my VisitorSession model:

This obviously makes it easier to call:

>> VisitorSession.first(:conditions => {:project_id => '2f5178'}).visits_for_project
=> 1

Tagged ,

fixing the rspec2 and rails3 “there is already a transaction in progress” error

I upgraded an app to rspec2 and rails3 recently and ran into a problem. I run each test in a transaction which rolls back after each test has finished. This works fine normally, but It seemed that after the upgrade the first failing test would cause this per-test transaction to not get rolled back so all the subsequent tests would run in the same transaction.

This caused the tests so fail with validation errors as test data was getting re-inserted, and a validates_uniqueness_of validation was complaining each time, note the second failing test, this test should have passed but failed with a “Validation failed: Name has already been taken” error:

It turns out that what was actually happening was that an exception raised in the after(:each) block was causing the rollback to fail and the failing test was just masking the error. This was my after(:each) block:

This code fails when there is no directory to remove and though this worked fine in the older version of rspec (rspec just ignored the error) with rspec2 it caused the issues discussed above. The solution (because I don’t care if the file exists or not before I delete it) is as simple as adding a condition:

Tagged , ,

Flushing memcached servers from Ruby

In Flushing memcached servers the easy way I highlighted a way to flush a memcached server without restarting it:

$ echo ”flush_all” | nc localhost 11211

However I almost never use the actual shell version of this, mostly I do the equivalent in Ruby by opening up a socket and communicating through that. Here’s a simple example:

socket = TCPSocket.new( '127.0.0.1', 11211 )
socket.write( "flush_allrn" )
result = socket.recv(2)
puts "Success!" if result == 'OK'
socket.close

Tagged