Tag Archives: unicorn

Fixing the Gemfile not found (Bundler::GemfileNotFound) error

I was working on an app (bundler, unicorn, Rails3) that had a strange deploy issue. Five deploys (using capistrano) after our unicorn processes had started unicorn would fail to restart, this is the capistrano output:

* executing `deploy:restart'
* executing `unicorn:restart'
* executing "cd /u/apps/dash/current && unicornctl restart"
servers: ["stats-01"]
[stats-01] executing command
** [out :: stats-01] Restarting pid 15160...
** [out :: stats-01] PID 15160 has not changed, so the deploy may have failed. Check the unicorn log for issues.

I checked the unicorn log for details:

I, [2011-08-02T15:59:32.498371 #11790] INFO -- : executing ["/u/apps/dash/shared/bundle/ruby/1.9.1/bin/unicorn", "/u/apps/dash/current/config.ru", "-Dc", "/u/apps/dash/current/config/unicorn.conf.rb", "-E", "production"] (in /u/apps/dash/releases/20110802155921)
/opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler/definition.rb:14:in `build': /u/apps/dash/releases/20110802152815/Gemfile not found (Bundler::GemfileNotFound)
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:136:in `definition'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:124:in `load'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:107:in `setup'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler/setup.rb:17:in `'
from :29:in `require'
from :29:in `require'
E, [2011-08-02T15:59:32.682270 #11225] ERROR -- : reaped # exec()-ed

Sure enough there’s the exception, “Gemfile not found (Bundler::GemfileNotFound)“, and the file referenced (/u/apps/dash/releases/20110802152815/Gemfile) didn’t exist. The directory that was being looked in (20110802152815) was from a previous deploy and had been rotated off the filesystem. We keep five historical deploys so that explained why the problem only happened five deploys after a full unicorn restart.

I suspected an environment variable was getting set somewhere, and never updated, so I added some debugging to our unicorn.conf.rb file:

ENV.each{ |k, v| puts "#{k}:tt#{v}" }

I then restarted the unicorns fully and tailed the unicorn log file while deploying the app. Sure enough one of the environment variables stuck out:

BUNDLE_GEMFILE: /u/apps/dash/releases/20110802165726/Gemfile

I deployed again, it remained the same, still pointing to /u/apps/dash/releases/20110802165726/Gemfile. I continued to deploy until release 20110802165726 was rotated off the filesystem and up pops the error again. This looks like the problem.

I committed a change to our unicorn.conf.rb that set the BUNDLE_GEMFILE variable explicitly in the before_exec block:

before_exec do |server|
ENV['BUNDLE_GEMFILE'] = "/u/apps/dash/current/Gemfile"

end

Over 5 deploys later and the env var is still set to /u/apps/dash/current/Gemfile and there are no more errors. Let me know if you found this useful!

Notes

  • There may be other issues that cause errors of this type, this was just the solution that worked for us, YMMV.
  • There may be better places to set the environment variable other than the unicorn.conf.rb, I’m open to suggestions (we’re using bluepill, I may be able to set it there for intance).
Update: I’ve changed this on our systems so the environment variable is set in bluepill, it works the same.
Tagged , ,

Unicorn and bundler issue

I am using unicorn with a rails3 app that uses bundler and I hit a weird error when trying to start a server:

`rescue in load_spec_files': git://github.com/odorcicd/authlogic.git (at rails3) is not checked out. Please run `bundle install` (Bundler::GitError)

The background to this is that I am starting unicorn_rails as root then dropping privs in the after_fork block, this is the relevant section from my unicorn.conf file:

after_fork do |server, worker|
  # …
  uid, gid = Process.euid, Process.egid
  user, group = 'will', 'will'
  target_uid = Etc.getpwnam(user).uid
  target_gid = Etc.getgrnam(group).gid
  worker.tmp.chown(target_uid, target_gid)

  if uid != target_uid || gid != target_gid
    Process.initgroups(user, target_gid)
    Process::GID.change_privilege(target_gid)
    Process::UID.change_privilege(target_uid)
  end
end

This is basically the same as the GitHub unicorn config.

Back to the problem. It works fine for all the standard rubygems.org gems, but was failing on the dependencies I had specified as ‘:git’ deps, for instance authlogic:

gem 'authlogic', :git => 'git://github.com/odorcicd/authlogic.git', :branch => 'rails3'

I checked where these gem/git dependencies were being stored and saw the problem:

[will@server current]$ bundle show activerecord
/usr/local/lib/ruby/gems/1.9.1/gems/activerecord-3.0.0

[will@server current]$ bundle show authlogic
/home/will/.bundler/ruby/1.9.1/authlogic-a087ad0

The :git dependencies were being bundled to a subdirectory of my home dir (the same user that does the deploy, and therefore the initial bundle install. The unicorn_rails process starts as root and has already run Bundler.setup before it forks, so Bundler.setup gets run as root, and the root user has no idea where the :git dependencies are, hence the error.

I solved this by adding a call to Bundler.setup from within the unicorn.conf after_fork block:

after_fork do |server, worker|
  Bundler.setup
  # …
  uid, gid = Process.euid, Process.egid
  user, group = 'will', 'will'
  target_uid = Etc.getpwnam(user).uid
  target_gid = Etc.getgrnam(group).gid
  worker.tmp.chown(target_uid, target_gid)

  if uid != target_uid || gid != target_gid
    Process.initgroups(user, target_gid)
    Process::GID.change_privilege(target_gid)
    Process::UID.change_privilege(target_uid)
  end
end

It seems to work fine, better solutions welcome!

Tagged , ,