Will writes

I was working on an app (bundler, unicorn, Rails3) that had a strange deploy issue. Five deploys (using capistrano) after our unicorn processes had started unicorn would fail to restart, this is the capistrano output:

* executing `deploy:restart'
* executing `unicorn:restart'
* executing "cd /u/apps/dash/current && unicornctl restart"
servers: ["stats-01"]
[stats-01] executing command
** [out :: stats-01] Restarting pid 15160...
** [out :: stats-01] PID 15160 has not changed, so the deploy may have failed. Check the unicorn log for issues.

I checked the unicorn log for details:

I, [2011-08-02T15:59:32.498371 #11790] INFO -- : executing ["/u/apps/dash/shared/bundle/ruby/1.9.1/bin/unicorn", "/u/apps/dash/current/config.ru", "-Dc", "/u/apps/dash/current/config/unicorn.conf.rb", "-E", "production"] (in /u/apps/dash/releases/20110802155921)
/opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler/definition.rb:14:in `build': /u/apps/dash/releases/20110802152815/Gemfile not found (Bundler::GemfileNotFound)
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:136:in `definition'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:124:in `load'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler.rb:107:in `setup'
from /opt/ruby/lib/ruby/gems/1.9.1/gems/bundler-1.0.15/lib/bundler/setup.rb:17:in `'
from :29:in `require'
from :29:in `require'
E, [2011-08-02T15:59:32.682270 #11225] ERROR -- : reaped # exec()-ed

Sure enough there’s the exception, “Gemfile not found (Bundler::GemfileNotFound)“, and the file referenced (/u/apps/dash/releases/20110802152815/Gemfile) didn’t exist. The directory that was being looked in (20110802152815) was from a previous deploy and had been rotated off the filesystem. We keep five historical deploys so that explained why the problem only happened five deploys after a full unicorn restart.

I suspected an environment variable was getting set somewhere, and never updated, so I added some debugging to our unicorn.conf.rb file:

ENV.each{ |k, v| puts "#{k}:tt#{v}" }

I then restarted the unicorns fully and tailed the unicorn log file while deploying the app. Sure enough one of the environment variables stuck out:

BUNDLE_GEMFILE: /u/apps/dash/releases/20110802165726/Gemfile

I deployed again, it remained the same, still pointing to /u/apps/dash/releases/20110802165726/Gemfile. I continued to deploy until release 20110802165726 was rotated off the filesystem and up pops the error again. This looks like the problem.

I committed a change to our unicorn.conf.rb that set the BUNDLE_GEMFILE variable explicitly in the before_exec block:

before_exec do |server|
ENV['BUNDLE_GEMFILE'] = "/u/apps/dash/current/Gemfile"
…
end

Over 5 deploys later and the env var is still set to /u/apps/dash/current/Gemfile and there are no more errors. Let me know if you found this useful!

Notes

• There may be other issues that cause errors of this type, this was just the solution that worked for us, YMMV.
• There may be better places to set the environment variable other than the unicorn.conf.rb, I’m open to suggestions (we’re using bluepill, I may be able to set it there for intance).