Fixing the chef “Error Syncing Cookbooks – EOFError: end of file reached” error

I ran into a problem recently with some corrupt chef cookbooks after a recent chef upgrade, I was getting an “Error Syncing Cookbooks – EOFError: end of file reached” error on every chef run on all nodes:

================================================================================

Error Syncing Cookbooks:
================================================================================

Unexpected Error:
-----------------
EOFError: end of file reached

[2013-01-21T18:17:27+00:00] ERROR: Running exception handlers
[2013-01-21T18:17:27+00:00] FATAL: Saving node information to /var/cache/chef/failed-run-data.json
[2013-01-21T18:17:27+00:00] ERROR: Exception handlers complete
[2013-01-21T18:17:27+00:00] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
[2013-01-21T18:17:27+00:00] FATAL: EOFError: end of file reached

I’d already deleted the cookbooks:

knife cookbook delete cookbook_name

and re-uploaded them, but this didn’t work. It took a purge to completely delete the cookbook from the chef server:

knife cookbook delete --purge cookbook_name

In the end I had so many cookbooks that were somehow corrupt that I bulk purged them and re-uploaded them all:

knife cookbook bulk delete --purge '.+'

Tagged

3 thoughts on “Fixing the chef “Error Syncing Cookbooks – EOFError: end of file reached” error

  1. 22blaze says:

    I have been struggling with a very similar issue for over 2 days. The key was the –purge. After purging and reloading with cookbooks, I am back in business.
    *Thank you for posting this.*

  2. dvb says:

    Ruby and chef 10.x are full of nasty little bombs…

    This EOFError problem being one of them…

    In my case, I pull cookbook changes into a local repo on our chef-server and tried to upload them to chef-server, we’re trying hard to keep things in sync…. Well, I didn’t mind disk utilization before doing so and found it at 90% afterward, which caused rabbitmq to complain that it didn’t have enough working spacing to handle the largish files resulting from the cookbook upload.

    Spinning up a new instance with more space, bring chef-repo up to date and uploading all the changed cookbooks to chef-server should have been the cure… After all I got no errors doing any of this… but launching a new machine with knife, I found EOFError and also navigating cookbook code in chef-server-webui. I stopped and restarted the variety of services but still the error occurred.

    First it took time to diagnose…

    Then I found that I couldn’t quickly upload all the cookbooks into chef-server after purging!! Two complexities: (a) accounting for cookbook versions (b) dependencies kept getting in the way no way to really turn them off! — `knife cookbook upload –all` still checks dependencies even though I did not specify –include-dependencies.

    My resolution (for now) create/commit a Makefile organizing the work so I can recover more smoothly next time…
    (a) knife cookbook bulk delete ‘^[a-z]*’ –yes –purge
    (b) stop services: chef-solr chef-expander chef-server
    (c) rm -Rf /var/lib/chef/cookbook_index/* – so I get a fresh index
    (d) restart the stopped services
    (e) load the cookbooks and their versions
    (f) knife index rebuild
    (g) navigate chef-server-webui – error should be gone
    (h) ls -l /var/lib/chef/cookbook_index/ – new folders with current timestamps should appear

    I hope this helps somebody cause it’s too easy to waste alot of time on stupid chef stuff…

    • chefman says:

      Thanks for posting this–absolutely helped me out of a jam tonight! I’m saving your post just in case this bites me (or someone I know) again.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.