Fixing the chef “Error Syncing Cookbooks – EOFError: end of file reached” error

I ran into a problem recently with some corrupt chef cookbooks after a recent chef upgrade, I was getting an “Error Syncing Cookbooks – EOFError: end of file reached” error on every chef run on all nodes:


Error Syncing Cookbooks:

Unexpected Error:
EOFError: end of file reached

[2013-01-21T18:17:27+00:00] ERROR: Running exception handlers
[2013-01-21T18:17:27+00:00] FATAL: Saving node information to /var/cache/chef/failed-run-data.json
[2013-01-21T18:17:27+00:00] ERROR: Exception handlers complete
[2013-01-21T18:17:27+00:00] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
[2013-01-21T18:17:27+00:00] FATAL: EOFError: end of file reached

I’d already deleted the cookbooks:

knife cookbook delete cookbook_name

and re-uploaded them, but this didn’t work. It took a purge to completely delete the cookbook from the chef server:

knife cookbook delete --purge cookbook_name

In the end I had so many cookbooks that were somehow corrupt that I bulk purged them and re-uploaded them all:

knife cookbook bulk delete --purge '.+'


6 thoughts on “Fixing the chef “Error Syncing Cookbooks – EOFError: end of file reached” error

  1. 22blaze says:

    I have been struggling with a very similar issue for over 2 days. The key was the –purge. After purging and reloading with cookbooks, I am back in business.
    *Thank you for posting this.*

  2. dvb says:

    Ruby and chef 10.x are full of nasty little bombs…

    This EOFError problem being one of them…

    In my case, I pull cookbook changes into a local repo on our chef-server and tried to upload them to chef-server, we’re trying hard to keep things in sync…. Well, I didn’t mind disk utilization before doing so and found it at 90% afterward, which caused rabbitmq to complain that it didn’t have enough working spacing to handle the largish files resulting from the cookbook upload.

    Spinning up a new instance with more space, bring chef-repo up to date and uploading all the changed cookbooks to chef-server should have been the cure… After all I got no errors doing any of this… but launching a new machine with knife, I found EOFError and also navigating cookbook code in chef-server-webui. I stopped and restarted the variety of services but still the error occurred.

    First it took time to diagnose…

    Then I found that I couldn’t quickly upload all the cookbooks into chef-server after purging!! Two complexities: (a) accounting for cookbook versions (b) dependencies kept getting in the way no way to really turn them off! — `knife cookbook upload –all` still checks dependencies even though I did not specify –include-dependencies.

    My resolution (for now) create/commit a Makefile organizing the work so I can recover more smoothly next time…
    (a) knife cookbook bulk delete ‘^[a-z]*’ –yes –purge
    (b) stop services: chef-solr chef-expander chef-server
    (c) rm -Rf /var/lib/chef/cookbook_index/* – so I get a fresh index
    (d) restart the stopped services
    (e) load the cookbooks and their versions
    (f) knife index rebuild
    (g) navigate chef-server-webui – error should be gone
    (h) ls -l /var/lib/chef/cookbook_index/ – new folders with current timestamps should appear

    I hope this helps somebody cause it’s too easy to waste alot of time on stupid chef stuff…

    • chefman says:

      Thanks for posting this–absolutely helped me out of a jam tonight! I’m saving your post just in case this bites me (or someone I know) again.

  3. Lamont Granquist says:

    EOFErrors are almost always due to networking issues. The nginx that chef-client is talking to should always respond with valid HTTP one way or another and not just tear down connections — no matter what is going on with erchef or the cookbook repository behind the scenes. I suspect that whatever you did with deleting all your cookbooks just wasted time until someone fixed the network problems.

    • Will Jessop says:

      Firstly, this was January 2013. That’s nearly three years ago, it’s probably not relevant now.

      Secondly, no-one fixed any network issues, I would have known about it. This was a persistent issue that was solved by the action taken. I don’t know what chef was doing back in 2013, but perhaps it was returning some, but not all, of a JSON file.

      Thirdly, Perhaps you didn’t mean to come across as condescending, but you did and I didn’t appreciate it.

  4. lukegesior says:

    I know not a lot of people will be dealing with chef 11 anymore, but I had a similar error of a corrupted cookbook and wanted to share my experience as well.

    I was getting the same “error syncing cookbooks” message on my chef runs, without any great insight into what was causing it. I ran a couple chef runs with the debug flag, but it wasn’t very helpful. Finally I was able to solve it by looking in the nginx access.log on the chef-server, and grepping for the node name. I saw some 404 calls from that node, the url included a checksum, I was able to search on the problem nodes chef run (with debug on) to find what files, and thus cookbook that checksum correlated to. I deleted the cookbook from the chef server with the purge command, and reuploaded and it worked.

    Thanks for the info guys, it sent me on the right track!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s