In my last post I linked to some animated gifs I made on my mac. It’s pretty easy if you have imagemagick installed.
Simply get a bunch of images together and use the convert command:
convert -resize 900x -loop 0 -delay 10 DSC_8264.JPG DSC_8265.JPG DSC_8266.JPG DSC_8267.JPG DSC_8268.JPG DSC_8269.JPG DSC_8270.JPG jon_walk.gif
The last argument is the output filename. The options should be self explanatory, but you can check them up in the man page if you want to tweak the values.
I put up some photos of Railsberry here, but Flickr doesn’t work with animated gifs, so here they are. Click the images to get the animated versions.
I got the following error installing the latest pg gem on Lion:
Installing pg (0.12.2) with native extensions Unfortunately, a fatal error has occurred. Please report this error to the Bundler issue tracker at https://github.com/carlhuda/bundler/issues so that we can fix it. Thanks!
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:482:in `build_extensions': ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
checking for pg_config... yes
Using config values from /usr/local/bin/pg_config
checking for libpq-fe.h... yes
checking for libpq/libpq-fs.h... yes
checking for PQconnectdb() in -lpq... no
checking for PQconnectdb() in -llibpq... no
checking for PQconnectdb() in -lms/libpq... no
Can't find the PostgreSQL client library (libpq)
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
--with-pg
--without-pg
--with-pg-dir
--without-pg-dir
--with-pg-include
--without-pg-include=${pg-dir}/include
--with-pg-lib
--without-pg-lib=${pg-dir}/lib
--with-pg-config
--without-pg-config
--with-pg_config
--without-pg_config
--with-pqlib
--without-pqlib
--with-libpqlib
--without-libpqlib
--with-ms/libpqlib
--without-ms/libpqlib
Gem files will remain installed in /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/pg-0.12.2 for inspection.
Results logged to /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/pg-0.12.2/ext/gem_make.out
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:445:in `each'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:445:in `build_extensions'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:197:in `install'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/source.rb:90:in `install'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/rubygems_integration.rb:82:in `preserve_paths'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/source.rb:89:in `install'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:73:in `install_gem_from_spec'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/rubygems_integration.rb:97:in `with_build_args'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:72:in `install_gem_from_spec'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:56:in `run'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:55:in `run'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:12:in `install'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/cli.rb:220:in `install'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/task.rb:22:in `send'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/task.rb:22:in `run'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor.rb:263:in `dispatch'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/base.rb:386:in `start'
from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/bin/bundle:13
from /usr/bin/bundle:19:in `load'
from /usr/bin/bundle:19
I fixed it by passing in a build option to the gem install command:
env ARCHFLAGS="-arch x86_64" gem install --no-ri --no-rdoc pg -- --with-pg-config=/usr/local/Cellar/postgresql/9.1.2/bin/pg_config
To make bundler use this option whenever it's installing a gem you can do:
bundle config build.pg --with-pg-config=/usr/local/Cellar/postgresql/9.1.2/bin/pg_config
See build options in the bundler docs for a description of this.
All Yahoo groups have public metadata. The number of members, the category, various email addresses etc.
Yahoo doesn’t provide an API to this publicly available data (you can see it by visiting one of the group pages). Getting information about any particular group in your programs is hard.
I’ve filled this gap by releasing a third-party API to get the publicly available Yahoo groups metadata.
The API itself provides a really simple interface for getting group data in JSON format, just stick the urlencoded URL of the Yahoo group you are interested in on the end of the (Unofficial) Yahoo groups public data API URL and request it. You get JSON back.
The URL you request looks like this:
http://yahoo-group-data.herokuapp.com/api/v1/group/http%3A%2F%2Ftech.groups.yahoo.com%2Fgroup%2FOneStopCOBOL%2F
…and the JSON you get back looks like this:
{
"private": false,
"not_found": false,
"age_restricted": false,
"name": "OneStopCOBOL",
"description": "OneStopCOBOL - Official COBOL group",
"post_email": "OneStopCOBOL@yahoogroups.com",
"subscribe_email": "OneStopCOBOL-subscribe@yahoogroups.com",
"owner_email": "OneStopCOBOL-owner@yahoogroups.com",
"unsubscribe_email": "OneStopCOBOL-unsubscribe@yahoogroups.com",
"language": "English",
"num_members": 151,
"category": "COBOL",
"founded": "2008-06-24"
}
You can try it out and get sample code over at the homepage of the (Unofficial) Yahoo groups public data API.
I run the Recycling Group Finder, a site that makes extensive use of Yahoo Groups data. The (Unofficial) Yahoo groups public data API is an abstraction of the functionality I wrote to get group data for that site. I just figured it might be useful to other people.
Sometimes I need to test an http server that is listening on a unix socket. It’s really easy to do this using socat, but the socat man page is pretty long. Here it is for anyone who needs it in the future, and me when I inevitably forget.
In this case the server is unicorn, but this will work for any http server listening on a socket, for instance thin. The lines beginning with “–>” are lines I typed (the 4 lines at the start), remove the “–>” when you try this.
$ socat - UNIX-CONNECT:/u/apps/app/shared/sockets/unicorn.sock,crnl -->GET /session/new HTTP/1.1 -->Host: thehostname.com -->X-Forwarded-Proto: https --> HTTP/1.1 200 OK Date: Fri, 02 Dec 2011 14:37:23 GMT Status: 200 OK Connection: close Strict-Transport-Security: max-age=31536000 Content-Type: text/html; charset=utf-8 X-UA-Compatible: IE=Edge,chrome=1 ETag: "2346c47c7cb3bc37729e42fc8b20c821" Cache-Control: max-age=0, private, must-revalidate Set-Cookie: _x_session=blablabla; path=/; HttpOnly; secure X-Request-Id: c0a374f460d1b1205df450ab77dd2328 X-Runtime: 0.159219 <!DOCTYPE html> <html lang="en" data-behavior="wallpaper"> <head> etc.
For those interested in the relevance of the crnl option at the end of the socket path, this from the man page:
Converts the default line termination character NL (‘n’, 0x0a)
to/from CRNL (“rn”, 0x0d0a) when writing/reading on this
channel (example). Note: socat simply strips all CR characters.
Larger interactive versions of all the graphs on this page are available here.
Update: Added Mailgun to the graphs.
Earlier this year I posted a price comparison between Sendgrid, and the then newly available Amazon SES.
Tim Falls commented on the post saying that Sendgrid had updated their pricing:
Since this post was published, we have released a new pricing structure *and* a new service tier that offers more email for less + a feature set and pricing model that you will find very competitive with SES.
That was back in June, so it’s about time I produced an updated comparison. First, lets look at the difference between the old and new Sendgrid prices:
Comparison of old and new Sendgrid prices, click for a larger version
Overall the up-front plan prices, and prices for email over allowance have remained the same, but email allowance within each plan has increased. The exception is the Silver plan where email over allowance has increased by $0.0001/email. New to the lineup is the Lite plan.
More interesting is how these new prices compare to the competitors. I’ve added in Amazon SES, and Postmark too:
Sendgrid, Postmark and Amazon SES price comparison, click for a larger version
The most notable differences here are the inclusion of Postmark, and the the Sendgrid Lite plan that shadows Amazon SES. I’d guess this was added purely to compete with Amazon. As in my last post it is hard to see what is going on with smaller numbers of emails being sent, here’s a zoom on the origin:
Price comparison for small numbers of emails sent, click for a larger version
Here you can see the Sendgrid Lite plan shadowing Amazon and the Postmark costs heading up rapidly.
It seems Sendgrid have just added an ‘Amazon SES’ plan to pull back any customers that would have chosen SES based on price. It’s probably a good move, and it will allow easy transition into their more ‘premium’ plans if you sign up and later decide to change plan.
Given the advertised features of Postmark compared to the price it seems hard to consider using them. They seem to have some fairly well known customers though, so if anyone has used Postmark leave a comment with how that is working out for you.
So which email cloud provider should you use? Use the graphs I made, but price is only going to be one factor, so check what each provider offers. I’ve linked to all the pricing pages below.
I was writing a Ruby script recently that needed to download 43 2GB chunks of a database backup from a remote source, then decrypt each chunk, then finally concatenate the decrypted files together.
I knew I wanted to use threads to do this as it would speed up the overall process a great deal, and the downloading and decryption can be done in any order, it doesn’t matter if chunk 5 is downloaded before or after chunk 35, and the same with decryption. Those processes all operate on discrete files on the filesystem.
Where order does matter however is when the script is concatenating the files together into the final output file (in this case an lzop archive).
While looking how to handle this I discovered Ruby’s Queue class which “…provides a way to synchronize communication between threads“. Great, that’s exactly what I needed.
In my script I set up two thread-pools, one for downloading and one for decrypting, each with it’s own queue. At the start of the script I push all the download jobs on the download queue. The download thread pool workers download them then push them onto the decrypt queue. The decrypt queue can then get to work. It flows a little like this:
[download queue] -> [download pool] -> [decrypt queue] -> [decrypt pool]
However one last step remained, the concatenation. I used a queue again for this but needed to handle the jobs in order or I would end up with a useless lzop archive, so I came up with the following code to help with this:
You can see from the output that though the work units appear on the queue in any order, they will always be processed in the correct order:
[1.9.2] ~ $ ruby queue.rb
popping the stack
vals is now [16]
popping the stack
vals is now [1, 16]
popping the stack
vals is now [1, 11, 16]
popping the stack
vals is now [1, 11, 16, 19]
popping the stack
vals is now [1, 6, 11, 16, 19]
popping the stack
vals is now [1, 6, 11, 16, 18, 19]
popping the stack
vals is now [1, 6, 11, 15, 16, 18, 19]
popping the stack
vals is now [1, 6, 8, 11, 15, 16, 18, 19]
popping the stack
vals is now [0, 1, 6, 8, 11, 15, 16, 18, 19]
Processing 0
Processing 1
popping the stack
vals is now [5, 6, 8, 11, 15, 16, 18, 19]
popping the stack
vals is now [3, 5, 6, 8, 11, 15, 16, 18, 19]
popping the stack
vals is now [3, 5, 6, 8, 11, 14, 15, 16, 18, 19]
popping the stack
vals is now [3, 5, 6, 8, 10, 11, 14, 15, 16, 18, 19]
popping the stack
vals is now [3, 5, 6, 7, 8, 10, 11, 14, 15, 16, 18, 19]
popping the stack
vals is now [3, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 18, 19]
popping the stack
vals is now [2, 3, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 18, 19]
Processing 2
Processing 3
popping the stack
vals is now [4, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 18, 19]
Processing 4
Processing 5
Processing 6
Processing 7
Processing 8
Processing 9
Processing 10
Processing 11
popping the stack
vals is now [14, 15, 16, 17, 18, 19]
popping the stack
vals is now [14, 15, 16, 17, 18, 19, 20]
popping the stack
vals is now [13, 14, 15, 16, 17, 18, 19, 20]
popping the stack
vals is now [12, 13, 14, 15, 16, 17, 18, 19, 20]
Processing 12
Processing 13
Processing 14
Processing 15
Processing 16
Processing 17
Processing 18
Processing 19
Processing 208
Processing 19
Processing 20
I recently installed jRuby via RVM and got a weird error:
I tried installing jRuby via brew, and this worked, but I got the same error when installing a gem, so it looked like a Rubygems issue. On a hunch I figured it would be the https entries in my gem sources:
I removed them all and replaced them temporarily with one non-SSL “http://rubygems.org/” entry and I was then able to install jruby-openssl. Once that was installed I deleted the plain old http rubygems.org URL and added back in my SSL URLs. I can now install ruby gems with no error.
The hard-disk in my sisters Macbook laptop finally emitted it’s last death rattle and shuffled off to silicon heaven so I found myself needing to install Snow Leopard (it’s too old for Lion) on a new disk drive. This was complicated by two factors.
I had read that SATA II disks sometimes worked on SATA controllers, but also a number of reports that the Macbook didn’t recognise some SATA II disks. I took a risk and bought an OCZ Vertex 2 60GB SSD from Amazon. So far the Macbook has recognised it. This was the easy bit.
Next I needed to install Snow Leopard. This was going to be harder (I thought) because the DVD drive in the machine was broken. I checked a bunch of tutorials on the Internet on how to create a Snow Leopard install USB drive but most assumed the USB drive would be created on Snow Leopard itself (I’m using Lion) and included a fairly large number of steps.
I gave up on these guides and decided to do it freestyle and it turned out to be incredibly simple. Here’s how in-case someone ever needs to do this in the future.
That was it. I tried it out on my sisters Macbook and it booted from it just fine and is now installing Snow Leopard on the OCZ SSD.
