Creating BERT dicts in Go

I’ve been learning Go recently and have written a program to connect to an existing service (written in Ruby) that sends and receives messages serialised as BERT terms.

I’m posting this partly because I had quite a lot of fun figuring it out and partly to document creating BERT dicts in Go should anyone else need to do this in the future and hit the same issues I did.

Why BERT?

I’m a big fan of BERT. It’s compact, flexible, and there are good libs available for serialisation/de-serialisation. So far I’ve exclusively been using the bert gem (written by Tom Preston-Werner, author of the BERT spec).

Creating BERT dicts

One of the great features of BERT is the complex types it supports, including dicts. The equivalent to a dict in Ruby would be a hash, in Go a map. They are really simple to create in Ruby:

require 'bert'
BERT.encode({"key" => "val"})
=> "\x83h\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02m\x00\x00\x00\x03keym\x00\x00\x00\x03valj"

We can pull this apart and see exactly what the bert gem did to our data. Let’s dump the string to an array of 8-bit unsigned integers:

BERT.encode({"key" => "val"}).unpack("C*")
=> [131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 1, 104, 2, 109, 0, 0, 0, 3, 107, 101, 121, 109, 0, 0, 0, 3, 118, 97, 108, 106]

It’s hard to see exactly what happened, but with the BERT docs and the erlang External Term Format docs we can see how the hash got encoded.

magic| tuple |  atom   |       bert     |            |    dict        |  list 1 elem    |      list      |   atom  |      key      |   atom  |      |  val       | nil | nil
131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 1, 108, 0, 0, 0, 2, 100, 0, 3, 107, 101, 121, 100, 0, 3,       118, 97, 108, 106, 106

If the formatting of that breakdown is messed up here’s a raw gist that may be clearer.

What you can see here are what the bytes represent (you can see the breakdown of each data type on the External Term Format docs). This is great, but why write a blog post just about dicts? Well, they’re easy to create in Ruby:

BERT.encode(:complex => {"key" => [:data, {:structures => "are easy to serialise"}]})
=> "\x83h\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02d\x00\acomplexh\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02m\x00\x00\x00\x03keyl\x00\x00\x00\x02d\x00\x04datah\x03d\x00\x04bertd\x00\x04dictl\x00\x00\x00\x01h\x02d\x00\nstructuresm\x00\x00\x00\x15are easy to serialisejjjj"

but it’s not so obvious in Go, and I hit some issues when trying to create them.

Serialising to BERT in Golang

Serialising data to BERT/BERP in Go is pretty easy for simple cases using the gobert lib:

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, bert.Atom("foo"))
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

This gives us:

0 0 0 7 131 100 0 3 102 111 111

If we run that through the Ruby lib decoder we get:

> BERT.decode([131, 100, 0, 3, 102, 111, 111].pack("C*"))
 => :foo

(The Ruby bert lib decodes atoms to symbols).

Serialising to BERT dicts in Golang

However, there is a little more effort involved serialising more complex data structures, in particular dicts, as I found out.

You might have thought that you could just pass in a map:

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    message := map[string]string{"key1": "val1", "key2": "val2"}

    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, message)
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

We get the output:

0 0 0 1 131

Well, that doesn’t work. What you end up with is a one byte long BERP. It seems that gobert doesn’t automatically serialise maps. No problem, we’ll build one up manually. A quick look at the BERT documentation shows the format of a dict:

“Dictionaries (hash tables) are expressed via an array of 2-tuples representing the key/value pairs. The KeysAndValues array is mandatory, such that an empty dict is expressed as {bert, dict, []}. Keys and values may be any term. For example, {bert, dict, [{name, <<“Tom”>>}, {age, 30}]}.”

So let’s create this special structure manually.

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    message1 := []bert.Term{bert.Atom("key1"), bert.Atom("val1")}
    message2 := []bert.Term{bert.Atom("key2"), bert.Atom("val3")}
    keys_and_values := []bert.Term{message1, message2}

    dict := []bert.Term{bert.BertAtom, bert.Atom("dict"), keys_and_values}

    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, dict)
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

The result:

0 0 0 51 131 104 3 100 0 4 98 101 114 116 100 0 4 100 105 99 116 104 2 104 2 100 0 4 107 101 121 49 100 0 4 118 97 108 49 104 2 100 0 4 107 101 121 50 100 0 4 118 97 108 51

It looks better, but it doesn’t decode, using Ruby:

> BERT.decode([131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 104, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 51].pack("C*"))
TypeError: Invalid dict spec, not an erlang list

We’re still missing something. Let’s compare the output of the Ruby bert lib to the output of gobert for the same data structure:

> BERT.encode({:key1 => :val1, :key2 => :val2}).unpack("C*")
 => [131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 50, 106]

We’re definitely missing some data in the gobert output.

If you follow along the byte sequences you can see that they start off the same until the 18th byte. In the Ruby output this is ‘108’, or LIST_EXT. In the gobert output it’s 104, a SMALL_TUPLE_EXT. We can see where this difference happens in encode.go in the gobert lib (in the writeTag func):

case reflect.Slice:
    writeSmallTuple(w, v)
case reflect.Array:
    writeList(w, v)

Let’s decode the BERT data to see where the diversion happens in the underlying data structures:

magic| tuple  |  atom   |       bert       |   atom   |    dict
  131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116

We can see that the “bert” and “dict” atoms are encoded the same, but the keys_and_values array is getting encoded as a SMALL_TUPLE_EXT by gobert when we wanted a LIST_EXT. If we look back at the gobert code we can see that the decision to use SMALL_TUPLE_EXT over LIST_EXT is dependent on a slice or array being present. We can use the go “reflect” package to look at the arrays/slices we are creating and see what they are:

package main

import (
    "fmt"
    "reflect"
    "github.com/sethwklein/gobert"
)

func main() {
    array := [2]bert.Term{}
    slice := []bert.Term{}

    array_val := reflect.ValueOf(array)
    slice_val := reflect.ValueOf(slice)
    fmt.Printf("array is a: %v\n", array_val.Kind())
    fmt.Printf("slice is a: %v\n", slice_val.Kind())
}

array is a: array
slice is a: slice

The fix

So, in order to fix our data structure to get gobert to correctly encode the dict we need to change the keys_and_values slice to an array:

package main

import (
    "fmt"
    "bytes"
    "github.com/sethwklein/gobert"
)

func main() {
    message1 := []bert.Term{bert.Atom("key1"), bert.Atom("val1")}
    message2 := []bert.Term{bert.Atom("key2"), bert.Atom("val3")}
    keys_and_values := [2]bert.Term{message1, message2} // Now an array

    dict := []bert.Term{bert.BertAtom, bert.Atom("dict"), keys_and_values}

    var buf = new(bytes.Buffer)
    bert.MarshalResponse(buf, dict)
    for _, b := range(buf.Bytes()) {
        fmt.Printf("%d ", b)
    }
    fmt.Println()
}

The result:

0 0 0 55 131 104 3 100 0 4 98 101 114 116 100 0 4 100 105 99 116 108 0 0 0 2 104 2 100 0 4 107 101 121 49 100 0 4 118 97 108 49 104 2 100 0 4 107 101 121 50 100 0 4 118 97 108 51 106

But more importantly, can we decode the data we encoded?

> BERT.decode([131, 104, 3, 100, 0, 4, 98, 101, 114, 116, 100, 0, 4, 100, 105, 99, 116, 108, 0, 0, 0, 2, 104, 2, 100, 0, 4, 107, 101, 121, 49, 100, 0, 4, 118, 97, 108, 49, 104, 2, 100, 0, 4, 107, 101, 121, 50, 100, 0, 4, 118, 97, 108, 51, 106].pack("C*"))
 => {:key1=>:val1, :key2=>:val3}

Yes!

================================================================================

[2013-01-21T18:17:27+00:00] ERROR: Running exception handlers
[2013-01-21T18:17:27+00:00] FATAL: Saving node information to /var/cache/chef/failed-run-data.json
[2013-01-21T18:17:27+00:00] ERROR: Exception handlers complete
[2013-01-21T18:17:27+00:00] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
[2013-01-21T18:17:27+00:00] FATAL: EOFError: end of file reached

I’d already deleted the cookbooks:

knife cookbook delete cookbook_name

and re-uploaded them, but this didn’t work. It took a purge to completely delete the cookbook from the chef server:

knife cookbook delete --purge cookbook_name

In the end I had so many cookbooks that were somehow corrupt that I bulk purged them and re-uploaded them all:

knife cookbook bulk delete --purge '.+'

Dear Will,

I understand that you have questions about downloading filler for the plot-holes in [the movie] Battleship. My name is Pauline and I will be more than happy to help you resolve this.

I am very glad that you have been able to view the rental “Battleship”, I hope you enjoyed this film and had fun watching it. While I cannot comment or assist with filter for the content of the movie, I would be more than happy to look in to any other account issues you may be experiencing.

It has been a pleasure assisting you today. Should you have any other concerns you wish to raise, please let me know. Have a nice day!

Sincerely,

Simply get a bunch of images together and use the convert command:

convert -resize 900x -loop 0 -delay 10 DSC_8264.JPG DSC_8265.JPG DSC_8266.JPG DSC_8267.JPG DSC_8268.JPG DSC_8269.JPG DSC_8270.JPG jon_walk.gif

The last argument is the output filename. The options should be self explanatory, but you can check them up in the man page if you want to tweak the values.

Riding the Railsberry Unicorn:

Jon Leighton getting closer:

Josh Kalderimis misbehaving:

Installing pg (0.12.2) with native extensions Unfortunately, a fatal error has occurred. Please report this error to the Bundler issue tracker at https://github.com/carlhuda/bundler/issues so that we can fix it. Thanks!
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:482:in `build_extensions': ERROR: Failed to build gem native extension. (Gem::Installer::ExtensionBuildError)

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb 
checking for pg_config... yes
Using config values from /usr/local/bin/pg_config
checking for libpq-fe.h... yes
checking for libpq/libpq-fs.h... yes
checking for PQconnectdb() in -lpq... no
checking for PQconnectdb() in -llibpq... no
checking for PQconnectdb() in -lms/libpq... no
Can't find the PostgreSQL client library (libpq)
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
	--with-pg
	--without-pg
	--with-pg-dir
	--without-pg-dir
	--with-pg-include
	--without-pg-include=${pg-dir}/include
	--with-pg-lib
	--without-pg-lib=${pg-dir}/lib
	--with-pg-config
	--without-pg-config
	--with-pg_config
	--without-pg_config
	--with-pqlib
	--without-pqlib
	--with-libpqlib
	--without-libpqlib
	--with-ms/libpqlib
	--without-ms/libpqlib


Gem files will remain installed in /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/pg-0.12.2 for inspection.
Results logged to /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/pg-0.12.2/ext/gem_make.out
	from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:445:in `each'
	from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:445:in `build_extensions'
	from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/installer.rb:197:in `install'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/source.rb:90:in `install'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/rubygems_integration.rb:82:in `preserve_paths'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/source.rb:89:in `install'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:73:in `install_gem_from_spec'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/rubygems_integration.rb:97:in `with_build_args'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:72:in `install_gem_from_spec'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:56:in `run'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:55:in `run'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/installer.rb:12:in `install'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/cli.rb:220:in `install'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/task.rb:22:in `send'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/task.rb:22:in `run'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/invocation.rb:118:in `invoke_task'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor.rb:263:in `dispatch'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/lib/bundler/vendor/thor/base.rb:386:in `start'
	from /Users/will/.rvm/gems/ruby-1.9.3-p0@status/gems/bundler-1.1.rc.7/bin/bundle:13
	from /usr/bin/bundle:19:in `load'
	from /usr/bin/bundle:19

I fixed it by passing in a build option to the gem install command:

env ARCHFLAGS="-arch x86_64" gem install --no-ri --no-rdoc pg -- --with-pg-config=/usr/local/Cellar/postgresql/9.1.2/bin/pg_config

To make bundler use this option whenever it's installing a gem you can do:

bundle config build.pg --with-pg-config=/usr/local/Cellar/postgresql/9.1.2/bin/pg_config

See build options in the bundler docs for a description of this.

All Yahoo groups have public metadata. The number of members, the category, various email addresses etc.

Yahoo doesn’t provide an API to this publicly available data (you can see it by visiting one of the group pages). Getting information about any particular group in your programs is hard.

I’ve filled this gap by releasing a third-party API to get the publicly available Yahoo groups metadata.

JSON API

The API itself provides a really simple interface for getting group data in JSON format, just stick the urlencoded URL of the Yahoo group you are interested in on the end of the (Unofficial) Yahoo groups public data API URL and request it. You get JSON back.

The URL you request looks like this:

http://yahoo-group-data.herokuapp.com/api/v1/group/http%3A%2F%2Ftech.groups.yahoo.com%2Fgroup%2FOneStopCOBOL%2F

…and the JSON you get back looks like this:

{
    "private": false,
    "not_found": false,
    "age_restricted": false,
    "name": "OneStopCOBOL",
    "description": "OneStopCOBOL - Official COBOL group",
    "post_email": "OneStopCOBOL@yahoogroups.com",
    "subscribe_email": "OneStopCOBOL-subscribe@yahoogroups.com",
    "owner_email": "OneStopCOBOL-owner@yahoogroups.com",
    "unsubscribe_email": "OneStopCOBOL-unsubscribe@yahoogroups.com",
    "language": "English",
    "num_members": 151,
    "category": "COBOL",
    "founded": "2008-06-24"
}

You can try it out and get sample code over at the homepage of the (Unofficial) Yahoo groups public data API.

Motivation

I run the Recycling Group Finder, a site that makes extensive use of Yahoo Groups data. The (Unofficial) Yahoo groups public data API is an abstraction of the functionality I wrote to get group data for that site. I just figured it might be useful to other people.

In this case the server is unicorn, but this will work for any http server listening on a socket, for instance thin. The lines beginning with “–>” are lines I typed (the 4 lines at the start), remove the “–>” when you try this.

$ socat - UNIX-CONNECT:/u/apps/app/shared/sockets/unicorn.sock,crnl
-->GET /session/new HTTP/1.1
-->Host: thehostname.com
-->X-Forwarded-Proto: https
-->
HTTP/1.1 200 OK
Date: Fri, 02 Dec 2011 14:37:23 GMT
Status: 200 OK
Connection: close
Strict-Transport-Security: max-age=31536000
Content-Type: text/html; charset=utf-8
X-UA-Compatible: IE=Edge,chrome=1
ETag: "2346c47c7cb3bc37729e42fc8b20c821"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: _x_session=blablabla; path=/; HttpOnly; secure
X-Request-Id: c0a374f460d1b1205df450ab77dd2328
X-Runtime: 0.159219

<!DOCTYPE html>
<html lang="en" data-behavior="wallpaper">
<head>
etc.

For those interested in the relevance of the crnl option at the end of the socket path, this from the man page:

Converts the default line termination character NL (‘n’, 0x0a)
to/from CRNL (“rn”, 0x0d0a) when writing/reading on this
channel (example). Note: socat simply strips all CR characters.

Update: Added Mailgun to the graphs.

Earlier this year I posted a price comparison between Sendgrid, and the then newly available Amazon SES.

Tim Falls commented on the post saying that Sendgrid had updated their pricing:

Since this post was published, we have released a new pricing structure *and* a new service tier that offers more email for less + a feature set and pricing model that you will find very competitive with SES.

That was back in June, so it’s about time I produced an updated comparison. First, lets look at the difference between the old and new Sendgrid prices:

Comparison of old and new Sendgrid prices, click for a larger version

Overall the up-front plan prices, and prices for email over allowance have remained the same, but email allowance within each plan has increased. The exception is the Silver plan where email over allowance has increased by $0.0001/email. New to the lineup is the Lite plan.

More interesting is how these new prices compare to the competitors. I’ve added in Amazon SES, and Postmark too:

Sendgrid, Postmark and Amazon SES price comparison, click for a larger version

The most notable differences here are the inclusion of Postmark, and the the Sendgrid Lite plan that shadows Amazon SES. I’d guess this was added purely to compete with Amazon. As in my last post it is hard to see what is going on with smaller numbers of emails being sent, here’s a zoom on the origin:

Price comparison for small numbers of emails sent, click for a larger version

Here you can see the Sendgrid Lite plan shadowing Amazon and the Postmark costs heading up rapidly.

Conclusion

It seems Sendgrid have just added an ‘Amazon SES’ plan to pull back any customers that would have chosen SES based on price. It’s probably a good move, and it will allow easy transition into their more ‘premium’ plans if you sign up and later decide to change plan.

Given the advertised features of Postmark compared to the price it seems hard to consider using them. They seem to have some fairly well known customers though, so if anyone has used Postmark leave a comment with how that is working out for you.

So which email cloud provider should you use? Use the graphs I made, but price is only going to be one factor, so check what each provider offers. I’ve linked to all the pricing pages below.

Price sources

• Sendgrid prices
• Postmark prices
• Amazon SES prices
• Mailgun prices