Eliot's Ramblings

10gen's New Office

Monday was a big day for 10gen in New York; we moved into our new offices on West 43rd Street. The last time we moved (about 16 months ago), our then new office seemed quite spacious and impressions were that it would last quite a while. That turned out to be a bit short sighted. By January of this year we were bursting at the seams, with every desk full, expansion space taken, and competition for conference rooms straining everyone’s patience.

Our new office is one we built ourselves, and I’m happy to say that because of that, it represents more than just an end to the constraints on our resource scheduling for the moment. It means we had the opportunity to build the type of space that suits our culture – an environment for serious work, but with enough comforts to make life at the office very enjoyable. In some future posts I’ll cover some of the choices we made and why, but for now I’d just like to say “phew!”

Streaming Twitter Into MongoDB

curl http://stream.twitter.com/1/statuses/sample.json -u<user>:<pass> | mongoimport -c twitter_live

One thing that you can do with mongodb is have 1 streaming master and 1 read/write master

server A:

./mongod —master —dbpath /tmp/a

server B:

./mongod —dbpath /tmp/b —master —slave —source localhost:27017 —port 9999

You can then pipe the stream into server A, and it will only process the live stream.

Server B will replicate all changes. You can also write to it, query on it, etc… This way you can do operations that block writing on server B, but server A will never backlog.