Eliot's Ramblings

Why Fly to London for 48 Hours

I visited London a few weeks ago to attend and speak at MongoDB London. The event was very successful, and I enjoyed many conversations with attendees and staff during the event. But having the opportunity to spend time with our 10gen London team makes the value of the trips far exceed my contribution to the conference.

Although my time with the team was relatively short since my entire trip to the UK lasted only two days, it provided yet another example of “no substitute for in-person collaboration”.

While I was in the office with the team, some of us began discussing a particular technical topic (related to mutability vs immutability for a specific class hierarchy). This discussion had actually started several weeks before, when a working group was attempting to get a specification for a new feature finalized. However, the geographical distance and time zone differences between the participants had meant that the discussion was drawn out and hard to finalize. During this phase, I had been persuaded of a particular viewpoint.

Working together in person, however, means more than just lower latency. It means better instantaneous understanding. When we met face-to-face, we were able to move rapidly from discussion to quick prototypes and, rather surprisingly, I found myself changing my point of view (as did one of the engineers in London). We therefore changed the spec.

10gen is a very distributed company, with offices in 7 cities and more to come. Maintaining our agility would not be possible without the benefits of teleconferencing in all of its forms; yet as useful as it is, I find no replacement for being in the same room with someone. It may be that I am particularly bad at remote communication. Regardless, I know my frequent trips to other 10gen offices are well worth the air time.

10gen's New Office

Monday was a big day for 10gen in New York; we moved into our new offices on West 43rd Street. The last time we moved (about 16 months ago), our then new office seemed quite spacious and impressions were that it would last quite a while. That turned out to be a bit short sighted. By January of this year we were bursting at the seams, with every desk full, expansion space taken, and competition for conference rooms straining everyone’s patience.

Our new office is one we built ourselves, and I’m happy to say that because of that, it represents more than just an end to the constraints on our resource scheduling for the moment. It means we had the opportunity to build the type of space that suits our culture – an environment for serious work, but with enough comforts to make life at the office very enjoyable. In some future posts I’ll cover some of the choices we made and why, but for now I’d just like to say “phew!”

Streaming Twitter Into MongoDB

curl http://stream.twitter.com/1/statuses/sample.json -u<user>:<pass> | mongoimport -c twitter_live

One thing that you can do with mongodb is have 1 streaming master and 1 read/write master

server A:

./mongod —master —dbpath /tmp/a

server B:

./mongod —dbpath /tmp/b —master —slave —source localhost:27017 —port 9999

You can then pipe the stream into server A, and it will only process the live stream.

Server B will replicate all changes. You can also write to it, query on it, etc… This way you can do operations that block writing on server B, but server A will never backlog.