Eliot's Ramblings

Dengue Fever

Last week I went to Las Vegas for MongoDB’s sales kickoff. The night before I left, Sunday, I came down with a decently high fever. I got a bit nervous, as it came on strong and fast, but I took some Advil, went to bed, and the next morning felt ok to get on a plane. That whole Monday was pretty good with the help of some more Advil. On Tuesday morning the Advil was giving ground, on Tuesday evening it was in full retreat, and Wednesday at 5am I found a helpful MongoDB employee in the hotel to take me to the ER.

Apparently, while in the Dominican Republic for a family vacation, a.k.a playing with my kids in the water, I was bitten by a mosquito carrying Dengue fever. So I have now officially crossed “Get a tropical disease” off of my bucket list. I’m very excited about that.

I have two take-aways from this experience:

First, I don’t recommend getting Dengue fever. It’s not pleasant. Use a lot of bug spray, really.

Second, if you do have to get Dengue fever, make sure when you get really sick, and are given a fair amount of morphine, that a) you do not write any code, and b) you be administered said morphine in the presence of co-workers, who then have blackmail material for life.

I’m still a bit under the weather, but at least I’m not contagious.

Seriously, though, don’t get Dengue fever.

MongoDB 3.0: Seizing Opportunities

MongoDB 3.0 has landed.

The development cycle for 3.0 has been the most eventful of my entire career. As originally planned, it would have been great, but still incremental in nature. Instead, we wound up acquiring our first company, integrating their next-gen storage engine, and by capitalizing on that unlooked-for opportunity, delivering a release so beyond its original conception that we revved its version number.

Renaming a release in-flight is out of the ordinary, so I wrote about our reasoning when we announced the change. We had originally planned to deliver document-level locking built into MMAPv1, and a storage engine API as an investment in the future, not part of a fully developed integration. That would have been our incremental improvement, in line with our storage engine efforts throughout the 2.x release series. We had already added database-level locking, iterated over many improvements to yielding and scheduling behavior, and refactored a ton of code to decouple components.

At the outset of this development cycle we did several things in parallel. We carved out the code layers to support our storage engine API, started building collection-level locking into MMAPv1, and started designing document-level locking. At the same time, we worked with storage engine builders to put our API through its paces. By the summer of 2014, we had a MMAPv1 prototype for document-level locking, which we demonstrated at MongoDB World. While this was not going to make our use of disks more efficient or solve other MMAPv1 problems, it was nonetheless a huge improvement, and exactly what we were aiming for.

Then the WiredTiger team called us and demonstrated a working integration with MongoDB’s storage engine API. Before long, we realized we had before us an opportunity to shoot the moon. We would have to scale back our plans for MMAPv1 to just collection-level locking, but by doing so, we could completely leapfrog our roadmap and supercharge our team. By delivering MongoDB with WiredTiger, we could offer our users everything we had promised, along with performance MMAPv1 will never match, and features it would take years more to build in. After all, WiredTiger was developed with laser focus on the raw fundamentals of data storage in a modern environment, allowing it to support massive concurrency and other great features like compression.

For all its magnificence, WiredTiger is not yet the default storage engine. We have every confidence in its ability– it is a shipping product in its own right, and has proven its mettle to customers with the most demanding production environments, such as Amazon. We are using it ourselves in production to back MMS. However, the use cases for MongoDB are so broad and varied, we need to gather a wide range of feedback. With that data, we’ll be able to optimize and tune the integration and provide robust guidance on the role of specific metrics in capacity planning, leading to better, more predictive monitoring, and a healthy collection of best practices.

The acquisition of WiredTiger marks an important transition for me as well. Storage engines are incredibly interesting components of a database, but as much as I might like to dig further into them, our goal to make MongoDB the go-to database requires me to be more pragmatic. With a team of world-renowned experts available, that know more about (for example) how to implement MVCC than I ever will, it makes sense to leave storage engines in their capable hands so I can focus on other areas.

MongoDB 3.0 is a great release. I am very proud of the massive team effort that produced it. We will not be resting on our laurels though. There is still a long list of features and improvements our users need to be successful, and with MongoDB 3.0, we expect MongoDB to be used in even more demanding and mission critical projects. Many of those projects will surprise us, and these surprises will create new demands. We are excited to get started on these challenges, further optimizing MongoDB, and extending its capabilities so the pioneers can continue to surprise us.

LiveScribe vs. Phone Camera Update: The NOOP Edition

In my first post on this topic, I said I’d post an update in a week or so. Ok, so that was about 7 weeks ago.

I abandoned the trial of both of these techniques because 2.8.0 is, frankly, more important than my experiments in productivity. I’m going to get back to it, but this is actually an opportunity to say something important about getting derailed from productivity projects by urgent items.

This happens to all of us from time to time when the pressure mounts, and that’s a good thing. The key is to keep your head, focus on the most urgent thing while it’s urgent, and remember to revisit those productivity projects. They are important in the long run, or you wouldn’t want to start on them in the first place. If you find yourself constantly saying “I had to drop that, I got too busy”, it’s time to re-evaluate.

MongoDB 2.8.0-rc0

Today our team made public our first release candidate of MongoDB 2.8, rc0.

Since June, beginning with MongoDB World 2014, I’ve been speaking publicly about MongoDB 2.8, and its headline features: document level locking and pluggable storage engines. What I haven’t said until now is just how related these two features are.

We’ve been working on our storage API for roughly a year, and with MongoDB 2.8 rc0, we’re rolling out the first fully supported and working storage engine integration: WiredTiger.

WiredTiger is a modern storage engine designed from the ground up and optimized to support high write performance, compression, and vertical scalability. By integrating the WiredTiger storage engine, MongoDB 2.8 will add document-level locking and high performance writes.

Migrating existing MongoDB instances to the new storage engine can be done with a rolling upgrade, the same process as is used to upgrade MongoDB versions. And as demonstrated at MongoDB World 2014, 2.8 supports mixed-mode deployments, so teams can test out the new engine before migrating entirely.

Our original storage engine, MMAPv1, will remain the default for this release, and is going from database-level to collection-level locking, offering a major win, completely maintenance free, to teams for which a storage engine upgrade is not a priority.

Please remember, rc0 is a release candidate! We can’t wait for our amazing community to take it for a spin and start giving us feedback, but we do not recommend you use it in production!

With every release of MongoDB, we try to offer our community the most important things they’ve been asking for. With v2.8, we hope we’ve done exactly that.

MongoDB London 2014

On November 6th, I’ll be delivering the keynote address at MongoDB London 2014. I’ll be talking about the upcoming 2.8 release, the future of storage engines in MongoDB, and Automation. Since our last conference (MongoDB Boston 2014), the revamped MMS with Automation has gone from soft launch to wide release, and the response from the MongoDB community has been fantastic. We’re seeing tons of adoption and getting lots of great feedback. We’ve also been hosting meetups in our offices, to demonstrate how easy it is to use Automation to deploy a MongoDB infrastructure at any scale.

So if you’re at MongoDB London, keep an eye out for me… I’ll be walking around, trying to meet as many of you as possible. I’d love to get your feedback on Automation if you’ve tried it out already, or hear what’s preventing you from using it.

The Road to MMS Automation

“MongoDB is as easy to operate at scale as it is to develop with.” From the very beginning of MongoDB, I’ve envisioned making that bold claim. Until today, it’s been a dream.

We just brought it firmly into the realm of the realistic. Today we rolled out a completely revamped MMS built atop Automation, our cloud service for deploying and running MongoDB. Automation works with any infrastructure, from AWS to private cloud to bare metal. It deploys brand new replica sets, adds new shards to clusters, adds replica set members, deploys version upgrades… all at the push of a button. Monitoring and Backups are maintained seamlessly via Automation as well. It also makes julienne fries.1

You can read about the new MMS on the MongoDB blog, or visit mms.mongodb.com to check it out for yourself. What I would like to explain here is the road that led to this point, which began with our mission to create a database that enabled the greatest possible productivity and horizontal scale.

The Consequences of Horizontal Scale

There are many possible designs for horizontally scaling data stores. As with all development, every design choice incurs trade-offs. Sometimes developer productivity comes at a cost of operational complexity. Moreover, MongoDB’s chosen mechanism for horizontal scaling results in an infrastructure where maintenance tasks must be performed in a specific order. We understood the consequences to these choices, and envisioned a layer of tooling above the raw MongoDB processes that would offer an operator a robust yet simple mechanism for maintenance.

The trouble is, no tool existed which could serve as that layer.

Existing Provisioning Tools

General purpose provisioning tools are not designed to manage non-homogenous, distributed resources for which ordering of maintenance operations matters. Every resource is treated as a stand-alone system, so operations which require ordering of events require complex and error-prone custom code. One example of this is upgrading a sharded cluster. The process is to upgrade the mongos processes (possible in parallel), then the config servers (serially), and then all the replica sets (sets in parallel, nodes within sets serially). While it is technically possible to express this within the frameworks of today’s provisioning tools, it is far from easy, and the tools do not provide primitives to address the problem domain. An operator with these needs is essentially on their own.

We’ve said all along that the complexities of operating a MongoDB cluster could and would be handled with tooling, but unfortunately, several years after MongoDB became by far the most popular non-relational database, that tooling still did not exist… until today.

MMS is not meant to replace provisioning tools like Chef, or Puppet… in fact many people will deploy the MMS automation agent via these tools, after provisioning and configuring their VMs with those same tools. Rather, it is meant to take over where they leave off.

MongoDB ♥ Ops

This is a love letter to operators everywhere. Our new MMS is a huge step forward for running MongoDB infrastructures, making even the most tedious and risk-fraught tasks simple.

Our mission is to take things that are overly complicated and make them simple, even enjoyable. First we did it with data storage programming, and now we’ve done it with our own infrastructure. We hope you enjoy this new chapter of MongoDB. Now go forth and deploy!

  1. Does not actually make julienne fries

LiveScribe vs. Phone Camera

I’ve been using this new toy. Well, it’s for work, but until the novelty wears off, it’s definitely also a toy.

I like taking notes in meetings on paper as much as possible. It’s less distracting, and more friendly. I’ve tried various ways of doing this, but nothing has stuck yet. The closest has been a regular notebook. The biggest problem is that I don’t like carrying things to and from work, or to different places. So I invariably end up with 4 notebooks, and then I can’t get notes from a trip when I’m at the office.

I’ve been checking out Livescribe. They make a special pen with a camera in it (I have the Livescribe 3), which works with special paper that has a teeny (nearly invisible) grid of dots printed on it. The pen talks to an app on my phone via bluetooth, capturing what I write in real-time. The notes sync automatically to Evernote, where I file them.

I can leave a notebook in every location I might want to take notes, and then view them from anywhere. Kind of nifty.

Once I started trying this out, though, I started thinking about the problem in general, and wondered if there wasn’t a way to accomplish what I needed without carrying around another dedicated piece of hardware. Like I said, travelling light is top priority for me.

So I tried making do with hardware I already carry around everywhere – my phone. It’s pretty quick to save a new “snapshot note” in Evernote, which is where my notes wind up, anyhow. That actually works much better than you might think. There are two issues, though. LiveScribe will transcribe handwriting to text, which I can then go back and edit, or copy for pasting. Even more of an issue for me is I have to remember to take the picture. I’m not great at remembering things when I’m done with one thing and starting to think about the next.

So… remember the pen, or remember to take the picture? Is it going to be worth it enough to me to carry around a special pen all the time? I’m not sure, but I’ll let you know in a week or so where I land.

The Staff Meeting

Everyone with a staff knows they need a staff meeting on a recurring basis, often weekly. And those who don’t have staff are themselves in other people’s staff meetings, making it one of the most common meeting types for anyone to attend. Sadly, there is often ambiguity around what they are for, making them annoying and inefficient.

What I Want out of Staff Meetings

The purpose of these meetings is twofold: 1) status updates, and 2) key decision making or the precursor conversations for decision making. I want status updates to be very efficient, well thought out, and delivered only for items that are really required. I want thoughtful conversation on key topics, and I want to cut short conversation when the topics are sufficiently discussed.

Meeting Format

A meeting should be formatted based on what you want out of it.

After years of iterating, my team and I have landed on a format that makes staff meetings useful and efficient for all attendees. We tried a lot of our own ideas, and took good concepts from others (Amazon’s study hall). This isn’t the last iteration (there’s room for improvement), but we’ve finally reached the point where they are highly functional.

Each meeting has a shared google doc. They live in per-team folders, and have the meeting date in the title. They contain a status update section, and proposed agenda for the meeting.

Before the Meeting

Before the meeting, team members fill out their status update section and also put agenda items in a list with their initials. Others can “second” an agenda item with their initials and we will only cover items that have received at least a second vote. There is a strong social contract enforcing preparation such that it’s rare that prep work is not done in time.

The First 10 Minutes

For the first 10 minutes of the meeting, everyone gathers in the room (real or virtual) and begins reading the status updates, silently, together. There are two main benefits of doing it this way. Firstly, the deadline for writing is clear, and the time to start reading is clear. The second is that this reading period ensures that everyone reads the full material, as it is often otherwise skimmed or skipped in the interest of time.

While reading the status updates, meeting participants comment on the updates (using google doc comments), add more agenda items, and second other items. This is all done silently.

The Discussion Section

After the silent portion of the meeting, the team begins the discussion portion of the meeting. First comes the commented portions of the status updates. Then we proceed down the list of agenda items (but only those that have at least two sets of initials by them).

During the discussions, notes are taken in the document, in-line. They are written in red text, and are classified rigidly into 3 types of note: action items (with responsible personnel), key facts, and decisions.

The meeting ends on time (usually early), and anything unattended to becomes automatically added (without need for seconding) to the top of the agenda for the next meeting.

Sample Agenda

Here’s an example of an agenda:


  • Person 1
    • I did X
    • My team did Y
    • I’ve been thinking about Z Person 2: What about trying Zeta?
  • Person 2
    • More status stuff
    • Cannot get Flooble to compilePerson 1: Have you tried –just-work-already?


  • Agenda item 1 (initials of proposer)
  • Agenda item 2 (initials of proposer, of supporter, [initials of more supporters])
    • Decided: we will use a producer/consumer model for this component
  • Agenda item 3 (name or proposer, name of supporter, [names of more supporters])
    • Key Fact: We will not be able to have this QA’d until QA finishes with this other team’s crunch project
    • Action Item: Bob will create an ASCII art logo for MongoDB while waiting for QA to free up.

MongoDB 2.6 and the Future

MongoDB 2.6 has been released. For my thoughts on many of the features of the release, please see my blog post on mongodb.org.

Beyond the features, this release means a lot to me. In five years, we’ve gone from four people trying to figure out if a document database was a viable concept, to the fifth most popular database in the world. MongoDB 2.4 and all previous releases proved that the document model can transform how modern applications are developed and deployed. Despite this, we knew many of MongoDB’s core components were imperfect. It was time to address these shortcomings.

MongoDB 2.6 is the first release of the next generation of MongoDB. To smooth out the rough edges, we’ve ripped out and re-written large portions of the code base. We’ve built an entirely new set of tools to complete our vision of how easy it could be to manage a database cluster comprised of thousands of machines1. We’ve grown the team tremendously, both in terms of size and expertise, and are confident we can continue innovating for years to come.

MongoDB 2.6 is the beginning of the next generation, but is in no way the culmination. We’ll continue rebuilding concurrency, storage, networking and anything else that gets in the way of scale and performance. MongoDB 2.8 will have document level locking and MMS Automation will take control of the most challenging MongoDB deployments.

Personally, helping the database world re-invent itself is incredibly rewarding. Even if MongoDB isn’t right for everyone’s project, knowing that we’ve made a contribution to the way people think about data storage in modern applications is truly gratifying. I hope we at MongoDB can make the database that will continue to push the boundaries, and that helps teams focus on making great products rather than worry about storing data.

None of this would be possible without the contributions of the MongoDB community. Their feedback, code, and support is invaluable to the MongoDB team.

Thank you to the entire MongoDB team, whose hard work and dedication to everything that goes into the success of this project and company inspires me to work harder and smarter.

Debugging the Boss: The Martyr

Like The Superhero, The Martyr does their team’s work to make up for not managing. However, whereas The Superhero insists on hogging all the interesting work, The Martyr does work that no-one wants to do.

When a deadline is looming and things are looking down, they will pull all nighters to finish it themselves rather than do what a manager should do, such as motivating their team, or fixing the deadline. When things go wrong, they will take all the blame instead of teaching their team.

In the grand scheme of things, The Martyr poses less of a threat to an organization than some other buggy bosses, more analogous to inferior lubrication on moving parts than the active sabotage of The Glory Hog. Nonetheless, they do have negative effects, and often would love to do better if they only knew how.

Subtypes: Some Martyrs won’t delegate because they don’t want to – their personality craves the sacrificial behavior. Others do so because they can’t bring themselves to delegate unpleasant work, and doing it themselves is their only option.

Behavior in meetings: The Martyr will never miss an opportunity to call attention to their sacrifices. This overt, calculated attention-seeking is the opposite of inspiring.

Impact on team: The good news is that The Martyr does not persecute their team, belittle them, or in any way create an environment that punishes excellence. They do, however, lower the bar. As the team learns that the boss will just cover all the gaps, team members will not feel the need to stretch, or put in an extra, last-mile sprint when it’s crunch time. If The Martyr covers enough of the grunt work, some on their team may become downright spoiled, internalizing the idea that they should never have to do work that they dislike.

Impact on product: The Martyr refuses to delegate crappy tasks, which can bottleneck the team when those tasks are the ones that matter. When The Martyr inevitably reaches the end of their capacity, or even breaks down, product releases will be delayed.

Trait gone wrong: A sense of duty.

Debugging: The right approach will depend on what kind of Martyr you’re dealing with. For those who are simply squeamish about delegating work they know is unpleasant, you can focus on the negative impact their behavior has on the team. As with The Best Friend, realign their view of what’s good for their team members from the short-term dislike of certain tasks to the long-term needs of their careers.

The self-appointed Martyr is a harder debug. While they do care about the happiness of their team, they are most of all driven to seek acknowledgment and approval through sacrifice. They will never miss an opportunity to go unhealthily above and beyond, so they can moan about how hard it was to everyone. They need help to believe that their peers will value them even if they are not burning themselves to cinders. This is a job for a therapist; but peers, reports, and managers of Martyrs can best help by facing this head on with a private, compassionate, but frank conversation. Tell them straight out that they are doing themselves harm, and that they do not need to fear being judged as inadequate, for “only” putting in a full week’s work. If they appear receptive to this feedback, you should continue to reinforce this over time. If they do not, and instead become defensive, you can try again, but if you gain no traction, you will have to evaluate if they are doing more harm than good.

Not to be mistaken for: The Superhero, who wants all the work and doesn’t care if they don’t leave it for team, or The Best Friend, who might do work rather than having their team do it because they want to be liked.