Eliot's Ramblings

Debugging the Boss: Intro

Managing in a technology company is one of the charter topics of this blog. I cannot think of any single thing that represents a greater risk to a growing tech firm than the damage that can be done by bad management. A really bad employee can waste resources and time, and lower the morale of those around them; a really bad manager can do more serious harm, in a wider range, that lingers on even after they have been removed or corrected.

I don’t think I’m exaggerating. Of all the things that make people dislike their job, having a bad boss has got to be in the top two or three. By contrast, “extremely difficult working conditions” isn’t even on that list. Check out all the stories of New Yorkers getting work done under whatever conditions they could during and after Hurricane Sandy for proof of that. Of every single complaint you’ve heard about someone’s working environment, “My boss is an idiot” has got to be the one that makes them the most unhappy.

Because of this, and because MongoDB’s engineering team is growing so rapidly, I have been thinking a lot about management, and some of the traits you find in people that, when found in the extreme, make them bad managers. Recently, my thinking on this has crystallized into a framework that I’m calling “Debugging the Boss.”

It’s basically a field guide to identifying these traits in the wild, along with some advice as to how to cope with these types of managers, both for those who report to them, and for those to whom they report. I think that every boss embodies some of these traits, and having one doesn’t make you a bad boss. The key is making sure you moderate, so that a trait which is ordinarily positive does not become negative. For this reason, my field guide includes a section on what positive trait gone wrong requires debugging.

An example of this, which I will expand on when I post my first entry in this series, would be how a sense of responsibility can become “hyper-ownership”. With this trait, a manager prevents their team from being meaningfully connected to responsibility and decision making.

The Series So Far:

Email Needs Embedded Apps

Email is my prefered method of communication (besides in person). At the same time, I get a lot of it, so making it better via tooling is very important to me. A large proportion of the emails I (and many other people, I’m sure – especially those in the tech world) receive are generated automatically, from LinkedIn notifications to Jira updates to monitoring alerts.

Email is good at receiving information, but then acting on the information is encumbered by the need to link out to a browser. In fact, the whole process is ungainly: receive email -> link out to browser for a single action -> return to inbox to delete message and move on, often to the next message generated by the same web app. Fixing this is just a matter of allowing mail clients a means of handling actions without linking out. They already format HTML and CSS, it’s time to add javascript as well. Let me give you a couple of examples.

When I’m monitoring a MongoDB cluster with MMS I set up email alerts, such as one for replication lag; I am emailed if lag crosses a certain threshold (2 minutes) because in that case I want to know about it immediately. What I read that email, there are a few things I want: to see current status (that email could be old), take an an action (acknowledge, escalate), or get more information. Current status is key, as there is often lag between event and email receipt. A list of one-click actions in the email would save considerable time when dealing with a full inbox. More information options embedded in the email would also save from switching context.

Another example is jira. I often get dozens of jira threads in my inbox between email reading sessions. Its great to be able to use gmail keyboard shortcuts to go between these threads and perform all the message management tasks common to email, such as viewing and archiving. What would be much better is being able to use keyboard shortcuts to make comments, edit fields, and take any other jira specific actions directly while processing these messages. Also having different views based on the device I’m using (desktop, mobile) would be very convenient.

One solution I’ve been thinking about is email apps. Emails could contain another mime section (text/email-json-application) that was a json document describing a minimal application. One option is for it to contain a URL for a javascript app (versioned for caching), along with the data payload particular to the message. The javascript API the app used would have to be consistent across all mail UIs and a pretty trimmed down system. Since it’s another mime type, this system would be totally backwards compatible since you could still have text/plain or text/html.

Google is working on something in this space. Google is the only company sucessfully doing real innovation in email, which definitely makes me happy. That being said, they’re trying to implement a limited set of features, and they’re doing it embedded in the the text/html segment of a mail message, which I don’t like. Overall, I think their solution is too timid, and a bolder idea is needed to really make a noticeable difference.

So what would have to happen for this to be real:

  • Someone crazy enough to take up writing a spec and api design
  • Write chrome extension for gmail as an experiment
  • Hook it up to some system that sends a lot of email


In Praise of SSH: The Cloud Print Edition

Problem: I’m at a playground in central park and I need to print something at home. I have a new printer that is online, but hasn’t been setup with Google Cloud Print yet.


  1. Set up an SSH tunnel from my phone, through my desktop to the printer (using iSSH on iPhone).
  2. Configure google cloud print from the browser on my phone.
  3. Print from chrome.

p.s. I was at a playground with my kids, not just to print something.

Coding Time for Managers

Update: I’ve written a much more in-depth article on this topic that was published in Dr. Dobb’s Journal on 1/7/2014.

The typical expected path of an engineer goes something like this:

  • Individual Coder
  • Project Lead
  • Team Lead
  • Manager
  • Director

At each step, their expected coding time looks something like

  • 90% (Individual Coder)
  • 80% (Project Lead)
  • 50% (Team Lead)
  • 1% (Manager)
  • 0% (Director)

This seems wrong. It is not shocking that engineering management is often found to be out of touch with the tech when they aren’t working on it themselves.

I prefer this view:

  • 90% (Individual Coder)
  • 80% (Project Lead)
  • 50% (Team Lead)
  • 30% (Manager)
  • 30% (Director)

I think managers and directors (and higher up) should be coding 30% of the time. That’s a lot of time out of a schedule, so I’ll grant, it’s unclear how to make it happen. I have a notion, though, that I’ve been trying.

The notion piggy-backs on the idea that one of the goals of a management job is to make yourself obsolete. It’s bad if things stop when you are on vacation.1 So to prevent that from happening, you should be working hard to create strong independent leaders under you.

Once you’ve done that, you should be able to spend 30% of your time coding without any ill effects. It could be 1 day a week, or blocks of time, or whatever works in your calendar. But if you’re not making the time, I think you’re doing yourself, your team and your company a great disservice.

How am I doing with this? Maybe a B-. (I’m only managing to code with about 20% of my time.) I’ve been using blocks method for a while, but am seriously considering switching to the day method and hiding for that time.

  1. I’ve come to think of this manager anti-pattern as the “hyper-owner”.

Email Aging Report

I get a lot of email. I used to think I got a lot of email, but that was before 10gen. Maybe one day I’ll remember writing this and laugh because comparatively today’s load is light. I hope not, because that thought is frankly scary.

There are a number of programs I’ve written to help me deal with email. One of them is less about helping me, and more about letting the people around me know that I don’t have a special desire to ignore them.

I used to respond very fast to incoming email… fast enough that I was a little notorious for it. That was back when I thought I got a lot of email. Then, I actually started getting a lot of email, and people started asking me more and more, “hey, did you get my email?”

I never like hearing that question. It puts me on the spot, and I feel bad. So, I decided to write an email aging report. It’s a very simple tool that shows unread email counts in my inbox by day. Then rather than ask if I read their email, they can get a sense for how behind I was. I’m pretty sure this doesn’t solve the deeper problem, but I get asked “hey, did you get my email?” less often, which makes me feel better.

Not everyone has this sort of issue to cope with, but plenty of people might still find it either useful or nifty to see this information on their inboxes, so I present to you my email aging report. The code is ripped out of a larger library that as I sanitize I’ll make more fully available.

Phone Screens

Doing a technical phone screen has always been a challenge for me. My preferred in person technical interview, especially for more junior engineers, is to take a relatively simple programming task, and dive deeply into it. The code will be simple enough to commit completely to a whiteboard, or a piece of paper, but I’ll lead the conversation to edge cases, performance, and how to test that code, for example. The coding task is really just the framework within which the interview happens.

What emerges from this conversation goes beyond seeing if the candidate can solve the problem. It lets me determine if the solution the candidate provides is a rote answer, or if they are able to analyze the whys and hows of the solution. Writing Merge sort, or Quicksort, for example, isn’t that hard, and many engineers can code it up in a couple of minutes. But when to use one or the other can be a subtle thing, and only discussion can make evident if someone gets that.

Also, engineers don’t work in a vacuum, they work on teams, which are sometimes quite large. A candidate’s ability to engage in quality discourse regarding the pros, cons, and other ramifications of sample code is also demonstrative of their ability to engage in similar discourse with their teammates regarding the real-world problems they will encounter.

I’ve found translating this to the phone doesn’t work very well. I’ve tried a variety of collaborative editing tools for doing programming, but all the nuances of an in-person interaction are lost, and I just can’t get a good of feeling for how someone is thinking about a problem. This is similar to the latency and throughput issues which make in-person visits to offices in other cities necessary.

Over time, my phone interview methodology has shifted more and more to discussions about engineering theory. I use questions that start off high level, but let you dive deep into a variety of technical areas. One interesting question is how you should go about choosing what language to build a project in. Something like “If you were going to build X, of the languages you know, which would you use and why?” The discussion can go through all sorts of trade offs of various memory models, concurrency, team composition, and type systems, any of which can lead to revealing conversations. I particularly enjoy when they end up in the bowels of different garbage collection algorithms.

The end result is similar to that of my in-person technique, in that a discussion of why gives me more insight than simple “how” answers.

None of this, of course, is to downplay the importance of the ability of an engineer to write clear, performant, bug-free code. But for me, an interview is not about that. Interviews are about learning how a person thinks, what kind of decision making process they use, and how they absorb and use new information.

MongoNYC 2013

MongoNYC 2013 is on Friday, 6/21, and I’m really looking forward to it. This is our 4th conference in New York City, and we’re expecting over a thousand attendees.

I’m delivering two talks one on Data Safety, and another on Full Text Search, which we added in 2.4. I’ll also be presenting the MongoDB Roadmap at the end of the day, during which I’ll both preview the short-term aims of the upcoming 2.6 release, and discuss how we think about the roadmap for the next few years.

While presenting is all well and good, that’s not what I enjoy most about our conferences. What I love most about our conferences are the opportunities to listen to our users. As I said in my last post, I’m obsessed with being in tune with the MongoDB community. The questions and harangues I get are a part of that process, only much more immediate than what I get out of reading tickets.

At MongoSF in May, for example, I had a couple of conversations with an engineer, whose name I sadly cannot remember, about his MongoDB use cases and needs. He is using MongoDB in a number of applications, none of which need to scale. This led a discussion of our general policy of not including features that don’t scale. From here, we quickly were able to go back to some first principles, and come up with some feature concepts that both scale and meet his requirements. I find this sort of iterative interaction very hard over email or chat, but highly effective in person.

Another thing I enjoy are whiteboard sessions, where I can focus on interaction with users’ real-world issues. There, I can get that two-way-handshake interaction with someone that confirms for me that I’ve successfully explained something to them, and hopefully helped them with a problem. So if you want to discuss a feature (current or future), or have a question you need answered, please come and spend some time with me (or another engineer) in one of the whiteboard sessions.

And lastly, I must say that another thing that I’m looking forward to is the enthusiasm of the increasingly mature MongoDB community. There is nothing like being in a room with a thousand people all interested in learning and talking about the project you’ve been thinking about daily for almost six years.

Reading Hundreds of Tickets at Once

I’m trying a relatively new thing these days: working through huge lists of open MongoDB JIRA tickets using a pencil and a big printout. This turns out to be a better way for me to handle this workload than sitting at a browser and doing it interactively. To explain this, I suppose I have to explain why I’m reading all these JIRA tickets.

I’m reading all these JIRA tickets because I don’t want to lose touch with the needs of MongoDB users, in spite of the ever increasing volume of related articles, blog posts, and yes, JIRA tickets. By reading all of these things, I am trying to keep an “on the ground” sense of use cases, issues, complaints, needs, and desires, which is invaluable in decision making. Knowing that this gestalt sense is honed and working well is crucial to my peace of mind. However, this sense is dampened and feels unfocused when the information comes through layers of delegation and summarizing.

Over the years I’ve developed a good ability for pattern recognition in the vast sea of open tickets. For example, while doing this very thing on a flight back from London recently, I saw that there were 10 open tickets related to one query parsing example that would be easy to fix in the new matcher. Reading all the tickets also prevents me from getting too distracted by new issues that attract attention, but aren’t as important as older issues.

So I read tons of JIRA tickets, and lately I’ve been using my favorite method so far: taking a giant printout of cases with me when I fly somewhere. My last batch was a single bucket of tickets, which printed out to 36 pages, containing about 600 tickets. I use a pencil, and mark tickets with which version to put them in, whether they are duplicates, if they should be closed, notes about implementation and what other tickets they might be related to. I generally mark about 20% of the cases in each pass.

During this process, I go into a bit of a zone, not the same as, but similar to, the zone I go into when coding. I’m not trying to triage particular cases or do meaningful work on an individual case, I’m trying to make mental links. So I move through the caseload quickly, keeping the overall view in memory, cross-referencing, looking for patterns. As I go, I’m building up a sense of areas that need more work, or where we can make headway quickly. This all-at-once method lets me observe patterns over time I might not otherwise notice. (If something comes up once every other week, it tends not to leave a dent in my thought process, but if it comes up once every other week for 4 years, that’s probably worth thinking about.)

Of course none of this works at all without 10gen’s amazing project managers that take these scribbles and do meaningful things with them.

So for those of you who use jira.mongodb.org, know that your tickets do actually all get read, and even if a ticket is really old, it doesn’t mean that we think it’s not important or that it’s being ignored.

In the end, this process gives me the confidence that when we have to rapidly shift plans, I can intuitively understand what the pros and cons will be. Given my role on the MongoDB project over the next few years, the ability to be agile will remain crucial.

In Praise of SSHFS

Emacs is the only editor I can use effectively at this point. It doesn’t matter if there are better choices (there aren’t ;-), it’s the one I’ve invested all of my muscle memory into. When working on files locally, I use normal emacs, and things are grand. Life, however, dictates that a great deal of my coding is done on remote machines. I had tried a variety of solutions to edit remote files (emacs in a shell, emacs of x, samba, nfs, etc…), none working terribly well for me.

Enter sshfs. I’m not entirely sure when sshfs crossed the divide between merely ok to solid, but it’s been more than a year that I’ve been using it happily. sshfs lets you mount a filesystem from any remote machine you can ssh into using FUSE. So when I’m at home, I can do

sshfs office2:work/mongo localworkmongo
to (and this will be intuitive for users of scp and mount) mount the ~/work/mongo directory on office2 onto the localmongo directory on my Mac. Then I can edit files in localworkmongo normally, and I just ssh into the work machine for compilation and execution.

This has been a life changer for me. It has made using a Mac desktop as my only workstation feasible, since most of my development is on a linux desktop sitting next to the Mac under my desk. It makes transitioning between work, home, or a different office painless, and even lets me do tricks like mount ec2 volumes just to poke around if I want to do something locally.

The key for me is that security is all ssh. I already use ssh for everything and go through painstaking efforts to make sure that is both secure and easy. So having to do no additional work to mount things is magic. And just to be clear, the target machine needs no additional software. If I can ssh into it, I can mount files on it.

Also, note that this solution isn’t about using a Mac everywhere, it’s about getting to use your favorite desktop apps to edit your work files no matter where they are… this solution applies just as well to editing Photoshop files mounted on a remote server.

sshfs is in package management on many platforms, including Homebrew for the Mac. As it says in the caveats, make sure you understand the info regarding the FUSE kernel extension before trying to use it.

Mongo's New Matcher

MongoDB 2.5.0 (an unstable dev build) has a new implementation of the “Matcher”. The old Matcher is the bit of code in Mongo that takes a query and decides if a document matches a query expression. It also has to understand indexes so that it can do things like create a subsets of queries suitable for index covering. However, the structure of the Matcher code hasn’t changed significantly in more than four years and until this release, it lacked the ability to be easily extended. It was also structured in such a way that its knowledge could not be reused for query optimization. It was clearly ready for a rewrite.

The “New Matcher” in 2.5.0 is a total rewrite. It contains three separate pieces: an abstract syntax tree (hereafter ‘AST’) for expression match expressions, a parser from BSON into said AST, and a Matcher API layer that simulates the old Matcher interface while using all new internals. This new version is much easier to extend, easier to reason about, and will allow us to use the same structure for matching as for query analysis and rewriting.

This matcher rewrite is part of a larger project to restructure query execution, to optimize them, and to lay the groundwork for more advanced queries in the future. One planned optimization is index intersection. For example, if you have an index on each of ‘a’ and ‘b’ attributes, we want a query of the form { a : 5 , b : 6 } to do an index intersection of the two indexes rather than just use one index and discard the documents from that index that don’t match. Index intersection would also be suitable for merging geo-spatial, text and regular indexes together in fun and interesting ways (i.e. a query to return all the users in a 3.5 mile radius of a location with a greater than #x# reputation who are RSVP’ed ‘yes’ for an event).

A good example of an extension we’d like to enable is self referential queries, such as finding all documents where a = b + c. (This would be written { a : { $sum : [ “$b” , “$c” ] } }.) With the new Matcher, such queries are easy to implement as a native part of the language.

Now that the Matcher re-write is ready for testing, we’d love people to help test it by trying out MongoDB 2.5.0. (Release Notes)