On November 1st, 2016, I gave a talk at the CTO Summit series hosted by NASDAQ. It was a 20 minute talk, updating and expanding on a topic I’ve both blogged and written articles about – how important it is for engineering managers to keep their hands in the codebase they make decisions about. Here it is:
Last week I was in Israel for the MongoDBeer meetup and an enterprise event, both hosted by Matrix, one of our partners, and a few really great client meetings. One of the things that I don’t get to do often enough these days is work directly with customers on interesting technical challenges, so those client meetings were really quite invigorating.
I was reminded of this recently when I was doing a fireside chat with Albert Wenger at NYCode, an event hosted by NextView Ventures. We were talking about some of the things we did early on at MongoDB that led to the great momentum we now have. Albert said that a major factor was how obsessed I was with making our users successful with their deployments. That’s true, I was completely obsessed. I had this thing about all the questions on our google group being answered as fast as possible. Day or night, if someone had a problem, I was trying to fix it with them.
As my responsibilities to my team grew, I had to leave that phase of my role behind me. It was hard to do. Albert joked on stage about how it became a board-level priority for me to stop handling support issues.
But my obsession produced more than successful users. In that formative period, those interactions taught me what people wanted from our product, enabling me to steer MongoDB towards where it should be better than I possibly could have without those direct relationships.
Of course, that “formative period” will never end, and my role remains to ensure that MongoDB is always evolving to meet the needs of as many users as possible. This is the sibling to my claim that engineering managers have to keep their hands in the code: a technology leader should never stop working directly on customer issues. If you do, and only get filtered information, you will not be able to help make good decisions, and in fact run the risk of making poor decisions. Does that take a lot of time? Of course it does. But spending that time isn’t a nice-to-have. It’s core to your job, and a well-run team works independently enough that you should have time for it. If you can’t find the time, you need to to reevaluate the rest of your commitments, as I periodically have to.
Are you getting enough exposure to your customers’ issues?
Back on April 25th I spoke at dotScale in Paris; I gave a talk called “The Case for Cross-Service Joins,” as in queries that join data across multiple 3rd party services. For example, analytics over data that comes from both SalesForce and Googe Analytics. I’ve been thinking a lot about this topic, because MongoDB sits at the middle of a lot of apps that utilize 3rd-party services, and the benefits of building your app on top of such services comes at the cost of that data being siloed away, and difficult analyze it in a holistic way. My thinking on this topic continues to evolve, and I’ll be writing more about that, as well.
DotScale was a great conference to speak at, with a lot of very insightful talks, and beyond that, the gorgeous Théâtre de Paris is easily the classiest venue I’ve ever spoken in.
The video is up now:
My overarching goal as CTO of MongoDB is to make building applications easier and easier. Given my day job, it’s actually been a little while since I did any work on the front end.
Sure, it’s not much to look at, but I was really excited about how easy it was to do full client rendering with just API calls for data compared to last time I tried. In addition, I finally took ES6 for a test drive, and have to say classes and modules live up to what I had hoped for.
I think next up is react.
One theme I kept harping on at MongoDB World a few weeks ago was knowing when to innovate around new ideas and when to just reuse what already works well for products that have been successful. This comes up continuously at MongoDB, because having a good understanding of it is a significant competitive advantage. I attribute a large extent of MongoDB’s success to our unbending adherence to this discipline.
When we started MongoDB, we had a clear goal - make data and databases easier for developers and operators, so that data and databases serve their users, not the other way around. To that end, there were two key things we wanted to change. First, we wanted to reduce the impedance mismatch between application code and data, which we addressed by building our database around collections of documents, rather than tables of rows. Second, we wanted to make distributed systems accessible and usable by most organizations, which we have done by making them first-class components and intuitive to use, instead of leaving them to higher layers of the stack to build.
Everything else we wanted to leave the same. We would ask ourselves, “Does this need to be different in a document based distributed database?” For example, indexes in MongoDB have the same semantics as they do in their relational counterparts, because they have worked quite well for decades, and their semantics match those used to query a document database. Query results composed of documents need the same filtering and/or sorting as those composed of rows, and indexes that cover an ordered set of fields (a, b, c) are redundant with indexes that cover a same-ordered subset of those fields (a, b). MongoDB provides a shell that mimics a relational shell, because in our world, the needs to explore databases and collections, do ad hoc queries, create indexes, get stats, and perform analytics is identical to those needs in a relational world. Even mongodump name came about because I had been using mysqldump for a decade, and making data easy to work with and distributed systems accessible would not be in any way furthered by changing that aspect of a database.
Today, we are constantly improving features and adding new ones to MongoDB. Every time we do, the question is: do we need to invent something new to make this feature fit into the document world or a distributed systems world? If the answer is yes, we innovate to try to make suitable for MongoDB. If not, we try to find the best solution out there and apply it to MongoDB (in a patent safe way of course).
Why is this so important? First off, it takes a lot more though to invent something than to copy something. Taking semantics or ideas from successful systems focuses design and architecture work where it is needed most. Focusing innovation also makes it easier for users to learn a new system. If everything is needlessly different, it will be more frustrating for your users, so there better be a good reason. And lastly, every innovation involves risk. You think you are improving something, but if you’re wrong, you’ve wasted time and have to do it all over again.
This is an important concept for all companies to master, both new and old. Like the adage “Is this a core competency” for helping decide if you should build or buy, all product teams adding features should be asking themselves “Does this need to be different in our domain?”
Almost all modern applications are composed of presentation layers, services executing business logic, and backing stores where the data resides. Developers could be more productive and agile if they could work more directly with the backing data without having to build specific APIs for every access type, but is quite a challenging problem. An emerging class of solution known as Backend as a Service (BaaS) has tried to address this problem over the last few years, but hasn’t become the norm yet.
In an ideal world, it would be great if your web or mobile app could talk directly to a database. In the real world, though, this is never done, for several reasons. Let’s start today with the security and privacy area: fine grained access needs to be built in from the ground up, and also be expressive enough to let any complex application to be built.
Security and privacy challenges are about allowing different users to have access to different data, different documents, and maybe even different fields. One might need to query on computed values of fields without being allowed to see those fields directly. A famous example of this is Yao’s Millionaires Problem in which two millionaires want to determine which one is richer without revelaing their net worth. Solving problems like that requires the kind of fine-grained access control to allow a user to run queries such as “show me all documents where a % 5 == 1” but not be able to see the actual value of a. A broad category of problems, of which Yao’s Millionaires is one, is called secure multi-party communication, and thier solutions all rely entirely on offering that kind of access control. If you are building your own REST api for your web app, building in that logic is trivial. If you are trying to build a generic BaaS, it’s a lot more complex.
There are a few BaaS providers working on this problem. Parse and Firebase are probably the best examples at the moment. They both definitely have pushed this along pretty well, but I think another big step function is needed. Further pushing the security and privacy model to allow apps to be more expressive will allow BaaS to radically improve time to market for many applications.
Last Thursday (4/7/2016) I spoke at the 2016 NYC CS Fair. Their number one goal is to encourage public high school students who study CS to stick with it, by showcasing all the great opportunities that await them should they pursue a career there. I talked about being a hacker, how to negotiate CS studies in higher education, the difference between CS and software engineering, and the importance of a good mentor. It was a great group of kids, and if even 25% of them go on to become CS students, I think the industry is going to see some positive change.
The CS Fair is put on by CSNYC, with the mission of “Computer Science for all”. One of the reasons I support csnyc is that I believe it actually has a chance of moving the needle for diversity in tech. My personal belief is the only way to make meaningful, lasting change in this area is to get kids excited about the field earlier and earlier. The more children think of CS as their thing, the more they can resist the fear, negative pressure, or bias they contend with, pushing them them to drop it.
In order to get kids excited about computer science, a few things need to happen, but the most interesting one to me at the moment is bootstrap. Bootstrap is a program that integrates a math and computer science curriculum that is meant to drop into an existing math class and be a better way to teach concepts like functions and variables, while also learning computer science. There are quite a few schools starting to use bootstrap, and I’m trying to help drive adoption. Others are doing way better and more than me, but I’ll keep trying anyway. If you’re involved in CS education, leave me a comment. How can I help?
I’ve always found that the fireside chat is a format with a lot of potential to be boring, but Matt is a great interviewer, and interacting with him on stage definitely adds to the event. For example, when I was talking about the headline features of our 3.2 release, I omitted a significant pair – the BI connector and Compass – and he reminded me to talk about them. It’s things like that which enhance the experience for the audience. At their best, a fireside chat interviewer takes care of the setup, makes sure you’re staying on track, and grabs opportunities to dig deeper.
One thing that Matt brought up (at around 12:40 in the video) was how, after an explosion of alternatives to relational databases, it’s starting to feel like things are converging again. Now, when you do one of these, you get a list of topics to prepare for in advance, but that’s a question that emerged organically from our conversation. I appreciated the opportunity to address that by citing a core tenet of MongoDB (at 14:45):
“We really want you to be able to configure yourself into different use cases, rather than having to use different kinds of products.”
All the other speakers were very interesting. I was particularly into the Dr. Kieran Snyder’s Textio presentation – that’s a cool product, with cool tech, and a lot of potential. Think about it: there’s an enormous amount of text in the world. You will never be able to read it all. Algorithms that understand those vast swaths of things you will never be able to personally consume have the potential to revolutionize knowledge. Or consider influential articles that wind up with thousands – or maybe some day millions – of comments… stuff like this can sift signal from noise. There’s a lot of promise there. Kieran and I had a really interesting chat about this stuff after the event, and I’m looking forward to see where they take Textio.
All in all, a really enjoyable evening, I would definitely recommend Data Driven NYC to anyone in New York with an interest in tech at all. They happen monthly, so it should be relatively easy to catch one.
Last week I talked about Parse shutting down and how unfortunate that was, but also how outstanding a job they have done providing a transition path for their current users. MongoDB also published a very detailed post on how to migrate a Parse app onto MongoDB Cloud Manager and AWS Elastic Beanstalk. Since that day, the amount of activity on the open source Parse Server has been phenomenal, and many have suggested, as did one commenter on my last post, that this means it’s time for MongoDB and Parse to work even better together.
All this discussion I’ve had about Parse has got me thinking about the nature of the Mobile Backend-as-a-Service space and MongoDB’s role there. I’m also interested in hearing directly from customers. If your MBaaS-backed application deals with a decent amount of data and load, leave a comment or shoot me an email (firstname.lastname@example.org), I want to talk about making MongoDB and the ecosystem around it even better for the MBaaS use case.
Updated 2/3/2015 to reflect the publication of MongoDB’s migration guide.
I was sad to hear about Parse shutting down last week.
Parse made a big push towards serverless architectures, which I think is a great goal. Serverless architectures are the ultimate in letting developers focus on making great products for their users and letting other people make the plumbing work.
In the early days of web and mobile application development, backends were a thing that every team had to write themselves from scratch. Over time, common patterns were encapsulated into application frameworks. Parse was a glimpse farther into the future, providing app developers an abstraction for an entire backend.
I’m a fan of this approach. MongoDB’s number one focus has always been on making developers more productive, so they don’t have to work so hard to use their databases. As more and more applications are built using a set of services, the next step will be around making those services composable, their data composable, and making those pieces really easy to build, deploy, and maintain.
I’m also really happy to see the way that Parse is handling their shutdown. They are giving users one year to move off, providing tools to move off, and provided an open-source version of the backend so the migration can be as painless as possible.
MongoDB has published a guide on how to use cloud manager and elastic beanstalk to help those migrating off of Parse. We’ll also offer a consulting package for those who want a bit more customized help.