The Slow Climb of Postgres and the Value of Persistence

Those intrepid individuals trying to bring a new innovation into the world, or a new start-up, should keep in mind the power of perseverance. Even if a technology is a hit, the creator must still face any number of hurdles to maintain and grow its success.

Attendees at the PostgresVision conference in Boston this week certainly heard a great testimonial for this idea, as database pioneer Dr. Michael Stonebraker offered an account of how he helped bring PostgreSQL database into the world, a 15 year journey of amazing highs and lows.

Stonebraker had already helped birth the first world’s first relational database, Ingres, a decade before, but he was back in the laboratory in early 1980s to create Postgres (later amended to “PostgreSQL” to reflect its SQL powers). Ingres was such a success. In fact, that users wanted to add their more arbitrary data types and operators to its schema, for use cases such as geolocation, he explained. The initial workaround to this problem with Ingres to encapsulate that logic within application code, often with the horribly unintuitive Prolog.

So the idea with Postgres, which debuted in 1984, would be to offer a relational database system with support for “advanced data types” (ADTs). ADTs set the stage for much wider use of relational databases, the new capability — built on object-relational mapping — saving untold hours in developer effort. Postgres was also the first database system not to overwrite data when an update was needed, and instead, just add a new entry and delete the old one, which opened the door to capturing a verifiable transactional history.

But Postgres was anything but an immediate success, no matter how obvious its value. First Stonebraker and his team had the seemingly good idea to write this new database in the best possible language of the day, LISP. LISP, however, turned out to be a terrible choice, because of its slow performance — not a characteristic you’d want in a database. So a rewrite into the much more performant C was necessary.

They also ran into a potentially-crippling catch 22. Early users wanted not just the crude ADTs created by the Postgres developers. They wanted sophisticated ADTs to work with the commercial applications they used, such as geo-mapping software from Arcvinfo. But when they approached companies such as ArcInfo and asked for their help, they were told the Postgres user base wasn’t then large enough to warrant such an effort. The lack of commercial ADTs would keep Postgres on the sideline, yet software vendors weren’t interested in creating ADTs for their own products no matter how useful they would be to Postgres end-users.

The database designers were seemingly stuck, at least until another use-case came into view, supporting the then-nascent but rapidly-growing Internet, and gave Postgres a bountiful user-base. In 1996, Stonebraker sold the company he formed, Illustra, to commercialize the database system, and went on to create other databases for other use-cases, such as analytics oriented Vertica.

These days, Postgres is the currently fourth most widely-used database system, according to the DB-Engines Ranking, just ahead of NoSQL MongoDB. EnterpriseDB, which offers commercial support for the open source database (and is the organizing sponsor for Postgres Vision),  sponsored an independently-executed benchmark test to compare Postgres and MongoDB, which was just enhanced with multi-transactional capabilities.

Despite MongoDB’s fast performance, the benchmarking company, Ongres, found that MongoDB had difficulty matching the transactional performance of Postgres. And, with more than 16 threads, Postgres left MongoDB in the dust entirely.

“PostgreSQL 11 was found to be faster than MongoDB 4.0 in almost every benchmark,” the report concluded. “Throughput was higher, ranging from dozens of percentage points up to one and even two orders of magnitude on some benchmarks. Latency, when measured by the benchmark, was also lower on PostgreSQL.”

Work still needs to be done to give Postgres multi-master capability, which would make it truly scalable, across multiple servers. Still, its presence in today’s business stack is not too shabby for software that is 35 years old.

Good ideas never come from “communing with nature on a mountaintop,” Stonebraker said. They come from talking with users, many users. Serendipity always helps (and anti-serendipity always hurts), but that is beyond your control.

And most importantly, Stonebraker reminds us that success comes from persistence, weathering the inevitable lows as well as the highs.

A Brief History of PostgreSQL

1973was the year that saw the end of US involvement in the Vietnam War, the year that English rock band Deep Purple released Smoke on the Water, and the year the US Supreme Court ruled on Roe v. Wade. People were sporting Farrah Fawcett hairstyles and computer technologies were changing the way businesses operated. The corporate sector was driving development of Database Management Systems (DBMS), using them to maintain information about clients, vendors, employees, inventory, supplies, product orders, and service requests. At the same time, in a lab at the University of California at Berkeley, Eugene Wong and Michael Stonebraker were beginning work on INGRES (INteractive GRaphics REtrieval System), one of the world’s first RDBMSs. Initially, the researchers had raised funds to build a geographic database system for Berkeley’s economics group (1). However, inspired by Edgar Codd’s 1970 seminal work on the relational model (2) and by white papers that IBM had just released regarding SystemR — the first implementation of SQL and an experimental database system built on the relational data model (34), the two researchers decided to use the money to seed their relational database project instead (1).

For the ongoing support of the larger INGRES project, the two received funding from the National Science Foundation and three military agencies: the Air Force Office of Scientific Research, the Army Research Office, and the Navy Electronic Systems Command (15). The context of the military’s interest in the development of such systems, together with the the confluence of the Cold War and the emergence of computer technologies to manage information, likely influenced the system’s concern with access control, protection, security, ownership, and the database “administrator.”