What Can the History of Data Tell Us About the Future of AI?

“Data is the fossil fuel of AI.” – Ilya Sutskever, co-founder and former chief scientist at OpenAI

“The best prophet of the future is the past.” – Lord Byron

“Show me the incentive and I will show you the outcome.” – Charlie Munger

I have decided to write about the history of data for several reasons. First, I work in data, and I like to know the history of my field. Second, I believe the best way to understand what might happen in the future is to understand what’s happened in the past. And third, I believe the trends we can learn from the history of data could tell us a lot about the future of AI. Data is the fossil fuel of AI, after all. When I get curious (or afraid) of what AI might mean for humanity, I look online to see what experts say, and I get confused.

“There is a 10 to 20 percent chance that AI will lead to human extinction within the next three decades.” – Geoffrey Hinton (“Godfather of AI”) — The Guardian, Dec 2024

“I’ve always thought of AI as the most profound technology humanity is working on—more profound than fire or electricity.” –Sundar Pichai (CEO, Google/Alphabet) – CNBC, Oct 2016

“There is some chance that is above zero that AI will kill us all.” – Elon Musk — NBC News, Sept 2023

“AI is the new electricity.” – Andrew Ng (Co-founder, Google Brain & Coursera) – 2017

“The development of full artificial intelligence could spell the end of the human race.” –Stephen Hawking — BBC interview, Dec 2014“

That’s why AI is exciting… What if we can have the kind of economic growth [we enjoyed in the early 20th century] only this time it’s much more even?” – Satya Nadella (CEO of Microsoft) – TIME, 2023

AI may be the end of the human race, or be as impactful and beneficial as fire or electricity. I am no AI expert, and I don’t even really understand what it is or how it works, but rather than throw my hands up in the air and say that the future of AI is somewhere between apocalypse and utopia, I started reading. My logic is that if I can understand the history and current state of data, I may have a better idea of the future of AI—at least better than the jokers I just quoted.

I break data into three types based on what it is about: personal, public, and enterprise. Personal data is data about individual people—all of the data stored on your personal computer and all of the click data that tech companies harvest from you. Public data is data about the world, which doesn’t necessarily mean it is free. Enterprise data is data about companies. It’s mostly stuff that doesn’t live on the public web, though it increasingly lives in the cloud. I know that there are additional ways to categorize data other than by what it is about. The type (text, images, video) of data, for example, can be equally important. We’re not going to talk about that here. 

My goal with this piece is to understand how data has changed over the past 40 years in terms of what is collected, how it is stored and what it is used for. To do that, I first had to explore the devices and architectures that shape those trends. Along the way, I found that what gets collected is only half the story; how that information is monetized is just as important. The SaaS business model and AdWords (the way Google began placing ads in search results) are just as impactful as any technological breakthrough, for example. I’m convinced the next wave of AI will be driven by exactly these forces: who captures the data, how they capture it, what kinds they capture, and the business models that turn that data into dollars. 

This article is meant for data practitioners who are interested in the future of AI but overwhelmed with articles by people claiming to know what the future of AI will look like. I have no idea what the future holds, but understanding how we got here is a good first step. My next piece will get into actual predictions about the future, which will be falsifiable claims so that I can be held accountable. I will use Philip Tetlock’s framework from his book, Superforecasting, to make these predictions. Here’s an overview of what this article will cover:

Part 1 is about Stewart Brand, my favorite person I learned about through this research. 🤘

Part 2 of this story is about the personal computer. Personal data really began with the dawn of the PC, which started in full-force in 1981, when IBM launched the IBM PC. The IBM PC ran MS-DOS, the operating system built and licensed by Microsoft. When “clones” of the IBM PC, like Compaq and Dell, popped up, they also used MS-DOS, benefiting Microsoft. Apple, on the other hand, never licensed their operating system. Microsoft remains, primarily, a software company, and Apple, a hardware company.

Part 3 is about how personal computers enabled enterprise data to move away from mainframes and mini-computers and to a client-server architecture—client PCs sharing data on a centralized database. This shift meant more people had access to enterprise data and apps, but created a nightmare of systems integrations and data alignment that persists to this day.

Part 4 is about how Tim Berners-Lee (TBL) invented the World Wide Web in 1993 and personal computers became portals to the Internet. The first “Browser War” began, mostly between Netscape and Microsoft’s Internet Explorer. It also goes into TBL’s original vision and the degree to which it has been realized with public data, notably Wikipedia.

Part 5 is about the rise of Google and Amazon in the 1990s. Google began scraping links off the Web and building a search engine. They eventually learned that the best way to make money on the Web was by harvesting click data (data about how people use the Internet) and using that data to serve targeted ads. They called this product AdWords. Amazon started as an online bookstore but quickly grew to an everything store. As they grew, they also built massive data center and started renting server space to other companies to run applications and store data. “The cloud” was born.

Part 6 is a deeper dive into the move to the cloud, using Nicolas Carr’s The Big Switch as reference. In his book, he draws a parallel between the growth of electricity as a utility in the late 19th century and the rise of cloud computing in the late 20th century.

Part 7 is about how enterprise data has started moving to the cloud, starting with Salesforce in 1999. The client-server architecture is replaced with “Web-based” architectures, using the technology of the World Wide Web, and then to a software as a service (SaaS) model, where the vendor hosts the entire architecture themselves and sells subscriptions rather than the software itself. Additionally, because of technologies like parallelization and virtualization, companies were able to store and compute data across multiple servers, leading the the rise of the “data lake.” I take some time here to highlight that the problem of integrated data that flared up during the client-server architecture era has still not been solved, but that Tim-Berners Lee’s vision of the semantic web might hold promise.

Part 8 is all about Facebook and the rise of social media. They took the business model that Google pioneered with AdWords and applied it to even more personal data.

Part 9 details the launch of the iPhone, which put computers in our pockets and changed the way personal data is captured. This led to entirely new industries like ride sharing and dating based on proximity. It was so successful, Apple became the first company with a half-trillion dollar market evaluation in 2012 and the first to a trillion in 2018 (Haigh and Ceruzzi 401). This also confirms Apple’s place as primarily a hardware vendor.

The conclusion goes through the major players business models and the data they collect. This helps to refine the kinds of questions that I will attempt to answer in part two: Will the business model of AI run on engagement-driven ads (Google & Meta), lock us into hardware ecosystems (Apple), or charge rent on subscription software and cloud capacity (Microsoft & AWS)? All of the above? Something entirely new? What will AI driven by different revenue models look like? Will ad-driven AI sacrifice accuracy for engagement? What tactics would AI use to drive engagement if it had all of our personal data? Outrage? Love? Will AI continue to focus on public datasets about the world (Wikipedia, Internet Archive, Common Crawl) or will personal and enterprise data become a bigger part of the training set? If AI is primarily trained on public data, how will it understand and integrate with personal and enterprise data? Retrieval-Augmented Generation (RAG)? Agents? Fine-tuning? Semantic layers? Small Language Models (SLMs)? Confidential computing? Will AI be packaged in a brand new device that harvests data in new ways like the iPhone did in 2007—and if so, who builds it? What kind of device?

Since I know you won’t read all of this, here are some major takeaways from my research:

First of all, there are not enough women in this history. Here are a few women that I want to highlight as being significant in the history of data and computers: Ada, Countess of Lovelace, was an artist and a mathematician and wrote the first computer program in 1843, a full fucking CENTURY before Alan Turing (Isaacson 33). Grace Hopper wrote the first compiler, wrote the first computer handbook, and championed COBOL, turning programming from arcane machine code into English-like instructions that anyone could learn (Isaacson 88). Larry Page and Sergey Brin didn’t start Google in their garage; they started it in Susan Wojcicki’s garage. Wojcicki became employee number 16 and oversaw their advertising and analytics products, including AdWords, “the most successful scheme for making money on the Internet that the world had ever seen” (Levy 83). She then managed the $1.65 billion acquisition of YouTube and became the YouTube CEO from 2014 to 2023. And Facebook never made a profit until Sheryl Sandberg showed up, ended the dorm room boys’ club, and turned Facebook into a real (and profitable) company (Levy 190). 

There is a lot more written about the personal computer era and the Steve Jobs/Bill Gates rivalry than any other part of this history. It is an interesting period, but we need more books and a biopic about Larry Ellison (starring Sam Rockwell) and the whole enterprise side of data.

There is also a lot written about the personalities of these billionaires. I am less interested in their psychology than the outcomes of their decisions, but it is hard to not see some patterns. Generally, the most common personality traits of these guys (Gates, Jobs, Ellison, Bezos, Zuckerberg, Brin, and Page) were that they are stubborn, relentless, and irreverent. 

The business model often followed the product. There’s probably a word for this that you learn in business school, but I didn’t go to business school. Often, the product becomes ubiquitous, and then the company figures out a business model and revenue stream to fund it. Google is the best example—it became the largest search engine in the world before they figured out they could use targeted ads to print money. Same with Facebook—they weren’t profitable until Sheryl Sandberg joined and informed them they were an ad company. 

Conversely, a product may become ubiquitous and a revenue stream never develops. Microsoft spent a lot of time and money (and became the plaintiff in an antitrust lawsuit) destroying Netscape. But once they had the most popular browser in the world, Internet Explorer, it didn’t matter. There’s not nearly as much money in browsers as other parts of the Internet. That being said, if you don’t win wars, you lose wars and die. The browser wars did have an impact on Netscape—it doesn’t exist anymore.

Established companies often do not embrace new technology fast enough because of their established success. This is known as the “Innovator’s Dilemma” and described in Clayton Christenen’s book of the same name. Basically, a company that has found product-market fit will incrementally improve their product to meet the needs of their existing customer base. An alternative product or architecture could cannibalize this existing revenue stream so they ignore that and focus on the thing that works. IBM invented the relational database but they didn’t commercialize it because they didn’t want it to encroach on the revenue of their hierarchical database business line. Similarly, Oracle was able to beat SAP to market with a web-based architecture (the E-Business Suite) because SAP didn’t HAVE to pivot—their client-server product (R/3) was massively successful. Barnes and Noble didn’t want to risk investing in an online store that wasn’t, at the time, as profitable as their brick and mortar stores (Stone 59).

The revenue model matters more than just dollars and cents. Companies actions can be better understood by understanding their underlying revenue model. Google didn’t create Chrome or buy Android to make money directly—they were tools to get more people to spend more time online and get served ads. Facebook’s content can be horrific and drive outrage, but outrage drives engagement, and engagement drives ad revenue. 

Moore’s law (the observation that transistor counts double about every two years) has held but slowed. Intel CEO Pat Gelsinger said in 2023 that the industry is now “doubling effectively closer to every three years.” And Butters’ Law of Photonics (that the data capacity of an optical fiber roughly doubles every nine months) held true through the 2000s, but advances have slowed to roughly every two years as systems near physical limits. Through much of the 2000-2020 period, Butters’ Law enabled fiber to replace legacy telephone lines. 

Data > Storage > Computation > Communication: The volume of data created has always been much greater than the total storage capacity. Storage capacity has always been greater than processing power. And processing power has always been greater than the ability to communicate the insights of those computations. I am not a brain doctor, but I think this is the same for humans: we perceive far more information than we can remember (store); we store more than we can think about at any given time (computation); and we think about more than we can effectively communicate.

There is a positive feedback loop between data, product, and AI. The best product gets market dominance, which allows it to collect more data which allows it to improve its algorithms which allows it to expand market share which…

Data is moving to the cloud. Duh. Enterprise data and apps are increasingly built on the hyperscalers—AWS, Google Cloud, and Microsoft Azure. There are even SaaS-native database companies built on this infrastructure like Snowflake and Databricks, which are the fastest growing database management systems (53 percent and 42 percent year-on-year revenue growth, respectively). For personal data, billions of users feed information into apps like Facebook, Instagram, and TikTok, on cloud-based collaborative tools like Google Workspace, and streaming services like Spotify and Netflix. Spotify has shut down its data-servers and runs everything on Google Cloud and Netflix completed its cloud migration to AWS in 2016. Even the CIA uses AWS.

Connecting enterprise data has been a headache through every architectural era. Whether in a client-server architecture or in the cloud, connecting data to make useful insights has been a challenge for decades. Oracle tried to solve this with their “one company, one database” initiative (Symonds 168) but realized that the “key to everything … was a shared data schema, allowing semantic consistency” (Symonds 188). With the rise of cloud computing, companies again tried to solve their siloed data problem by putting it all in one database, but this time called it a “data lake.” No surprise that this didn’t work because of the lack of a unified semantic layer. 

Graph analytics fueled the rise of Google and Facebook: From day one, Google’s PageRank and Meta’s social graph mined network connections to rank pages, notifications, and target ads, making graph analysis on metadata the engine of both companies’ meteoric rise.

Revenue models and data resources can tell us about where AI is going, or at least which questions to ask: At least, this is my theory.

Google and Meta are advertising companies. They are using AI to get users to engage with their products more so they can serve them more ads. They are creating devices (Meta’s Ray-Ban’s and Google’s Warby Parkers) to get people online more so they can serve them more ads. AI is a feature of their products to drive engagement. They also have a ton of personal data; Google knows our search history and Meta knows everything about us. Google also has a lot of enterprise data through their Google Workspace and Google Cloud Platform and a lot of public data because they are the largest search engine on the Web. What will AI built to maximize engagement look like? Will Meta and Google be able to use their data to fuel new kinds of AI? 

Apple is a device company and their revenue is driven by hardware sales. They are embedding AI directly into their devices so they can sell more of them. They have a lot of personal data too, though they don’t use it to sell targeted ads. Will they be able to integrate AI into our personal lives in a way that keeps them on top, or will OpenAI’s new device replace the iPhone?

Microsoft rents its software and servers, and makes most of its revenue on these subscriptions. It is incorporating AI into these applications (Copilot) to drive expansion. Other enterprise software companies (Google, Oracle, IBM, SAP, Salesforce, Workday, and ServiceNow) are doing the same. Microsoft’s Azure is also the second largest cloud computing platform behind AWS. Will they be able to integrate AI into the enterprise enough to stay on top and/or supply the servers that power the AI?

Amazon is a space exploration company funded by some terrestrial enterprises. Will Bezos be able to maintain dominance in the cloud with AWS enough to start building an O’Neill cylinder?

1. Acid Foundations

I know I just said we’d start in 1981, but I want to take a moment to recognize the coolest person I learned about in all of the reading I did for this project: Stewart Brand. The personal computer movement and bringing “power to the people” in the 1970s and 80s was a direct consequence of the hippies and the beats of the 60s, and Brand is the embodiment of this transition. “The counterculture’s scorn for centralized authority provided the philosophical foundations of the entire personal-computer revolution,” Brand himself wrote (Isaacson 269).

Brand was a part of the “Merry Pranksters” in the 60s—Ken Kesey’s LSD-fueled group who rode a bus driven by Neal Cassady (Dean Moriarty from On the Road) cross country, making pit stops to throw psychedelic parties and jam with the Grateful Dead. While tripping one day, he became convinced that seeing a picture of the whole earth from space would change the way people thought about protecting our home and petitioned the government to take and release a picture from space. Famed inventor, architect and futurist Buckminster Fuller offered to help, and some NASA employees even wore Brand’s pins that said, “Why haven’t we seen a photograph of the whole Earth yet?”

After NASA took the photo in 1967, Brand started the Whole Earth Catalog with the picture of the Whole Earth on the cover. The catalog was a do-it-yourself magazine teaching people how to use tools (including computers), be self-sufficient, share resources, and resist conformity and centralized authority (Isaacson 265). This magazine would inspire many young people, including Steve Jobs, who would famously quote it during his 2005 Stanford Commencement address: “Stay hungry, stay foolish.”

After starting the Whole Earth Catalog, he met Douglas Engelbart, an engineer running a lab focused on how computers could augment human intelligence. They took LSD together at the lab, and Brand parlayed his experience throwing psychedelic trip fests into helping Douglas Engelbart give the “Mother of All Demos” in 1968. This was the first time many fundamental parts of the personal computer were shown: the mouse, on-screen graphics, multiple windows, blog-like publishing, wiki-like collaboration, email, document sharing, instant messaging, hypertext linking, and video conferencing (Isaacson 278).

He realized that computers were the new drugs and “hackers” were the new hippies. He organized the first Hacker’s Conference in 1984. He started the WELL in 1985 (The Whole Earth ‘Lectronic Link), one of the first and most influential virtual communities. It was craigslist before craigslist (though its founder Craig Newman was a member of the WELL) and “AOL for Deadheads” (AOL founder Steve Case was also a WELL member).

The personal computer was not created by corporate suits. Yes, IBM brought the personal computer into the mainstream, but a lot of the pieces they put together had been invented by hippy hackers who read the Whole Earth Catalog. These innovations were driven by people fighting against straight-laced corporate conformity, trying to bring the power of computers to the individual. Think about how trippy it is that the words you’re reading are tiny flashing lights on a screen that you’re moving with your finger. That couldn’t have been envisioned in a board room; it was the function of anti-authoritarianism, irreverence, free love, and psychedelics. 

What’s wild is that Stewart Brand is still alive today and actively working on futuristic environmental problems like the Long Now Foundation, which is building a 10,000 year clock, and trying to bring the wooly mammoth back to life. He lives on a boat in California with his wife. Check out the documentary We Are As Gods (which comes from the Whole Earth Catalog’s statement of purpose: “We are as gods and might as well get good at it”) for more info on this awesome dude.

2. The Personal Computer

The year is 1981. Ronald Reagan becomes the 40th US president, Lady Diana Spencer becomes a princess, Indiana Jones prevents the Nazis from using the Ark of the Covenant for evil, and IBM releases their first personal computer, the IBM PC. 

The IBM PC is not the first personal computer. The real first commercial personal computer was the Altair 8800, built by Ed Roberts in Albuquerque and released in 1975. The Altair was wildly successful among hobbyists and inspired an entire wave of innovation, including a young Bill Gates to start a company called Microsoft to write and sell code for the Altair. While not mainstream successful, the Altair started the personal computer race. Two years later, in 1977, Radio Shack began selling its TRS-80, Commodore International unveiled the Commodore PET, and two Steves in Cupertino, California began selling their Apple II. While more expensive than its competition, the Apple II was far more popular (Ceruzzi 265). 

At the time, IBM was the dominant force in computing, focusing primarily on mainframes. The popularity of the Apple II forced IBM to take personal computers seriously and enter the market. To get a product to market as fast as possible, IBM used third parties and off the shelf components.

None of this would have been possible without the microprocessor, built by Intel in 1971. Intel was the product of Robert Noyce, Gordon Moore, and Andrew Grove. Noyce and Moore had left Fairchild Semiconductor due to differences with erratic founder William Shockley. “He may have been the worst manager in the history of electronics,” said Shockley’s biographer. Side note is that Andy Grove wrote a great management book (High Output Management), which I would recommend. Larry Ellison even said in his book, “Andy’s the only guy whom both Steve Jobs and I agree we’d be willing to work for” (Symonds 271). 

Our story starts in 1981 because, while the IBM PC was not the first personal computer, it was when PCs entered the mainstream. IBM was THE name in computing for decades, and when it launched its first PC, it meant that PCs could become part of the workforce in a way that machines built by startups like Apple never could. The launch of the IBM PC is also significant because of the software it used. It ran PC-DOS, an operating system licensed by Bill Gates at Microsoft. This is significant for several reasons. Let’s go through them one by one:

First, Bill Gates and his team at Microsoft were able to see the potential in selling software, specifically PC-DOS to IBM, even if it wasn’t that profitable on the front end. They got a flat rate from IBM for selling the OS to them (about $80K) and no royalties. But, they were free to sell their OS to other vendors as well. They kept the IP and licensed the right for IBM to use it, non-exclusively. That would become the standard way Microsoft would do business for decades.

Second, Microsoft didn’t have an operating system to sell to IBM when IBM asked. They told IBM to talk to Gary Kildall of Digital Research about his OS, but when Gary wasn’t available, Microsoft seized the opportunity and went and bought an OS from Seattle Computer Products for $50K. The initial success of Microsoft was fueled by a fair amount of luck and stealing products from others. 

This is also significant because it set the stage for DOS becoming “one of the longest-lived and most influential pieces of software ever written,” (Ceruzzi 270). IBM sold 750,000 of their PCs within two years but then the replicas started springing up, starting with Compaq in 1983 (Ceruzzi 277). “[…] companies like Compaq and Dell would earn more profits selling IBM-compatible computers than IBM would. IBM remained a major vendor, but the biggest winner was Microsoft, whose operating system was sold with both IBM computers and their clones” (Ceruzzi 279).

As Robert Cringely puts it in his documentary, “Microsoft bought outright for $50,000 the operating system they needed, and they turned around and licensed it for up to $50 per PC. Think of it. 100 million personal computers running MS-DOS software, funneling billions into Microsoft, the company that, back then, was 50 kids managed by a 25 year old who needed to wash his hair.” 

Finally, this is indicative of the lasting difference between computers running Microsoft software, which would become known as ‘PCs’ and Apple products. Apple products are vertically integrated—the hardware, software, and apps are all integrated and tightly controlled. Apple does not sell its OS separately. It wants complete control over the user experience. Apple is a hardware company; Microsoft is a software company. 

IBM dominated the PC market in the 80s, with Apple trailing behind. Remember the famous Super Bowl ad in 1984 where Apple positioned themselves as the challenger to the dominant “Big Brother” of IBM? Meanwhile, Microsoft pushed forward with DOS and then Windows. Windows 3 (Haigh and Ceruzzi 266) launched in 1990, bringing graphical user interfaces (GUIs) into the mainstream. Apple had been using GUIs for a while, which Steve Jobs stole from Xerox PARC, but Jobs was still upset at Gates for using them. 

By 1993, just 12 years after the IBM PC was launched, nearly 100 million American households (23 percent) had a personal computer, and this was even before the Internet. The majority of these computers were what became known as “PCs” which really meant “IBM PC compatible.” Because of its open architecture decision, however, IBM lost its lead in market share by 1994 to “clones” like Compaq and never regained it. 

IBM sold its personal computer business to the Chinese company Lenovo in 2005 for $1.3 billion. Hewlett-Packard bought Compaq in 2002 for $24.2 billion. In 2024, Lenovo (26 percent) and HP (22 percent) still dominate market share, and over 245 million personal computers are sold globally every year. 

The personal computer boom reshaped data in two ways. First, it forced enterprises to rethink how they stored and managed information, shifting from a few central mainframes to networks of individual PCs, i.e., the client-server architecture described in the next section. Second, once the Internet arrived, adoption exploded. Millions of personal computers were already wired and ready to go.

Tangent on the Gates/Jobs bromance: There’s a lot written about the young Gates/Jobs rivalry in the 90s. In terms of the personalities of Steve Jobs and Bill Gates, here’s my take: they were both entitled, bratty children who became entitled, bratty young men. They’d both throw fits when they didn’t get their way and bullied or manipulated those around them to get their way. And they both smelled terrible. The biggest difference in personalities between the two, as far as I can tell, is that Steve Jobs smelled like shit early on because he convinced himself, despite all evidence to the contrary, that by eating only fruit he didn’t have to shower, while Bill Gates smelled like shit because he’d stay up all night coding and forget to shower.

3. Client-Server Architecture

We shouldn’t judge IBM too harshly for completely flubbing the personal computer race, as it was busy dominating enterprise data and the relational database wars. Just kidding, they totally fucked that up too. IBM invented the relational database management system (RDBMS) and decided not to pursue it. 

In 1970, Edgar F. Codd, while working at IBM, wrote a paper called, “A relational model of data for large shared data banks,” which defined the relational database model. A relational database stores data as tables, with keys to uniquely identify each row. A structured query language (SQL) is a computer language to retrieve data from and insert data into tables. This is, to this day, the standard way data is organized for everything from medical records to airline schedules (O’Regan 274).

IBM built the IBM System R research project in 1974, marking the first implementation of SQL (Haigh and Ceruzzi 274). They decided not to commercialize their RDBMS because they wanted to preserve revenue from their existing hierarchical database, an example of the “Innovator’s Dilemma” I mentioned in the intro. Codd’s paper was public, however, and others read it and understood the commercial value. Michael Stonebraker of UC Berkeley created INGRES during the 70s using the framework described in the Codd paper (Haigh and Ceruzzi 275), and a young Larry Ellison read the paper and started Software Development Laboratories (SDL) in 1977 with Bob Miner and Ed Oates. They changed their name to Oracle Systems Corporation in 1983. 

Oracle’s first product, Oracle Version 2 (there was no Oracle Version 1 because they wanted their product to appear more mature than it was) was released in 1979. They beat IBM to market. IBM’s first commercial relational database management system, SQL/DS was released in 1981, a full 11 years after Codd’s article (Symonds 62).

During the 80s, database products were focused on either a mainframe architecture or minicomputers. By the way, the ‘mini’ in minicomputer meant that they were small enough to (hopefully) fit through a doorway, but they were still gigantic. The primary players in the database wars of the 80s were Oracle, Sybase (whose code base Microsoft licensed and later forked into Microsoft SQL Server), IBM, and Informix (Symonds 110).

Oracle came out on top in the database wars. “With the release of Oracle 7 and, in particular, Version 7.1 in 1993, Oracle had, for the first time in several years, unambiguously the best database on the market (Symonds 105). While Oracle won the database wars, there was a cost. Oracle was so focused on beating other RDBMS that they neglected the “applications” side of the business. The applications side are back office things like financial accounting and procurement (later called Enterprise Resource Planning or ERP), human resources and payroll (Human Capital Management or HCM) and sales and marketing (Customer Relationship Management or CRM). These are things that use the internal data stored in the relational database. Additionally, the world had moved towards personal computers and away from mainframes, even at the office. That meant a new architecture was required to manage enterprise data. 

In 1992, SAP, the German company founded by former IBM engineers, launched SAP R/3. SAP’s previous product, SAP R/2, released in 1979, was “widely recognized as the most complete and thoroughly engineered of the new breed of packaged applications” (Symonds 114). The R/3 version was built for a client-server architecture—capitalizing on the prevalence of personal computers. This is a significant event for many reasons. Let’s go through them one at a time:

First, R/3 used a three-tier model. Users work on their PCs, usually a Windows machine (client tier); this client communicates with SAP’s business logic, usually hosted on a Unix server (tier 2); then all of the data is stored in the third tier, a massive database. This was a fundamental architectural shift away from mainframes and towards personal computers. The idea of the client-server architecture was “custom corporate applications running on personal computers that stored their data in a relational database management system running on a server. This combined the best features of personal computing and traditional time sharing systems,” (Haigh and Ceruzzi 275). 

Second, it highlights the difference between enterprise data and enterprise applications. The way data is stored and the way it is used at an enterprise are very different things and products meant for one are not built for the other. They are also entirely different products, sold differently, marketed differently, and operated differently. 

Third, this loss would drive Oracle business decisions for decades, and they would never catch up to SAP. As Ray Lane from Oracle stated, “R/3 changed the game. Although we’d had some success in that area, we weren’t really an application company. Our sales force and our consultants didn’t really understand how to compete in the applications business. … Against SAP, we were a fraction. So we went on what turned into a four-year binge to try and catch up with SAP. From 1993 through to 1997, our entire application effort was devoted to trying to build features to compete” (Symonds 114-115). Oracle would struggle with applications and eventually buy PeopleSoft and JD Edwards in 2004, Siebel Systems in 2005, and NetSuite in 2016.

And finally, and partly as a consequence of the three-tier architecture, this led to a boom in “systems integrators,” or SIs, which are companies focused on helping with the transition to this new client-server architecture and digitizing internal systems. “SAP had carefully nurtured relationships within the Big Five consulting firms, especially with Andersen Consulting (now called Accenture), the largest integrator in the world. When companies were deciding whether and how they were going to implement an ERP system, they rarely started off by talking directly to the software vendors. Instead, they would ask one of the consultancies, usually one with which they had an existing relationship, to evaluate their business processes and then recommend the software that would best fit their requirements” (Symonds 116).

Andersen Consulting’s revenue from client-server-related projects grew from $309 million in 1990 to nearly $2 billion in 1993, employing 10,000 of their people. IBM Global Services, their consulting arm, grew from $4 billion in revenue in 1990 to $24 billion by 1998. In 1997 alone they hired 15,000 people. The dark side of the growth in ERPs and SIs is potentially best shown by looking at FoxMeyer—a $5 billion drug company that spent $100 million in 1993 to implement SAP R/3, failed, and went bankrupt. 

The cynical stance on SIs is that they are incentivized to make implementing enterprise software as difficult as possible because if anything worked out of the box they wouldn’t be needed. As Ellison said, “IBM recommends that you buy a lot of different applications from lots of different vendors. In fact, IBM resells applications from SAP, Siebel, i2, Ariba, pretty much everyone I can think of except Oracle. Then IBM makes a bundle by selling you guys with glue guns to stick it all together” (Symonds 281).

The potential nightmare of systems integrations and ballooning IT costs is best captured in Dave McComb’s book Software Wasteland (McComb). In his book, McComb explains how most enterprise software is middleware and requires integrations with other software. Not only does this mean huge IT costs, but it also leads to tons of siloed apps. “An estimated ‘35 to 40 percent’ of programmer time in corporate IT departments was spent keeping data in files and databases consistent” (Haigh and Ceruzzi 276).

Integrating enterprise data became a bigger problem with the rise of the client-server architecture and persisted through web-based and SaaS architectures as we’ll see in the next sections. Time and again, the proposed solution was to put all of your data in the same place, physically or in the cloud, but the differences in underlying schema still prevented a unified database. A potential solution came from outside of the enterprise data world and on the other side of the Atlantic.

4. The World Wide Web

While Ellison was battling SAP, a young man at the European Organization for Nuclear Research (CERN) was devising a way for different computers at his research center to communicate with each other. The Internet had been around for a while, and was established at research centers like CERN, but none of the computers “spoke the same language.” Tim Berners-Lee (TBL) built the World Wide Web in 1993, wisely choosing an acronym with more syllables than the words themselves. 

The World Wide Web laid the foundation for people to navigate the web by establishing things like URLs and html, but users still needed a browser to actually surf the web. Netscape was founded by Jim Clark and Marc Andreessen in 1994 and launched the first popular web browser. Sixteen months later, in August 1995, they went public and had a market value of $4.4 billion, the largest IPO in history, and they had yet to show a profit (Berners-Lee and Fischetti 106). Microsoft, so consumed by the personal computer, didn’t see the importance of the web early enough. “Microsoft saw the importance of the web and open standards, but its leadership could not imagine solutions that did not center on the personal computer” (Muglia and Hamm 28).

Bill Gates did realize the magnitude of the Internet in 1995 and issued a now famous memo to his company where he stated that the Internet is “crucial to every part of our business” and “the most important single development to come along since the IBM PC was introduced in 1981.” One way he planned to dominate the browser wars was by packaging their new browser, Internet Explorer, with their new operating system, Windows 95. This triggered an antitrust lawsuit—United States vs. Microsoft Corp. Microsoft LOST the case and was ordered to be broken up into two companies: one for producing the operating system Windows and one for producing other software components. They appealed and won, largely because the judge improperly spoke to the media about the case, violating codes of conduct. 

Netscape released its source code and started the Mozilla Organization in 1998 to permit open source versions of its browser. It was acquired by AOL for $4.2 billion one year later. Part of the acquisition required Andreessen become the CTO of AOL, reporting directly to former WELL member Steve Case. Microsoft, however, was dumping $100 million into IE every year and there were 1000 people focused on it, which eventually paid off. In 2003, just five years after the AOL acquisition of Netscape, IE held 95 percent of the market.

Microsoft won the first browser war, at a huge cost, but this was before anyone really knew how to make real money from the Internet. Netscape sold their browser directly to consumers and Microsoft gave theirs away for free (to kill Netscape). By the time the second browser war rolled around, the business model for Internet companies had become clear—collect user data for targeted ads, something Google had pioneered. This is why, despite veteran CEO Eric Schmidt’s reluctance after witnessing the brutality of the first browser war, Google entered the second browser war. Google knew there wasn’t money in browsers themselves, but the more people on the web, the more they search and the more ads they see, and the more money Google makes. “Chrome was always thought of as an operating system for web applications” (Levy 213). 

The source code released by Netscape in 1998 was turned into a new browser, appropriately named Phoenix. The browser was renamed Firefox in 2003 due to trademark claims. Firefox never beat IE but rose to a peak of 32 percent of market share in 2009. Google launched Chrome in 2008, which is now the most popular browser, accounting for 68 percent of market share. Apple’s Safari is the second most popular at 20 percent, and the successor to IE, Edge, is in third with just 5.7 percent.

4.1 Tim Berner’s Lee’s Vision

In his book, “Weaving the Web,” Tim Berners-Lee describes his vision in two parts (Berners-Lee and Fischetti 157). Part one is about human collaboration on the web. This required standards and protocols so that everyone could access all parts of the web. That was realized by the invention of the URI/URL, HTML, and XML. Because of those standards, browsers like Netscape and Internet Explorer could flourish. But he also saw the web not just as a place to read web pages, but to contribute to them too. This part was never realized in the way he envisioned—a popular browser was never invented that allowed editing capabilities on html directly. 

The idea of people participating on the web, of course, has been successful. This part of the vision is related to ‘Web 2.0’, a term popularized by Tim O’Reilly of O’Reilly books at the Web 2.0 conference in 2004. If Web 1.0 was about reading static HTML, then Web 2.0 is about users actively contributing to the web. Wikipedia, the online encyclopedia, contains 65 million articles, receives 1.5 billion unique visits a month, and 13 million edits per month. Social media sites like Facebook also allow people to contribute directly to the web, though the data is more personal than public (more on Facebook later).

TBL’s vision was grander. The second part of his vision is about computers collaborating on the web. “Machines become capable of analyzing all the data on the Web—the content, links, and transactions between people and computers. A ‘Semantic Web,’ which should make this possible has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy, and our daily lives will be handled by machines talking to machines, leaving humans to provide the inspiration and intuition” (Berners-Lee and Fischetti 158). There is often called “The Semantic Web” or “Web 3.0,” not to be confused with Web3, the idea of a decentralized web built on the blockchain. 

The idea behind the Semantic Web is that people would connect structured metadata to their html so computers can interpret web pages. The format of the metadata (or semantics) is Resource Description Framework (RDF). RDF data is often called “triples” because rather than storing data in columns and rows, RDF stores the data as a series of statements of the format: subject – predicate – object. These triples allow users to make information on the web machine-readable. For example, instead of saying “Kurt Gödel died in Princeton, New Jersey,” you could say: Kurt Gödel (subject) – died (predicate) – Princeton, NJ (object). Likewise, Albert Einstein (subject) – died (predicate) – Princeton, NJ (object). A machine could then infer that Albert Einstein died in the same town as Gödel. In addition to RDF data, languages for describing the RDF metadata exist, allowing users to create ontologies. For example, we could describe the predicate ‘died’ as being restricted to one location per subject, i.e., you can only die in one place. With rich ontologies and RDF data, users can create large graphs of knowledge, i.e., Knowledge Graphs, which computers can reason over. 

The Semantic Web never truly took off but its core principles are alive and well in pockets of the web. For example, there is a counterpart to Wikipedia called Wikidata that stores Wikipedia data as a structured knowledge graph and provides facts for Wikipedia pages. They have a public SPARQL API (SPARQL is like SQL but for triples) where you can query the data directly. Here is an example of how you can find all people who died in the same town as Gödel. Most websites do not offer public SPARQL APIs however. These technologies (SPARQL, RDF, OWL, SHACL, etc.) are all open source and the standards are maintained by the World Wide Web Consortium (W3C), the non-profit TBL started to ensure interoperability on the web. 

While the technologies haven’t exactly taken off on the public web, they have had success for enterprise data management. The idea of creating a rich metadata layer to keep track of and query all of the data on the Internet is a bit overwhelming, but the idea of building a rich metadata layer for a company, an Enterprise Semantic Layer—a graph of rich metadata linking systems, documents, and policies—is more reasonable.

5. Amazon and Google

In February 1994, a Senior Vice President at hedge fund D. E. Shaw & Co. read in a newsletter that the volume of information transmitted on the Web had increased by a factor of about 2300 between January 1993 and January 1994 (Stone 25). Jeffrey Bezos would claim that this was the reason he quit his hedge fund to start a website to sell books. He would claim in interviews that he “came across this startling statistic that web usage was growing at 2300 percent a year.” This is incorrect—a factor of 2300 means a 230,000 percent increase. Luckily for Jeffrey, he was incorrect in the right direction. 

Bezos considered names like makeitso.com (a Star Trek reference) and relentless.com but eventually landed on Amazon.com. They grew quickly without making a profit, competing with existing brick and mortar bookstores who were also selling books online: Barnes and Noble and Borders. Barnes and Noble struggled to pivot—another case study of the “Innovator’s Dilemma.” “The Riggios were reluctant to lose money on a relatively small part of their business and didn’t want to put their most resourceful employees behind an effort that would siphon sales away from the more profitable stores” (Stone 59). Bezos knew this. In response to a Harvard Business School student who told him he would fail and that he should sell his company to Barnes and Noble, Bezos said, “I think you might be underestimating the degree to which established brick-and-mortar business, or any company that might be used to doing things a certain way, will find it hard to be nimble or to focus attention on a new channel. I guess we’ll see” (Stone 65). 

Amazon started as an online retail store similar to eBay but without the auction component. It started spreading into CDs and DVDs and even digital books (tablets) but it wasn’t until 2006 with the launch of Amazon Web Services (AWS) that it truly became a tech company and not just another dot com startup. There is a popular story that AWS was started because Amazon needed to build infrastructure to support the holiday shopping season but that those servers sat idle the rest of the year. That seems to be untrue. Werner Vogels, the Amazon CTO even said so. There are a bunch of reasons Amazon started AWS: they were struggling with allocating server space internally fast enough to keep up with growing demand for experimentation; Tim O’Reilly of O’Reilly books made a personal appeal to Bezos to share their product catalog with a broader community so he could better predict trends in the market; and Bezos read the book Creation by Steve Grand (Stone 208-211).

Bezos listened to O’Reilly preach about Web 2.0 and the mutual benefit of sharing data and built APIs as a way for developers to better access the Amazon website (Stone 210). Around the same time, the Amazon executive book club read Creation, by Steve Grand. Grand created a video game called ‘Creatures’ in the 1990s that allowed you to guide and nurture a creature. No, not like a Tomagotchi. This game, apparently, allowed you to “code artificial life organisms from the genetic level upwards using a sophisticated biochemistry and neural network brains, including simulated senses of sight, hearing and touch”. 

“Grand wrote that sophisticated AI can emerge from cybernetic primitives, and then it’s up to the ‘ratchet of evolution to change the design,’” (Stone 213). The Amazon team wanted to use this framework to inspire developers to create new and exciting things without prescribing exactly what those things should be. The ‘primitives’ for the developer, they concluded, were storage, compute and a database. They released the storage primitive (Simple Storage Solution or S3) in March 2006, followed by the primitive for compute (Elastic Cloud Compute or EC2) a month later (Stone 213 – 214).

Comedy break: Here’s a video of Bezos in a documentary from 1998 talking about his “Internet idea” before he started cosplaying as Jean-Luc Piccard. And here’s Bo Burnham performing Jeffrey’s theme song. Come on, Jeff! Get ‘em!

While Bezos was starting to sell books online, two young PhD students at Stanford were looking for dissertation topics. Larry Page thought that he could devise a better way of ranking the importance of web pages—by counting the number of other pages that referenced them. An important web page would be referenced by many web pages, like how an important academic journal article is cited by many other articles. The problem is that web pages only tell you what they reference (hyperlinks) but not what references them. Links on the web only go in one direction. To know the number of times a page is linked to from other webpages you need all of the backlinks, which means you have to scrape the entire web. Page teamed up with another PhD candidate and math prodigy, Sergey Brin, who specialized in this kind of data mining. They called their project ‘BackRub’ because it was all about harvesting these backlinks. They named the algorithm, a variation of eigenvector centrality, PageRank, after Larry Page (Levy 16-17). “We take advantage of one central idea: the Web provides its own metadata…This is because a substantial portion of the Web is about the Web…simple techniques that focus on a small subset of the potentially useful data can succeed due to the scale of the web” (Wiggins and Jones 213). 

Jon Kleinberg was a postdoctoral fellow at IBM in 1996 and was also playing with the idea of exploiting the link structure of the Internet to improve search results. Through mutual friends, he got in touch with Larry Page and learned about BackRub. By this time, IBM finally learned their lesson and moved quickly on a technology that would define the next generation of tech companies. Just kidding, they boofed it again. Kleinberg encouraged Page to write an academic paper about the technology, but Page declined. Kleinberg went on to a successful academic career, while Page founded Google but never got his PhD (Levy 26).

Page and Brin eventually realized that this ranking would make for a good search engine, and they created a company they called Google, a misspelling of the word for the large number ten to the hundredth power, googol (Levy 31). They started a search company “even though there was no clear way to make money from search” (Levy 20). Soon, they figured out a way to make money, and it was through a technology that was arguably more important than PageRank: AdWords. They kept their revenue a secret because they didn’t want anyone else to use the same method for generating revenue. They had to reveal it as part of their IPO in 2004 (Levy 70). 

“Google launched the most successful scheme for making money on the Internet that the world had ever seen. More than a decade after its launch, it is nowhere near being matched by any competitor. It became the lifeblood of Google, funding every new idea and innovation the company conceived of thereafter. It was called AdWords, and soon after its appearance, Google’s money problems were over. Google began making so much money that its biggest problem was hiding how much” (Levy 83). 

The idea is relatively simple: put sponsored ads at the top of users’ search results. But it was different from existing online advertisements in several ways. First, the ads were based on the user’s search words—the products or services a user would see an ad for would be relevant. Second, the price of the ads would be the result of an auction—advertisers could bid against each other to determine the price of the ad related to the keyword. And three, the advertiser would be charged by the number of clicks, not the number of times their ad was seen. Because Google had so much data about how people searched and were so good at getting users the best results possible, they were also experts at putting the appropriate ads in front of the right people. This benefitted the advertisers, who got more clicks, Google, who got ad revenue, and often the users, who (hopefully) got ads for exactly what they were searching for. 

Before they figured out AdWords, they assumed they would have to rent their search engine out to an Internet portal like Yahoo! or Excite to generate revenue, now they could make money directly. Their entire business model changed, and they eventually expanded to advertising on more than just search results (Levy 95). AdSense was launched three years later, in 2003, and allowed websites to embed ads directly on their pages. Google was able to ensure that ads would be relevant to the content on the site by matching key themes on the site and matching them to ads. They acquired a startup called Applied Semantics to do this (Levy 103). If you ran a webpage, you could sell a portion of it to Google, who would place relevant ads there and give you a percent of the revenue. Matching ads to keywords on a webpage doesn’t always work, however. An early version of AdSense put an ad for Olive Garden on an article about someone getting food poisoning from Olive Garden (Levy 105). 

One year later, in April 2004, Google launched Gmail, a free email service which included a gigabyte of storage for every user. For reference, the largest existing email services were Microsoft’s Hotmail and Yahoo!, who only offered 2 and 4 megabytes of storage, respectively (Levy 168). To accommodate the massive amounts of data storage from websites and Gmail, along with all of the computations required to index and provide search results for over 200 million queries a day, Google had to build a ton of data centers. 

This information is not public, and Google doesn’t disclose numbers on how many servers it runs, but Steven Levy, in his book, In the Plex, said, “According to an industry observer, Data Center Knowledge, there were twenty-four facilities by 2009, a number Google didn’t confirm or dispute. Google would not say how many servers it had in those centers. Google did, however, eventually say that it is the largest computer manufacturer in the world—making its own servers requires it to build more units every year than the industry giants HP, Dell, and Lenovo” (Levy 181).

Following Amazon’s lead, Google launched Google Cloud Storage (the S3 equivalent) in 2010, allowing users to use their servers for storage and launched Google Cloud Compute Engine (the EC2 equivalent) in 2012. They remain one of the big three cloud providers currently (behind AWS and Microsoft Azure). The ability to use third-party servers to run applications and store data, along with increasing bandwidth, led to a fundamental architectural shift in the way applications are built and where data lives. The next section explores that architectural upheaval. 

6. The Big Switch

Nicolas Carr wrote a book, “The Big Switch: Rewiring the World From Edison to Google,” that’s so good, I sometimes even recommend it to people who are not data nerds. In it, he draws a parallel between the growth of electricity as a utility in the late 19th century and the rise of cloud computing in the late 20th century. Here’s a brief summary, but I definitely recommend this book.

Thomas Edison invented the lightbulb and built all the required components to demonstrate its use for the International Exposition of Electricity in Paris in 1881. There, he also showed blueprints for the world’s first central generating station (Carr 28). He got the generator working the next year. He then built a business focused on licensing the patented system and selling all of the required components. He thought an electric generator would be a substitute for gas utilities, that many would need to be built, and that currents would not need to travel far. In fact, because his system relied on direct current, they couldn’t be transmitted far. “Edison had invented the first viable electric utility, but he couldn’t envision the next logical step: the consolidation of electricity production into giant power plants and creation of a national grid to share the power” (Carr 30).

Samuel Insull, who worked for Edison, realized that electricity could be sold as a utility. The more you sell, the cheaper it gets, which lets you sell more. This plan required convincing business owners that they should stop producing their own electricity and buy it from a centralized power station—something that had never been done before. Eventually, and obviously, we all got electrified. Factories got bigger and more productive, and modern corporations were formed (Carr 90). Ice companies disappeared because of refrigeration. Ford created the electrified assembly line to produce the first mass-produced car, the Model T. To hire the factory workers, Ford offered higher wages, which others were forced to match, setting in motion the creation of the modern American middle class (Carr 93). As industries became more advanced, they had to hire scientists, engineers, marketers, designers, and other white-collar employees. This new group of “knowledge workers” incentivized investments in education—high school enrollment in 1910 was 30 percent max in the wealthiest areas but went up to between 70 and 90 percent across the country 25 years later (Carr 94). 

Let’s return to the client-server architecture of the early 90s. Remember in this setup, users have personal computers that they connect to their company’s centralized data centers. This is like a company running its own electricity generator to power its factory. The logical next step in this architecture is to treat data storage and computation as a utility. This happened (or is currently happening) but was facilitated by a few things.

First, the Internet needed to go from a DARPA research project into mainstream America. In 1991, Tennessee Senator Al Gore created and introduced the High Performance Computing Act of 1991, commonly known as the Gore Bill. Yes, that’s right. Al Gore did, to his credit, play a big part in making the Internet available to all. Before the Gore Bill, it was illegal for ISPs like AOL to connect to the Internet, they were “walled gardens” (Isaacson 402). The Gore Bill allowed AOL to give its users access to the broader Internet. The Gore Bill also put $600 million into Internet infrastructure, including funding the National Center for Supercomputing Applications (NCSA) at the University of Illinois. An undergrad at the University, Marc Andreessen, worked at the NCSA and learned about TBL’s World Wide Web. He created a browser called Mosaic, which he commercialized into Netscape after graduating. As Vice President, Gore pushed forward the National Information Infrastructure Act of 1993, making the Internet available to the general public and commercial use (Isaacson 402). 

By the way, he never said he invented the Internet. Here’s the interview where he said, “During my service in the United States Congress, I took the initiative in creating the Internet.” He misspoke and should have phrased that better, but Vint Cerf and Bob Kahn, who did invent the Internet’s protocols said, “No one in public life has been more intellectually engaged in helping to create the climate for a thriving Internet than the Vice President” (Isaacson 403). Even Newt Gingrich said, “Gore is not the Father of the Internet, but in all fairness, Gore is the person who, in the Congress, most systematically worked to make sure that we got to the Internet (Isaacson 403). Al Gore had great ideas, but as Jared Dunn from Silicon Valley said, “People don’t want to follow an idea, they want to follow a leader. Look at the last guy to create a new Internet. Al Gore. His ideas were excellent, but he talked like a narcoleptic plantation owner, so he lost the presidency to a fake cowboy and now he makes apocalypse porn.”

The other reason computing power could become a utility is that Amazon, Microsoft, and Google built a shitload of data centers. Amazon started AWS and started renting out its servers. Google launched GCP in 2010. But renting out servers required some additional technologies, specifically virtualization and parallelization. Virtualization is the ability for a machine to run multiple operating systems—one server can contain a ‘virtual’ PC running Windows and a ‘virtual’ Linux OS (Haigh and Ceruzzi 368). Amazon’s system runs on virtualization. “When you rent a computer—through Amazon’s EC2 service, you’re not renting real computers. You’re renting virtual machines that exist only in the memory of Amazon’s physical computers. Through virtualization, a single Amazon computer can be programmed to act as if it were many different computers, and each of them can be controlled by a different customer” (Carr 76). Parallelization is the ability to run a task on multiple different servers simultaneously (in parallel). Google pioneered this technology with their product, MapReduce.

But there was still a problem: the Internet was strung together with phone lines. There was no way to transmit computing power very far. The benefits of computing could only be realized by having a data center in-house. This would be like if we were stuck with direct current (DC) electricity, which couldn’t be sent long distances. But we weren’t stuck with DC; we had alternating current (AC), which could be sent long distances. Thanks, Tesla (the man, not the company). And we were soon no longer constrained by telephone poles. Moore’s Law met Grove’s Law. Remember Andy Grove, who both Larry Ellison and Steve Jobs would work for? These two laws coincided. “Moore’s Law says that the power of microprocessors doubles every year or two. The second was proposed in the 1990s by Moore’s equally distinguished colleague Andy Grove. Grove’s Law says that telecommunications bandwidth doubles only every century” (Carr 58). This is not true at all, by the way. Telecommunications bandwidth increases much faster than that. Grove said that more as a criticism of telco and regulator progress than as an actual prediction. 

Nevertheless, telecommunications was finally catching up. With the rise of fiber-optic cables, Internet bandwidth has become fast enough for data to stream like electricity. “When the network becomes as fast as the processor, the computer hollows out and spreads across the network,” Eric Schmidt (Carr 60). We are now moving on-premise data centers to the cloud, just like we moved electricity generators to the power station. But transitioning computing and storage to the cloud doesn’t just mean we don’t need on-prem data centers any more. The idea of renting these resources enables an entirely new business model: Software as a Service, or SaaS. 

There are a few things to point out in the comparison between electricity and cloud computing. First, the “rebound effect” is real. Lower costs don’t shrink workloads; they increase them. Electricity was supposed to lighten household chores, yet cheaper power led families to run more appliances, and rather than reducing the effort to iron clothes, people just expected to iron them every day (Carr 99). Cloud promises to cut IT overhead, but as storage and compute get cheaper, companies spin up more micro-services, datasets, and integrations than ever. In both cases the rebound effect turns savings into surging demand. The same pattern is emerging with AI: while it’s marketed as a way to ease our workloads, its availability is already raising expectations and workload volumes faster than it reduces effort.

The second take away from the electricity metaphor is that it led to a golden age of prosperity, but it took a while. Edison invented the lightbulb in 1879, but Henry Ford didn’t create an electrified assembly line until 34 years later, in 1913. Only decades later, after WWII, did the American middle class hit its post-war peak. If AWS was the lightbulb, and we assume the same time-delay, a Ford-scale cloud assembly line won’t appear until 2040, and a new middle-class boom will be a generation after that. 

7. SaaS / Cloud Computing

7.1 Enterprise Data Moves to the Cloud

As more and more people began using the Internet, an Oracle employee saw the writing on the wall and decided to start his own company focused on enterprise applications hosted entirely in the cloud. Marc Benioff describes the way he started Salesforce in his book, Behind the Cloud, which contains advice like how you should take a year-long sabbatical and talk to the Dalai Lama about your business idea before starting a company (Benioff 2) and how you should listen to your customers (Benioff 13).

Salesforce was founded in 1999 and surpassed one billion in revenue in five years. Benioff wasn’t the first to think of this, of course. Oracle had been investing heavily in Internet technology since it got wrecked by SAP’s R/3 in 1992. “Client/server might be all right for departmental use, but for any company that wanted to unify its operations over a number of different sites, it was a nightmare” (Symonds 143). But while Oracle’s E-Business Suite, launched in 2001, was using web-based technologies, like browsers, it was still hosted on the customers infrastructure (on-prem). Salesforce was SaaS from the start—they hosted all of the infrastructure themselves and sold subscriptions to their product. Their first “mascot” was SaaSy, which is just the word “software” with a red line through it, indicating the end of software. 

Other enterprise application companies caught on, but not as fast as Benioff. ServiceNow was founded in 2004 and Workday in 2005, both SaaS-based ERP solutions. To start, Salesforce hosted its own servers, but eventually began moving to the hyperscalers, along with the other ERP vendors. In 2016, Workday selected AWS as its “primary production cloud platform”, and Salesforce selected AWS as its “preferred public cloud infrastructure provider”. In 2019, ServiceNow chose Azure as its preferred cloud provider.

7.2 Semantics Tech in the Enterprise

Connecting enterprise data has been a headache through every architectural era. When personal computers entered the workforce, the number of applications, databases, and integrations increased. Because you’d have multiple apps, it became impossible to ask even basic questions about a large company like, “How many people work here?” Oracle pushed for “one company, one database” in the 2000s as a way to address this pain point (Symonds 168) but soon realized that to run applications off of this database, you need a unified data structure or schema. “The key to everything was the seemingly esoteric concept of a common data model uniting every piece of the suite. Every module—and there were about 140 of them—would be written to the same shared data schema, allowing semantic consistency (for example, the definition of a customer remained the same no matter from which application the information was coming and could thus be shared by all the other applications in the suite) as well as a complete view into every transaction” (Symonds 188).

We didn’t learn that lesson when a new architecture presented itself. The parallelization technology, MapReduce, that allowed Google to run computations across millions of servers was described in several papers by Jeffrey Dean, Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung in 2003 and 2004. This technology was turned into an open-source project called Hadoop which allowed anyone to implement cloud computing (Levy 202-203). This essentially allowed companies to store and compute large datasets across multiple servers, and led to the term ‘data lake’. In contrast to data warehouses, which had to follow a predefined schema, data lakes could be data of any format. Unfortunately, the ability to dump anything into a giant lake without a standard schema or metadata management layer did not work out, as Oracle knew too well. 

Data lakes became data swamps. Enterprises stored wastelands of data in the hopes it would be useful in the future. More recently, Databricks, a cloud-native data management platform has pushed the idea of a “data lakehouse.” The idea is to take the benefits of a data lake (ability to store data without a predefined schema) with the benefits of the data warehouse (assurance that transactions are complete, correct, conflict-free, and safely stored, aka ACID).

Bias alert: I work in this space. 

While the architecture has changed from mainframes to minicomputers to client-server to cloud to SaaS, the underlying problem hasn’t changed: it is difficult to connect disparate datasets because they don’t speak the same language. That could mean they follow a different metadata structure (schema), are of a different format entirely (JSON vs relational vs text), or are in different servers. This is very similar to the problem that TBL solved with the World Wide Web. The move to the SaaS/Cloud architecture has only helped with the third problem—keeping data together in the same virtual server. But colocation doesn’t really help you connect datasets. It’s like if I put a bunch of people who spoke different languages in the same room and expected them to collaborate—you’re going to need some shared vocabulary or translators or something to bridge that language barrier. 

This is where the semantic technologies inspired by TBL come in. While annotating the entire web with structured metadata may be impossible, it is doable at the enterprise level, at least for the most important data. This is often called the enterprise semantic layer, and, I believe, it will become more important as we start trying to get AI (which wasn’t trained on enterprise data) to interact with enterprise data. AI agents need to understand your data to make use of it. They need to know the meaning of the data, not just the numbers. Semantics is the layer of meaning that connects data and makes it understandable to humans and machines.

8. Facebook

While the world was starting to use Google as a verb and Bezos was expanding Amazon to a full-on empire, a 20-year-old Harvard student saw the social implications of the web. Mark Zuckerberg, trying to be as cool as the lamest version of Justin Timberlake, started Facebook in his Harvard dorm room. 

Facebook started as a way for Harvard students to find each other. A facebook is a hard copy book of students’ (and faculty) faces that many schools use to help students get to know each other. It’s basically a boring yearbook that is distributed at the start of the school year. Zuckerberg allowed Harvard students to make their own online facebook page, a photo of themselves along with some additional data like relationship status. He then expanded to other campuses and eventually opened it to everyone.

Before making thefacebook, Zuckerberg scraped photos of all of the undergraduate female students at Harvard and built an app that allowed users to rate their ‘hotness’. He got in trouble for this and learned two important lessons. One: don’t steal data, let users give it to you and Two: people are more voyeuristic than you’d think (Levy 52). 

Social networking sites do just that: they allow users to upload their own data and they allow you to look at pictures of your friends. Other social networking sites like Myspace and Friendster already existed, but one thing that made thefacebook different from the start was exclusivity—originally it was only for users with a harvard.edu email address. Even after expanding to everyone, the idea of exclusivity remained in the sense that only people you “accept” can view your profile. This was different from other social networking sites at the time. Everything you put on Myspace, for example, was visible to everyone, at least when it started. By putting this barrier in place, people were more willing to give even more information about themselves. 

As sociologist Danah Boyd said, “Zuckerberg made it interactive. It had a slight social stalking element too. It was addictive. And the fact that you could see only people on your network was crucial—it let you be in public but only in the gaze of eyes you want to be in public to,” (Levy 67). Eventually, Facebook built a “News Feed” where you can see updates about your friends. They quickly realized that the users responded most to stories about themselves. The secret of Facebook’s success isn’t a secret at all—people just want to stalk their crushes online and see “news” about themselves. I have a theory that the reason the movie The Social Network is so good is that director David Fincher understands this. As Fincher has said, “I think people are perverts. I’ve maintained that. That’s the foundation of my career.”

Facebook collected data on each user and each user’s friends but didn’t have a clear business model. They knew they could sell ads but didn’t want to spend engineering resources on that so they outsourced all ads to Microsoft (Levy 179). Zuckerberg said, “We don’t want to spend a single resource here working on advertising…It’s not something we care about. Microsoft wants to build an advertising business here…and so we’re going to give our inventory to them and they’re going to pay us,” (Levy 179).

Eventually, however, Facebook needed to become profitable. Just like Google hired Schmidt to be the “adult in the room” to a company founded by young people, Facebook hired Sheryl Sandberg in 2008. She came from Google and understood that Facebook, just like Google, was in the advertising business. As Sandberg explained to everyone on her first day, advertising is an inverted pyramid with a wide top of demand and a narrow bottom of intent. Google dominates the bottom—when people go online intending to buy something, they search for it, and Google delivers the relevant ad. Facebook could dominate the wider top of the pyramid, by creating and monetizing demand. Advertisers can get in front of people before they even know they want the product (Levy 195). So Facebook became an ad company, and its overall goal became to get its users to spend more time on Facebook and share more personal information so it could serve more ads (Haigh and Ceruzzi 375).

The Dark Side of Facebook

When TBL created the Web and put forth a vision of a utopia where we all come together, the assumption was that more sharing and more openness was an inherently good thing. Websites should share data and allow others to contribute, and we can all learn more about the world. This is true when it comes to public data, and it’s how we have something like Wikipedia. Millions of people are coming together to build the largest encyclopedia in the history of humankind. But when it comes to personal data, it is not so easy. “Walled gardens,” platforms where the admin controls access to data, went against the original tenets of the World Wide Web. But when that data is about people’s personal preferences, habits, family and health, walled gardens are a necessity. By building a platform that allows users to create content that can go viral or pay for targeted ads at specific demographics, however, Facebook enabled propaganda machines.

Understanding a company’s data and revenue model can tell us a lot about their actions. Facebook (and now Instagram, which they own) collects personal data on people so it can serve targeted ads. The metrics for success, then, are growth in users and engagement on the site. The more people log in to the apps regularly, the more ads they see and the more revenue for Meta. Unfortunately, a big driver of engagement is outrage—people are more likely to engage with content if it upsets them, even if it is untrue. “Humans are more likely to be engaged by a hate-filled conspiracy theory than by a sermon on compassion. So in pursuit of user engagement, the algorithms made the fateful decision to spread outrage” (Harari 199). While not doing anything inherently evil, Facebook’s algorithms set the stage for viral misinformation which has led to hate speech and violence. 

What does this mean for the future? The OpenAI revenue model right now, along with most AI companies, is based on subscriptions. What if the revenue model changes to targeted ads like Google and Meta? Then the information AI gives us will not be aimed at giving us the most ‘accurate’ or ‘truthful’ answer, but the answer that keeps us engaged the longest, interacting with our friends (or enemies) on their platforms, and that encourages us to reveal more personal information about ourselves. In Yuval Noah Harari’s book “Nexus,” he describes a man who tried to kill the Queen of England in 2021 because his AI girlfriend encouraged him to (Harari 211). If Facebook could be turned into a propaganda machine that contributes to genocide because of the data it collects and the algorithms serving its business model, then AI can too. The most dystopian AI future I see is not Terminator but one where AI girlfriends convince packs of incels that genocide is cool. 

9. The iPhone

The popularity of social media would not have been possible without handheld computers that we can carry with us everywhere we go. Improved bandwidth and cloud computing technologies have allowed the computer to “hollow out and spread across the network” as Eric Schmidt said (Carr 60). But the computer has also shrunk and ended up in the pockets of billions of people. 

The iPhone was launched in 2007, and there really hasn’t been a more significant or impactful single products since the dawn of the personal computer in 1981. Yes, there were smartphones like the Blackberry before the iPhone, but the iPhone changed everything. It was a phone, an iPod, and an internet communications device. “Are you getting it? These are not three separate devices. This is one device. And we are calling it: iPhone”, Steve Jobs said during the product launch. It had a complete touchscreen with the ability to do multi-finger gestures, something that had never been done in a mass-produced product. And it had a 2 megapixel camera. It also had a full operating system (OS X). It was a device you could keep in your pocket that you could use to view webpages, something that had never existed before (Haigh and Ceruzzi 395). The operating system also meant that apps could be built for it.

The iPhone didn’t really invent anything new, but it put all of those pieces together in a way that had never happened before. As Jobs said, “We have always been shameless about stealing great ideas”. The idea of having a device in your pocket that you could use to listen to music, watch videos, make phone calls, and browse the internet was the stuff of science fiction. In many ways, the iPhone is a fulfillment of Stewart Brand’s vision of personal computing. It makes sense that Jobs—a reader of the Whole Earth Catalog, which espoused individual empowerment, decentralization, and access to tools—would turn Apple into the largest company in the world by building the most personal computer ever made.

Here are just some of the ways the iPhone fundamentally changed the tech industry and everyday life for most humans.

Having a computer with an operating system in your pocket meant that apps could be developed. Apple controlled the app store of course, meaning they could control the apps users got to use. Gaming were some of the first popular apps. You could play games like Angry Birds and Candy Crush, which disrupted the gaming industry.

Soon, all kinds of new and creating apps could be built that took advantage of iPhone features that weren’t possible before. iPhones had a built-in GPS which meant a restaurant booking site like OpenTable or Resy could now become a restaurant booking site for restaurants near your physical location. Likewise, apps for dating based on physical proximity were created. Grindr was launched in 2009 and the hetero version, Tinder, was launched in 2012. GPS also enabled ride share apps like Uber (2009) and Lyft (2012).

Facebook caught on and invested in a mobile version of their product, which quickly became one of the most popular apps. iPhones had cameras so you could take pictures with your phone and directly upload them to your Facebook page. As the popularity of taking pictures using phones increased, Instagram was started in 2010 so people could add artsy filters to pictures of their food.

In 2011, the iPhone launched with Siri, an AI-powered virtual assistant (Haigh and Ceruzzi 394 – 400). Then Google created an AI Assistant, Microsoft created Cortana, and Amazon created Alexa. By 2011, Apple sold more smartphones than Nokia and made more in profits than all other cell phone makers put together (Haigh and Ceruzzi 401). Apple became the first company with a half-trillion dollar market evaluation in 2012 and became the first to a trillion in 2018 (Haigh and Ceruzzi 401). They remain one of the largest companies in the world by market cap to this day.

While there have been many attempts to replace the iPhone as the device of choice, so far no one has succeeded. Not even Apple, with its watches and glasses, can get people to trade their iPhones for something else. However, OpenAI recently acquired Jony Ive’s (the designer of the iPhone) startup for $6.5 billion and has said they will release a device in late 2026. 

10. Conclusion

In my next post I will go through an accounting of the different sources of data and the major players in each sector. For now, here is a high-level overview of who owns different kinds of data and their revenue models.

Google and Meta are advertising companies. They make money by collecting personal information about people and serving them targeted ads. About 78 percent of Google’s revenue comes from ads and nearly 99 percent of Meta’s revenue comes from ads. Because of this, they want you online so they can serve you ads. The top four most visited websites in the world, as of June 2025 are Google, YouTube (owned by Google), Facebook, and Instagram (owned by Facebook). Google also has a 21 percent market share of the collaborative software industry through Google Workspace and owns Android, the most popular phone OS in the world. Yet, these are really just tools to get people online to view ads. Google is also the third largest hyperscaler company in the world with their Google Cloud Platform, which accounted for over 10 percent of their total revenue in 2023. 

Apple is primarily a hardware company—over half their revenue is from the iPhone and about a quarter from other products like MacBooks, iPads, Wearables, etc. Nearly a quarter comes from “services,” which means the AppleCare, cloud services, digital content, and payment services. They claim that they only collect user data to “power our services, to process your transactions, to communicate with you, for security and fraud prevention, and to comply with law.” 

Microsoft is primarily a cloud computing and software company. Azure (and other server and cloud products) accounts for 43 percent of revenue. The second largest money-maker is Office, followed by Windows. Their revenue model is based on subscriptions to their software or cloud computing resources. They also own LinkedIn, the 17th most visited website in the world in June 2025, Bing, the 24th, and GitHub. 

Amazon is a space exploration company that is funded by an online store and a cloud computing service on Earth. That is not a joke—I genuinely believe that. Zuckerberg and Gates were coders who loved building things; Jobs and Woz turned their love of tinkering into a company that sells computers. Page and Brin were Stanford PhD students who had a passion for math and data and turned a dissertation idea into a business. They all followed the thing they were passionate about, and it led them to riches. Bezos didn’t spend his childhood dreaming of online retail—he spent it dreaming about space exploration and science fiction. He didn’t start selling books online because he loves books, he started selling books online because it was the most practical and lucrative thing to sell online. With Blue Origin, he is finally starting to realize his vision. Congratulations, Jeff! 

Amazon online sales (including third-party vendors) accounts for the largest portion of their revenue (39 percent), but AWS is a bigger share of their operating income (because of the higher margins). AWS is the leader in cloud computing since they got there early—they have 29 percent of the market for cloud computing, followed by Azure (22 percent) and Google (12 percent). 

Let’s return to our framework of personal, enterprise, and public data:

For personal data, Meta and Google dominate and generate revenue from targeted ads. Apple and Amazon also capture a ton of personal data through devices, they just don’t use it for targeted ads. 

For enterprise data, we can look at both database vendors and applications. When it comes to database management systems (DBMS), the leaders are Amazon, Microsoft, Oracle, and Google, accounting for three quarters of the $100 billion market. IBM and SAP are behind them at the 5 and 6 spots and Snowflake and Databricks are the fastest growing challengers. For applications, Microsoft still leads collaboration with its Office suite (38 percent market share), followed by Google (21 percent). Salesforce leads CRMs (over 20 percent market share). SAP and Oracle are still the ERP leaders but they also play in Human Resource Management (HCM), competing with Workday, and Supply-Chain Management. ServiceNow leads IT/Customer Service Management.

Google owns the largest repo of public data in the world—Google’s search index contains over 100 million gigabytes of data. While Google is proprietary, there are truly public data sources. The three big ones are the Internet Archive / WayBack machine which has over 100 petabytes of data, Common Crawl which has more than 9.5 petabytes of data, and Wikimedia projects which is about 30 terabytes of data. GPT3, and other large language models were trained on these public data sources. 

I’m convinced the next wave of AI will be driven by the companies that capture the data, how they capture it, what kind of data they capture, and the business models they use to monetize it.

In my next post, I will formalize a list of questions about the future of data, the Web, and AI. I will use the framework that Philip Tetlock proposes in his book, Superforecasting, and implemented in his Good Judgement Project. These will be predictions with percentages about falsifiable claims about the future with dates. This way, I will be able to validate my predictions and improve over time. For example, a question might be, “Will a mass-market smartphone (or comparable personal device) ship with a ≥ 10 billion parameter language model by the end of 2025?”. I will place my prediction against this question, 20 percent maybe, and then use a Brier score to calibrate my answers. If a device with an LLM is shipped this year (the outcome of the question is a probability of 1) then the Brier score for this question would be (0.2 – 1) ^ 2 = 0.64. The goal is to get a Brier score as close to zero as possible.

I will create a list of relevant questions, my predictions, along with explanations for my predictions. I’d also like to make this as collaborative as possible by allowing others to make their own predictions so that we can collectively come to a better understanding of the future of AI.

Works Cited

Benioff, Marc. Behind the Cloud. Jossey-Bass, 2009.

Berners-Lee, Tim, and Mark Fischetti. Weaving the Web : the original design and ultimate destiny of the World Wide Web by its inventor. Edited by Mark Fischetti, HarperCollins, 1999.

Carr, Nicholas. The Big Switch: Rewiring The World From Edison To Google. W. W. Norton, 2013.

Ceruzzi, Paul E. A History of Modern Computing, 2nd Edition (History of Computing). ebrary, 2003.

Gorelik, Alex. The Enterprise Big Data Lake: Delivering the Promise of Big Data and Data Science. O’Reilly Media, 2019.

Grove, Andrew S. High Output Management. Knopf Doubleday Publishing Group, 1995.

Haigh, Thomas, and Paul E. Ceruzzi. A New History of Modern Computing. MIT Press, 2021.

Harari, Yuval N. Nexus: A Brief History of Information Networks from the Stone Age to AI. Random House Publishing Group, 2024.

Isaacson, Walter. The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution. Simon & Schuster, 2014.

Isaacson, Walter. Steve Jobs. Simon & Schuster, 2011.

Levy, Steven. Facebook: The Inside Story. Penguin Publishing Group, 2021.

Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. Simon & Schuster, 2021.

McComb, Dave. Software Wasteland: How the Application-centric Mindset is Hobbling Our Enterprises. Technics Publications, 2018.

Mirchandani, Vinnie. SAP Nation: A Runaway Software Economy. Deal Architect Incorporated, 2014.

Muglia, Bob, and Steve Hamm. The Datapreneurs: The Promise of AI and the Creators Building Our Future. Skyhorse Publishing, 2023.

O’Regan, Gerard. Introduction to the History of Computing: A Computing History Primer. Springer International Publishing, 2016.

Stone, Brad. Amazon Unbound: Jeff Bezos and the Invention of a Global Empire. Simon & Schuster, 2022.

Stone, Brad. The Everything Store: Jeff Bezos and the Age of Amazon. Little, Brown, 2014.

Symonds, Matthew. Softwar: An Intimate Portrait of Larry Ellison and Oracle. Simon & Schuster, 2004.

Tetlock, Philip E., and Dan Gardner. Superforecasting: The Art and Science of Prediction. Crown, 2015.

Wiggins, Chris, and Matthew L. Jones. How Data Happened: A History from the Age of Reason to the Age of Algorithms. W.W. Norton, 2024.