Popular markets commentator Citrini7 recently published a compelling and popular piece of AI doomer fiction, admittedly with some small probability of occurring. But I am old enough to have seen many cycles of economic doomsaying. I want to present a critique of Citrini’s work and show a much likelier, more positive view of the future.

1. Never Underestimate Institutional Momentum

In 2007, people thought the US was geopolitically done under peak oil. In 2008, they thought the US dollar was just shy of collapse. In 2014, they thought AMD and NVIDIA were done. Then came ChatGPT, and they thought Google was done... Every time, existing institutions with momentum have proven themselves far more durable than onlookers thought.

When worried about institutional turnover and rapid labor replacement, it is very funny that Citrini writes:

Even places we thought insulated by the value of human relationships proved fragile. Real estate, where buyers had tolerated 5-6% commissions for decades because of information asymmetry between agent and consumer...

People have been calling for the end of the real estate broker for 20 years! You don’t need superintelligence for this! All you need is Zillow or Redfin or Opendoor. This example actually shows the very opposite of Citrini’s point: we have a type of labor that most people consider obsolete, and yet, market inertia and regulatory capture have made the real estate broker far more resilient than anyone would’ve bet a decade ago.

My wife and I bought a house a few months back. The transaction required us to have an agent, ostensibly for the above reasons. Our buyer’s agent made about $50,000 on the deal, for about ten hours of form-filling and party-coordination that I could’ve done myself. This market will eventually be efficient and price this labor fairly, but it takes a long time to get there. I know a lot about inertia and change management: I built and sold a company that focused on moving insurance brokerages from service to software, and the main thing I learned is the iron rule of dealing with human reality: everything is always more complicated and takes much longer than you think it will, even if you already know about the iron rule. That doesn’t mean that a meaningful change in the world won’t happen, but that the change will be more gradual, giving us the time to respond and adjust.

Subscribe now

2. Software Has Infinite Demand for Labor

The software sector has been struggling in recent months as investors fear that companies like Monday, Salesforce, Asana, etc. can now be easily replicated and that the value of their backend systems is indefensible. Citrini and others talk of AI coding as spelling the end of jobs at SaaS companies as (1) the products become obsolete/zero-margin and (2) the jobs themselves disappear.

What everyone seems to be missing is this: these products fucking suck. I can say this, because I’ve actually spent hundreds of thousands of dollars on Salesforce and Monday. Sure, maybe AI enables competition to replicate their products. But more importantly, AI enables competition to deliver better products. It’s no surprise to see the stocks drop: an uncompetitive, sticky lock-in sector filled with dogshit incumbents is becoming competitive again.

More generally, it is uncontroversial that virtually all current software is garbage. Everything I use and pay for is littered with bugs. Some software is so broken that I can’t even pay for it. I have not been able to send a wire using Citibank’s online banking in three years. Most web apps can’t even get mobile vs. desktop right. Nothing has the functionality that you want. Everything is deficient. Silicon Valley darlings like Stripe and Linear have built massive followings just by not being as insanely unusable and horrendous as their competitors. Ask tenured engineers “show me a piece of good software” and you’ll get long silences and blank stares.

There is a deep and important truth here: even if we get something like the Software Singularity, the level of demand for labor here is practically infinite. Famously, it is the last few percent of completion that take the most work, and by that token, virtually every software product could probably scale up its complexity and features by something like 100x before beginning to saturate demand.

I have the feeling that commentators on the imminent demise of software don’t have much intuition for making software. We’ve had software for about fifty years now. Though it has improved meaningfully over the years, it has always been inadequate. As a programmer in 2020 I was able to do what would’ve taken hundreds of man-years in 1970; the leverage gained is incredible, but the results still leave massive space for improvement at every step along the way. People underestimate Jevons Paradox. Importantly, this does not mean that software engineering is a forever-resilient source of jobs. Of course not; nothing is. But my point is that again, the sector has more momentum and ability to absorb labor than people give it credit for, and saturation of this will be a slow process, giving us time to respond and adjust.

3. Re-Industrialization

There will be some labor displacement, of course. Driving stands out. Many types of white-collar work, as Citrini suggests, will undergo some gyration as some jobs disappear and others change meaningfully. AI may be the straw that breaks the camel’s back for jobs like the real estate broker, where the job had actually already disappeared a long time ago, but the pay was still there.

The saving grace here is that in the US, we have a virtually limitless capacity and need for re-industrialization. You may have heard about bringing back manufacturing, but it’s more than that: we largely no longer know how to create, and don’t have the facilities for, making the core building blocks of modern life: batteries, motors, small semiconductors — the whole electric stack is something we are almost entirely dependent on China and other countries for. What if there’s ever a military confrontation? Actually, it’s much worse than that: did you know China makes 90% of the world’s ammonia? If there’s a war, we can barely make fertilizer. We’d just starve.

Once you start looking at the physical world, you see a virtually endless scope for work on job-creating, nation-benefiting, fundamental infrastructural work that is politically bipartisan.

We’ve seen the economic and political milieu slowly make their way in this direction — talking about re-industrializing, manufacturing, deep tech, American dynamism, and so forth. My prediction is that as AI challenges white-collar labor, the political path of least resistance will be funding large-scale re-industrialization in the form of employment megaprojects which, thankfully, are not subject to a singularity but rather move at the friction-heavy speed of getting things done in the physical world. We’ll build bridges again. People will find it gratifying to see the fruits of their labor in the real world, not in digital abstractions. The Senior PM at Salesforce that loses their $180K job might find a new job in the field at the California Desalination Works, to finally, finally, end the 25-year drought. And it shouldn’t be good enough, but excellent. And once it is built, it must be maintained! Once more, Jevons Paradox can apply, if you allow it to.

4. And Beyond

The outcome of industrial megaprojects is of course that we move toward abundance: America will once more be independent, and make things at large scale and low cost. Transcending material scarcity is the key: in the long run, if we do lose almost all the white-collar jobs to AI, we have to be able to provide people with a continued high quality of life. Part of this we get automatically, just because AI taking margins to zero means that those consumer products will become equivalently cheap.

My view is that different parts of the economy will “take off” at varying speeds, and virtually all the areas are slower than a piece like Citrini’s might suggest. To be clear, I am extremely bullish on AI, and expect that one day, my labor too will be obsolete. But it’s going to take a while to get there, and that time gives us the opportunity to make good policy.

On that front, preventing a market meltdown the way Citrini imagines is actually pretty easy, and the federal government’s response during Covid showed how proactive and aggressive it is willing to be. I’d expect large-scale stimulus to kick in quickly once needed. It slightly irks me to say that it won’t be efficient, but that’s also not the point. The point is material prosperity for people in the course of their lives — broad consumer well-being that legitimizes the state and carries forth the social contract — not satisfying the accounting metrics or economic norms of the past. If we are nimble and responsive to this slow but sure technological revolution, then we will be fine.

These changes also extend to open-source software, our great collective project that underpins technology everywhere. Over the last few weeks, I’ve seen projects like tldraw change how open-source software is written, and openclaw change how it is utilized. These are important changes, not just for open-source aficionados like myself. In this blog post, I’ll make some predictions: the future will have a lot more open-source software, written by machines but funded by people, giving better experiences and greater freedom to the consumer.

Creation of Open-Source Software

The term open-source means the source code of such software is published for anyone to read, modify, and redistribute. This key tenet provides several important benefits:

• Trust: anyone can audit the software and raise issues;

• Free as in freedom: anyone can edit their copy of the software as they please;

• Collaboration: anyone can publish their modified versions of the software;

• Free as in beer: it doesn’t cost any money!

This puts open-source software (OSS) in a neat intersection of the personal and the professional. People contribute to it in areas of their interest; OSS is a patchwork of passion projects. While those engineers are not doing it for money, the experience gained by this work, and the respect it confers, becomes a professional advantage.

Many junior engineers, myself once included, took their first steps into “real” engineering by making small contributions to open-source projects. But today, many open-source projects are considering closing themselves to contributions:

As the author of tldraw explains, AI coding tools are leading to a proliferation of very inexperienced developers using AI to generate massive, poorly-formed PRs (requests to contribute code to a project), which drain time and attention from the maintainers, who need to review those requests.

Previously, low-quality contributions had always been an issue, but they were kept in check by the fact that writing them still took significant time and effort, much more than reviewing them. For tldraw, Pradyumna Prasad ran the numbers: PRs by external contributors went from 90% acceptance in 2021 to 43% in 2025 — nearly a 6x increase (10% → 60%) in bad submissions, an overwhelming quantity.

There are a few clear ways to address this problem.

• Projects could use AI tools like Greptile to automatically review PRs, and stem the tide of junk. But those cost money, which OSS projects usually don’t have.

• Projects can do some gatekeeping and set a more requirements-heavy process for first-time contributors.

• Projects could close their doors to code contributions altogether and only accept monetary contributions.

The third bullet here might sound crazy, but it’s not that far off. Many skilled engineers already report basically no longer writing code, but just interfacing with AI to define the job to be done, and then letting the AI run with it. This means that the leverage provided by money (to purchase AI tokens) on top of the time of a skilled developer is growing significantly. SemiAnalysis reports that already 4% of public GitHub commits are written by Claude Code today.

Additionally, the open-source development model where anyone can try to contribute is also under strain from supply chain attacks, in which malicious contributors try to hide sophisticated malware in open-source software. Similar to the deluge of low-quality PRs: we should expect this to increase due to the ease of AI code generation.

Given these trends, we may expect a world in which projects are closed-by-default to outside contributions, and a small group of trusted maintainers mostly direct AI to make contributions by spending donated funds. This would also make donations more appealing for donors, because they could direct the funds more precisely — instead of a general-purpose $20 donation, you could direct those dollars to buy AI tokens to make a specific change.

If the future of open-source software is therefore basically crowdfunding, then this would be a tremendous accelerant: right now, the creation of open-source software is constrained by the spare-time throughput of a small number of skilled humans. But if it becomes easy and accessible to contribute small amounts of money to creating the software that everyone2 wants, then this ecosystem will grow by orders of magnitude overnight. There is no way that proprietary systems would be able to keep up. It will finally be The Year of Linux on the Desktop.

Utilization of Open-Source Software

Open-source came about in the earliest days of computing, when all software, and the data it used, lived on your computer. But around 2010, mobile and the shift to cloud changed this dynamic: suddenly your software was online, and the data not on your disk, but on some faraway server.

This made it harder for open-source software to succeed, because even if you wanted to use it, you might not be able to get your data from a proprietary service and use it in an open-source client. The shift to cloud meant an emergence of walled gardens, where users are not free to port their data to competing services.

Some enthusiasts3 wished for a return to the prior paradigm, in which users could possess all their data on their computers, and provide that data, only as needed, to third-party cloud services. There are a lot of good things about this — privacy, security, ownership — but most users don’t care much, and the level of engineering required to implement API interfaces for every single service makes it impractical.

However, in our burgeoning AI era, this dream lives:

• Many cloud software products that would operate on your data can be easily replicated with AI coding tools;

• AI is amazingly good at implementing on-the-fly API interfaces to services, making it easy to pull all your data from a service to your disk, or to conversely upload it.

What we saw with openclaw is that it is viable to pull your entire context to your machine, give it to an AI agent, and let it rip. A variant of the bitter lesson strikes again: having everything in one place and letting the AI munge it is both more general and more performant than painstakingly curating a bunch of special-cased behavior.

There is now a point to having all your data on your disk that goes beyond abstract trust, privacy, security, or long-term ownership: having all your data will make your AI provide better results! I suspect that in coming years, we’ll see a great repatriation of user data from lots of fragmented services into just one place — either a cloud repository that they control, or to their own hardware.

Having overcome the walled garden problem, this creates opportunities for open-source software4 to operate on the user’s data. But what’s going to be operating most heavily on the user’s data? Well, AI models, of course. And those can be either proprietary (like OpenAI or Anthropic today) or open-source — ones that you can run yourself.

Open-Source Models

What does it mean for an AI model to be open-source? A model is not just determined by its source code. The authors would have to make available the model weights, training code, inference code, and training data. The training data is a tall bar to clear, because it’s usually legally messy.5 For that reason, the most prominent “open” models like Kimi, Qwen, and Llama are predominantly open-weight.

On the one hand, this violates some core principles of open-source: if you can’t reproduce it, then you can’t audit it. There’s a significant problem in not knowing how your model was trained: what if it was built to suppress certain types of information? However, this may not be that severe an issue in reality, because you can build evaluations to test the quality of the model. For example, it would be very easy to figure out if a model has been trained not to speak of Tiananmen Square: just ask it.

But on the other hand, an open-weight model still is enormously empowering from an open-source perspective: you can run it privately, trustlessly, on your home computer.6 And between algorithmic and hardware improvements, the quality of home model performance will become better every year.7 It’s well possible that this ends up sufficient for personal purposes.8

Conclusion

There’s a good chance we are moving into a golden era of open-source software. It is faster and easier than ever for people around the globe to collaborate in calling great software into reality, and the accessibility of this creative process — all you need is taste and a little bit of money — is increasing dramatically.

Further, the explosive capability of AI to both pull data from, and make obsolete, traditional cloud software, means that we may see a paradigm shift: data back to the user’s own disk. The user will benefit greatly from having all their personal context in one place for AI to consume and analyze. Finally, for many cases, the AI that runs on this data may take the form of a fully locally hosted open-weight model.

All in all, this is tremendously hopeful for open-source values. What we’ve described above will return sovereignty, freedom, privacy, and trustlessness to consumers everywhere. It looks tremendously empowering for the individual. It is bearish for proprietary software that will struggle both financially to compete in this paradigm, and in terms of feature parity with these open-source projects that we should expect to grow in scope by orders of magnitude.

And yet, in its entire fifty-year history, Silicon Valley has never had political representation: no congressman, no senator, no governor has ever come out of Silicon Valley’s great technology industry. At best you might count one mayor2 and one guy on San Francisco’s eleven-person Board of Supervisors.3 This is paltry, and the lack of representation is now putting Silicon Valley at risk of being destroyed by political looters. Timing could not be worse: the global AI race is on, and bad local policy could destroy the key to America’s technological dominance and sovereignty. This places extraordinarily high, national-level stakes on keeping Silicon Valley alive.

In this essay, I will make the case that:

1. Financially tough times are ahead for California;

2. California’s government will almost certainly try to loot Silicon Valley;

3. Silicon Valley will flee;

4. This will create a severe loss for the Bay Area, for California, and for the USA;

5. The only way out is for successful technologists to run for and win office at every level in the 2026 mid-term elections, especially for governor;

Most importantly: if you are a successful person in Silicon Valley, I want you to think very seriously about running for office. Someone needs to step up.

1. Tough Times Ahead for California

California is going through an economic boom. Tax revenue has basically doubled over the last ten years. However, literally all of it is being spent as it comes in!

It’s a strange picture: revenues are on a generational tear — chiefly because tech is succeeding more than ever — but the state is running a deficit.

Worse, these high-level statistics actually look better than the reality, because the General Fund constitutes only about 60% of the state’s total expenditures. Conventional state budget discussions do not include all the details. For example, Governor Newsom took out a $20B loan for pandemic-era unemployment programs, which is now being slowly repaid by additional federal payroll tax. And California’s pensions are underfunded to the tune of a head-spinning $260B, for which the taxpayer is ultimately responsible.

Sometimes, when someone gets a financial windfall, it doesn’t encourage prudent saving and reinvestment, but puts them on a track of profligate, eventually-ruinous spending. This is what’s happening for the state of California. The California Legislative Analyst’s Office has projected that the state will run a $120B deficit in five years. (Simon Berens published a great interactive tool for exploring these projections.) California has the simple but hard problem of runaway spending and out-of-control entitlements.

Today, things are okay. Industry is booming, and the 2025-2026 deficit is most recently projected at only $3B. But as the business cycle naturally gyrates, severe pressure will be exerted on California’s public finances, and on the next state administration to plug the gaping hole in the budget.

2. The Looting of Silicon Valley

This essay follows a recent proposal for a wealth tax on billionaires,4 which is already misfiring spectacularly. But what stands out is that this heavy-handed, controversial tax is being proposed during a time of relative prosperity.

We must ask: if extraordinary taxes are being considered during good times, then what will bad times bring? What happens when the budget is actually under pressure?

You might hope that the state will reduce its spending, but that’s the last thing that will happen. Reducing the budget is very unpopular, because people get used to spending the budget! Nobody wants to spend less: if there were any demand for efficiency of government spend, then California’s financial catastrophes — spending $70B on high-speed rail with little to show for it, $24B on homelessness only for it to go up, or up to $31B on fraudulent unemployment claims — would’ve been righted long ago. The unfortunate reality is that the state budget is a gravy train for millions of people, and nobody has had the strength of will to contain it.

Rather than cutting spend, the state will raise taxes. Of course, it is easiest to raise those taxes from a small minority of people and institutions with lots of money: Silicon Valley is a sitting duck.

After all, how many people are actually personally interested such that they would vote against a tax on, for example, capital raised by venture funds or startups? A few thousand? Why do a one-time billionaire wealth tax when you could do it more frequently, or lower the bar to $100M or perhaps even $10M?

This might sound far out, but bear in mind: this is the same state that was radical enough5 to pass Prop 13, and because of which, it does not have property tax hikes available. By California’s own design, wealth, income, and sales taxes are easiest to reach for — which will end up being borne by the middle class, not billionaires.

3. Silicon Valley Will Flee

Like in any great tragedy, the sin itself, the attempted looting of Silicon Valley, is pointless: these are some of the smartest people in the world, who will see it coming from miles away. Many of their jobs are literally predicting the future. The public-sector unions will not outsmart them.

We saw this just over the last few weeks with the proposed wealth tax on billionaires: though the tax is to be voted on in November 2026 and implemented in 2027 if it passes, the tax obligation would be backdated to January 1, 2026 — so if the tax passes, billionaires are locked-in to paying it and can’t relocate it to escape it.

Unless, of course, the billionaires relocate in the few weeks between the ballot initiative being announced in October, and the tax threshold date of January 1. Which is exactly what happened: over a trillion dollars of wealth fled California in a few weeks, over a ballot measure which may not even pass.

The people of Silicon Valley are highly mobile. Their work can famously be done from anywhere. They’re also the most childless people in the country — conventional obstacles to moving, like nearby family or kids in school, apply much less here than elsewhere. The network effects of Silicon Valley are already weakening:

Tragically, despite harvesting great tax dollars from these people, California has failed to put those tax dollars toward building things that would keep them there: the quality of public schools is highly variable, the public infrastructure is poor, and public safety is bad. If California used its taxes to offer an exceptionally high quality of public services, then perhaps it’d have more pull. But it doesn’t. The divorce between taxes charged and public service provided has occurred a long time ago.

I have never met anyone in California who is excited for a new tax because it means a new local benefit, like a road or a sports center. The going assumption is that any new tax will be wasted in byzantine graft, no matter how noble the objective. We pay taxes like a club membership fee: the club will charge as much as it can, that’s the price of living here, and you’ll get nothing in return except admission. But the downfall of any club is that it arrogantly forgets that it is a product, and when the product gets worse and more expensive, the customers will go elsewhere — slowly, then suddenly.

It bears emphasizing that during Covid, many people thought Silicon Valley was done. Overzealous stay-at-home mandates, remote work, and an absolutely bonkers level of local crime had technologists leaving in droves. It felt apocalyptic: by June of 2021, nearly everyone I knew had left. The California exodus was real. My entire social network had dispersed to Miami, Austin, Denver, New York, Nevada, and so forth. And then I left, too.

Silicon Valley was saved just in time by AI, which reignited the local network effect. I returned, but many of my friends remained elsewhere. Though it’s boom-time in the Bay Area once more, this episode laid bare that Silicon Valley, like any product of network effects, is fragile: it is very hard to reverse the doom loop once it starts.

4. Loss for the Bay Area, California, and the USA

Potential losses to the Bay Area and California are obvious. Once executives start moving, their companies start moving, too. Google’s 80-acre San Jose megacampus is on hold indefinitely. Meanwhile, employees are moving into Google’s offices in Austin. Citadel fully relocated to Miami, after 30 years in Chicago. It can be done.

California is uniquely vulnerable because ~45% of its tax revenue stems from personal income tax, and ~35-50% of that revenue6 comes from the top 1% of earners.7 The mere proposition of the wealth tax is already going to lose a tremendous amount of tax revenue, more over the next few years than it ever would’ve gained had it passed. The new shortfall of tax revenue means that the burden will now fall on the next tranche of productive residents. The vicious cycle of trying to loot a small minority group, it fleeing, and then needing to go loot the next one, happens almost automatically, can destroy unlimited value, and capture none.

In some sense, this is a classic resource curse, where the state of California has historically had a seemingly never-ending spigot of money, totally irrespective of any actual work that the state does, and only has to spend it. This creates dependency, waste, and hollow institutions. And when it looks like the spigot will turn off, the state will go into overdrive destroying it to get every last drop out. Truly great mismanagement requires truly great resources.

Cracks show elsewhere in the armor: Hollywood is no more. And the homes in the Pacific Palisades that burned down a year ago still haven’t been rebuilt. The lack of California’s state capacity is clear. It now has the most expensive electricity in the continental United States, which reminds of de-industrializing countries like Germany and the UK, currently undergoing economic doom loops themselves. Californians would be wise to look to those countries, for which the last 25 years hold just one lesson: it is possible to fall from prosperity.

California would not be the first state to lose its fortune. Boston, once a hotbed of American entrepreneurship, managed to kill its startup ecosystem. Many countries in Europe have experimented with wealth taxes and seen their most productive citizens leave. Too-clever-by-half countries like Norway slapped on exit taxes on top, only to thereby choke off local entrepreneurship altogether. It is a bitter irony that nearly every great technology with European founders is started in America. California may be beautiful, but so is Europe. People move on.

But the story does not end here. The stakes are greater: we are living through the AI revolution. America must develop leading AI capabilities to keep our technological position, and — if AI truly does diffuse everywhere — to keep our sovereignty. We are in the midst of an AI race, and we would be well-advised to stay ahead! To that end, we have the greatest possible advantage: pretty much the plurality of the world’s greatest AI researchers and engineers all living in the same place, knowing each other, exchanging information in their tight-knit community. We have an organic, private-sector Manhattan Project. We have firms raising tens of billions of dollars to invest in the smartest entrepreneurs building AI in America. This is a priceless national asset, far beyond the capabilities of any other nation-state but China. Squandering this advantage — and that includes the dispersion of Silicon Valley residents, crushing the productive in-person network effect — because of the short-sightedness its own political representatives would be an unspeakable historical loss.

5. Successful Technologists Must Run for Office

The stakes are great. We as technologists need representation because we will otherwise be governed by unwise forces, which may spell our destruction.

The greatest of penalties is being ruled by a worse man if one is not willing to rule oneself.
Plato

Silicon Valley figures have historically avoided politics, for a variety of reasons: they have great companies to run, the financial payoff from entering politics tends to be poor, and the quality of life tends to be worse, with all kinds of people mad at you all the time. They have instead stuck to donating and delegated the thick-skinned job to representatives like Ro Khanna, who have unfortunately shown that they cannot be trusted to represent these constituents: as so often in financialization, we cannot pay someone else to do the job, but we must do it ourselves. Someone needs to step up.

It is an unthinkable sin that the work of the greatest innovators and savviest capital allocators of our time is given as tribute, placed on the high altar of government, only to be frittered away on waste and fraud. Only when the waste and fraud is cataclysmically bad has the tech community stepped up to respond. But being reactive is, of course, a poor way to participate. We need tech to be proactively and affirmatively involved in governance — imagine how prosperous our ecosystem would be if we worked to support it, rather than having it administered by fools who see it as a cumbersome cash-cow to be milked. In a wiser land, the government would throw everything it has at supporting Silicon Valley and making sure that the technology sector keeps compounding.

If you feel any love for where you live, at any level — whether for San Francisco, California, or the United States — and you’re successful enough to command the respect of the public, then you should run for office.8 We need representation. I will do whatever is in my power to support you. I love California and want it to succeed. Reach out to me and I will put you in touch with like-minded people.

As a parting thought, while I have focused on state and local issues in this essay, I’d like to highlight that perhaps even greater trials lie ahead: as AI begins affecting labor markets and changing our society more broadly, it is easy to foresee some public uncertainty and “reining in AI” becoming a hot-button political issue at the federal level, and characters like Newsom or Khanna turning on Silicon Valley to advance their own national-political ambitions. Neutering Silicon Valley would, of course, leave the US losing the AI race to China. We need courageous technologists to step up and guard the goose that lays the golden eggs.

Years later, he connected the dots: the school was at the bottom of a hill, in a little valley, and the playground right by the busy main road. All the exhaust fumes pooled and hung in the air there. And these were the 1970s: literally all the gasoline was leaded.1 This was lead poisoning. Over the years, the children were getting brain damage.

Nobody knew. There was no pediatric lead testing.2 Later pilot studies in Birmingham, Manchester, and Glasgow would eventually confirm this: children were found to have average blood lead levels of 3-5x the safe maximum. Just imagine what the severe cases looked like.

This story has stuck with me. It features the shocking and tragic loss of healthy lives — condemned to live in functional disability — brought about by many well-intentioned people doing their best, trusting that the status quo is safe and normal. But it often isn’t — what you hope and trust to be fine is secretly killing you.

The world has come a long way on this. Standards have improved substantially: houses are no longer being built with asbestos, lead paint is no longer permitted (though chances are your house has some), public water is mostly clean, and so forth. But better codes don’t go all the way: if the municipal water is fine but my house’s pipes are made of lead, that’s still a big problem. If mold is silently growing in my walls, nobody’s looking out for that — I’m on my own.

Subscribe now

And there are new dangers. Globalization means a world where nobody knows what’s in anything anymore because the supply chains are so complex, the financial incentives are to bring the costs down as much as possible, and when something is full of poison, you have no recourse. Somehow we wound up with steaks from Whole Foods being chock-full of BPAs — yes, even meat has microplastics!3 And regardless of whether you shop on Amazon or at Restoration Hardware, pretty much everything4 is sourced & manufactured far outside your control.

Is the furniture I sit in every day made with harmful substances? I don’t know. Are my plates, pots, and pans safe to eat from? No clue.5 And if they aren’t, there’s no way for me to assert my rights or collect a single penny from some faceless factory in Cambodia. If you think there’s any kind of quality control, there’s zero — nothing is getting inspected. Every nine months it turns out my protein powder contains heavy metals. The border can’t even stop counterfeit Rolexes from getting through, and the bar for listing a product on Amazon is the floor.6 I am sorry to say that for consumers, the buck stops with no-one but you. And your position is totally helpless.

Two years ago, I kept seeing these ads on the NYC subway. It’s so crazy: in one of the wealthiest cities on the planet, babies eating from lead-contaminated glassware is so pervasive a problem that a private company has to step up to do basic quality control.

NYC babies are not the only ones silently getting their IQs nuked because of careless manufacturers. Afghan children probably have the catastrophically highest levels of blood lead — even in the diaspora abroad — because virtually all the manufacturers of traditional Afghan cookpots were using lead-contaminated metals. Even when this was found out, it took Amazon over a year to take down the listings for the damn things. The level of public harm is off-the-charts. Chances are you don’t own one of these, but when’s the last time you might have eaten in a restaurant that does?

The problem is so overwhelming that you almost can’t engage it. There’s just too much stuff to check on your own. This is catnip for neurotic Type-As. You’ll drive yourself crazy if you try to fix it. And, in fairness, none of these hazards are big and likely enough on their own to warrant your deep-dive attention. It’s in aggregate that they’re impactful: in your life, most risk factors aren’t an issue at all, but there’s probably something that needs to be found out and fixed. The only solution is to delegate it to a third party that you can trust to do a really thorough job. Only a business with this as its core competency is capable of the breadth and depth required for this Herculean task.

In Germany, there’s a popular nonprofit which tests consumer goods for safety and publishes the results. When I was a baby, my mom followed their publications, and only bought the baby foods, diapers, etc. that had been deemed high-quality. Those ratings alone directed many thousands of dollars of high-margin spend for her. Consumer goods are as big as markets get, parents are willing to spend virtually any amount of money for the benefit of their children, and the product scope is endless. There’s going to be a generational company that uncompromisingly creates trust and will charge a hefty premium for never breaking that trust. Providing infallible peace of mind is the strongest of moats.

I am seeing the latent demand. Technology is empowering citizen scientists: consumers are taking charge of their health. They’re buying Whoop,7 Mira, Levels, Eight Sleep, Nucleus, Ezra, Function, etc. to understand their bodies, optimize their health, and catch potential issues way ahead of time. They’re starting to want things like Blueprint, where the manufacturer is staking their credibility on the work they’ve done to own the whole supply chain.

Soon the penny will drop with the public: health is not just about your body, but about your environment. People are starting to pay attention to air quality.8 They’re realizing that the “premium” consumer brands are full of microplastics. They’re waking up to the fact that life can and should feel better. Everyone wakes up congested, everyone gets headaches, everyone gets a rash sometimes — but these “normal” experiences are your body telling you that something is wrong. It’s just so common that it’s normalized. And so many health outcomes that people talk about in terms of luck are actually deterministic, but people gloss over there being causality at work.9 The problem is large. And we have the science to do better.

Health is the final frontier. The idea of luxury was once conferred by design, materials, and manufacturing — but today, even the highest-end goods are now instantly replicated for pennies on the dollar. The question that remains is what lurks inside: the peace-of-mind escape from hidden hazards is not just necessary, but offers infinite optimization.

This will be a big business. It has been on my mind for many years now. I’ve seen all the startups that have taken a stab here — Yuka, Oasis, Tap Score, you name it. But while I admire their missions, I don’t think anyone’s historically gotten this right as a business. Now I’ve finally met the right founders taking the right approach: empowering people as citizen scientists, and taking on the big task of monitoring for and remediating hazards at home. This is very important to me, and I am excited to help them succeed. If this mission sounds interesting to you, email me at contact@johnloeber.com and I’ll put you in touch.

But this has not been the case at all. LLM adoption has been everywhere, for everything, at all once. Ordinary people use LLMs as their therapists, students use them for papers, lawyers use them for case filings, executives use them to draft their public statements, politicians use them to draft policies — from official memorandums down to tariff proposals. The format is incredibly seductive: the answer comes instantly, it’s usually good, the prose is nicely written, it feels authoritative, and you can get it at any level of detail that you want.

The reason why LLMs have made their way so quickly into even the most sensitive contexts is that there’s an analog hole problem: even if an organization forbids the use of LLMs, at the end of the day, a task is given to an individual person, that person has broad latitude to complete the task, nobody will know if they ask an LLM for the key questions, and the outputs almost always easily clear the “good enough” bar with minimal effort.1

Voters use LLMs too, of course. Being an informed member of a democracy is hard: there’s so much going on. Even politicians who are in the thick of it all day every day are stretched thin for attention. I’m sure that voters and policymakers alike are finding LLMs to be a real godsend, helping them make sense of complex issues and hundred-page policy proposals. There’s no doubt that this is an improvement.

Centralization

What this points toward is a future where everyone is outsourcing their knowledge and reasoning, all the time. And I agree with Noam Brown that this largely isn’t going to be a multi-model future: raw scaling seems to obviate the need for distinct models or architecture that routes between them. Between returns to scale in usage2 and economies of scale in hardware/compute, the future of technology looks much more centralized than it has historically. There may only be a few companies providing state-of-the-art LLM services.

This raises an obvious concern about centralization. Societies today rely on a vast network of digital products and services, provided by all kinds of people with all kinds of backgrounds. The same is true for knowledge — we distill our views and opinions from a global jumble of resources. But the future looks different: both will be far more centralized.3 Some large, large percentage of all your interactions with a computer, or questions about the world, might just be with OpenAI.4 Remember how your school teachers told you not to use Wikipedia as a primary source, and yet practically everyone does anyway? Dial that up by a factor of ten.

Manipulation

We’ve also seen how vulnerable people are to LLMs. There are plenty of stories of lonely people falling in love with LLMs, or falling into what some call LLM Psychosis.5 There’s even a very recent public example of this happening to a prominent investor with billions of dollars under management.6 These technologies are still in the very early stages of rolling out, but it’s clear that psychological dependency on LLMs will be a big theme of the coming years.

But you don’t just need to worry about explicit cases like AI companions pushing their users toward unsavory actions. Maybe you’re worried about LLM companies parroting the views of their executives. Or nation-states pushing propaganda through their LLM firms. Or censoring/rewriting history for controversial events. Maybe you’re worried about subtler, just asking questions-style propaganda designed to sow doubt and discord, rather than push any particular view.

While those risks are real, they can be much subtler yet. Imagine asking an LLM what the causes of World War Two were, and consider these answers:

• The main causes of WW2 were…

• The mainstream theories about the causes of WW2 were…

Even if the factual content that follows is identical, the latter answer softly implies that the questions are not settled — these are just theories — and while there are mainstream ones, there must also be contrarian, more intriguing ones. You know exactly what the user is going to ask next. That’s all it takes. The wording here is totally benign — it would pass any safety test. And yet this kind of nudge, applied over and over again to any number of questions, across a whole population, would surely move public opinions and beliefs. Remember all the concerns about nation-state misinformation on social media? Again, dial it up by a factor of ten.

Democracy

The core principle of a democratic nation is that of popular sovereignty: the power of the government comes from the people, it governs only with their consent, and it is independent of any other power.

Conventionally, the people give power to the government by weighing their views in a voting process. But if everyone is ultimately getting their views from an LLM, and that LLM is biased in some way, then your core apparatus for bestowing consent has been compromised.7 Popular sovereignty has broken down. If you take a more aggressive position, you might say that sovereignty has been lost to the LLM provider: they’re now dictating the facts and views that permeate your culture, and everything is downstream of that.

Sovereignty by LLM

This suggests that in the future, a country won’t be self-assuredly sovereign unless it is able to train its own LLM8 from scratch.9 That may seem like a tall order, but it’s necessary10 — if the essence of your state is a consensus mechanism for navigating the facts of the world, then you can’t have someone else dictate the facts. You can’t even risk the possibility of this. Importantly, this concept is not new: in the US, foreign media ownership used to be heavily restricted, and still is elsewhere.

When it comes to LLM sovereignty, the US and China11 are both in fine positions. Virtually all the leading LLM providers are American or Chinese companies. Saudi Arabia and the UAE are also well-situated in this respect: they may currently not have their own LLM providers, but they (are going to) have huge data centers on which LLMs are trained and run. Owning the hardware helps.

Europe is, as usual, in trouble. Unwise energy policy has saddled many European countries, particularly Britain and Germany, with some of the highest electricity costs in the world, making it hard for them to operate competitive data centers. To boot, they’ve been so focused on regulating rather than owning these assets that they have no frontier LLMs and therefore no seat at the table at all.

Conclusion

LLMs are going to be everywhere, informing all decisions, even for the most sensitive political matters. If democracies come down to information, and information comes down to LLMs, then any country will need its own LLM, probably even running in domestic data centers,12 as a simple matter of sovereignty. Few politicians currently seem to understand this, and many countries appear to be sleepwalking into a rapidly changing world. We may see supra-national alliances become established, or expand their scope of responsibilities, to include pooling resources for this purpose. The big question on my mind is how quickly this will happen, or if it will take some kind of incident to invoke concern and spur nations into action.

In recent months, many other people have also shied away from Google: ChatGPT is eating into Google Search, and Google’s public response has been tepid. Is this a textbook example of the Innovator’s Dilemma? Will Google’s empire crumble?

Such fear and doubt is reflected in the stock: Google is now trading at a 19x P/E ratio, when its historical average over the past decade is 28x, and today’s S&P average is 26x. In other words, the street ascribes a much lower value to Google’s profits than to those of other companies, implicitly anticipating a collapse in Google’s profitability.

But this is myopic, a view far too fixated on legacy conceptions of Google’s Search and advertising business. While the near term is anyone’s guess, the street substantially undervalues the totality of what Google has built, and how that positions Google for the future. My view is this:

1. AI poses threats to Google’s Search business, but they are overrated and solvable;

2. In fact, AI may supercharge Google’s existing Search business;

3. Google is best-positioned to win the AI race;

4. If Google wins the AI race, it may become a $20T+ company in 5-10 years;

5. Oh, and, by the way, Waymo is a trillion-dollar company hiding in plain sight.2

Points three and four in bold are the ones that really matter. The AI community’s AGI timeline is now only seven years out. Most people do not understand:

• These years will pass quickly;

• As we get closer to AGI, trillions of dollars in potential revenue become unlocked. The first firm(s) to the finish line will win the largest economic prize in history.

• Google is in the lead to win.

It’s easy to miss the value. Many investors are bearish on Google because they are fixated on Search as an immutable one-trick-pony, and Search appears paralyzed in a changing world. But Google’s position for AGI is wildly underrated, and it presents opportunities that make questions like whether Search makes money or not unimportant. There is a much larger game in play now. My bet is that Google slowly but surely turns the ship, and in this essay I’ll chart their path from here to a $20T+ world.

Part 1: Search

Many commentators view AI as disruptive to Google Search: people are going to ChatGPT rather than to Google Search because it provides better answers,3 and the answers are exhaustive such that no monetizable click on an advertisement can occur. But this misses a few things:

• Net search volume is still growing. Google’s Search volume increased by 20% from 2023 to 2024. This may feel like a mature industry, but in some respects it is still early! People are still coming online. Software continues eating the world.

• If Search becomes more like a ChatGPT-style experience, that may decrease link clicks, but not necessarily ad clicks: only ~20% of searches show an ad, and fewer yet result in an ad click. Today, most searches are not monetizable at all.

• ChatGPT-style queries and answers may turn out more monetizable than traditional searches because the questions are higher-intent, and the answers surface far fewer links, which better nudges the user toward any displayed link.4 The prose of the answer can further nudge the user. As this matures, I’d expect higher click-through-rates/overall value for advertisers.5

• Google has the world’s best dataset on queries, ads, and user behavior, and Google’s ads are already partially AI-generated today. The advertiser only has limited ability to provide guidance. Advances in AI further empower Google’s existing advertising flywheel.

• Finally, Google may eventually capture far more value by not getting paid for an ad click, but by closing the loop and offering the product or service that the user is looking for. This enables Google to capture the full amount the user is willing to pay, rather than just the partial margin ceded to an ad click. More on this later.

In short, the future of Search seems to come down to two questions:

1. If ChatGPT offers a superior form factor, can Search move toward that form factor and avoid disruption? I think so, and it seems to already be happening.

2. Can advertising work just as well in that form factor as in traditional Search? Early results suggest yes, and it may work even better. The ChatGPT form factor is more powerful in how it can present the result to persuade user action.

Finally, if there’s a lesson from the last twenty years: whether for countries or big tech companies, betting on the collapse of an incumbent with great momentum rarely works out. Google has colossal momentum — old user habits die hard, and Google’s services are among the most deeply entrenched in the day-to-day lives of consumers.

Part 2: Positioning for the AI Race

But forget about Google’s Search business for a minute, and consider what Google has:

• The most visited website on earth, the default entry-point to the internet for most humans for 25 years and counting;

• The #1 consumer brand in the world;6

• Gemini: arguably the best AI models;7

• YouTube: the world’s biggest repository of video data;

• Google Search: the world’s biggest store of internet data, having scraped the entire internet for the past 25 years;

• Google Books: the world’s biggest store of published words;8

• GMail: the most popular email client with 1.8B active users;9

• Google Drive/Docs/Sheets: the most popular workplace suite in the world;10

• Android: the most widely used mobile phone operating system on earth;

• A mature devices business including phones, laptops, watches, home assistants…

• Google Chrome: the most popular web browser in the world;

• GCP: their own cloud, behind AWS and Azure;

• TPUs: their own chips for machine learning, now used by OpenAI;

• Global data centers representing about $200-290B in investment-to-date11 and another $75B committed;

• $100B on their balance sheet;

• ~$110B in annual operating profit that they could plow into AI if they so wished;

• ~180,000 employees including some of the very best and brightest machine learning researchers and engineers on the planet;

• A truly massive amount of user behavior and ad performance data;

• Endless weird dark horse projects that aren’t even on the public radar right now.

Don’t be distracted by existing revenue or product-in-market. The more you think about Google’s structural advantage in AI, the more staggering it is.12 They own the whole vertical stack required to win.

The full strength of this competitive advantage against Anthropic, OpenAI and others is yet to become apparent: where other firms top out, Google can keep pushing. Right now, the big AI labs are all focused on making better use of their not-fully-exhausted resources in terms of data, capital, and compute. Therefore, model performance is pretty competitive, and the perceived market leader switches every few months. But eventually, these firms will fully saturate the data, capital, or compute available to them. And however much they may have, Google has a lot more. Similar to how Mistral, Cohere, and others once looked competitive and then couldn’t keep up against superior resources, the same fate may play out at much larger scale — companies worth tens or even hundreds of billions of dollars exhaust their resources while Google’s products and distribution keep improving.

For the last few weeks, Meta has given us a taste of what it means for a trillion-dollar company with conviction to flex its weight: raiding competing labs to the point that OpenAI shut down for a week. Google has barely begun seriously competing; the world will look different when it does.

Part 3: Winning the AI Race

What does it mean to win the race to AGI? There are three important parts to it:

1. Distribution: AGI replaces white-collar labor, which means that many services and products will become commoditized. Commodities differentiate themselves by the quality and efficiency of their distribution (marketing). Without a doubt, Google has the greatest distribution machine in the world.
   
   Today, companies that offer commodity products figure out their gross margins, decide how much money to give up,13 give it to Google, and Google provides them some number of clicks, hopefully yielding a positive return to the company. The more lopsided the importance of distribution, the more margin gets reallocated to Google. There’s an argument that marketing is the final industry — human attention the final scarce resource — and Google is not just the leader, but has a powerful structural network effect and moat. That’s a winning position.

2. Verticalization: one of the biggest unsettled questions in AI is to whom the profits will accrue. At every level of the stack, there is margin uncertainty: for example, do profits accrue at the application level? Or will applications be commoditized, and the pricing power will rest with the fundamental model providers? Or with the chipmakers? Who’s going to squeeze whom?
   
   In light of this, I’m inclined to bet on the player that can truly own the full vertical, from chips to end users, rather than on players in never-ending battle with their partners over margin and control. This verticalization doesn’t just provide greater financial safety, but also superior efficiency from coordinated economies of scale than any vertical stacking of patchwork competitor products.

3. End Products: this is the one that people are really missing. In Distribution we spoke about the importance of Google’s distribution machine in a world of commoditized products and services. But if we assume anything close to AGI, then Google shouldn’t collect a fee for making a referral to a third party service provider: AI enables Google to simply provide that service themselves.
   
   For example, if you Google for a hotel today, you encounter a rich cascade of middlemen and economic interactions: you are served an ad to a booking website, which in turn serves ads to numerous hotels, you look through them, you click through screens of primitive upsells for car rentals, trip insurance, etc.
   
   But nobody wants to browse all these websites and rifle through all the options.14 Google’s AI will obviate this experience. It already knows all your preferences from years of GMail and Search data: it will pick the optimal hotel, discuss with you the services you’re most likely to want, cut out all the middlemen and collect the fees directly. Talk about platform risk — many highly profitable products and services have been built on Google’s distribution platform, but as AI advances, Google will launch their own, superior offerings, and eat those markets overnight. People are already saying that ChatGPT competes with lawyers in the margins: reflect on what AGI really means, then take that to its logical conclusion.

As we get closer to AGI, trillions of dollars of revenue — every white-collar professional service,15 every software product — will be up for grabs, and Google owns distribution. If you’re a Google exec and you don’t have a vision for Google providing most of the world’s digital services by 2035, then you’re not being ambitious enough.

Importantly, I’m not suggesting that being first to AGI is all that matters. There’s a ton of value unlock along the way. For example, as coding models and computer use models keep improving, they will perform valuable labor at scale, presenting trillion-dollar revenue opportunities even before AGI.16 If Google has the AI capabilities to seize those opportunities, owns distribution, and owns the full vertical stack such that they have economies of scale and are totally independent of other players, then that seems like a clear winning position. AGI would only make the position stronger yet.

Part 4: The Challenge

Given all this great potential, why is Google not winning more? Why is the adoption of Gemini models so limited? How is ChatGPT taking market share from Google Search? Why are investors not more bullish? From my outsider’s perspective:

• Internal product ownership is far too fragmented. Different teams own poorly divided parts of the same product.

  • Former Googlers have told me that while they have all the resources, the organizational structure makes it far too hard to ship great products.

  • Therefore, large products have incoherent, low-quality user experiences;17

  • Internal rivalries lead to sub-par, team-over-company outcomes;

• Lawyer-driven-development: excessive fear and caution around launching products due to operating in so many geographies, and AI having unpredictable outputs;

• Managing for near-term shareholder outcomes creates a demand for caution not to jeopardize search;

What’s missing is courage. The past few years have been enormously rewarding to Mark Zuckerberg, Sam Altman, and Elon Musk — these Nietzschean characters with tremendous will to power, who will bet big and hard and take huge risks with asymmetric payoff no matter the scale. These are wartime CEOs, true live players who will reconfigure reality around themselves and who would not hesitate to fight their competitors to death in hand-to-hand combat if necessary.

First and foremost, Sundar must pick up the wartime mantle and act far more aggressively. It is time to truly compete, and the recent quasi-acquisition of Windsurf is a good first step. Beyond that, there are four key steps:

1. Google’s Gemini needs to be front-and-center. It needs to be on the google.com homepage, perhaps where Google Images is now.18 This needs to launch as soon as possible, no matter what the armies of internal worrywarts say.19 Google must push, push, push Gemini to supplant ChatGPT.20

2. Google must reorganize to enable itself to ship great AI software quickly. Too many disparate teams are pulled in to work in loose concert, so speed and overall quality suffers. One approach would be to take the best and brightest, fully mirror the OpenAI structure internally, and then aggressively keep growing that team, drawing talent from other divisions as it succeeds.
   
   This is hard! There is lots of corporate inertia against it. And I suspect it is particularly challenging for Sundar because he rose through Google’s complex political culture — and now he must smash and reorganize the structure that once enabled him to succeed. That’s hard on many levels. But it is necessary.

3. The mandate must be explicit and come from the top. The Google bears are fundamentally right that Search over a Big Corpus of Hyperlinks has an expiration date on it. Larry and Sergey were wise to retain super-voting shares — when push comes to shove, they can do whatever they want. They now need to exercise their legal and moral authority as founders to turn the ship.

4. Brace the shareholders. Google’s near-term financial results need to take a backseat to winning the AI race. This is a very simple priority as a matter of expected value. If Search traffic dips, if CapEx gets expensive, if operating profit temporarily shrinks — none of these things matter given the stakes of the game. (And losing the game is a much worse outcome.)

I’m long Google because I believe that these four steps are readily attainable21 and will unlock trillions of dollars in value. I don’t know how quickly Google will get there, but its momentum and resources are so great that they should, even if there’s some near-term bleeding. Google’s position is so favorable that they would have to mess it up immensely not to win.

Conclusion

If you believe that AI will be everywhere, then you should bet on the player that already is everywhere. Google is the winner-by-default in this arena,22 and the moment this starts to crystallize, the public markets will react. Surely there may be near-term volatility,23 and the task ahead of reconfiguring the company around AI requires bold and uncomfortable action, but the long term looks good:

1. Google has everything required to win;

2. They just have to not mess it up;

3. Winning unlocks trillions of dollars in revenue opportunity;

4. Winning provides a moat via a positive feedback loop.24

Right now at $2.1T, none of this is priced in.25 Many investors, when looking at particularly large assets, will implicitly believe that they’re fully valued. It’s hard to believe that a trillion-dollar-thing, with so many smart analysts looking at it, is actually undervalued — it doesn’t pattern-match how we think of a bargain, the small diamond in the rough. And it feels a little crazy to suggest it could be 10x or 20x larger.26 But I’ve seen this play out enough times in my life to know that it happens. We live in an era of returns to scale, and software continues to eat the world. Progress in AI remains rapid, and the economic consequences are, in some respects, simple and undeniable. The prize ahead is the most valuable one there has ever been, and the public has not yet fully internalized this. The race to AGI is the greatest competition of our lifetimes. Good luck to all!

In alphabetical order, thanks to Anon, Archie, Chris, Coby, John, and Paul for discussions over the year-and-a-half that led to this piece.

17

Let me vent: just look at this! On Google’s homepage, there’s no way to get to a ChatGPT-style interface. However, in the top-left corner, there’s a little icon for their Search Labs.

What happens if I click on it?

There are pre-seed companies that wouldn’t make this mistake! If you’re going to do user feature-flagging, then hide the button based on feature-flag status, rather than showing the button, letting the user click it, and then get to a disappointing “sorry!” screen. I’m sure that hundreds of thousands of people are hitting this every day. This is some of the most valuable screen real estate on the planet. It is unbelievable that it is wasted like this.

This week, the President of the United States has promised to make America the “Bitcoin superpower of the world” and established a Strategic Reserve that is never to be sold. I’ve been thinking a lot about this, not just because it’s amazing news for Bitcoiners, but because the world has turned out surprisingly differently from what Bitcoiners expected ten or fifteen years ago.

Setting the Scene

In the late 2000s, American institutional power crested an all-time high. The country rallied around the flag after 9/11, the government expanded significantly, and wars in Iraq and Afghanistan made clear that America would defend its interests, no matter what. The Subprime Mortgage Crisis of 2008, resolved by a $700B+ bailout for banks, once more drove the message home: true free-market economics be damned, American institutions must survive.

But the Subprime Mortgage Crisis showed millions of Americans that the institutions had, in some way, failed. And few people were satisfied by how it resolved: on the left, people thought the banking and mortgage industries hadn’t been punished enough, and on the right, people thought that the government had failed to uphold the free market. Occupy Wall Street on one hand, and End the Fed on the other.

Bitcoin as Political Instrument

Bitcoin launched into this uniquely charged political zeitgeist, right around the very peak of the financial crisis. Many people felt screwed by the system. The idea of using Bitcoin for savings and financial transactions — be your own bank! — sounded like a cool way of sticking it to the banks. But Bitcoin went much further: giving people trustless, decentralized money was a great way of sticking it to the central banks. Many early Bitcoiners saw a vision, far in the distance: if you can separate money from the state, then that takes away a core power of the state.

Satoshi himself was careful to be politically neutral, though his libertarian, cypherpunk1 leanings shone through. But practically, many early adopters had very strong views on (monetary) politics: Bitcointalk in those days had a lot of discussion about Ending the Fed, anarcho-capitalism, and generally pushing back on the state in favor of individual privacy and liberty. Just by being a powerful tool for political causes, Bitcoin was necessarily political.

Subscribe now

Surely Governments Will Try to Kill It?

In the long term, weakening the monetary autonomy of states is a tremendous threat to them. Back then, I wondered if Bitcoin could survive: in those days of strong American institutions, it seemed only like a matter of time until the statists in charge would nip this little experiment in the bud.

In May 2013, the US killed Liberty Reserve, an offshore centralized digital currency service — imagine PayPal without KYC/AML.2 The CEO got 20 years in prison for money laundering. At the same time, governments everywhere seemed to be slowly cracking down on cash (that they issued!) just because it’s not traceable.

Those were ominous times. After the big MtGox crash of early 2014, the price went sideways for three years. High-profile early Bitcoiners quit. Simple legal questions, like whether a Bitcoin wallet operator was a money transmitter, were unresolved. The US financial system is so rife with opaque regulations that many early Bitcoiners were worried that they were doing something that would one day be deemed illegal.

Even years later in 2017 and 2018, I remember speaking with people who were sincerely worried about one day being persecuted for their Bitcoin evangelism. They saw the totality of what large-scale Bitcoin adoption implied, and the challenge it would pose for governments and their fiat currencies. It was a common view that there was a Big Fight ahead, that one day the regulators would wake up from their slumber and fire every weapon they had.

Surprise!

The Big Fight never came. Regulators slowly clarified the important questions, and the law turned out to be favorable. Cryptocurrencies became mainstream via several hype cycles of ICOs, NFTs, and memecoins. Those hype cycles got a lot of people interested, and created a “pro crypto” political constituency. Most of these people had very little in common with the Bitcoiners of 2012, but that didn’t matter — they thought the coins were cool,3 and they had the right to vote.

As crypto became a voting bloc, the Big Fight became more and more unlikely: shutting it all down would really require bipartisan consensus, but that wasn’t going to happen — if one party rejected this constituency of voters, the other party would welcome them.4 The Biden administration rejected crypto voters, pressured the industry under Operation Chokepoint 2.0, but Trump took the crypto voters as allies, and today we have a Bitcoin Strategic Reserve.

American Institutions Have Changed

But the success of Bitcoin is not just about voting bloc dynamics. Over the last fifteen years, the formerly strong American institutions have also pulled back significantly, and created an opening of permissibility for Bitcoin to succeed. Things have changed:

• Covid: the Fed pulled out all the stops, printed 3 trillion dollars, and caused significant inflation. If they’re willing to do that, then how sacred is the dollar?

• Defense: the US has pulled out of its wars, and pursued isolationist foreign policy.

• Government: there is widespread dissatisfaction among Americans with their leaders and institutions. Donald Trump has been elected President twice on the articulation of that dissatisfaction, and the promise to shake up the system.

• Trust: the trust that Americans have in their institutions and authority figures is at an all-time low. People don’t trust their doctors or schools, let alone banks.

The last decade has been a time of relatively weaker institutions. Donald Trump’s promise in his first term was to drain the swamp and shake up Washington. Biden’s term didn’t change American institutions much one way or another, and Trump’s second term so far has been characterized by mass firings and dissolution of certain government agencies.

Today, there isn’t as much willpower to defend the institutions as there once was. If Bitcoin had gone mainstream in 2003, the Bush administration would’ve probably tried to nuke it just like Liberty Reserve. But Bush-era conservatives would be surprised by politics today in general. Imagine a cabinet official in 2004 openly talking about the dollar losing reserve currency status and hedging against it!5

Speculation De-Fanged Bitcoin

Early on, there was a common belief that Bitcoin would be viewed as a threat, and banned in many countries. It would be a slow grind for expansion over time, working its way up from overtaking the weakest financial systems. But in reality, almost no country has truly banned it.6 To the surprise of many early Bitcoiners, governments mostly didn’t care.

What had happened was that all the speculators came in, and changed the tone. They made it much easier for governments to get comfortable with it. Early Bitcoiners were talking about buying groceries with Bitcoin, using Bitcoin ATMs, etc., framing it as an immediate fiat replacement.7 But people buying memecoins, or even saving Bitcoin as “digital gold” aren’t threatening the dollar in the same way.

If the vibe of crypto had stayed like the early days — people talking about being their own bank and separating money and state — it might have faced more scrutiny.8 But instead, the crypto community was overrun by people saying things like “my dentist bought a Lambo after making 100x on Dentist Coin”. This apolitical appearance of harmless get-rich-quick-schemes is much easier for governments to ignore.9 Ironically, this made it easier for Bitcoin to succeed as a political project long-term.

The Fork in the Road

Looking back, probably the most important thing for Bitcoin was that Donald Trump won in 2016, and Hillary Clinton lost. Clinton was a serious statist who would’ve taken a hard-line pro-American-institution position on every issue. Donald Trump, by contrast, was tapping into an electorate that wanted to shake up the institutions.

While Trump made some disparaging comments about crypto during his 2016-2020 term, his administration mostly left crypto alone. By contrast, Clinton’s views on crypto remind me more of Elizabeth Warren’s views, who has been one of crypto’s fiercest critics, always pushing legislation to hobble the crypto markets.

2016 was probably the last time that it was really possible to kill Bitcoin. It ended the year at ~$800 a coin, a $15B market cap. It was small, hadn’t run through a public hype cycle, and didn’t have many investors who would stand up for it. A serious attack from Clinton-Warren-Gensler types, something on the order of Operation Chokepoint 2.0, could’ve killed Bitcoin adoption in the US, and maybe even pushed some international consensus among G8 Countries.10

Where Bitcoin Goes from Here

Bitcoin’s position finally seems secure. At long last, its market cap dominance has been steadily growing. There’s no longer the risk of getting “flippened” by higher-tech alternatives; the positive feedback loop of confidence in Bitcoin is winning. Regulatory uncertainties have been blown away, and Bitcoin is now positioned favorably to and distinct from all other cryptocurrencies.

It’s probably too big to fail now. It has institutional support at the highest levels, and will grow into an even stronger position by the end of the Trump presidency. Four years from now, it will be too deeply intertwined with the financial system and political interests to undo. The possibility of the eventual Big Fight now seems remote.

In some ways, it looks like an easy trade. Bitcoin is “expensive” today relative to the past, but it’s far more de-risked than it has ever been before. For once, the future is somewhat predictable. It seems like all the obstacles have been cleared, and it’s a straight shot for digital gold to overtake physical gold — that’s a 12x from here.11

Reflections

• When I was looking at Bitcoin early on, I was 19 or 20. I viewed the positions of American institutions as fixed, because they were all I had ever seen. It turned out these positions weren’t fixed at all. Over the course of ten years, things can actually really change. If I had been a little older and seen more change, maybe I would have understood this at the time. This is my biggest lesson.12

• Early Bitcoiners viewed the state as a bipartisan, self-interested monolith, that they would eventually conflict with. It turned out that the state is not monolithic, and that a controversial idea can succeed when it becomes a wedge issue, with different parts of the state on either side of it.13 Technologists stereotypically have little patience for the political process, but it turned out to work fine.

• Bitcoin never had to go underground. Bitcoin’s decentralization never really got tested in practice — most activity is still via exchanges. Perhaps just having a credible defense is good enough to never see it get stressed? People say “you can’t ban bitcoin, it’s decentralized” and so nobody really ever tries.

• Speculators not only brought massive amounts of capital to the ecosystem, but also took the political edge off its appearance, without compromising the long-term vision. Early Bitcoiners worried about many obstacles to adoption: is it not private enough? Will it be illegal? What if people don’t want to be their own banks? But almost everyone underestimated how easily Bitcoin could go mainstream: give people a way to get rich, and they will move mountains for you.

To a meaningful extent, Bitcoiners have won. And it all happened without a fight. The chance of getting to this point seemed so small thirteen years ago. But connecting the dots looking backwards, the simple market dynamics at work did not leave much up to chance. Perhaps the odds were always very good. The power and elegance of a system bootstrapping its own value just from provable scarcity seems to have been, and perhaps still is, undervalued and misunderstood.

• President Trump announced Project Stargate, a $500B investment in infrastructure for OpenAI.

• Dario Amodei, founder of Anthropic, claimed1 we are two to three years out from Artificial Superintelligence.

• OpenAI released Deep Research, which is capable of creating in-depth, graduate-level research reports — thousands of words — in minutes.

• OpenAI released Operator Mode, a computer use model that can use your browser to execute tasks just as a person would.

• DeepSeek released an open-source model that is roughly on par with OpenAI’s O1, and claimed (misleadingly[2][3]) that it cost only $6M to train.

Many well-respected AI industry leaders are now expressing fairly short timelines: on the order of artificial general intelligence (AGI) within 5 years.2 These aren’t outlier views, either. On the technologist prediction market Metaculus, 1484 people have cast their bets, and the median one is for May of 2030. The upper bound has been steadily coming down. “Later than 2038” is coming to be a fringe view. And of course, some think there’s a good chance we’ll get there next year.

Whether it’s five, ten, or fifteen years — take that seriously, and let it sink in for a moment. That’s soon. We might be just a few years out from sharing the planet with software that is infinitely replicable and can do the exact same mental tasks as you and I. The implications are immense. For a start, it’d be the biggest job turnover cycle in history. Even if the chance of this happening were small — not zero — that still seems worth thinking seriously about, in the same way that we prepare for other contingency events. But despite industry leaders being very clear about where things stand,3 there’s basically no public discussion about this. Not a peep from policymakers;4 the Overton window isn’t there yet. This seems like a real miss.

This is a long article, so I’ll put the take-aways explicitly up front:

1. Progress in AI is much faster than you think.

2. How to deal with AI-driven social and economic change is probably the only public policy question that matters right now.

   Subscribe now

Speed

The first thing to recognize in AI progress is that it has already happened much faster than people expected. I remember very well the 2015-era hype cycle for machine learning:5 breakthroughs using neural networks and generative techniques were finally returning rapid progress to a field that had stalled for years. But researchers still quoted AGI timelines for the year 2050 or 2100, with many shying away from saying that it’s possible at all.

At that time, the idea of AGI by 2030 would’ve been received as a little kooky. Now it is mainstream. Only famously-out-there-futurist Ray Kurzweil seems to have gotten it right: he bet that the Turing Test would be passed by 2030. There’s a prediction market on this: in early 2020, bettors were pricing this at 20%. Now it’s at 80%.6

During this time, we’ve kept coming up with AI tests and benchmarks, blasting through them — attaining parity with human experts — and then coming up with new ones. You know you’re making headway toward AGI when it’s getting harder and harder to come up with tests that distinguish human from AI performance.

There’s now the cutely-named Humanity’s Last Exam, a collection of 3,000 difficult questions across a hundred fields. While I think the exponential plot of its performance improvements is misleading,7 the progress is undeniable:

What’s going on here is that progress toward AI follows an exponential growth curve.8 Unfortunately, even highly intelligent, technical people have no intuition at all for exponential growth. Most people’s imaginations are stuck in linear growth — and so, time and time again, people underestimate how quickly growth can compound.9

Why Aren’t People Talking About It?

Though the headlines are everywhere, to most people they are still abstract. If they’re not using AI products, then they won’t develop intuition for those coming changes. Moreover, those changes may still feel far away because technology has historically been slow to percolate into the real world:10 plenty of things are still done manually even though software exists to do it. Even the internet took years to go mainstream, and many more years to start truly reconfiguring the world around itself.11

But these may not be the right analogies. Change in the physical world comes slowly, where logistical constraints exist. But change in the digital world — where we all now spend key parts of our lives — can come very, very, quickly.

People already underestimate the speed and scope of present change because “AI” has been such a buzzword for over two years now.12 People aren’t noticing how much better these technologies have gotten during that time, because the form factor of interacting with them (i.e. for most people, the ChatGPT window) hasn’t changed. It still feels the same to the consumer,13 and the cycle of consumers being amazed by new technology to them totally taking it for granted is very short.

Finally, perhaps the very idea of AGI is still too “out there” for people to discuss. In a similar way, in most of the world the idea of a self-driving car is still space-age science fiction. But in San Francisco, people are taking self-driving cars every single day. The technology which always seemed forever away is finally arriving.

It Gets Personal

All these changes seem abstract and far-away until they knock on your door. It’s really hard to appreciate or develop intuition for this progress until you see software doing the tasks that you take pride in: then, suddenly, it’s crystal-clear. In the next few years, most knowledge workers will have an uncanny realization when they’re looking at a screen and saying “wait, this thing can do what I do.”

Everyone has their Lee Sedol moment.14

I had mine recently.

Deep Research

I’ve spent a lot of time in my life doing research. To that end, OpenAI’s Deep Research is amazing. It’s basically a longer-running version of ChatGPT — instead of visiting one or two websites and returning a short answer in thirty seconds, it visits dozens of websites and returns an essay-length answer in ten minutes. Objectively, it does a great job. (Example research projects in the footnotes.15)

Deep Research hugely decreases the intellectual “activation energy” required to learn about something new: instead of having to muster the effort and spend many hours combing through resources to synthesize details, OpenAI can do it all for me. This is greatly empowering16 — but of course, at the same time, clearly the clock is now ticking for lots of highly paid knowledge work. If we’ve gone from hallucinating-all-the-time early ChatGPT to this in just two years, where will be in another two?17

Cursor AI

I build software. When I was nineteen years old, I decided to make that my primary intellectual and professional quest. Twelve years later, I’ve become quite good at it. I’ve made things that I am proud of, and in some cases have been used by millions of people. Yet my experiences using AI to write code have put the writing on the wall: I have probably already written the majority of code that I will write in my lifetime. The skill that I have spent many years cultivating will slowly but surely become antiquated.

I had previously tried writing software using ChatGPT or Claude as an assistant to generate and modify code for me. But using Cursor with Claude 3.5-Sonnet was a much better, more fluid experience. Like Deep Research, it dramatically decreased the “activation energy” for starting a new project, and it easily generated hundreds of lines of complex-but-shallow code.18 These are the parts of the work that would be most time-consuming to me, and it handled them with relative ease. This is even though Cursor is still early as a product: there are lots of bugs, it makes plenty of mistakes, and doesn’t even leverage standard programming tools like linting or typing by default. There’s a lot of room for it to get a lot better very quickly.

Funnily enough, many programmers don’t get this. In what seems to be a head-in-the-sand-defense, many are looking at Cursor as it is today and saying “well, that’s not as good as me!” unwilling to imagine just another three years of improvements. I’m hearing things like “well, you have to learn how to prompt it…” and “it can’t handle deep, complex problems” and “it can never understand a codebase as deeply as me…”

But this is obvious and willful cope from people who should know better: of course these are solvable problems. Each of these objections will fall as AI-assisted code editors integrate better tools and deploy more powerful, longer-context, models that leverage more test-time compute. This seems inevitable to me. Once more, even smart, technical people do not necessarily have the intuition for exponential growth.

Beware Dismissing the Questions

Until 2022, a common attitude was that AI could never do creative work — that creativity was the domain of humans; that it required some special element19 that a machine could not replicate. This was popular to say and comforting to think, but now we know it also wasn’t true.20

But maybe we always knew it wasn’t true: if you had really pressed this question ten or twenty years ago, you wouldn’t have gotten a good answer. People were dismissing these AI questions with lazy, hand-waving answers. The reality they were trying to avoid was very simple: if a human activity really boils down to thinking and AI can think, then that activity can be done by AI. Understandably, people want to believe that what they do is special and not possible to automate. They won’t believe otherwise until they see a machine do it. We will see this play out many times over the coming decades: people wishfully asserting that something is unique to humans, and then eventually finding that may not be so.

As we think seriously about the impacts of AI, there are plenty of these dismissive answers that are worth pressing. For example, people say not to be concerned about AI taking over jobs, because we’ve historically always created more, higher-leverage jobs as we’ve innovated old ones away. But there’s just no reason for that to hold ad infinitum. Why couldn’t AI perform those higher-leverage jobs, too?

What Should We Be Thinking About?

In the long run, AGI will transform our world, just like the industrial revolution did long ago. I think this can go very well for us, and the right attitude is cautious optimism. But no transformation of this size is without friction. While it’s not worthwhile to try to predict second- or third-order consequences (there are just too many variables at play), there are a few things that are straightforwardly predictable:

1. Many jobs will cease to exist, and be completed by AI instead.

   1. Even if some of these jobs are replaced by new jobs, the cycle of job replacement still means transient unemployment en masse.

2. People will generally have more and more free time.21

3. On the internet, AI activity will vastly outnumber human activity.

   1. If we believe that the distinction between human and machine matters, then working on proof-of-humanity seems important.

I’m not an AI Safety guy. I think AI will be fine. But I think that there is some danger of conflict between humans if we mess up the transition. For example, it’s easy to imagine a case where a rapid loss of jobs turns into social unrest, which spirals into something really ugly. It wouldn’t be the first time that people have felt threatened by automation, or that economic and social volatility is exploited by demagogues.

What Should Policymakers Think About?

First, please note that public policy moves on slow timelines. Change takes years to enact. If we’re taking seriously the idea of AGI by 2030, to a policymaker that’s basically tomorrow. We need to get started. To me, there are several tasks ahead:

1. The impacts of AI are mostly absent from contemporary policy discussions. This needs to change as quickly as possible.

   1. For a start, most policy issues will be affected in some way by AI. Simply asking “how will AI affect this topic?” will be a good way to start introducing the discussion at all levels, while also preparing for upcoming changes. If you’re debating policy X, it’s always worth reframing: what does X look like in a world where we have intelligence too cheap to meter?

2. Secondly, we need to begin discussing the impacts of AGI by themselves, not just as a framing for contemporary issues.

   1. We need to solve for the economic challenges of the transition to AGI. This mostly means preparing for rapid turnover within the labor market, and (in my opinion) for a rapidly shrinking labor force participation rate.

   2. We need to solve for the social challenges of the the transition to AGI. On the one hand, there will be some adversarial behavior to remedy. (Think AI-enabled spam and scams.) On the other hand, people will need to feel security, significance, and connection in a world where their labor or intelligence is suddenly no longer so special.

3. We need to internalize that these questions are way more important than almost any other contemporary public policy debate. This is where we need to spend our time and effort. Topics like immigration, climate change, income taxes, or whatever else, do matter, but not nearly as much as the big change coming up. And all these topics will look different in a world with AGI, anyway.

4. Finally, we need to think deeply and carefully about what’s going to happen. Given the difficulty of predicting the future: if in doubt, under-regulate rather than over-regulate.

I don’t have all the answers. My purpose with this essay is to get people thinking about these questions and appreciating their urgency. While there are some clear first-order action items, like needing to provide additional dignified ways for people to off-ramp from economic production,22 I remain suspect of easy answers that dismiss the questions: you can’t just say “UBI” and call it a day. You might be right in the long run, but it would gloss over the very real near-term challenges of getting there.

These are big topics — almost overwhelming in scope. Getting started won’t be easy. And stated in the abstract, they might not feel so urgent. But we live in a world that is built around the unique value of human intelligence and labor everywhere you look: from the thirty-year mortgage to people’s identities being wrapped up in their vocations, there are meaningful changes incoming for fundamental aspects of our society. They are manageable changes, of course, but they deserve careful preparation. I can’t think of anything more important than getting the AI transition right.

Appendix: Clearly We Have Not Hit The Wall

A few months ago, there was a moment of doubt about whether we had hit “the wall” in improving AI models: some people speculated that the scaling laws might not continue to hold, and that we might be in for an era of diminishing returns.

Right now, this doubt seems to be refuted. Progress is as fast as ever, though big gains seem to be coming from areas outside pure scaling. It’s worth revisiting Leopold Aschenbrenner’s Situational Awareness: while most of his text focuses on scaling laws, he points out that several orders of magnitude could be gained just from “unhobbling” the models by algorithmic improvements, and moving more resources to test-time compute. He cautiously suggests that it might be possible to reach AGI just via these two. My hunch is that this is about right. Andrej Karpathy once said that we might observe intelligence in a <1B parameter model, and while it’s still early, practical discoveries are pointing this way, too.

Some people are (maybe wishfully) assuming that the big burst of progress is done and will level off from here. I don’t think that’s true. At least right now, it looks like the exponential growth is uninterrupted, and there is no reason to expect it to slow down.

You could read that as a bullish thesis for such an AI insurance product: imagine a world of widespread AI adoption, where every AI deployment is underpinned by an insurance policy. Or imagine a world where insurance products act as the critical enabler for widespread AI adoption in the first place.

But the thesis is not that easy! While I won’t present a slam-dunk-view either way, I want to discuss some of the nuance and complexities that make this market tricky, and probably smaller than it appears at first glance.

Insurance for (Software) Errors

In the history of business, AI isn’t the first thing to make mistakes. Humans have been making mistakes for a long time. For that reason, accountants, lawyers, real estate agents, etc. all carry insurance — specifically, an Errors & Omissions or Professional Liability policy that covers them if they make a costly mistake on the job and get sued by a client.

In recent decades, a significant amount of rote human labor has transitioned to being completed by software instead. This software transition was subject to the same concerns as the current AI transition: can you really trust accounting software not to make mistakes? Won’t there be edge-cases in mortgage underwriting that software might miss, but an experienced underwriter would catch? The proof is in the pudding: the world runs on software now. And similar to Professional Liability, many software companies carry Technology Errors & Omissions insurance, in case their software messes something up and their customer goes after them.

You would think that the market for such insurance is massive. Software handles every button-press in your car, it manages industrial control systems in factories, it monitors the life-or-death status of patients in hospitals. The stakes are high. And we know most software is broken in the margins: every day I visit websites of big, respected companies, and they’re full of bugs.

But most software companies haven’t even heard of Tech E&O insurance. It’s considered a specialty product, often included as an add-on to cybersecurity insurance. Because it’s so niche, it’s hard to estimate the market size, but that’s an indicator of just how small it is: accounting for under $5B in global annual premiums seems like a very safe bet to me.1 For comparison, in the US, Workers’ Compensation runs around $55-60B a year in premiums, and Personal Auto insurance over $300B.

This should give you pause. The handing-over of professional duties to software feels riddled with liability, even today. The thesis for Tech E&O would be very similar to the thesis for the AI insurance product we started out with. (Let’s call it AI E&O.) And yet the market for Tech E&O is small, even in the face of software carrying weighty responsibilities in every nook and cranny of our world.

AI E&O and Tech E&O

Taking this one step further: you could consider AI E&O as a new form of Tech E&O, or — depending on the details of the contract — as included by Tech E&O policies. After all, AI software is still software. It may not be quite as deterministic as software before LLMs, but you’re still trying to insure the same type of risk: software mistakes.

Then, in what sense does AI E&O expand the Tech E&O market? Before LLMs, software could make devastatingly expensive mistakes. After LLMs, software can still make devastatingly expensive mistakes. The LLM aspect may increase the potential frequency and severity of those mistakes, but you have a needle to thread: if frequency of severe mistakes increases too much, then insurance becomes moot. People are not going to use a software product that breaks all the time, regardless of whether any damages are covered or not. It’d just be a nuisance.

This puts insurance entrepreneurs in a structurally tricky position. The Tech E&O market is so small that for a venture-scale thesis, you’d need to forecast AI E&O increasing the size of the Tech E&O market several-fold, probably 10-20x+. To get there, you’d have to:

1. Overcome any structural market issues2 that may inhibit growth;

2. Bet on severity of claims shooting up, much more so than frequency. AI-enabled software would have to become tremendously more dangerous to deploy, with multi-million-dollar-loss glitches lurking. The risk scenarios you’d be insuring would be cases like “I’m Chevrolet, and my marketing AI promised new trucks to 163 customers”3 or “I fired all my accountants, replaced them with ChatGPT, and when I woke up this morning I owed a customer a million dollars.”

Maybe I’m being unimaginative, but the maneuvering room to get to widespread AI E&O adoption seems tight. I think the likelier path is that businesses will adopt AI while maintaining some risk-reward equilibrium: steering clear of the use cases with the most severe downside risks, and leaving humans in the loop where appropriate. You may well be right to argue that there is still more risk in the system than before, but I don’t know if there’s so much risk that it gives rise to a major new class of insurance product and satisfies a venture-scale thesis.

Information Asymmetry

An important detail of insurance markets is that insurance carriers must be better at evaluating the risk than the purchasers. Otherwise you get adverse selection problems: consumers who know they are more likely to incur claims purchase insurance, the insurance carriers take losses, and the market eventually collapses.4

This takes you to a practical concern: how would AI E&O products be underwritten? There would be two parts to it:

1. The insurer would evaluate the characteristics of the AI company — industry, size, safety and testing practices, etc., and look at their service agreements with customers to figure out what kind of risk they’re on the hook for.

2. The insurer would run a large battery of tests against the AI offering of the company, seeing how it holds up under a variety of adversarial scenarios, and what the variability of outputs is.

The first part is a classic point of strength for insurers: given a large portfolio of businesses underwritten, they can figure out how these factors affect pricing. But I expect that for an AI E&O insurance product, it’s really the second part that determines the risk. Here’s the problem: why would an insurer be better at testing a company’s AI outputs than the company itself?

Revisiting our earlier example, the folks at Chevrolet would have a much better understanding of their own business, all the ways in which they could deploy AI, and the most dangerous, error-prone areas, than any insurer looking in from the outside. Specifically, there are two related problems:

• As an outsider, it’s extremely hard to get a full understanding of all the ways in which AI will be deployed, and what risks that implies downstream. Hard to price!

• There is a massive information asymmetry between companies utilizing/selling AI software, and insurers seeking to insure the consequent risks. Trying to insure AI applications looks like a hotbed of adverse selection.

Concentration of Risk

Another classic detail of insurance markets is that insurers need to diversify the risks that they underwrite: for example, if you provide flood insurance, then you wouldn’t want to write all your policies in a single town by the river: when one house gets flooded by a storm, chances are that all the houses get flooded, and you go out of business. That’s concentration of risk, and insurers strive to avoid it.

The trouble is that the ecosystem of AI software products currently has enormous concentration of risk. There’s a single-digit number of major LLM providers. AI infrastructure, whether for RAG or data labeling, etc. has a similar concentration of activity, with many small providers and a few major ones. Practically speaking, if you’re insuring mostly GPT wrappers, and the newest GPT model has some kind of safety regression, then your entire portfolio of policies is in trouble.

For any insurer, it will be tricky to maintain adequate diversification of the underlying risks. In practice, this means your portfolio might simply be constrained to a small size, as you can never grow such that you’d be over-exposed to any particular underlying provider.

Underwriting for the Year Ahead

The final challenge is that insurance policies are usually written for the full year ahead, and AI software is evolving with great speed. In our own AI deployments at Limit, we found surprising differences in behavior and quality from different models. It’s hard to trust software updates from outside vendors to be strict improvements.

Further, at the speed at which businesses are iterating on their AI software, or deploying it in new contexts, makes the underwriting problem even harder. It’s tough enough to test the AI software at any one point in time. There’s no good way to make assumptions about how else it will get used in the next few months, or how well-tested the next software release will be. The remedy for an insurance underwriter will be to prescribe what kinds of updates are in scope for the policy, what level of testing must be done, etc. This helps limit the risk, but it also greatly increases the complexity of the insurance contract, and makes it more cumbersome to purchase.

What You Need for AI E&O

My skepticism above doesn’t mean there’s no case for AI E&O. There certainly is. But it’s tricky. You’d have to bring the following conditions together:

1. There must be rare, hard-to-mitigate, severe risks from AI deployment;

2. The purchasers of such insurance are the actors in the market (such as software providers and consumers) that are stuck with the risk, i.e. not able to contractually transfer it to other parties;

3. The insurers would need to be better than the policyholders at figuring out the riskiness of the AI deployment.

   • Could AI E&O insurers partner with AI testing/safety/QA service providers, similar to how cyber insurers partner with cybersecurity providers? Yes, but those services are already readily accessible to potential insurance customers on the open market!5 The information asymmetry remains.

   • An insurer wouldn’t need to know how to underwrite every such company, but could constrain their appetite to certain types of businesses where they feel they can exhaustively understand the AI risks;

4. Diversification of underlying risks (technology vendors) would have to be maintained, which practically implies limiting the portfolio size of the insurer;

5. The insurance policies would need to prescribe guardrails around software updates.

It is certainly possible to bring all these conditions together — it’s just not easy, and even when you do, it implies a very selective, small portfolio of underwritten risks. I suspect that at least for the next few years, the set of such opportunities will be pretty thin, making it a way but not the best way to attack the AI liability problem. Furthermore, you would need this risk environment to scale up dramatically to give rise to a venture-scale insurance thesis. For now, if you’re really good at evaluating AI model safety, that’s probably better sold as a standalone service than used to underpin an insurance product.

This piece was inspired by conversations over the past weeks with Rune, Bala, Zack, Alex, and others. Thanks for your thoughts!

A good pitch deck makes it easy for the reader to write an investment memo.

Classic advice on writing is to remember your audience, to empathize with your reader, and make your writing accessible to them. In this way, the advice to founders writing pitch decks is to simplify, simplify, simplify — assume the investor doesn’t have much time to read the deck, focus on the key points, etc. This is all true, but it misses what’s next in the process. Keep in mind that the investor wants to learn and understand your company, but they also have work to do: a process to follow, and documents to write. You want to make it easy for them to do their work.

The investor reads your deck, they may take a call with you, and then they may share your deck and their notes with other people on their team. Early on in the process, the notes might be informal, like some bullet points in an email. Later on in the process, they will draft an investment memo to recommend to the rest of their team — and to memorialize for the investors in the fund — why to invest in your company. At that point, you will have supplied many other materials that they may use to write the memo, but the deck is the anchor that people will most frequently refer back to.

When you pitch, you are taking on a process that should result in an investment memo on the other side. This means that two things are very important:

1. You should know what an investment memo looks like.

2. You should make it as easy as possible for the investor to draft their memo.

Subscribe now

What Does an Investment Memo Look Like?

Depending on the firm, sector, and stage, investment memos have different formats and styles. Thankfully, there are enough public examples1 that you can get a feel for them. Some examples below:

• 1Confirmation’s memo for Bridge.xyz (2022)

• Sequoia’s memo for Youtube (2005)

• Canaan’s memo for On24 (1999)

• Lightspeed’s memo for Snapchat (2012)

• Redpoint’s memo for Ramp (2021)

• Greylock’s memo for Roblox (2018)

• Bessemer has published 17 memos for investments between 2005 and 2015.

Additionally, some firms write about their processes and how they draft their memos. NextView has a fine blog post on this, for example. As you dig into these examples, you’ll notice a few things:

• Venture firms tend to examine businesses from a different perspective than how a founder might present them — paying a lot of attention to areas that a founder might not, and vice-versa.

• Writing a memo is not easy!

It’s a good exercise to try yourself. Write a memo for a company that you’re familiar with. Write a memo for your own company. (One of the underrated benefits of investing in startups is that it gives you more of an investor’s perspective when writing the pitch for your own company.) Having some empathy for and familiarity with the work that happens on the other side will make your presentation of your company much better.

Make it Easy for Them to Write their Memo

I usually recommend the following process. First, put together your pitch deck organically — there are many good templates and structures you can adopt — and present your business however you feel is right. Then examine your deck from the investment memo perspective:

• For every slide: are you presenting information in the way that you’d expect it to be reflected in the memo? What would the memo version of the same content say? If there’s a big difference, consider presenting the information another way.

• For every piece of information: does it get in the way of the reader as they’re doing their work? Does it distract? If it’s certainly not going to make it into the memo — maybe it’s a detail, but still important — consider moving it to an appendix or a supplementary document.

• Are you missing any information that the investor would normally mention in a memo to their colleagues? Include it.

Applying this framing should help take you out of the bubble of your own perspective, and clarify your work. In practical terms, this normally simplifies your pitch, moves content from abstract to concrete, and helps reduce downstream games of telephone as investors try to interpret your work.

Tactics and Pitfalls

Expectations Cascade
Sometimes it’s temping for a founder to skip a slide that an investor might expect, like on TAM, revenue projections, or competition. The founder may have a rational reason for doing so, and write it off as an unnecessary hoop to jump. But the investor knows that their colleagues will want to know those things as they try to understand the business.2 If you don’t provide the information, the investor will have to try to put it together on their own. Usually, you’re better off presenting that information up-front and guiding their research.

Answer the Simple Questions
Try to anticipate the questions that you’re going to get in a slightly adversarial reading of your deck. For example, if you only have a few months of revenue progress and some LOIs, reading that can be confusing for outsiders. I have often had to ask “okay, who are your top five customers, how much are you they paying you this year, and what do you think they will pay you next year?” as I try to wrap my head around what’s going on inside a company. A simple table in the appendix can do wonders.

From another angle: imagine an investor casually bringing up your pitch to their colleagues internally for a gut-check, and answering their basic questions like “what does the company do?” and “why is this interesting?” Your deck should equip the investor with all the key talking points and make it super easy for them to advocate on your behalf, before having spent many hours on a deep-dive.

Use a Supplementary Memo
One of the main challenges in writing a deck is figuring out what’s superfluous and can be struck. Everything feels important! To sidestep this, some founders like to keep their pitch decks brief, and write a comprehensive document as a supplementary memo. Personally, I like that format. It lets you keep the deck as an overview, and gives you a place to put all the details. That document should provide to the investor everything they need to understand your business, and to write their investment memo. (Of course, they will consider many other materials as they do their diligence, but the materials that you provide will always function as the entrypoint.)

Provide Good Resources
As investors dig in on your business, they will want to consult lots of external resources. They will start from your deck (and/or supplementary memo): they’ll look up and double-check the industry statistics you cite, research the competitors you mention, etc. You can make it easier for them by using footnotes in your deck to link to resources they can read. Ideally, a resource that you link to doesn’t just back you up on one particular fact, but generally gives the investor high-quality information that feeds well into the case for your business.

Testimonials
Founders sometimes play coy with the identities of their customers. They might write things like “Amazing product, I’d pay lots of money for this” - Fortune 500 COO in their deck and then voice-over the identity of the person in a live pitch, or provide logos of big customers, but not name who specifically at the firm bought the product. There’s nothing quite as useful for an investor as a discussion with your customers — make it easy! If you’re confident that your customers will provide good testimonials, and they’re willing to act as references, then always provide their names.3

Competitors and Comparables
Founders often focus on competition in their slide decks, trying to point out why what they’re doing is distinct and special. But competition isn’t important in a deck just as a threat to the business, but also as a reference point: for example, your legacy competitors in the public markets are useful to look at, because they show how such businesses are valued at scale, how their market is evolving, whether they might be good acquirers of your company one day, etc. Understanding how to value your business can be nontrivial, and clear enumeration of the comps will, just like everything else in this essay, save the investor time as they dig in on your business.

Conclusion

If you want to work productively with other people, then a good principle is to ask how can I unblock them? and how can I make it easy for them to do their work? Raising capital is no exception. Appreciate that people on the other side have a process they must run, think about the boxes they have to check, and make that easy for them. By contrast, going with unorthodox presentation/data formats, insisting on doing the process your way, or pitching your company such that it raises many questions that are hard to answer, are good ways to get shuffled to the bottom of the priority list. The best thing that you can do for busy people is to help them be efficient with their time.

If you’re thinking about putting together a deck for something: good luck! I hope these notes are helpful. If you ever want a second pair of eyes, I’m happy to help. You can reach me at contact@johnloeber.com.

Thanks to Mike and Evan for their feedback and comments on this piece.

But these companies are devouring capital with no profitability in sight, and are caught in a game of margin-eroding lockstep competition.1 Most importantly, the limited revenue being created downstream of these businesses calls into question whether these businesses may grow into their valuations: David Cahn called this AI’s $200B, and later, $600B Question.

Taking this logic a step further, some people are calling the current level of investment in AI a “bubble”. I am now old enough to have heard many things referred to as “bubbles” in my lifetime, and have generally found this framing to be misleading. As with many cynical views, talk of bubbles is a good way to sound clever but miss the value. In this essay, I will:

1. Debunk some major historical bubbles;

2. Make the case that the only true, permanently-collapsing bubbles are frauds;

3. Suggest how things will play out in AI: we are still very early.

4. By analyzing Dot-Com capital investment trends as an analogy, make the case that we should expect much more capital to flow into AI.

5. Discuss how and why we should expect investment activity in AI to shift away from traditional venture capital and into the public markets.

2000: The Dot-Com Bubble

From about 1995 to 2000, investors were hugely excited about telecom, computing, and internet businesses, with early-stage startups routinely going public and getting enormous valuation multiples. It was an incredible time to be a technologist — until it all collapsed, and countless paper millionaires were wiped out entirely.

But how irrational was all this exuberance? The largest companies of today — Microsoft, Google, Amazon, Apple — were all significant players in the Dot-Com era. Companies like Pets.com and Webvan did go out of business, but were reincarnated a decade later as firms like Rover and Instacart, which ended up being successful in their own rights. Marc Andreessen and Bret Taylor make good cases2 that there was no categorical tech bubble, but rather a market distortion due to the Worldcom and Enron Frauds, which caused collateral damage as they collapsed.3

The dot-com bubble was the little blip on the left of this chart. Even if you had bought the top on March 30, 2000 and sat through a catastrophic drawdown — over the next two years you’d face 80% losses in most of your positions — you would’ve done extremely well in the end. Between the “top” in March 2000 and today, your Amazon position would return 203x, your Apple position 66x, etc.4 Even if your portfolio was mostly losers that went to zero, you would’ve covered your losses and then profited many, many times over.

All you had to do was nothing. But to dispassionately wait it out is no easy feat. You had to be a technology investor with a well-diversified portfolio and true long-term conviction. When most people take an 80% loss and all the pundits are sneering at them like “obviously a 200 P/E ratio was unsustainable”, their natural reaction is to sell and wash their hands of it, not to hold for another 20 years. But holding would’ve been right, in retrospect in a very predictable way: of course technology is the future, and some of the giants of the future are born today.

2008: Housing Bubble

In the lead-up to the 2008 financial crisis, prices of housing in the US exploded for a variety of reasons, one of which was the easy access to (subprime) credit. Contemporary commentators spoke of a “housing bubble”, as they still do today.

In March 2020, Jesse Colombo pulled together some statistics to argue both that 2000-2007 constituted a fiscally-driven housing bubble, and that the US was due for a 2020 correction. His chart below:

The timing of this argument could not possibly have been worse! Within a few weeks, the Fed would kick off the mother of all growth cycles, and housing prices would boom even faster than before. When you zoom out, talk of a “bubble” becomes unpersuasive:

To me, this looks pretty steady. The ‘08 recession barely shows up — a short-lived drawdown in an exponential trend. All you had to do was wait through it. In retrospect, this trend seems very predictable: without profound regulatory changes that you’d be able to see coming from years away, the price of housing in the US will continue to grow just as it has for decades.

2013, 2017, 2021: Crypto

In late 2013 and early 2014, a flurry of trading activity on MtGox — the main cryptocurrency exchange of the time — briefly took the price of Bitcoin past $1,100. At the time, it seemed extraordinary. $1000 per coin? A ten billion dollar market cap? Unbelievable amounts of money. Months later, MtGox collapsed, and so did the price of Bitcoin with it. Bitcoiners soon found out that the price had been fraudulently pumped on MtGox by a trading bot that researchers named Willy.5 The price of Bitcoin fell by over 80% at its lowest, and took three-and-a-half years to reach back to $1,100. Countless early Bitcoiners lost faith during this time and thought it was over. Today, nobody remembers any of this.

On January 13, 2018, the New York Times published a memorable article: Everyone’s Getting Hilariously Rich, and You’re Not. It made fun of the wild exuberance in cryptocurrencies, with Bitcoin trading as high as $19,300 and Ethereum at $1,100. It was the first time that crypto had really hit the mainstream. People at the time thought this was a crazy bubble; I remember taxi drivers telling me what ICOs to buy.6 The market collapsed shortly thereafter, and Bitcoin would once more come down to around $3,000 in the intervening years.

Then, in 2021, following the massive COVID-era financial stimulus, crypto had another incredible run, lifting Bitcoin well past $60,000, and countless smaller cryptocurrencies along with it. This cycle too wound up with truly spectacular financial collapses, with billions of dollars incinerated at shops like Terra/Luna, Celsius, FTX, Three Arrows Capital, etc. Bitcoin would collapse down to $15,000 in the aftermath.

It is now October 2024, and Bitcoin is once more trading at $60,000, with not a peep in the popular press. The 2013 and 2017 episodes no longer look like bubbles, but mere blips on a chart. Even 2021 no longer looks like a bubble, insofar as the $60,000 price ballpark appears to be well-supported today. Similar as to the Dot-Com bubble, many positions will pan out as losses, but long-term investors with carefully diversified7 portfolios have done well.

1630: Tulips

Both crypto and the Dot-Com era have been compared to the Dutch Tulip Mania of the 1630s, since that’s the one crazy bubble story that everyone knows. There are only two problems with it:

1. That story is 400 years old;

2. It’s mostly made up.8

I’ve always been surprised that the Tulip story has had so much staying power: you’d think that a hundreds-of-years-old, niche-market story would not be highly valued as an analogy for reasoning about massive modern markets, even before historians figured out it was mostly fiction. But it’s been a very sticky — and damaging — meme. Mass enthusiasm for any kind of new technological paradigm is often metered by skepticism, and invoking Tulip Mania is always an easy dismissive reach.

The Role of Fraud

The most disastrous of all historical market bubbles was the South Sea bubble of the 1710s. But it was not just driven by speculation, but rather by massive fraud: prominent politicians were paid to pump the stock, and financial games like large-scale debt-for-equity swaps created an illusion of value.

Indeed, in virtually all other famous bubbles, fraud played a meaningful part: Enron and Worldcom set the pace for the Dot-Com bubble. FTX and others misappropriated funds to pump the prices of cryptoassets in 2021, creating fraudulent demand. The financial crisis of 2008 followed widespread mortgage and ratings fraud.

This creates some interesting framing. Frauds can create bubbles of their own, and draw a lot of speculative excess,9 but eventually collapse permanently. And it is not uncommon for big waves of technological/financial opportunity to get mixed up with some level of fraud; hot markets tends to attract grifters. However, after these frauds collapse — temporarily dragging down the market with them — the fundamental value remains. Investors in technology post-Enron, crypto post-FTX, US real estate post-2008, all have done very well.

Takeaways

Once you discount the singular story of Tulip Mania, and try to account for the impact of fraud, you are left with market histories that don’t look so crazy. It’s hard to find examples of irrational, speculative excess at scale. The popular narrative of a market bubble as a madness of the crowd, with people deluding themselves into losing their last shirt betting on magic beans — is, like a lot of other popular psychology, much more of an attractive story than grounded in fact. I think this hurts the people who believe this framing, as it excludes them from participating in episodes of great long-term wealth creation. A “bubble” is rarely a bubble.

Of course, the market can be short-term irrational, and there are fads that blow up and then collapse permanently, like the Beanie Babies craze. Those things happen all the time.

But the difference is in scale. Beanie Babies were a retail craze, not a multibillion dollar market. When there are billions of dollars of (sophisticated) capital moving into a space, there has usually been something to it. If you invest in groundbreaking technological innovation, avoid frauds, and you’re patient to hold for the long term, that tends to do well. Disruptive value creation tends to come with some volatility, and the paper-handed get shaken out over a few boom-bust cycles. The returns are captured by those who hold on and double down.

Looking back at the Dot-Com crash, or even the Railway or Automobile investment manias10 of the late 1800s and early 1900s: temporary market crashes tend to flush out many of the competing firms, but some of them survive and go on to capture the market’s long-term value. The tricky question for investors, and the reason to be well-diversified, is that it is very hard to predict ahead of any crash which firms will be the long-term winners.11

AI

By our framing of prior bubbles, the most important question is whether there is fraud. I don’t think there is. There’s no FTX injecting fake liquidity. There’s no Enron luring people with blockbuster returns. In fact, reported revenues are still quite low.

In that sense, I don’t think there’s a “bubble” in AI. In fact, I think we are still extremely early. Right now, there are still many questions about deployment, monetization, margins, commoditization, and where long-term value accrues. As those questions get answered and revenues from other parts of the economy start substantially transitioning into AI, mainstreet investors will begin to internalize just how much change is coming down the pike. Then you’ll see a much larger scale of capital inflow.

AI in the Public Markets

When I mention a much larger scale of capital inflow, what do I mean? Isn’t $6.6B already the largest venture round of all time? Wasn’t venture investment even at the peak of the Dot-Com era just $35B a year?12

People forget that private markets are much smaller than public markets, and equity markets are much smaller than debt markets. So far, all investment activity has been in private, equity markets.

During the Dot-Com era, telecom companies were spending $135B a year — $250B in 2024 dollars — on infrastructure. In aggregate, they spent around $700B that decade. These were incredible levels of raw capital expenditure on the belief that there were network effects to owning internet infrastructure. To support this extraordinary level of spend, they raised over $2T in the public markets,13 at least $600B ($1.1T in 2024) of which was in bonds (debt).

That debt came to be the big problem. Virtually all of these telecom firms were massively overlevered.14 You know how that story ended.

But the Dot-Com telecom story makes it clear that there’s a much larger scale of capital available. Just as for the advent of the internet, AI offers a globally transformational opportunity. There are some lessons to be heeded with respect to debt and leverage, but modern technology giants are much smarter about this.15 When Sam Altman said he wanted to raise 7 Trillion Dollars, the telecom era puts it into historical perspective: do I believe that the opportunity in AI is seven times larger than the opportunity in the internet? I’ll leave that question to you.

In the path of this opportunity, the capital demands are unending. Larry Ellison thinks that competing in LLMs will soon cost over $100B to enter, and he’s right. This means that some of these private firms may need to soon transition into the public markets to fundraise (and maybe even into the debt markets for those confident in their revenues and margins), just as a way of accessing sufficient capital at scale. In tech, we haven’t seen that kind of dynamic in a long time.

I think it would be interesting and good if that were to happen. Investment capital into AI would scale up significantly, but there’s also a pro-social effect. Bear in mind that the bulk of innovation is currently in hard-to-access privately held companies. There are public companies like Google, Microsoft, and Nvidia for folks to bet on, but the vast majority of investors will not have any exposure to firms like OpenAI or Anthropic for the foreseeable future. These technologies will meaningfully reshape the economy, and the perception of all the economic gains being captured by a few private companies would draw significant political heat. This transitional phase will go much smoother if mainstreet investors can have, for example, at least a little bit of exposure through their 401Ks. Regulators would be smart to help make the public markets more accessible to “democratize” this big shift.

Conclusion

It’s going to take some time for AI to percolate and revenues to transition from other parts of the economy. Between then and now, I’m sure some investors will lose patience, many firms will be outcompeted, and unexpected innovations will hit many times over. The market will have some volatility.

But overall, it is clear that, just as in the Dot-Com cycle, we are witnessing the emergence of technologies that will become the dominant forces in our economies in the coming decades. Long-term, the biggest players in AI will be the biggest companies on the planet. If I’m making sector-level bets today, I’m not going to be worried by any 2025 drawdown, for example. As usual, it’s hard to predict which firms specifically will come out on top a decade from now after some boom-bust cycles, and where in the stack they will play, but I think it’s safe to expect that well-diversified investors16 in AI will do very well long-term.

1. Arithmetic problems are simple examples of reasoning tasks in general.

2. Arithmetic problems can be solved by simple algorithms, rules that must be consistently applied. You can arbitrarily increase the difficulty of these arithmetic problems, i.e. the number of times that a rule must be applied.

3. This means that arithmetic provides good windows into both single-step and multi-step (i.e. chain-of-thought) reasoning tasks. We can evaluate both individual calls to LLMs, as well as compositions of such calls.

4. State-of-the-art LLMs still struggle to do simple arithmetic problems, even while LLMs have scaled up dramatically in size and standard evaluation benchmarks.

5. Solutions are easy to evaluate.

In this article, I try to summarize everything that we know about LLMs doing arithmetic. I’ll point you to all the interesting papers that I’m aware of, and draw some observations of my own.

Subscribe now

1. The Problem

The first thing you’ll notice is that LLMs do great on short arithmetic problems, but struggle with long ones. For example, if you ask an LLM to add three two-digit numbers, it’ll do fine, but it will fail for large numbers or long sequences (e.g. adding many small numbers). I shared some experiments on this topic in a prior blog post.

These results tend to surprise people: LLMs seem smart and quite capable of abstract reasoning over complex texts. Why do they struggle with basic arithmetic? For example, in my experiments, I found that failures for large-number additions almost always occurred in the same way, with one particular digit being misplaced, usually in the thousands-place of the number. Why?

2. Issues with Tokenization and Position

You have to keep in mind that LLMs do not see text the same way you do. LLMs use two key abstractions for working with text:

• Tokenization: input words (or numbers) are converted into tokens that the LLM processes. For example, the number “483,647” could be converted:

  • Into seven tokens [“4”, “8”, “3”, “,” “6”, “4”, “7”]

  • Or into two tokens [“483,”, “647”]

  • Or into three tokens [“483”, “,364” “7”]

  • Or in many other ways.1 Of course, your tokenization scheme impacts how the LLM interprets the inputs.

• Positional Encodings: every token is related to the ones that came before and after. This allows the LLM to maintain an understanding of how the textual statements are ordered, and the meaning inherent in their relative positions. These encodings are usually one-dimensional, representing the text like one long line.

Your intuition might suggest that tokenization and positional encoding that works well for words may not work well for math. With words, there’s some leniency on precision — you can understand what someone means even when their writing is riddled with typos — but for arithmetic, you need to get every single character right.

Doing that can be harder than it sounds. Consider, as an oversimplified example, adding three numbers, written below in black. If you’re not allowed to rewrite the problem, then it’s tedious: following the grade-school right-to-left algorithm for addition, you have to figure out that positions2 8, 17, and 26 all correspond to each other, add those up, then add positions 7, 16, 25, etc. and then add the intermediate results. Relative to the length of the problem, this requires a lot of working memory, and it’s easy to make mistakes.

3. Papers that Attack these Problems

Several researchers have experimented with clever ways to solve these issues. Let’s go through them!

Reverse That Number! Decoding Order Matters in Arithmetic Learning

If the problem in adding long numbers is that they cannot be naively added left-to-right the way you read them, but rather have to be aligned right to add the digits column-by-column, then why don’t you teach the LLM to reverse the digits and then add them left-to-right? This technique works well, improving on state-of-the-art accuracy by about 11%.

Tokenization Counts: the Impact of Tokenization on Arithmetic in Frontier LLMs

Instead of reversing the numbers, you can also adjust the tokenizer.3 In this paper, the authors enforce commas to tokenize long numbers more reliably, and run the tokenization process from right-to-left rather than left-to-right. This performs much better: by contrast, left-to-right tokenization yielded systematic errors in certain digit positions. The authors improve on state-of-the-art accuracy by about 14%.4

Positional Description Matters for Transformers Arithmetic

In this paper, the authors find several results:

1. They asked whether transformers trained on arithmetic tasks in isolation could transfer this knowledge to arithmetic embedded in natural language. They found that just training on isolated arithmetic data isn't enough.

2. They changed how addition tasks were represented, e.g. by adding spaces randomly between digits. They found that this helped models better generalize to adding longer numbers than what it had seen during training.

3. They found that padding all numbers to the same length with leading zeroes, and using a reversed-digit approach (similar to other authors), would improve performance on multiplication tasks.

4. They found that introducing random noise to the positional encodings helped the model avoid overfitting to specific positions, and instead encouraged the model to learn more generalizable patterns. To me, this is the key result of the paper, though they don’t specify what percentage improvement this provides over the state-of-the-art.

Combining these techniques, they achieved 100% accuracy for multiplication tasks of 12-digit numbers, and 99% accuracy for 15-digit multiplication. This was a dramatic improvement over the baseline, which is roughly 0% after 5-digit multiplication.

Transformers Can Do Arithmetic with the Right Embeddings

In this paper, the authors replace the positional encodings altogether. Instead, they attach a positional embedding5 to each token, which tells the LLM the position of each digit relative to the start of the number. Using these so-called “Abacus Embeddings” delivered another colossal improvement over the state-of-the-art: when trained on adding 20-digit numbers and then tested on adding 100-digit numbers, they achieved 97.9% accuracy whereas prior models were in the 20-30% range.

Some additional tweaks — Input Injection6 and Looped Transformer Layers7 — added further improvements, raising the accuracy up to 99.1%.

4. Other Useful Work

We’ve seen that architectural tweaks can significant improve LLM performance for addition. There are more texts that can help us understand the underlying dynamics.

Teaching Arithmetic to Small Transformers

This paper stands out to me for two observations:

1. When teaching addition to their their transformer, they observe a “phase transition” from poor performance to nearly perfect performance. Unlike many other cases in machine learning, the model isn’t gradually improving. It accumulates information until it reaches a critical point at which it suddenly understands the underlying rule.

   

2. The authors suggest that the “phase transition” occurs because addition can be represented as low-rank matrix completion. Matrix completion has a similarly sharp threshold, where if you haven’t seen enough examples, it’s very hard to correctly complete such matrices, but once you’ve seen enough, you can complete them almost perfectly. The authors suggest that both addition and matrix completion have a sharp transition in capability after seeing similar amounts of training data. The authors conclude that this must be the same dynamic.

Shin Boson’s Twitter Thread

This is not a paper, but a Twitter thread where Shin investigates how LLMs multiply, reaching some of the same observations as the other papers above, with some neat illustrations. He makes two additional observations:

1. The fact that LLMs represent positional encodings one-dimensionally is a limitation, since we usually view mathematical text in two dimensions. Consider the example from earlier, where we added three numbers, and see how much easier it is to run the same operation when we rewrite it in two dimensions:

   

   I suspect there are certain types of math problems where the relevant positions are not just what’s to the left or right of a given token, but also what’s above or below. Linear algebra problems involving matrices seem like a good example. For these problems, I wonder if two-dimensional positional encodings could help. Two-dimensional positional encodings would be much closer to how humans see the world, anyway.

2. He ascribed the failure-states of LLMs to an extremely short working memory, which is worth digging into further. There’s an interesting open question as to whether LLMs suffer from:

   1. Not having enough working memory to store and transform intermediate results — for example, “carrying” in addition, or adding up intermediate sums for multiplication;8

   2. Not being able to apply enough attention to every token in the sequence, thereby losing some information that then causes arithmetic error. (There’s a related paper, Working Memory Capacity of ChatGPT, that’s not hugely informative, but I’ll cover it in the Appendix.)

5. Conclusion

Over the last two years, many people have claimed that LLMs can’t do math. I’ve seen some folks even suggest that this is a limitation of the transformer architecture; that it can’t learn to generalize arithmetic operations. This seems false.

Run-of-the-mill LLMs appear hobbled in their arithmetic accuracy due to how they implement tokenization and positional encodings. We’ve seen several approaches that adjust those, and then deliver near-perfect accuracy. In particular, Abacus Embeddings seem to generalize well, and I am very curious how far their length generalization goes. An LLM should be able to learn the addition rule with limited data — could we train a model on adding 10-digit numbers and then have it accurately add 10,000-digit numbers?

Going further, the key open question is whether an LLM can learn arithmetic perfectly. LLMs deliberately use some randomness: in spite of this, is it possible to train them to execute arithmetic computations deterministically? Going even further, would it be possible to formally prove correctness of such networks in the same way that programs can be provably correct?9 Such a bridging between deterministic and non-deterministic behavior would be not only be extremely interesting, but would powerfully extend the capabilities of such neural nets.

I’ll leave you with a few more questions that are on my mind:

1. Are there benefits to using two-dimensional positional encodings? Not just for arithmetic problems, but very generally. Naively, I think two-dimensional positional encodings are more representative of how humans view the world.

2. To what extent is dilution of self-attention a fundamental limitation of transformer architectures, and can it be overcome?10

3. The examples of length generalization in these papers have been very modest. Nobody’s tried to add numbers with 100,000 digits. Are further architectural changes needed to scale? It’s not clear from the research specifically how11 these nets are handling intermediate computation: e.g. adding numbers requires some “carrying” from one place to the next, multiplying numbers requires adding intermediate computational results. How does this scale? I thought the use of looped layers in the Abacus Embeddings paper was interesting, and wonder if recurrent techniques12 could help scale these nets to arbitrary input lengths.

Appendix 1: Other Papers

Below are some other papers that I’ve read and that you might consider part of the literature. I haven’t listed them in the sections above because their findings are more tangential to this discussion; most of them are about limitations that now appear at least partially overturned. (They do remain relevant insofar as many questions remain about limitations for larger scales of inputs.)

Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks

The authors assert that other papers have focused too much on architecture (e.g. positional encoding) and not enough on the fundamental difficulties of doing arithmetic. They describe a number of properties that make arithmetic tricky, focusing on multiplication and modular arithmetic. However, at least for multiplication, these difficulties are probably overstated, since architectural solutions in some of the papers above brought multiplication tasks to near-perfect accuracy.

Faith and Fate: Limits of Transformers on Compositionality

The authors suggest that LLMs struggle with arithmetic because they memorize simple patterns, rather than truly learning the rules of arithmetic: they claim that transformers fail to generalize these patterns. Errors in reasoning compound as problems become more complex. This may all be true for naive LLM approaches, but the papers above have shown that LLMs can, with architectural tweaks, actually generalize these patterns to a much more significant extent.

Transformers Can Achieve Length Generalization But Not Robustly

Similar to the Faith and Fate paper, this one focuses on limitations. They observe a number of issues with getting transformers to generalize learned patterns beyond ~2.5x the input length. However, this paper similarly looks overturned, e.g. Abacus Embeddings were able to generalize well beyond this suggested limitation.

What Algorithms can Transformers Learn? A Study in Length Generalization

This paper studies what algorithms transformers can learn and generalize, i.e. perform on inputs longer than what they’re trained on. To that effect, the paper introduces a domain-specific programming language, RASP-L, which contains operations designed to mimic those that are natural to transformers, e.g. creating attention matrices and applying them to sequences of tokens.

The central idea is that if a problem or operation can be expressed as a RASP-L program, then a transformer is more likely to learn and generalize the underlying pattern effectively. However, the authors found it difficult to express arithmetic operations in RASP-L.

Working Memory Capacity of ChatGPT: An Empirical Study

After all, why is it that LLMs are so good at short arithmetic problems and then get confused when the problems get long? Surely the difficulty isn’t just in “carrying the 1s”? This paper runs memory tests (n-backs) with ChatGPT. The “memory test” is a loose analogy here since ChatGPT doesn’t really have memory but rather the user is supplying longer prompts of historical context on each iteration.

In any event, the experiments found that ChatGPT struggled with tasks beyond n=3, similar to humans. The paper posits that LLMs struggle to maintain context over long input sequences, specifically because self-attention is computationally expensive to maintain. As the sequence gets longer, the self-attention is spread thinner across it: each token gets less focused attention on average, effectively causing the model to lose track of certain tokens.

Appendix 2: Chain-of-Thought Reasoning

If you want LLMs to do arithmetic, then you may consider chain-of-thought reasoning to sidestep the length generalization issues.

I haven’t yet covered this because I don’t think it’s interesting anymore: the technique is well-understood. Of course you’ll get better performance from an LLM when trying to do arbitarily complex tasks if you break them down into tiny sub-tasks and then solve them one by one. (You could go one step further and just have the LLM call a Python console…) I’m interested in whether an LLM can learn perfectly an arithmetic operation.

Regardless, in the spirit of covering relevant papers, consider reading these:

• Show your work: scratchpads for intermediate computation with language models

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

• An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs

The EU takes an aggressive stance toward American Big Tech. Citing concerns about privacy and monopolization, it has enacted countless regulations, and fined Google and Meta for billions of dollars. In the last six months, EU regulators have kicked this motion into overdrive:

• They adopted the Digital Markets Act (DMA), which they used to immediately open investigations into Apple, Google, and Meta.

• They adopted the AI Act to constrain AI applications.

• They slapped Apple with a $2B fine.

• In July alone, they opened antitrust proceedings against Nvidia, antitrust investigations into Google, and threatened to fine Twitter over seemingly-trivial Blue Checks.1

The posture is clear: the EU is not satisfied with the bloodletting-to-date and is raising its demands from Big Tech. The AI Act and DMA both may assess penalties as a percentage of global turnover, and are so broad in scope that European regulators are emboldened to pursue tech giants for practically limitless amounts of money.

But the EU is overplaying their hand. They have been able to make their demands to-date because it is easier and cheaper for Big Tech to pay and comply than to resist. Of course, this encourages the EU to make even more demands, but eventually those will become too onerous and expensive. We’re about to be at that point.

Kissinger wrote that relations between states are best understood in terms of power and legitimacy:2 in this essay, I will argue that the EU’s actions against Big Tech lack, and have always lacked, substantial legitimacy. Furthermore, the EU vastly overestimates its own power in this relationship: when push comes to shove, it is Big Tech that actually holds the cards, and the EU would do well to change its course. Indeed, the current course cannot, and will not continue.

A Brief History of Fines

For quick background, below is a non-exhaustive list of $100M+ fines that the EU has assessed on American tech companies over the past decade:

• June 2017: Google fined $2.7B (Antitrust over Google Shopping)

• July 2018: Google fined $5B (Antitrust over Android)

• March 2019: Google fined $1.5B (Antitrust over advertising)

• September 2021: Meta fined $266M (GDPR over WhatsApp)

• July 2021: Amazon fined $887M (GDPR)

• January 2022: Google fined $169M (GDPR)

• September 2022: Meta fined $427M (GDPR over Instagram)

• November 2022: Meta fined $275M (GDPR)

• April 2023: Meta fined $1.3B (GDPR)

• March 2024: Apple fined $1.95B (Antitrust over Apple Music)

That’s a little under $14.5B over seven years, or just over $2B a year. For scale, the US exports $592B to the EU annually, and EU-originating annual revenues for these companies sum up to over $100B.3 So, there’s still plenty of room for the EU to escalate upwards. This is what the EU appears to be preparing.

Context on EU and US Relations

The EU’s aggressive position on Big Tech does not exist in a vacuum. There’s important historical and contemporary context:

Asserting Leadership

Many international treaties and regulations were led by Europe. That’s true both in the distant past (see the Geneva or Hague Conventions) as it is more recently. The EU led the charge on international standards for chemical safety and carbon emissions trading, and its stringent regulations on consumer products, tobacco, and anti-money-laundering have set the pace for the EU’s global trading partners.

The EU is attempting to similarly set the pace with GDPR, the AI Act, the DMA, and other technology regulation. However, the international community generally isn’t following. That is an important break from history. It is no surprise that when the EU tries but fails to assert leadership, it will try again more forcefully. I read the EU’s mounting efforts to shape global norms to both assert and seize power and relevance.

Trade Generally

The US-EU trading relationship, while valuable and productive, is fraught with many sharp-elbowed negotiations at the margins. For example:

• Disputes over Boeing (US) and Airbus (EU), each accusing the other side of providing anti-competitive subsidies to their home aerospace giant, leading to billions in retaliatory tariffs, including in unrelated industries.

• Disputes over a 2018 set of US tariffs on steel and aluminum imports. The EU retaliated with tariffs, again including unrelated industries.

The US-EU trading relationship involves continuously testing boundaries and tit-for-tat exchanges. What’s different in Big Tech regulation is that it’s entirely one-sided: the EU doesn’t really have Big Tech of its own that the US could regulate in retaliation. Ironically, if it did, the EU’s regulatory efforts would probably seem fairer — just part of a tit-for-tat between roughly equally strong market participants.

The EU may view its Big Tech regulations as just another small negotiating chip in a much larger $1.3T annual trading relationship. However, the one-sided optics encourage the rest of the world to see it as unfair.

Bad Precedent

The EU’s framework goes so far as to assess fines on a percentage of global turnover:

• GDPR: up to 4% of global turnover (top-line revenue);

• AI Act: up to 7% of global turnover;

• DMA: up to 20% of global turnover;

These just keep getting more expensive! The idea of issuing fines based on global revenue for local violations of law is a brazen stretch of legal convention:

1. Penalties must be commensurate with damages;

2. Courts may assert their authority only over subjects in their jurisdiction.4

The legal convention would be for the EU to assess fines based on EU revenues, not global revenues. Permitting fines based on global revenue would set disastrous precedent: if the EU can set fines based on global revenue, why can’t any other country? Any other big market with a little bit of leverage could try to extract a slice of the pie. Why shouldn’t India, which has ~500M Meta users, start fining Meta for 10% of its global revenue? Why shouldn’t Brazil do the same? Or Nigeria? And why should they keep their fines to Big Tech? Why don’t they fine Exxon Mobil for a percentage of global revenue?

Permitting this scope would set terrible precedent, and it has no legal legitimacy. Not only must Big Tech refuse to comply, but the US must reject it as a matter of national interest and international order.

Illegitimate Fines

EU fines have been asserted under GDPR, and will soon be asserted under the AI Act and DMA. All three of these operate on the notion of consumer protection: that EU persons have rights, most often articulated around privacy or data, that are being violated by Big Tech. You can view this almost like a class-action lawsuit, where some compensation is sought for harm done to a large group of people.

But for a class-action lawsuit to be legitimate, it must reward the consumers! Consider the $1.3B Meta fine: the legal decision cites 309M daily active users in Europe. Assuming the EU asserts that all such persons were violated, this means the fine is equivalent to $4.21 per person. As an EU citizen, I have two problems with this:

1. I have not seen a single penny of my $4.21. The EU regulators alleged that my rights have been violated, pursued a legal case on my behalf, paid the legal fees with my tax dollars, recovered the damages, and then… pocketed them? Why should the tax rate on recovered damages for violations to my rights be 100%?

2. It is difficult to square the minute damages of $4.21 with the posturing of the EU. European regulators and media constantly express incredible scorn toward Meta; the narrative is so bombastic and severe as if Meta’s transgressions were wild violations of fundamental rights. But after all this drum-beating about privacy rights and the great value of my personal data, when the damages finally got priced, it turned out all this stuff is worthless? $4.21 can’t even buy me a sandwich.

Fundamentally Unserious Actions

As I wrote in a related prior blog post:

If they can’t ban products, then it’s not consumer protection: it’s just wealth extraction. European politicians always drum on about supposed violations from Google and Facebook, fine them for some totally unimpactful amount of money, and then pipe down for six months before starting over again. If those politicians had real grievances, they would try to ban those products, or build local alternatives. They do neither. They’re just selectively enforcing regulation to extract what I view as a bribe to operate.

The great discrepancy between the public posture and the minute per-capita fines suggests a triviality to the grievances pursued. Practically, it seems that regulators are pricing and pursuing very small negative externalities imposed by Big Tech firms — but under that regime, there should be higher priorities. European consumers suffer much more greatly from other firms and practices: e.g. air pollution is priced at roughly $110B per year in negative economic externalities (~55x of Big Tech fines), while the long-term loss of economic competitiveness, relatively weak tech industry, and failure to transition away from Russian oil are incalculably damaging to EU citizens. By contrast, the European Commission is dabbling in trifles.

The issue with dabbling in trifles is not just that it’s relatively unimportant and not a good use of lawmaker time, but also that trifles are plentiful: this makes it difficult to enforce the law in a consistent manner. Indeed, it encourages selective enforcement, which saps the appearance of legitimacy. When the EU threatened to fine Twitter over Blue Checks, that did not seem like an impartial application of the law, but as a personal grievance. Such capriciousness is no standard for any respectable authority.

The Realpolitik of Technology

Having gone over issues of legitimacy, let’s talk about power. Thierry Breton, the relevant leader in the European Commission, states:

And a market of 450 million customers is simply unthinkable for anyone not to be there.

Where the digital giants could pay fines of several billion dollars without batting an eye — by the way, when they had to pay them, after long years of procedures, which was not systematic, far from it... — today none of them can afford not to be in our market.

This is the reality of the balance of power of the world in which we operate.

Mister Breton, you may be mistaken! Specifically, in two ways:

1. Overstatement of EU Market Size

As a global market, Europe is not as big as it used to be. And the EU is smaller yet. If you take a look at Apple’s financial statements by geography, “Europe” accounts for 25% of global revenue. But in Apple’s consolidation, “Europe” includes the UK, Norway, Switzerland, Russia, Turkey, and the entire Middle East! None of these are EU members. The EU might account only for 7% of Apple’s global revenue.

7% is still a big market, but Apple is by no means dependent on it. Especially considering the exceptionally high level of operational headache in complying with European requirements, if it comes to be Apple’s view that the fines-as-percentage-of-global-revenue cannot be avoided, then it may be rational to pull out. John Gruber makes a similar argument for Meta.

2. Consumer Dependency

Where the rubber really hits the road is that consumers are dependent on Big Tech. They like using iPhones. They like using Instagram. They like using Amazon. And European businesses everywhere use Google services. The EU would struggle severely to go up against the most popular and widely used products on the planet.

This is the nature of true power: the firms have captured the hearts and minds of the public. Even if the EU were to ban these companies, consumers wouldn’t want to lose access to these services. Their digital lives are on them. Rather than moving from WhatsApp back to Skype and losing all their contacts, they would use VPNs to access software products, and purchase Apple devices on grey markets.

Moreover, the EU doesn’t have true local alternatives. If it pursues Nvidia on Antitrust grounds: does it really want Nvidia GPUs to be replaced by, say, Huawei GPUs? Does it want Facebook to be replaced by VK? If EU regulators are motivated by concerns over unaccountable, outside influences, I might suggest that American Big Tech is still their best option.

Power

To Europeans, Big Tech products are hard-to-replace public services. In past nation-tech disputes, like Facebook and Google in Australia (2021), Facebook in Canada (2023), or Google News in Spain (2014), tech firms pulling out of markets in just a small respect generated significant backlash, and allowed the tech firms to flex negotiating leverage.

Never forget: these Big Tech products are, for the most part, cloud services. They can simply be turned off remotely, from one minute to the next. Hypothetically, if Big Tech were to coordinate, play true hardball, and shut off EU-facing products, the EU economy would grind to a halt overnight. Imagine the fallout from hundreds of millions of people suddenly not having email anymore. Without AWS, GCP, Azure, etc. things simply wouldn’t work. We live in a digital world; the dependencies are everywhere. It’d be like when OPEC constrained oil supply in the 70s, except percolating much more deeply and instantaneously throughout economies.

Falling Behind

Of course, it’s very unlikely for Big Tech to withdraw from the EU entirely. That would be drastic. The reality is subtler, and we’re seeing it play out right now: Meta is not making its multimodal Llama models available in the EU. Apple isn’t going to bring Apple Intelligence to the EU. These are important, state-of-the-art products. If you believe at all that AI is promising or important, then EU businesses and consumers will suffer from not having access to them.

We may see the EU and US technology markets bifurcate, similar to how automobile markets have separated. Cars are widespread in Latin America, but the models are overwhelmingly older, smaller, less safe, and less fuel efficient than ones in the EU or US. This is due to cost, but all things being equal, consumers would be better off with newer models. EU consumers might similarly end up with older, lower-quality technology everywhere — but not due to cost, just due to regulation!

Because technology builds on itself, omissions compound. Maybe multimodal Llama AI is not important for EU consumers today. But what if the best radiology AI assistant gets built on Llama AI — and EU patients can’t have access? Or an EU business needs the best-in-class AI to remain globally competitive? What if Apple Intelligence can automatically call an ambulance for you if you have a heart attack — but not in the EU? If only second-tier technology is available, then after several years of falling behind, the democratic masses will realize they have been deprived of real things of value, and they will undo these policies.

Suggestions

The EU must compete or cooperate. Either one is fine. But it would be ill-advised to continue the current regime of low-grade economic harassment of its nominal allies by syphoning off fines and imposing obnoxious requirements.

• It is inappropriate for EU regulators to assess fines based on global revenue. Such precedent cannot stand. The US must, and surely will, flex its overall trading relationship with the EU to put an end to this.

• The EU must recognize that the US will eventually push back, and find a way to ignore or sidestep5 these regulations. At that point, they will cease to impose anything useful on foreign businesses, while constraining EU businesses.

• The EU’s regulatory gamesmanship will also be under long-term pressure by its own citizens, who will not appreciate access to lower-quality goods and services. If the single reason why their businesses are less competitive and their quality of life is lower is not one of cost but of abstract regulation, they will push back.

• The EU would do well to recognize that it is broadly dependent on the products and services of Big Tech, while being only a minority revenue originator. In my view, the true push-come-to-shove balance of power rests with Big Tech more so than with EU regulators.

• The EU would be well-advised to grow its own tech industry, which it currently lacks. No Big Tech company has anything even close to an EU competitor. More free-market competition would be good, both generally and for EU consumers.

The last point bears repeating. I view all of this regulatory squabbling as a tragic and futile misdirection of effort. The EU recognizes that it is dependent on Big Tech, recognizes that it doesn’t have any ownership, and is using a combination of fines and regulation to claw back some power in this relationship. But that’s not sustainable; the relationship will be parasitic at best, never one of equals. The practical reality is that the EU must create and foster its own tech industry to be globally competitive. That is the only way for the EU to have a seat at this table.

FURTHER READING

Some work from other writers that acted as partial inspiration for this post, which I haven’t linked to already:

• Stratechery: The E.U. Goes Too Far

• Daring Fireball: European Commission Opens DMA Non-Compliance Investigations Against Google, Apple, and Meta

• Mostly Borrowed Ideas: Tweet

ACKNOWLEDGEMENT

Thanks to Evan Zimmerman for helping me refine my notes on Antitrust and US-EU trade relations.

For the last few years, Waymo has been operating fully driverless cars in Phoenix and San Francisco. They’re commercially available. Anyone can download the app and hail one: I was recently passing through SF, and took the opportunity to make a few trips by Waymo. I found the experience hugely impressive, to the point that it made very clear what some parts of our future will look like.

In this essay, I will talk about my experience riding Waymos, predict the impacts that self-driving vehicles will have on our society, sketch out Waymo’s unit economics and discuss its competitive positioning vis-a-vis Uber and others, and finally argue that Waymos will totally overhaul how we think of public transit — offering a rare technological “leapfrog” opportunity to urban America.

The Experience

I hadn’t thought much about how it would be to ride in a self-driving car. Surely it’d be like taking an Uber, but without a driver — functionally, it’s all the same, right?

It’s actually totally different. And it’s wildly superior.1

1. The feeling of the drive is much better. Waymos accelerate very gently, and drive in a slow, defensive way such that they will never suddenly speed up or hit the brakes. In a city with lots of hills and stop-and-go traffic like SF, this is a godsend. If I were stuck with an impatient Uber driver who floors gas and brake, I’d feel nauseous after a few minutes. On the other hand, the gentleness of the Waymo means that I can read for the entire drive,2 and not even notice any traffic.

2. The Waymo is your own personal space. It’s really nice to have it entirely to yourself. That makes it much more relaxing than taking an Uber. When you share a small, confined space (like a car) with a stranger, it causes a very subtle stress: you’re paying attention to what they’re doing, and you think about your own actions so as not to bother them. It’s slightly taxing. Waymo obviates this.

   1. This is really noteworthy, because it means that even having a great driver is worse than having no driver in this respect. One’s first-glance intuition might be that driverless is better than a bad driver and worse than a great driver, but this is not so. Driverless is better in both cases.

3. A Waymo is clearly a much safer driver than a regular human. Waymos actually keep their legal minimum distance of three feet to bicycles on the road! As a cyclist, I’ve had trucks speed past me at over sixty miles an hour with maybe six inches of clearance. I’d feel much safer sharing the roads with Waymos that are actually programmed to follow the law.

In short: the Waymo experience is great. It’s hard to imagine how good it is if you’ve never been inside one, because you’ve been used to a different paradigm your entire life. Taking a Waymo is relaxing in a way that taking an Uber (let alone driving yourself and having to actually navigate traffic) simply never is. Even if a Waymo travels a little slower than a human driver, the ride is so gentle that you’re better off on net. It turns into quiet personal time or reading time, instead of being waiting time.

They’ll be Everywhere

There’s a saying in surfing: slow is smooth. Smooth is fast. Google has played it right: fifteen years of slow-is-smooth work on Waymo, delivering an impeccable safety record and a magical experience. They are far ahead of any competition,3 ideally positioned to roll-out nationally as quickly as regulators will allow.

I’d expect for large fleets of Waymos to become commercially available in major US cities over the next few years. Approval for roll-out will become easier and easier as the cumulative safety record speaks for itself.

They’ll be Cheap

Waymo is still a rare luxury today. The SF fleet has only about 250 cars, of which 100 are on the road at any time. This makes Waymo slightly more expensive than Uber. As more Waymo vehicles become available, you might expect Waymo to undercut Uber on pricing, and take the rest as margin. But I think Waymo will race to drive down its pricing much further. Why? Because Uber is not nearly as big as the terminal market size. For comparison:

• In the US, there are 1.1 billion car trips every day;

• In the US, I estimate Uber and its competitors complete 19 million rides a day;4

In other words, there are ~58x more personal rides than rideshare rides in the US. The true TAM is that Waymo replaces all travel by car (and perhaps even all travel by bus, subway, etc.). To get there, Waymo has to bring its price down to the point that anyone with a car would pay to not have to drive themselves. Maximizing business value here is a volume game: the winning recipe is massively scaled deployment, with prices lower than any competition, and taking the thin margin.

Unit Economics

Operating Expenses

Not having a driver significantly lowers OpEx. Take a look at Uber:

• Uber claims that its drivers get roughly 70% of the ride fare whereas drivers claim they get about 50% of the ride fare;5

• Uber’s cost of revenue is about 16% of gross bookings, and their cost of operations and support is about 1.8% of of gross bookings;6

Clearly there is an opportunity to reduce the marginal cost of operating the ride by between 50 and 70%, perhaps more if the removal of driver operations, payouts, etc. further reduces costs. But Waymo’s safety record makes for even larger savings. Fewer crashes mean:

1. Less need for expensive repairs;

2. Lower insurance rates;

3. Longer vehicular lifespans over which they depreciate.

Long-term, you’d expect the price of a Waymo ride to be the sum of:

1. The cost of gas (or electricity);

2. The per-mile depreciation of the vehicle;

3. The marginal cost of any supporting software or operations (e.g. cleaning);

4. Whatever the average consumer is willing to pay to not drive themselves, which might be a few dollars an hour, or a little more if the experience is really great and they win back some of the opportunity cost of driving.

I think a decent first-order approximation, before factoring in any next-generation technology advances or economies of scale, is that Waymo could bring down operating costs 70% relative to Uber. The world looks very different when a $40 half-hour Uber ride suddenly becomes $13 — that’s a lot of increased mobility.

Capital Expenses + Payback Period

Waymo vehicles are not cheap! Each one costs about $200K to deploy. Recapturing this CapEx is a tall order, and it makes Waymo’s prospects less clear.

Suppose a Waymo vehicle runs a gross profit of $10 per hour,7 and it drives for 20 hours a day on average. That’s $73,000 a year in gross profits. Even if the cost of deployment comes down to $150K, it takes 2.1 years to recapture the CapEx.8 If it drives an average of 25 miles per hour, then it’s putting on 182,500 miles per year, or about 383,000 miles by the time the CapEx is recouped. That’s a lot of miles! I don’t know what the life expectancy of the vehicle is after 383K miles, but it’s probably not great. Further, if any of the expensive self-driving hardware (Lidar, sensors, etc.) has to be replaced on any regular schedule, it’s possible that an individual Waymo would take many years to recapture its upfront cost, if it can do so at all.

This suggests that Waymo would have to make very big, long-term bets, risking significant CapEx with uncertain payoff profiles. Waymo might be severely unprofitable at significant scale before it arrives at viable unit economics.9 Importantly, Waymo kind of has to do this, because it must (1) use its current market-leader advantage to build distribution while competition is scant and (2) scale up in order to have economies of scale that can bring the costs down.

A True Google Bet

But never forget: Waymo is owned by Google, and making very big, very long-term CapEx bets is the Google style. This is the true nature of the moat in Google’s search engine:10 Google has invested heavily for twenty-five years in building web crawling infrastructure. If you wanted to compete with Google on its turf today,11 and create a web crawler and search engine on par with Google’s, the level of capital expenditure would render it basically impossible: it would take billions of dollars over many years of engineering just to get to par with Google. Then you’d have to spend tens of billions of dollars trying to take market share away from Google, and only then would you be in a position where you can fight a price war on ads against Google, at the end of which neither of you will make any profits. It’d be economic insanity to try. So nobody does it, and Google’s search engine enjoys a natural monopoly and prints money.

The reason to tell that story is that there’s an analogous case for Waymo. Google could perfectly rationally deploy many billions of dollars into Waymo to obtain:

1. Proprietary vehicles that are a superior form factor, both in terms of rider experience and unit economics (i.e. minimizing maintenance costs);

2. Massive scale, creating superior ride liquidity — i.e. when a customer requests a ride, they are able to obtain one faster than with any competing service;

3. Superior economies of scale in purchasing, outfitting, and maintaining vehicles;

4. Superior technology and data, creating a better and safer drive experience;

5. A superior safety record translating into regulatory capture.

Google has plenty of money: $108B in pure cash on hand, and $69B in free cash flows for 2023. Self-driving cars are among the few markets with TAMs big enough to move the needle for Google, so it would make sense to burn a few billion dollars a year on Waymo for the next decade, and to start capturing profits only once the barriers to entry are too tall for competitors and Waymo is in a position of natural monopoly.12

There are two important notes here:

1. To believe there can be a natural monopoly, you must believe this is a winner-take-all or winner-take-most market. We’ve had a great live experiment with that over the past fifteen years: even though there is little differentiation among rideshare services, it has been very hard for new ones to compete. Ten years ago, I expected many vendors to emerge, but the reality has been that acquiring customers is so expensive and the margins are so thin that Uber has been able to take over the market, with small (shrinking) slices for Lyft and for local alternatives (primarily outside the US). By analogy, this suggests that the market for self-driving vehicles will have winner-take-most characteristics.

2. Google has already shown tremendous long-term thinking: they have been investing in Waymo for fifteen years now. Consider the level of conviction required to keep up this level of investment for so long, especially in the early years when deployment seemed ever-so-far away. Now that the goal is in sight, massive spending is more rational than ever.

What Happens to Uber?

Uber might go down as a strange story in business history. They’ve burned $31.4B in operating losses since 2014 to win this market. For many years, Uber (Travis Kalanick) recognized that human drivers were a transitional step, and that autonomous vehicles would eventually gobble up the space. TK had the right instinct to focus on Uber ATG starting at 2015, when it was still possible to catch-up. It’s not anymore. I expect that as soon as Waymo deploys a thousand cars in a big taxi market like LA, Vegas, or NYC, the public markets will put an expiration date on Uber.

Startups are a tough game, and the path from disruptor to disrupted can be surprisingly short. Uber was founded in 2009. If Waymo disrupts Uber by 2029, then it will have been a twenty-year-journey of posting huge losses to seize a market, scoring a few years of comparatively small operating profits, and that’s it. Over a twenty-year journey, you really do have to keep innovating — in fact, it’s so long that you may be forced to bet on, and pivot, the very definition of your business. This will have been the case for Uber. TK and team saw this; I’m not sure if their successors did.

Commutes will be back

The once-dreaded long commute is about to come back in a big and pleasant way. I would have no issue at all sitting in a Waymo for 45 minutes each way every day. It’s just a nice time to myself that I can use to nap, work, or read.

This means that the suburbs and exurbs stand to benefit meaningfully. Working in a high-cost-of-living city and then commuting from a lower-cost-of-living suburb is a common strategy, but it comes at a certain lifestyle tax: you lose a lot of time on the commute, and you have to plan around availability of transport. Inexpensive, widely available Waymos fix this.

Leapfrogging Public Transit

This brings us to public transportation: so far in this piece, I have chiefly compared Waymo to Uber. But Waymo is most compelling not as an alternative to a taxi or Uber, but as an alternative to driving yourself — or to being on public transit.

Waymo becomes most interesting as an alternative to public transit. It favorably changes the economies of scale: more riders per vehicle imply better economics, and deals with local governments can stabilize Waymo financially with long-term contracts.

Waymo as public transit is particularly attractive, because virtually all cities in the US have wound up in a position where building public transportation infrastructure is well-nigh impossible. In San Francisco, it cost $346M over 6 years to install a new set of north-south bus lanes on a straight stretch of two-mile road.13 In NYC, a new subway line is costing $2.5B per mile. Trying to “build” anything, even if it’s just sectioning off an existing part of a road with some paint, invites months or years of local political debate, environmental litigation, and runaway costs far beyond any reasonable imagination.

But we have tons of roads. And thankfully, self-driving cars do not require any more infrastructure. This sidesteps the quagmire of trying to build new public transit infrastructure, and allows us to provide great public transit in the form of driverless cars/small buses without all the administrative overhead. It’s an easy winner:

• The fact that these cars can be called by app to any destination and routed efficiently by algorithm means that they offer huge accessibility and environmental advantages over public transit that always follows a fixed route and schedule.

• Many people who currently drive themselves would probably be happy to carpool in a self-driving vehicle if it’s reliable and easy, which would improve congestion.

• Unlike all other public transit, maintaining the infrastructure for self-driving vehicles (simple roads) is relatively easy and inexpensive.

It is ironic that for so long, every lane of road for cars was seen as zero-sum competition for public transit, and now we might achieve broad, high-quality public transit precisely via all the road space that we’ve already made available.14

In fact, this might bring American public transportation to a leapfrog moment. Many pundits have lamented that developing cities elsewhere have “leapfrogged” the US on public transportation — building subways and rail networks that put ours to shame. Over a hundred years ago, we built first-generation public transit.15 Over the last forty years, other countries built second-generation public transit. Now we have the opportunity as a nation to lead the world on third-generation public transit, and in that course develop products and expertise that can be exported.

More granularly, cities all over the US are about to have a fantastic opportunity to redirect budget from artificially expensive transit infrastructure projects toward driverless cars and small buses as next-generation public transit. A billion dollars doesn’t buy you a lot of subway stops these days, but it’ll probably buy you a terrific long-term, at-cost contract for the future of transportation.

BEST-PRACTICE DISCLOSURES

I don’t have any financial positions in Uber or Google other than indirect investments in their common stock via low-cost index funds.

• CRMs take unstructured data like emails and phone calls, and create a schematic interface to them: letting you group, filter, sort, and take actions on your contacts.

• Business intelligence tools take structured data, like your application database or Google Analytics, and give you new, visual interfaces for this combined data.

• Special-purpose OCR tools take raw documents and extract key data from them, exposing them by API, again turning unstructured data into structured data.

This is a very broad category. It basically includes any kind of data transformation task, which is common across all kinds of management systems,1 document parsing tools, data visualization software, and so forth. It has been a successful industry because data transformation has (1) historically been tricky and labor-intensive, with many edge cases that you have to guard against and (2) usually been integrated deep in the bowels of a business: once you install Salesforce, you’re not tearing it out again.

So, here’s a question that might have seemed crazy two years ago:

What happens when data transformation is no longer valuable?

Let me take a step back and provide two examples.

Documents and APIs

Historically, many businesses have interacted like this:

And for a long time, the engineers’ answer was: these businesses need APIs. Instead of having employees push documents back and forth, the systems should talk to each other directly. Many good companies have been built on this proposition.

But in a world in which OCR is really good, it may not matter much anymore. Suppose someone emails me a document that contains some useful data. I can pass the document to an application that extracts the text, has an LLM extract the relevant data, and prompts it to coerce that data into the schema my database has. It no longer practically matters whether the data comes as an e-mailed document or by API. It winds up in my database either way: effortlessly, and without the two companies having to spend months agreeing on an API schema and then building out some maintenance-needy interoperability layer.

Salesforce

Suppose you have a small team, ten thousand clients, and a couple hundred thousand emails across all those clients. The conventional solution to keep track of everything is a CRM like Salesforce.

But a couple hundred thousand emails isn’t actually very much data. You can put all those emails into a database, compute the embeddings, and then run RAG across that. None of this is hard, and LLMs then allow you to query your dataset:

• Where is Bryce on the deal with the Fisher Account?

• How are we doing with Van Patten, do we need to follow-up?

• What did I talk about last with McDermott?

• List all the accounts that I spoke to last month, but not this month

That functionality is similar to what Salesforce provides. Adding basic automation that handles prompts such as “Send a follow-up email to any client that I haven’t spoken with in the last 6 months” is easy. Would I rather have this, or spend months building out and maintaining a costly Salesforce implementation? I lean toward the former.

The Attack

There are three components coming together:

1. Large Language Models are very good at data transformation;

2. Continuously improving data store technologies and the falling cost of compute make it feasible to run arbitrary data transformations on-the-fly;

3. Business data is growing, but more slowly: documents, emails, spreadsheets, etc. just aren’t very big.

Not too far out in the future, businesses will gain the option to choose between (1) schematic software that lets them organize their data into a very precise traditional interface, or (2) LLM applications that let them organize and interface with their data on-demand as necessary. In the near future, the schematic software might have a precision advantage over the LLM application, but as time goes on, I’d expect the LLM application to become better and better at making precisely the right transformation based on the user’s demand.

The LLM Advantage

The LLM approach has one big advantage: it’s much more flexible, so it’s not annoying to set up and maintain.

By way of example: most companies are in Dashboard Hell. They have a bunch of internal dashboards that provide a gazillion different views into the business data. Most employees are familiar with maybe two of these dashboards, and are otherwise totally lost when trying to get to a data-driven answer. They have to go ask someone on the data team the question, and then the data person will file a ticket and eventually return some kind of view that the employee would’ve never found. Not only does such dashboard software (Looker, Tableau, Sigma, etc.) run you at minimum $10,000 a year, but because they also require people to continuously maintain them2 and help regular employees get answers from them, the fully-loaded cost easily and quickly climbs into six figures a year and beyond.

But the competition is coming! There are now many, many startups trying to attack exactly this problem with LLMs: they enable companies to connect their database and other data sources, and then let the user ask natural-language questions. Behind the scenes, an LLM-enabled application makes sense of the schemas in real time, designs the right queries, and returns the output. That’s accessible and useful in a way that current business intelligence tools just aren’t.

My Perspective

Many years ago, I once worked for a company that had to handle and reconcile user data from a great many sources, some of which would conflict with one another. My new-engineer instinct was to carefully design precise schemas for everything that we could possibly expect, and then define carefully-thought-out rules for when to prioritize persisting which data.

More senior engineers on the team suggested a different path: they thought it was too hard to predict all the weird data schemas we might encounter, and so they designed one fairly simple, permissive schema that could easily accommodate all such data, and then they put all their effort into a fast harmonization algorithm that would run on the fly and pick out the best pieces of data in response to a query.

I often think about harmonization. It was one of those classic don’t-let-perfect-be-the-enemy-of-good techniques that make for practical engineering: instead of building a brittle, special-cased system, build something that’s easy to maintain, works well in the vast majority of cases, and then you can tune performance up over time. You’re not going to have perfect accuracy up-front, but for most applications, you don’t really need perfect accuracy and the trade-off is worth it.

I suspect that there is a large number of software businesses that constitute useful but brittle, special-cased systems for schematizing data, and that they will soon give way to LLM applications simply sitting right on top of the data, harmonizing on the fly.

But these arguments tend to leave out what exactly they mean when they say “intelligence”. This is no coincidence, because these arguments are often not actually about intelligence when you examine them closely. They are about related, but distinct concepts (like consciousness or emotion) that are often conflated for anthropological reasons. In that way, because intelligence is such a nebulous concept, it is easy for people to make misleading arguments about it. My objective in this blog post is to:

1. Explain why intelligence is often conflated with other concepts;

2. Try to disentangle intelligence from these concepts;

3. Discuss what the current generations of LLMs are showing us about intelligence;

4. Predict what the next generations of LLMs may look like, and what we may learn about human theory of mind in the process.

Anthropic Bias

Out of all the life on planet Earth, we humans are spectacularly unique and capable. For example, we possess the following qualities:1

• Intelligence: the ability to perceive, infer, acquire, retain, and apply information

• Consciousness: the awareness of our own existence

• Emotion: the capacity to alter mental state due to physiological change;

• Reason: the ability to logically draw true statements from other true statements;

This is in contrast to all other animals, which either do not possess these qualities at all, or to vastly smaller extents.2 This leads to what some call Anthropic Bias:3 for example, if we view ourselves as the only truly intelligent creatures, and as the only truly conscious creatures, then it is tempting to think that intelligence and consciousness are tied together; perhaps even that they are the same thing.

This is the conflation that I mentioned earlier: people talk about intelligence when they really mean reason or consciousness. From a normal human perspective, because these qualities are so intertwined to us, they get lumped together as one. But they shouldn’t be, and contemporary progress in LLMs makes this clear.

What We’re Learning from LLMs

If you’ve played with ChatGPT, your immediate response was probably something like: wow, this is intelligent. There’s no doubt it passes a 1980s-style Turing Test. You can ask it about the significance of the Peace of Westphalia or what a Monad is, and the answers will be better than what I can tell you.

But what’s also apparent from playing around with ChatGPT is that it doesn’t seem to meet any standard of emotion or consciousness. In fact, I’d argue it’s not even capable of reasoning: it fails to compute simple arithmetic. Furthermore, the core architecture of ChatGPT — it’s a transformer-based LLM — is already at huge scale: it’s trained on a dataset that most likely constitutes a significant percentage of all human text in existence, so it’s not clear whether scaling it up another 10x or 100x will cause its abilities to converge toward “true” reasoning. Reading the tea leaves of the contemporary literature reinforces my impression: precise reasoning may require some different or additional architecture.4

In the previous section, I listed “intelligence” and “reason” as separate bullet points, which may have looked unusual. This is the reason I did so: one of the things that’s been enormously surprising to me in my work with LLMs is that conventional information-learning-and-recall notions of “intelligence” may be mostly distinct from the ability to reason.5 Fascinating!

So What is an LLM?

Well, in the context of artificial intelligence, it doesn’t really resemble humans at all. It’s a totally new thing. No consciousness, no emotion, arguably no reasoning, just pure informational intelligence in the form of a gigantic interpolative datastore.

In theory of mind, there is the popular thought experiment of a P-Zombie, short for a “philosophical zombie”: a creature that is externally identical to a human, but does not have an internally conscious experience. For example, if you were to poke a P-Zombie with a stick, it would say “ouch” and “get that away from me” and step backwards, but it would not internally feel pain or think “I will tell my sibling about the maniac with the stick” as a person might.

Similarly, if you were to prompt ChatGPT with the phrase “your beloved family pet has died. How do you feel?” and then “now tell your teenage daughter about this,” you will get perfect human outputs: a somber reflection on mortality, an expression of grief, and a delicately worded parental message. It reads like emotion, but it isn’t. ChatGPT does not have a dog or a family or a daughter or continuously running internal dialogue,6 all of this is just P-Zombie simulacrum.

Dismantling AI Safety Arguments

Now that we’ve built up some precision in discussing intelligence, disentangled it from other notions like consciousness, and pointed out some of the singular weirdness and limitations in LLM behavior, we’re better-equipped to cut through the noise of some AI Safety arguments. Specifically, the broad class of argument that I’d like to address goes something like this:

Encounters between species or civilizations with a great imbalance of power tend to end in the destruction or conquest of the weaker one. If we invent a digital super-intelligence that is hostile to us, we would be outmatched and face existential risk.

There are a lot of subtly-buried assumptions in there that make this argument misleading:

• The analogy to inter-species or inter-civilizational conflict is clearly inappropriate. While it might be temping to look at AI through the interpretative lens of things that we know well — ourselves — it doesn’t fit. We have shown that LLMs are nothing like any existing thing.

• Hostility or conflict of any kind follows some impulse for self-preservation, and acting on that, subject to an environment of scarce resources and zero-sum competition. This is natural to virtually all animals on planet Earth (including ourselves): Darwinian evolution results in creatures that protect themselves and reproduce. This is such a powerful historical filter that this trait is deeply ingrained in us, but a priori there is no reason why any kind of artificial intelligence — not subject to the same evolutionary filter at all — would share this characteristic.

• What does “super-intelligence” mean? Would that be a thing that is extremely good at learning, retrieving, providing, and applying information? Doesn’t GPT-4 already meet that definition? The conflation is that “intelligence” here implies the capability for reasoning too — but we still seem to be very far away from that.

Being Precise about Intelligence

This is not the first time in history that a widely-used concept, when closely examined, has turned out to be opaque and instead give way to an assembly of many other concepts: the ancient Greeks called everything philosophy; this has now been broken down into hundreds of scientific disciplines. In the 20th century, psychologists diagnosed everything as schizophrenia; today that’s an exceedingly rare diagnosis, but the DSM-5 lists over 300 more precise conditions.

The same may occur for intelligence: as the field of artificial intelligence keeps advancing and we work practically with more forms of AI, we will be pressed to split many of these nebulous, broad concepts up into precise characteristics. This may cause us to re-shape our own theory of mind as well. It seems likely to me that in a few years, the model of human cognition will not be as a monolith, but rather as the outcome of many distinct systems interacting in complex and perhaps surprising ways.78

Continued research in LLMs may take us to an unprecedented position, where we will have invented these things that are hyper-intelligent in a very specific way — extraordinarily powerful at statistically compressing knowledge and returning approximations to it — but otherwise totally inert. This breaks how most people currently think about intelligence, and that’s why I wrote this blog post — it’s important to start thinking differently about these constructs if you don’t want to make premature or dumb policy decisions.

The Intelligence Revolution

I wrote at the start that we are in the very earliest stages of an intelligence revolution. What did I mean by this?

Consider the industrial revolution. Some historians define it by the inventions of certain machines and processes — but others see it as defined by a break from history: the amount of energy harnessed and consumed by humanity began to increase super-linearly per capita. The availability and energy density of fossil fuels was a tremendous unlock, probably a necessary condition for bringing our civilization to where it is today.

Similarly, I have little doubt that future historians looking back on our time will again see it as defined by a break from history: the amount of intelligence available to humanity began to increase super-linearly per capita in the early 2020s, and will have been a tremendous unlock. But it’s yet unclear what exactly that intelligence really comprises, or what the retrospective measurement of it will be.9

The result looked plausible. But I had a moment of doubt: why should GPT be good at addition? So I double-checked the sum myself. It turned out that GPT was wrong; the right number was $660.44. Ever-so-slightly off.

This struck me as a very strange result, for two contradictory reasons:

1. It’s not surprsising that GPT is wrong. There’s nothing really about a language model that implies learning to sequentially apply the strict rules of arithmetic. While it’s possible to argue that GPT can reason if you give it some memory and many iterations over which its language inference turns into (fuzzy) symbolic inference,1 that’s not happening in a single ChatGPT query.

2. But then why is the result so close? It’s only wrong by 0.3%! Clearly there’s some kind of fuzzy symbolic inference getting made here that approximates addition pretty well.

This piqued my interest. My hypothesis was that the GPT-4 training set was so vast that it simply included many strings of arithmetic, which could then be recited verbatim for correct results, or used as a base for some token-level inference. Especially if there’s memoization at play, then this would explain good arithmetic performance even without the LLM actually conducting symbolic reasoning.

However, that’s a fragile process. If the above was true, then GPT should become less accurate as the arithmetic expressions became longer. I was curious what the relationship would look like, so I took an evening to run some experiments.

Experiment: Plain Arithmetic

You can follow along by looking through the code and data in my open-source repo.

Addition

I created sequences of random integers between 1 and 100. I varied these sequences in length from 2 numbers all the way through 24 numbers, and joined them up with addition symbols.2 I used a prompt in which I asked GPT-4 to solve the arithmetic problem and to return strictly a number. I then evaluated the GPT-generated sum against the true sum, and measured what percentage matched exactly. For every sequence length, I took 20 samples, so the chart below represents 460 datapoints.

Subtraction

Same methodology. I expected addition to perform better than subtraction because addition is a much more common operation, but the results were quite similar.

Multiplication

Same methodology. Again, my guess was that accuracy would drop off faster than for addition or subtraction because there’s likely fewer such text samples online, and multiplication blows up the number of tokens: if you’re multiplying 20 numbers between 1 and 100, the result is going to be a long number — lots of digits — which increases the room for GPT to make mistakes.

As predicted, performance was much worse: GPT-4 had 95% accuracy for multiplying two numbers under 100, 30% accuracy for multiplying three, 5% accuracy for multiplying 4, and zero thereafter.

Large Addition

I wanted to further investigate the idea that GPT-4’s addition is good because there are many solved examples in its training data. I assumed that numbers under 100 are much better represented than large numbers, and was curious if performance would degrade if I used larger numbers instead. For this set, instead of generating random integers between 1 and 100, I generated random integers between 812,300 and 812,400. I figured these numbers were big enough that it’s unlikely for there to be lots of solved examples online.

The results bear this out: accuracy drops off much faster than for regular addition, but it still stands out as remarkable to me that it’s scoring 100% accuracy on 2- and 3-number sequences. (We’ll come to why this is important in a later section.)

Addition by Sum

We observed that changing the operator from addition to multiplication, for example, is accompanied by degradation in performance. But what if you just change the syntax? Does “sum of 2, 3, 4” perform differently from “2 + 3 + 4”? Again, my hunch was that “sum” would perform worse because it’s a less commonly used syntax.

It turns out that the sum syntax performs much better than regular addition. If you’re surprised, I was too!

The implication is that “+” and “sum” are not represented the same way internally by GPT-4.34 If the results were the same, or close, then there would’ve been a conclusion to draw about GPT internally transforming different syntaxes of equivalent inputs into the same format — but that is not so. This is noteworthy to me because syntax/format transformation5 is a task that GPT is usually strong at. But I suppose that transformation for output is not the same as transformation for internal processing.

How Far Off?

We know that GPT becomes likelier to make an arithmetic error the longer the expression is. But how severe are these errors? The example from the very start, which motivated this entire investigation, was off by only 0.3%.

I took the Large Addition dataset (adding numbers between 812,300 and 812,400), and took a look at how far off the GPT-generated result was on average:

These are pretty close! But looking at the average % deltas obscures some of the detail. Below are the actual results for the sequence of length 24:

What’s interesting is that these results are very consistent: the delta is either around 0.20% or around -16.67%. This suggests that there are two different ways that these prompts get handled by GPT. The 0.2% delta way is clearly both hallucinating but also approximating closely the results of actual6 addition.

Symbolic Reasoning for Addition?

Earlier I mentioned that even in the Large Addition case, it maintained 100% accuracy for 2- and 3-integer sums. But maybe those numbers were too small? I tested with even larger numbers.

Most of these are right. (70% correct.) The ones that are wrong are wrong in interesting ways — row 2 is just off by four, rows 7, 9, 10, 11 all have the thousands digit too low by one. Row 17 just has one extraneous digit. This feels similar to the “How Far Off” section just earlier; where maybe there are several internal paths by which GPT may handle these computations — some of them work, and others don’t.

Can I force GPT into the “right” path? My first hunch was that the “wrong” paths were probably downstream of some of the internal randomness of the LLM. There’s a way to control this: the temperature parameter.

So I set the temperature to zero, and ran a 400-sample experiment for the same number range. The results improved from 70% to 93.5%. The errors were the same types as above, mostly one digit too low in the thousands place.

Could I improve it further? I wrote a “stronger” prompt admonishing the LLM to use its best technique and fiddle with the results, and again ran a 400-sample experiment. The results improved further to 97.5%.7

It appears that the LLMs contain some pathways for achieving correct addition, and we are able to (approximately) unlock them by querying the LLM in particular ways. If such correct addition is purely generated by the LLM (and not any additional logic like built-in arithmetic tables), then this suggests that the LLM’s language inference somehow is conducting actual symbolic reasoning at small scale. Not reliably all the time, but the fact that it’s doing it at all is remarkable, in and of itself.

This is a sign of what’s to come in the future. The big limitation of LLMs at present is their inability to precisely and symbolically reason. But if they’re able to do any symbolic reasoning at all, then inductively, they should be able to do more symbolic reasoning. This suggests to me that the transformer-LLM architecture may actually scale to true (and correct) symbolic reasoning, which I regard as pretty much the Holy Grail of this branch of machine learning.

A Dumb Turing Machine

We observed that LLMs appear to have some limited capability for symbolic reasoning, particularly for adding two numbers. We saw performance degrade as we tried to add more numbers, but there’s a workaround.

I mentioned at the start that you may argue that GPT can reason if you give it some memory and many iterations over which its language inference turns into (fuzzy) symbolic inference. GPT will correctly compute long addition sequences, just like the ones we saw earlier, if you prompt it to proceed two numbers at a time.

It worked. You can check out the live example here, in which I had GPT compute the sum of a length-24 sequence — something we couldn’t do at all in the naive prompt.

This “scratchpad” technique is a popular technique in working with LLMs: you have the LLM write down intermediate results and then explicitly use those results to continue working. You can chain arbitrary scratchpad steps together, though it may require re-prompting.

Abstractly, the idea of keeping and adjusting a memory register for every computational step reminds me of a Turing Machine. You can imagine prompting the LLM as I did, using its language capabilities to break a symbolic reasoning task into tiny steps that it can handle symbolically, and using a memory register to store the intermediate values so it doesn’t get overwhelmed. Basically, you can prompt to implement a Frankenstein Turing Machine inside an LLM. This is sort of neat — a way to purely use LLMs to solve symbolic reasoning tasks, which is what we’re all after. But it’s also sort of dumb, because your computer is already a Turing Machine, simulating an LLM, in turn simulating a Turing Machine. My intuition is that the way to get LLMs to reason symbolically at scale is probably not by brute-forcing it via Dumb-Turing-Machine-style computation, but maybe it is. Intuitition for LLMs is tricky — I’ve been wrong several times just in the course of writing this blog post.

Appendix Bonus Material! Prime Factorization

This was another experiment I conducted that didn’t really fit well into the discussion piece above, but I thought might be of interest to some of you.

The theme of “close approximation” raised a question: when GPT-4 was incorrect, was this straight-forward hallucination, or something more subtle, like a token being dropped or double-counted? In my work with LLMs at Limit, this is something I’ve seen frequently: LLMs are biased toward interpreting the beginnings and ends of prompts, and the longer your prompt, the greater the probability that instructions from the middle are not followed.

There’s an arithmetic way to test this. It relies on the fact that every integer greater than 1 is the product of a unique set of prime numbers. For example, the prime factorization of 38 is 2 and 19, and the prime factorization of 8 is 2, 2, and 2.

Therefore, I could ask GPT to multiply a set of prime numbers, and then I could compute the prime factors of the number it returned, and check the difference. By way of example, suppose:

• My prompt is “multiply 2, 3, 5, 7, 7”

• GPT returns 2646

• The unique prime factorization of 2646 is 2, 3, 7, 7, 9

• Therefore, while GPT multiplied the digits 2, 3, 7, 7, it omitted the 5 from the prompt and included a 9.

However, the results were nothing like this. Even for a sequence of only 5 digits, any overlap in prime factors seemed no more significant than chance. An example result below:

• Prompt: 37, 37, 47, 47, 127

• True value: 384,063,367

• GPT result: 1,213,787,489

• GPT result prime factorization: 29, 41854741

Or:

• Prompt: 11, 13, 19, 43, 47

• True value: 5,491,057

• GPT result: 203,679,161

• GPT result prime factorization: 7, 4973, 5851

I’ll spare you the many other examples. Unlike the results in addition, when it comes to multiplication, the results are wildly off. There were no close approximations to be seen in the results, and I couldn’t find any relationship between the prime factors I was using as input and the ones I was getting as output. After looking into it more, I found a paper extending this topic, Positional Description Matters for Transformers Arithmetic, which you may find interesting.

In this essay, I’m going to make exactly that case: translation is the next big thing. Language translation as a concept has been so normalized for so long that progress in it and related fields is hugely underrated, especially in its future economic impact. While we are early on in an artificial intelligence boom and there are many open questions as to how and where exactly it will play out, I believe that one of the highest-confidence AI bets you can make is on ubiquitous language translation, which will have a significant globalizing economic and social impact, perhaps on par with the invention of the smartphone. Let’s dig in.

(1) The Tech

While traditional, text-based language-to-language translation has gotten much better in recent years,1 the key is that several other technologies have become really good in parallel, and are coming together productively:

• Speech-to-text (i.e. automated transcription)

• Text-to-speech (i.e. machine-generated voice)

• Optical character recognition (OCR)

• Audio and visual style transfer

When you put them together, magic emerges. For example, you can get perfectly translated videos,2 even with all the lip movements adjusted to match the sound.

Importantly, this is not some highly proprietary, secretive R&D effort: it’s so simple and accessible that solo researchers can put together open-source versions that get the job done. Part of the reason why these technologies will be so disruptive is that their building blocks are often easy-to-work-with pieces of open-source software. The technology is already here: we are on the cusp of it going mainstream.

From dubbing videos, it’s only a short leap to doing it in real time: instantaneous language translation. Imagine walking around a foreign city with your Airpods in, and the chatter of another language becomes English in your ears. It’s been a long time coming: Bose and Google tried it back in 2018, and Timekettle seems to have gotten it to work today.

It’s a safe assumption that not too far in the future, you will have automatic, instantaneous translation always on. If you’re on a Zoom call with folks who don’t speak your language, then Zoom will translate it in real-time. If you’re walking around a city where you don’t speak the language, but you’re wearing a pair of inconspicuous smart glasses (Meta Ray-Bans below), they will overwrite the street signs in English. You could go on a date with someone who doesn’t speak your language — and so long as you’re both wearing translation earphones, it could go just fine.

In general, I believe that people will interact with the world around them more and more through digital interfaces. Many use cases have been cited for smart glasses and earphones — navigation, digital assistant, recording, etc. — but it seems clear that translation is at the top of the list, both in terms of usefulness and ease of implementation. It is too easy and the benefits are too large for this not to become the case.

(2) The Impact

The key term for thinking about internet-age economy and society is innervation:3 to create a nervous system for a large organism. Historically, humanity was a large set of mostly-independent actors, and it took a very long time for information to spread from one group of humans to another. What the telegraph and radio started to do — and the internet kicked into high gear — is to give humanity a nervous system: a centralized, instantaneous way to transmit information across all people.

In this framing, ubiquitous translation is an extremely powerful agent of innervation. Most people do not speak one another’s languages, which means that the true extent of internet-enabled connectivity today is smaller than it seems. While it is true that 5.3 out of 8.1 billion people have been connected via the internet, they congregate in language-specific groups. For example, I don’t speak Russian, Arabic, or Japanese, so I’m mostly disconnected from those pockets of the internet. The below table shows how small these language groups are, relative to the size of the internet’s full population.

From the mid-90s through the early 2000s, the story of internet adoption was mostly about desktop users in developed nations getting broadband access, amounting to about two billion people. From about 2010 onward, the story of the next three billion people coming online was mostly about smartphones: the great agent of global innervation was the inexpensive smartphone with data, bringing the internet to mobile-first communities all over the developing world.

It is in this sense that ubiquitous language translation could have an impact on par with the smartphone: not by bringing more folks online, but by greatly increasing the connectivity of people on the internet. Simply put, given instant, always-on translation, everyone will be able to communicate better,4 with far more people than before.

Our cultural melting pot is about to become much larger, and the globalizing effects will be massive. There will be gains: for example, in exposure to a greater diversity of cultures and viewpoints, a higher degree of economic participation and exchange, and an easier ability to engage with one’s interests. There will be losses: for example, astroturfing and viral misinformation will be easier to spread, and language-specific internet subcultures will be lost as their users are amalgamated into larger communities.5

But as far as I can see, the largest impact will be on the global labor market. We are still very early when it comes to remote work. The tools for it have become better, and pandemic-times have greatly increased willingness of employers to hire remote employees. But there are two very important considerations:

1. Remote work is still hampered by language barriers. While there are lots of folks all over the globe who speak English at a professional level, far more do not. Additionally, even fluent speakers may still face discrimination against their accents. People are biased against accents, and this likely creates subtle reluctance about hiring remotely in general. Real-time translation can remove these barriers: not just by translating between languages, but also by localizing accents to the listener6 — thereby removing subtle but powerful sources of workplace harassment and discrimination.

2. Implicitly, most of this discussion is about English-speaking firms hiring internationally. But that’s a biased perspective: most firms are not English-speaking. Suppose that you’re a Greek entrepreneur who speaks only Greek, looking to hire internationally: it will be difficult. It doesn’t help you if international applicants speak English, while you and your office don’t.
   
   You may be thinking about always-on, real-time translation expanding the English-speaking internet, and US firms doing more international hiring. Those are true, but translation would provide a much bigger relative benefit — leveling the playing field — to those who are currently outside that ecosystem. Countries like Turkey, Brazil, and Bangladesh7 are the biggest winners from this kind of development.

In sum, the global labor market will become radically more efficient, and its current many national-linguistic subsectors will blend into one. The promise has been made for a long time, but at last, hiring someone five thousand miles away might become literally indistinguishable from hiring someone fifty miles away. How supply and demand will play out is hard to predict, but I suspect that that citizens of developing nations will gain hugely in employment opportunities, while elite knowledge work positions in wealthy enclaves will go remote at lower cost.

(3) Investing

We are on the precipice of a large-scale set of changes that, as a matter of technology and economics, seems inevitable. It raises questions about capital allocation: how do you best bet on this? That’s not easy to answer. The challenges are:

1. In predicting second-order consequences;

2. Greater efficiency does not always create opportunities to capture profits;

3. This is a long-term secular trend that may not take place overnight, and positioning today for a five-, ten-, or fifteen-year trend is not easy.

Regardless, I’ll provide some thoughts below.

Invest At the Fundamental Model Layer?

Probably not for me. My first hunch is that future generations of LLMs will outperform state-of-the-art machine translation systems. My second hunch is that open-source LLMs will probably perform just as well as proprietary LLMs in this respect in the long run. While I see some fields where proprietary fundamental models may maintain some perpetual utility edge, I don’t think that’s the case here.8

At the Fine-Tuned Layer?

Translations are contextual! As I suggested in footnote 1, one of the great perks of the architectures at play is the ability to apply context and stylize the output: the desired style of translation will differ depending on whether the input is, for example, a contract, a patent, a sales email, a poem, or a novel. If I’m listening to a Spanish podcast translated into English, I might want it narrated in the voice of Brian Blessed; or if I’m reading shareholder letters from Japanese companies, I might like them written in the crisp prose of Steve Jobs. While there will certainly be general translation models, you may expect to see thousands of fine-tuned stylistic models. I am uncertain whether this yields a single venture-scale opportunity or an artisanal cottage industry.

At the Interface Layer?

To me, this is the more likely bet. I mentioned that I expect to see more and more digital interfaces that people use to navigate the world around them. While that includes things like smart glasses and earphones, it also includes regular desktop and mobile software. Imagine an application that is always on and translates any foreign words that pop up on your screen, before you even see them. Imagine an audio plugin that translates any language coming out of your speakers.

• Software: there is a lot of diverse opportunity for these translation layers, and the field is fresh. Some competition is starting to crop up in the video translation space, but it’s still early. Due to the accessibility of the technology, there is some danger of commoditization, so startups seeking to sustain significant profit margins long-term will need to think about network effects and moats. Regardless, there certainly exists some opportunity.

• Hardware: this gets a little more interesting. The space is wide-open, and the user experience offered by these physical devices will be paramount. Surely it’s possible to make a nicer product than Timekettle. This may be a good arena for a talented product designer to build something excellent and establish a quality moat. It’s noteworthy that while Apple and Meta should be seen as formidable long-term competitors, Apple hasn’t really managed to integrate AI with its hardware productively. It is surprising that Siri is so far behind nowadays; perhaps this is an area of institutional weakness for Apple that can be attacked.

At the Geographic Level?

Another approach is to focus on the downstream consequences of translation. The effects on the labor market, accelerating globalization, etc. lend themselves well to betting on labor marketplaces, high-skilled education targeted at geographies with strong labor/wage arbitrages,9 remote work enablement, collaboration software, and so on. The main question on these opportunities is whether they are more ripe for new entrants, or for incumbents with existing network effects.

Finally, while we have discussed labor markets at length, I also suggested earlier that the largest relative benefit might accrue to entrepreneurs who are currently outside the English-speaking internet ecosystem. There are many geographies that currently do not have globally significant entrepreneurship, but do have the talent and the regulatory environment for it. They just need the global language access. For example, look at how successful South Korea has been in exporting culture (film, television, music), punching far above its weight on a global scale: but not yet in software. As language barriers fade, this too shall come.

Thanks to Evan and Gavin for their comments and feedback on this piece.

I grew up in Europe (mostly Germany, Denmark, Switzerland). I had never even set foot outside the continent until I was 18, when I moved to the United States. I have lived here for 12 years now, with most of that in San Francisco. This gives me some perspective on the disconnect between European attitudes and the creation of prosperity by technological advancement. In this essay, I will cover:

1. Why the AI Act is bad;

2. Providerism as the reason why Europe fails to build technology and create wealth;

3. What this means for Europe;

4. How to fix it.

(1) Why the AI Act is Bad

If you read a summary of the AI Act Briefing, one thing will stand out to you immediately: it’s all regulated at the application-level. The flavor of regulation is stuff like “you’re not allowed to use AI to exploit vulnerable groups,” “if you’re using AI for educational purposes, it has to pass this set of EU tests”, “you can’t use facial recognition for these forbidden purposes.”1 This style of regulating-by-patchwork-of-many-distinct-but-related-rules is infamously ineffective:

• Because it lacks unifying/general principles, new regulations will need to be added for every new future case.2 There’s a new application? Let’s spend a thousand hours of lawmaker time on adding one more rule set.

• The specificity of the rules ironically makes it unclear whether they apply or not, which invites adversarial lawyering: “is my application really using biometric data? No, it’s metadata…” or “does my application do forbidden social credit scoring? No, we’re really selling an actuarial risk assessment and you choose how to use it…”

• On the other hand, the AI Act regulates techniques so basic that most technology companies will be technically out of compliance in some way from day one.

This is regulation at its worst: broad, sweeping, everyone’s technically out of compliance in some way, but you can pay to fight it. Obedient, rule-following entrepreneurs will waste all their time and money trying to comply with every paragraph of the AI Act. More Realpolitik-minded entrepreneurs will position their business such that it’s in a gray area, assume that if they’re successful, they’ll get sued one day, and save some funds for that eventuality.

And if they get sued, what are the penalties? Up to 7% of global annual turnover. This is a farce. If the EU’s contention is that this technology is so dangerous that it requires EU-wide regulation, then the penalties should actually be a lot higher. If there’s a bad actor running a massively abusive AI business, and the biggest threat they face is a 7% revenue penalty that they can probably knock down to 2% after a few years of litigation — then that’s no deterrent at all! These businesses run at 75% gross margins. You have one year that runs at 73%? Doesn’t matter.

This puts the AI Act in the dismal middle ground of regulation: annoying enough to dissuade legitimate entrepreneurs, toothless enough to not prevent large-scale abuse. I am shocked that the AI Act does not include any capability for banning something that would actually be dangerous, like a nation-state-funded, AI-optimized propaganda machine masquerading as a social network. If they can’t ban products, then it’s not consumer protection: it’s just wealth extraction.3

(2) Providerism

It is extremely telling that the AI Act’s regulations are all the application-level. The AI Act is drafted from the perspective of a consumer rather than a producer.

I do not endorse regulating AI technologies at this point in time. But for the sake of argument, if you wanted to regulate AI, I think you’d want to regulate somewhere at the production level, not at the consumption level.4 Why is it that the EU regulators are focusing entirely on the consumption level?

Well, because they are consumers. Europe is the continent of consumption. This is deeply ironic, because Europeans will thumb their noses at America and call it a consumerist society: runaway fast food obesity, endless billboard advertising, hapless folks drowning in credit card debt. But while America may be consumerist at the micro-level, it is highly productive at the macro-level: the US makes tons of great stuff. From medicine to fundamental scientific research to technology to space travel, we’re leading the charge. European individuals may not be consumerists, but Europe is a macro-consumer: virtually everything of value comes from elsewhere.5

I didn’t really get this until I moved to San Francisco. I had never in my life met people who make stuff. In Europe, my parents worked for non-profits. The parents of my friends were mostly middle managers, financiers, or professional service providers. Living in Silicon Valley is profoundly different, because the people you meet are working on building things that you use. It is hard to articulate just how colossal that difference in exposure is. In Europe, I used computers all day — but never gave any mind to where computers actually come from: you buy them at the store and that glosses over the abstraction. It feels like I was sleepwalking through economic life.

I call this Providerism: the ability to ignore political-economic reality because everything is provided for you, and the underlying mechanics and costs are abstracted away. Europeans may not be consumerists, but they are hardcore providerists. Growing up, virtually every consumer good I interacted with was made in Japan or made in China, and in 18 years that never gave rise to more than 15 minutes of conversation. The goods that you want appear from far away in your local store: they are provided. And if you fall on hard times and cannot afford these goods? The state will provide the basics. If you’re 22 and don’t know what to do, the state will provide more time and will provide a master’s degree. Or two. And if there is war, the state will call NATO, and NATO will provide defense.

Just before the final draft of the AI Act, Thierry Breton of the European Commission posted a picture of the team hard at work:

How fitting: the European Commission assembled around an iPad. On the back, in fine print, it will read: Designed by Apple in California. Assembled in China. There is no European iPad. There is no European computer. There is no European search engine. There are only European consumers, to whom things are quasi-magically provided, and so they regulate the providing and consumption of those things.

The European Union is so deep in Providerism that it does not recognize how far removed it is from the production of things of value. This myopia is a great peril for the citizens of Europe. Every year that passes, Europe slips deeper into complacency as goods and services are provided from abroad, while regulators are writing missives and assessing fines in impotent, play-acting gestures of agency. This encumbers European technological entrepreneurship by weakening domestic entrepreneurial network effects6 and setting ever higher local barriers to entry. This is terrible, because for Europe, the main way to maintain and achieve long-term prosperity is obviously to innovate technologically and produce things of value.

(3) The Writing on the Wall

Europe is falling behind. It largely missed the internet and personal computing booms, and now it sits in danger of missing the coming AI boom. Today’s Europeans are not yet poor — they are still living off the prosperity created by prior generations, and that enables their passive consumption7 — but tomorrow’s Europeans may be.

Further, European regulator gamesmanship can’t go on forever. Foreign firms don’t just have to comply with the AI Act, but also with GDPR and many other EU standards. At some point it’s all just too cumbersome and foreign firms will play hardball.8 In that situation, the EU will lose because they mostly don’t have domestic alternatives to major foreign software products, and EU consumers9 are not going to be willing to go back to the stone age.

This is a poor setup, both for European consumers and for the prosperity of Europe. The EU needs to course-correct swiftly and decisively.

(4) How to Fix It

Europe needs to escape Providerism, discourage complacent reliance on outside goods and services, and encourage the virtue inherent in making new things. This will require tremendous leadership. The task ahead is no small feat, and it will be unpopular. People like having things provided to them, but this state of affairs cannot last. From a public-budgets perspective, you already see Providerism breaking all over the EU. The great task ahead for European Regulators is to facilitate wealth creation, not wealth consumption. My recommendations:

1. Deregulate Technology. Throw it all out and start over. In the future, regulate big negative externalities, not imaginary potential ones. Do not presume need.

2. Federate. The EU needs to make it easy for businesses to expand across the entire EU market. Right, now there are many legal and financial barriers to doing so.

3. Foster a Production Mindset. It is important for Europe to repatriate some production of cutting-edge goods and services: not even for economic reasons, but just culturally. Focus on making things.

4. Lean into the AI Boom. The AI boom is a unique opportunity for Europe because it is tied to academic research, and the EU’s many universities are graduating great talent with no debt. Mistral is an amazing company: there should be more!

I combed through the Internet Archive and OpenAI’s non-profit filings to try to make sense of OpenAI’s governance. Below, I have attempted to chronicle the composition of OpenAI’s Board over time, point out the conflicts, and you can see how we got to the earthquake yesterday. You can skip to the end for my summary perspective.

December 11, 2015

OpenAI is founded.

Board Directors:1

• Elon Musk (Co-Chair)

• Sam Altman (Co-Chair)

December 31, 2016

OpenAI’s Form 990 public filings for calendar year 2016 show the Board Directors:

• Elon Musk

• Sam Altman

• Chris Clark

• Jonathan Levy (?)

Chris was the initial COO of OpenAI, and still works there to this day. Jonathan Levy was listed as Secretary/Treasurer, and may have been a trustee rather than a Director. It’s unclear from the filings.

March 2017

Open Philanthropy donates $30M to OpenAI. Holden Karnofsky, the founder of Open Philanthropy, joins OpenAI’s Board of Directors.

December 31, 2017

OpenAI’s Form 990 public filings for calendar year 2017 show the Board Directors:

• Elon Musk

• Sam Altman

• Chris Clark

• Holden Karnofsky

• Greg Brockman

• Ilya Sutskever

February 20, 2018

Elon Musk is removed from the Board. The official press release proclaims a departure to avoid potential conflicts, but journalists report leadership disagreements culminating in Elon proposing a takeover and being rebuked.

Board Directors:

• Greg Brockman

• Ilya Sutskever

• Holden Karnofsky

• Sam Altman

We don’t know exactly when Chris Clark was removed from the Board.

March 2018

Reid Hoffman, founder of LinkedIn and General Partner at Greylock, joins the Board. I couldn’t find a press release or official announcement, but Reid’s LinkedIn profile has the dates.

April 24, 2018

Adam D’Angelo, CEO of Quora and former Facebook CTO, joins the Board. This follows the February Board changes, where the OpenAI blog post had noted the intent to add another Director to the Board soon.

September 2018

Sue Yoon joins the Board. Sue’s exact employment at the time was unclear — she was previously an EIR at First Round, and in the coming months would lead robotics projects at Google. Similar to Reid Hoffman, I couldn’t find an official announcement, but her LinkedIn profile has the dates.

December 31, 2018

OpenAI’s Form 990 public filings for calendar year 2018 list the Board Directors:

• Sam Altman

• Sue Yoon

• Holden Karnofsky

• Greg Brockman

• Ilya Sutskever

• Adam D’Angelo

• Tasha McCauley

I could not find anything in the way of a source on when, or under what circumstances, Tasha McCauley joined the Board.

March 11, 2019

This gets strange. There’s an OpenAI blog post listing the Board Directors:

• Greg Brockman

• Ilya Sutskever

• Sam Altman

• Adam D’Angelo

• Holden Karnofsky

• Reid Hoffman

• Shivon Zilis

• Tasha McCauley

Note the unannounced elevation of Shivon Zilis (previously an advisor) and the unannounced departure of Sue Yoon. Weirder yet, OpenAI published its new homepage just that day, still listing Sue Yoon as a Board Director, and not Shivon Zilis.

November 2019

Sue Yoon leaves OpenAI’s Board, according to her LinkedIn. The OpenAI Website still lists her (and not Shivon Zilis) as a Board Director.

December 31, 2019

Another year, another OpenAI Form 990 public filing, listing the Board Directors:

• Ilya Sutskever

• Greg Brockman

• Sam Altman

• Reid Hoffman

• Sue Yoon

• Holden Karnofsky

• Adam D’Angelo

• Tasha McCauley

Note that Shivon Zilis still doesn’t appear in the list of Board Directors. Was the March 11, 2019 blog post just wrong? Did someone in marketing make a mistake and no-one caught it?

December 31, 2020

Slow news year. The Form 990 public filing for calendar year 2020 lists the Board of Directors, finally including Shivon Zilis and not Sue Yoon:

• Ilya Sutskever

• Greg Brockman

• Sam Altman

• Reid Hoffman

• Shivon Zilis

• Holden Karnofsky

• Adam D’Angelo

• Tasha McCauley

I couldn’t find a public statement on when Shivon actually joined the Board, other than the March 2019 blog post that may have been in error.

May 3, 2021

Will Hurd, Republican member of the House of Representatives, and former CIA agent, joins the Board.

September 8, 2021

Helen Toner, Director at Georgetown’s Center for Security and Emerging Technologies, and formerly of Holden Karnofsky’s Open Philanthropy, joins the Board.

Fall 2021

Holden Karnofsky resigns from the Board, citing a potential conflict because his wife, Daniela Amodei, is helping start Anthropic, a major OpenAI competitor, with her brother Dario Amodei. (They all live(d) together.) The exact date of Holden’s resignation is unknown; there was no contemporaneous press release.

Between October and November 2021, Holden was quietly removed from the list of Board Directors on the OpenAI website, and Helen was added (Discussion Source). Given their connection via Open Philanthropy and the fact that Holden’s Board seat appeared to be permanent, it seems that Helen was picked by Holden to take his seat.

December 31, 2021

OpenAI’s Form 990 public filings list the Board Directors of the 2021 calendar year:

• Ilya Sutskever

• Shivon Zilis

• Greg Brockman

• Will Hurd

• Sam Altman

• Reid Hoffman

• Holden Karnofsky

• Adam D’Angelo

• Tasha McCauley

• Helen Toner

The fact that both Holden and Helen are listed here is not surprising; both of them were Board Directors at points in 2021. (It does not necessarily imply that they were both on the Board at the same time.)

2022?

There did not appear to be any Board events in 2022. The Form 990 does not appear to have been filed as of the time of writing.

January 2023

Reid Hoffman steps down from the Board, citing the need to avoid potential conflicts with his investments. While this was reported in March 2023, according to his LinkedIn profile’s dates it happened in January.

March 23, 2023

Shivon Zilis resigns from the Board for reasons unknown. (Commentators speculate that her resignation is over conflicts due to her bearing Elon Musk’s children, but that is ultimately just speculation.)

July 13, 2023

Will Hurd resigns from the Board, citing the need to focus on politics/his 2024 Presidential campaign. (Three months later, in October, he drops out of the race. I don’t know what to make of that.)

November 17, 2023

Sam Altman is fired from OpenAI and the OpenAI Board in a surprise meeting of the Board (except Greg). Minutes later, in a separate surprise Board meeting, Greg Brockman is removed from the Board (and as Board Chairman).

Board Directors:

• Adam D’Angelo

• Helen Toner

• Tasha McCauley

• Ilya Sutskever

Summary/Perspectives

The first thing that sticks out to me is that there have been, for several quarters, two significant conflicts of interest on the Board:

• Adam D’Angelo founded and appears to be spending all his time on developing Poe, an AI chat platform partially leveraging and partially competing with OpenAI. In my opinion, that’s too close. Reid Hoffman resigned over potential indirect investment conflicts; Adam’s conflicts are more direct. Best practice would’ve been for Adam to resign when he began working on Poe.

• Helen Toner and and Tasha McCauley are jointly participating in a highly ideological AI governance organization. As Alex Konrad noted: “McCauley currently sits on the advisory board of British-founded international Center for the Governance of AI (GovAI) alongside fellow OpenAI director Helen Toner.” It turns out that the advisory board is six people, and beyond Helen and Tasha, the other four include: one who currently works for Open Philanthropy, and another is the founder of GovAI, which was mostly funded by… Open Philanthropy.

  • For OpenAI’s six-person Board, it was inappropriate for two Board Directors to be this strongly associated with an ideological organization and therefore so strongly and predictably aligned in their voting. It calls into question the independence of their votes.

  • Due to Open Philanthropy’s link to major OpenAI competitor Anthropic, there’s also a hint of corporate conflict here. If I were on OpenAI’s Board, I would have requested at least for Tasha2 to relinquish her seat for a true independent Director.

Secondly, what the hell happened in Q1/Q2 2023?

• Reid, Shivon, and Will all resigned, and the Board did not line up replacement Directors? By comparison, when Elon resigned in February 2018, Adam joined two months later.

  • Were these seats just left vacant, with a deadlocked Board unable to agree on new Directors to appoint?

• They all resigned within a few months of one another despite OpenAI looking like the rocketship of the century? Something feels a little odd about that.

It seems less likely that the November firings would have happened if Reid, Shivon, and Will — or even just one of them! — had still been on the Board, or replaced with an appropriate representative. With this view, the outcome was almost predictable given these two facts:

1. The thinning-out of the Board from 9 to 6 members;

2. Half of those 6 members carrying conflicts in their relationship with OpenAI!

We will find out, in due time, the motivations of the Board in the November firings. Right now they aren’t clear. It isn’t known whether anyone acted inappropriately, and I am not accusing anyone (to be clear, even the Board Directors that I consider conflicted) of having acted subject to conflicts of interest. But the 2023 changes made drama likely, no matter what. A Board is a delicate balance of perspectives and interests. When a Board rapidly changes in size, rarely is the remainder left well-balanced. Potential conflicts only make the balancing act harder.

Final Thought

Governance can be messy. Time will be the judge of whether this act of governance was wise or not. But you should note that the people involved in this act of corporate governance are roughly the same people trying to position themselves to govern policy on artificial intelligence.

It seems much easier to govern a single-digit number of highly capable people than to “govern” artificial superintelligence. If it turns out that this act of governance was unwise, then it calls into serious question the ability of these people and their organizations (Georgetown’s CSET, Open Philanthropy, etc.) to conduct governance in general, especially of the most impactful technology of the hundred years to come. Many people are saying we need more governance: maybe it turns out we need less.