The obvious question is whether the Service Led Growth (SLG) approach can drive the rapid revenue scaling that is necessary to achieve venture returns. There’s a philosophical discussion here that we won’t get into about founder motivations, investor alignment, and the risks of short-termism to a business’s terminal value. You can be Bezos and “trade margin for moat” to build a multi-generational behemoth, or you can be a tourist-founder looking for a quick flip to land a fancy VP role in an acquihire once you’ve spent all of your seed money, or you can be somewhere in between. All are respectable life paths.

The interesting thing to me though, is that a second order effect of AI destroying the capital needs for product development may be that startups no longer need to pick one GTM. What if SLG could be paired up with PLG in a “greater than the sum of its parts” way?

In March, as we were going through our customer discovery and ideation process, I shared the following diagram with my cofounders to structure how we could think about our activities at this early stage:

SLG provides an edge in understanding enterprise problems deeply and spotting non-obvious productization opportunities. PLG provides rapid feedback, wide top-of-funnel, and brand identity to close large enterprise deals.

The popular maxim in startup-land is that trying to do both can kill your company so you should only stick to one. But what if “this time is different”?

1. The costs of building and distributing are falling precipitously. The scarce resource is a precise articulation of real underserved problems and a product positioning that becomes the obvious choice to solve them.

2. There is a lot of buyer fatigue around shallow products / wrappers that don’t add lasting value beyond shiny demos. People want stuff that solves their specific problems well.

3. In order to make the best strategic decisions, an early stage application layer team needs clarity about both the ground reality of enterprises (to know what can be sold) and the rapidly shifting contours of the AI research/tech/product frontier (to know what can be built). SLG helps with the former and PLG with the latter.

Obviously, time and bandwidth is the limiting factor. One may need to double down on a single GTM once further along to maximize burn efficiency. But at an early stage, the hybrid approach could uncover differentiated perspectives about the market that serve as the “earned secrets” to build your Thiel-esque monopoly on.

The core idea is that we are entering the AI Mainframe Era, where the most transformative AI systems are not those working under employees’ guidance, but those standing above them—acting as central hubs, orchestrating operations with a bird's-eye view.

Here’s the argument:

1. Real application-layer value in AI has yet to be created. See AI’s $600B Question.

2. All current product thinking around AI is basically some offshoot of the ChatGPT model: an AI assistant that helps users/employees accomplish tasks. In the B2B SaaS setting, this has morphed into the idea of an “AI Teammate” that replaces/augments certain job functions and does it faster/better/cheaper.

3. The “AI as human assistant” or “AI teammate” paradigm has two major problems: (1) it overestimates how close AI is to human way of thinking—leading to unmet expectations and uncanny valley, and (2) it underestimates how much AI could benefit us in other ways that better suit both AI’s and our strengths.

4. True advantages of AI come from scale and polymath-ism: aggregating large amounts of diverse context, extracting patterns from it, while crossing traditional boundaries of individual human expertise. This goes beyond what we expect a single human or groups of humans with homogenous skills to be able to do in an organization. Hence, by definition, an AI Assistant tasked with automating existing activities of individual employees is barely scratching the surface of what’s possible with AI.

5. To fully leverage the scale advantage of AI requires designing AI systems as the central hubs at the core of our organizations and networks: plugged into thousands of information streams, churning out insights, decisions and recommendations for allocating resources and directing efforts — with human oversight, of course. This is conceptually like the “mainframes” of the mid-20th century, except at a much much grander scale, enabled by several decades of Moore’s Law at work.

6. The concept of “AI Mainframes” is not new: it is widely present in fiction, and is parallel to the concept of AI Assistants. See this table on Human-like vs Alien-like AI from my previous post.

7. AI Mainframes will let us re-imagine vast categories of enterprise software: ERP, Business Intelligence, IT Consulting, Management Consulting, etc.: trillions of dollars of value creation/destruction. Most enterprise software, processes and business models are predicated on working around the inherent limits of the individual human brain to let large numbers of humans coordinate value-creating business activities. All of this must be re-thought when you place an AI brain that far surpasses the context and expertise limits of human brains at the center of an organization.

8. As pre-training scaling laws slow down, inference time scaling is the “hot shit” for the foreseeable future. [See Generative AI’s Act o1.] The AI Mainframe paradigm can more naturally absorb inference-time scaling of reasoning capabilities than AI Assistants can — imagine a central, thinking entity that quietly performs System 2 reasoning on vast data streams behind the scenes and “comes up to surface” when human contact is necessary. This is better than watching AI Assistants “think” for minutes while you stare at your screen.

9. We’re collectively drawn to the AI Assistant paradigm because it is uncomfortable to think of AI as a mainframe hub rather than an agent subservient to humans (see: Skynet). But as I quickly learned when I took a surfing lesson off the coast of Maui once—either you ride the wave or are ridden by it and left with a salty aftertaste. The steady march of technological progress and the gravity of profit-seeking efficiency maximization in capitalistic enterprises both point towards the AI Mainframe direction.

10. I don’t believe for a second that the dystopian nightmare is the default outcome along this path; rather we are capable of building AI systems that remain under human oversight while providing novel value to society.

11. The AI Mainframe era will create many more jobs than it takes—if you believe that the limiting factor to massive GDP growth is inefficient mis-allocation of resources by human managers, then an AI system making (some/most/all?) decisions should be a welcome upgrade.

12. We may have to pass through an AI Mainframe era in order to unlock the branch of the technology tree that gets us to the coveted vision of truly useful AI personal assistants — what Ben Thompson calls the Consumerization of AI — this task might be up to the next generation of AI-native humans.

13. This post opens up more questions than it answers; long road of building ahead and I’m excited.

Now, answer this: if this spectacle was the first broad public interaction with LLMs, instead of a friendly human-like chat interface, would we still collectively map LLMs onto the scale of "human/superhuman" intelligence, rushing to overhaul our B2B SaaS product roadmaps with AI features? Or would we regard them as an alien intelligence, something to be carefully studied and marveled at for a while before we even think of integrating them into all aspects of our personal and work lives?

Obviously, this timeline is pure sci-fi1, but it highlights a key point: LLMs are smart, yet their intelligence spikes in different areas from ours. This divergence has led to both fascination and confusion. The prevailing mistake, especially in Silicon Valley VC-land, is to frame AI as augmenting or replacing human output—an easy but misguided assumption that, at best, leads us on an off-ramp from the path towards optimal human-AI interface design, and at worst, creates unattainable expectations that will inevitably lead to bubble-bursting unhappiness.

Allow me to explain.

The Alien Intelligence Among Us

LLMs (and Foundation Models in general) are already superhuman in several areas:

• Memory: Can recall thousands of details from immense datasets, without blinking. Sometimes they confidently get things wrong or invent their own facts, but hey, aliens like to have fun too.

• Pattern Matching Across Oceans of Data: Can absorb data from the internet (including that novel you abandoned in high school), distilling patterns from everything they’ve read.

• Language Translation at Scale: Imagine being instantly fluent in every language ever and being able instantly to create essays, poems, code, websites—you name it.

• Handling Long Contexts: Can take in a sprawling, 200-page technical document or a 2 hour long YouTube video and maintain coherence across the entire source, while human attention context window sizes have only been decreasing (with a precipitous drop since the day TikTok was invented).

That said, there are some very human things they’re terrible at:

• Ingenuity and Creativity: Sure, they can remix patterns and generate convincing text, but they’re not going to come up with the next Lord of the Rings or solve world hunger. They’re only as creative as their training data allows.

• Common Sense Reasoning: Ask an LLM whether it's safe to use a toaster in the bathtub, and it might not respond with the level of panic that such a question deserves. LLM reasoning capabilities have been getting better over time, but it’s unclear if LLMs will truly reason in the way humans do (this is the blue dress/gold dress question for nerds).

Just like an alien species might possess strengths we can't fathom but lack traits we consider essential, LLMs excel in areas that are beyond human capacity, yet struggle with the very things that make our intelligence unique. Their strengths and weaknesses suggest a kind of intelligence that operates on a different wavelength entirely—something to be studied, leveraged, and complemented, but not mistaken for a mere extension of our own thinking.

Why the Concept of AI Teammates Is Flawed

Silicon Valley loves a good narrative, and the current favorite is the "AI teammate" concept. You’ve probably heard it in many forms: AI is going to be your co-worker, your assistant, maybe even your work best friend. If you haven’t, just drive up on 101 and look at any billboard:

But this analogy, while convenient in the short term, will be prove to be a hindrance in the long run.

Anthropomorphization: The Shortcut to Uncanny Valley

We humans have a habit of attributing human-like qualities to machines—sometimes to hilarious effect, sometimes to tragic disappointment. Remember Microsoft's Clippy? It was supposed to act like an intuitive assistant, but instead it quickly became the most infamous office troll, popping up with suggestions that no one needed.

More recent examples include chatbots that attempt to mimic conversational flow. Sure, they can string together sentences and may even tell you a joke, but when tasked with solving complex, context-heavy problems, they can veer off into strange territory, leaving users baffled.

History of Mismatched Expectations

We’ve been down this road before with almost every major advancement in computing interfaces. The late '90s and early 2000s saw AI assistants and speech recognition systems that were heavily marketed as "game-changing," but they weren’t nearly as capable as advertised.

Voice assistants and chatbots in the 2010s, for example, promised conversational fluency, but users quickly discovered that asking them to "play music" or "set a timer" was about the limit of their usefulness — I lived through a whole voice assistant product cycle at Google/Meta and have battle scars to show for it.

Similarly, voice bots introduced to replace human labor in call centers didn’t live up to the expectations, frustrating both customers and businesses alike with their inability to handle even mildly complex requests.

This is the danger of AI anthropomorphization: we expect too much, then we end up hating it. Like any toxic relationship.

The Gen AI Era: New Dog, Old Tricks

Since the arrival of ChatGPT, the way we interact with AI has rapidly evolved. We’ve gone from simple chatbots to "copilots" assisting in specific tasks, and now there’s talk of AI as "coworkers." Each step in this progression has seen AI systems gain more autonomy and context-awareness, but this one-dimensional view of progress misses the broader potential of what AI can really offer.

It's the same old mistake of attributing too much human-like behavior to our machines, with all the same risks. While it may seem that “this time is different” and we now finally have tech that can behave as a human, that’s not the fundamental reality of LLMs and sooner or later the cracks will show up in the seams of every product that pretends otherwise.

Chatbots: Basic but Helpful

Chatbots like ChatGPT, Gemini, Perplexity and Claude, were the first widespread AI tools, answering a wide spectrum of questions and capturing the zeitgeist. They mimicked conversation but quickly hit limits when faced with complex tasks. Useful, but ultimately constrained by their lack of deep context.

Copilots: Smart Assistants

Next came "copilots," like GitHub Copilot and Google's Smart Compose. These systems suggest relevant actions based on the task at hand—anticipating needs, completing code, or refining writing. This shift came with a key improvement: context-awareness. These AI systems have access to more information about the tasks you're performing, so they can offer smarter, more relevant suggestions—but the human operator still holds the reins.

Coworkers: Independent Agents

Recently, AI "coworkers" are being developed to take on more independent tasks, for example in DevOps, where they handle alerts and basic infrastructure problems autonomously. The idea is that AI coworkers can take some of the load off human workers, acting as reliable partners.

This transition from passive assistant to proactive coworker represents a significant increase in autonomy. These AI agents are expected to not only understand the context of tasks but to take action without waiting for human approval. However, this shift assumes that more autonomy and context awareness automatically lead to more useful and powerful AI systems. Reality, as always, isn’t that simple.

One-Dimensional AI Progress Limits Potential

It's tempting to view AI progress as a straight line from chatbots to autonomous coworkers—more context, more autonomy, more intelligence—but this oversimplifies AI’s role:

• More Autonomy Isn't Always Better: AI doesn’t need to fully take over tasks to be useful. Copilots that make suggestions often outperform autonomous systems because they amplify human abilities rather than replace them. In many cases, human judgment is critical.

• Not One-Size-Fits-All: AI roles—chatbot, copilot, or coworker—aren't hierarchical but situational. Sometimes a simple chatbot suffices, while in other cases, a copilot or coworker excels. The right role depends on the task.

• Sets the Wrong Expectations: Projecting AI onto a linear path creates unrealistic user expectations: when AI fails to act like a human "teammate," it breaks the magic of the product experience, leading to user disappointment. It also steers the industry in the wrong direction, wasting time and resources on misguided ambitions.

• Missed Opportunities: Focusing solely on more autonomy ignores hybrid forms of collaboration, for example AI could suggest well-thought out multiple pathways, leaving the human to weigh pros and cons, or assist with knowledge retention, or augment human decision-making in ways that go beyond automation.

• Human-AI Teaming is Complex: True collaboration requires nuance, flexibility, and trust—things AI isn’t inherently equipped to handle. Viewing AI as a teammate risks oversimplifying the complexities of human-AI interaction.

Embracing Complementarity Over Imitation

So, if AI isn’t going to be your next teammate, how should we think about working with it? The key is to embrace what LLMs actually are: powerful tools that can complement human ability but aren't built to replace it.

Designing Effective Human-AI Collaboration

Effective collaboration with AI requires an approach that leverages its strengths while acknowledging its limitations. Here are a few design principles for better human-AI interaction:

1. Task Specialization: AI isn’t going to invent the next iPhone, but it can analyze vast amounts of market data to identify trends and opportunities. The key is to match AI with tasks that align with its strengths: data synthesis, pattern recognition, and handling of large-scale text processing. Let humans handle creativity, strategy, and decision-making.

2. Defining Clear Boundaries: It's crucial to communicate clearly what AI can and cannot do. Setting realistic expectations avoids the uncanny valley of mismatched hopes. For instance, AI can generate content, but it won’t truly understand the emotional nuances behind it. Should you ask it to write your next novel? Probably not. Should you use it to analyze 10 years' worth of customer reviews to find hidden insights? Absolutely.

3. Symbiotic Workflows: What could a symbiotic workflow between humans and AI look like? One model is humans making high-level decisions while AI handles complex data manipulation in the background. Imagine product managers setting goals, and AI systems autonomously flagging potential issues or opportunities based on real-time data. Could this allow teams to focus on innovation, while AI does the heavy lifting of operations and monitoring? A flipped out model, on the other hand, is where AI suggests the most promising directions to explore, which in turn actuates teams of humans to go and do the work, verify the outputs and report back results to the AI for further directions.

4. Interactive Learning: Human-AI collaboration could also take inspiration from how we work with other human experts. In much the same way that two specialists might engage in iterative feedback, AI systems could prompt humans with new insights, questions, or summaries, and humans could refine and guide the system's direction. Imagine an AI assistant suggesting five different interpretations of data and asking the human operator which approach seems most relevant—turning the user into a sort of "AI coach."

5. Collaborative Creativity: Although LLMs aren't creative in the human sense, they can serve as idea generators, sparking new directions for projects. What if we treated AI as a brainstorming oracle, like a partner for Socratic thinking? This approach shifts AI's role from being a "doer" to an "idea generator"—but with lot more context than the current chatbots or copilots.

Questions to Ponder

• In the tasks/workflows you are building AI solutions for, where is an AI copilot—providing real-time suggestions and support—more beneficial than a fully autonomous coworker, and vice versa?

• How do we design workflows and interfaces that allow humans and AI to operate as partners, not competitors for control?

• What new forms of human-AI interaction can we explore that don’t fit into the "more autonomy" trajectory? Could AI serve as a reflective tool for humans to refine their own thinking, but also double up as an agent executing tasks when instructed?

• How can we prevent the next wave of AI hype from making the same anthropomorphic mistakes?

• What are the implications for AI Safety when we’ve truly embraced the notion that what we have in our hands is an alien intelligence?

Inspiration from Fiction: Human-like vs. Alien-like AI

Technological progress and product innovation is often inspired by science fiction: many innovators grew up with futuristic ideas from movies, shows, and books—like how Star Trek's communicator foreshadowed the flip phone. Myself and folks I’ve worked with to build voice assistant products were all inspired by Jarvis, Samantha, or similar AIs from books/movies, which heavily influenced our product thinking.

However, I’ve come to accept that we need to be careful about which fictional models we choose to be influenced by. Leaning too heavily on human-like AI depictions, like C-3PO or Samantha, can lead to unrealistic expectations. Alien-like portrayals, such as HAL 9000 or TARS, better reflect the reality of today’s AI: smart but fundamentally different from human thinking.

Here’s a list of AI in fiction, categorized by how closely they resemble human-like intelligence versus something more alien:'

There’s a diverse set of mental models to pick from when designing human-AI interfaces (would love to hear about any interesting ones that I missed). The danger lies in viewing AI through too narrow a lens, based on specific fictional depictions that either completely humanize or dehumanize it. When developing products and imagining how AI will interact with us in the real world, we’re better off with a more nuanced understanding—one that blends the right amount of intelligence augmentation with an appreciation for AI's "alien" nature.

Wrapping up

The progression from chatbots to copilots to coworkers may seem linear, but it’s a limiting view. AI isn't destined to become more "human" with each step—it’s a versatile tool with unique strengths that complement human work. Instead of focusing solely on greater autonomy, we should explore diverse ways for humans and AI to collaborate effectively.

While this wave of AI may end up unbundling human labor and bite off big chunks of value creation across industries, let’s not pigeonhole AI into a path that doesn’t suit its nature—or ours.

It's been an incredible few years seeing AI go from research projects that barely work to capturing the popular imagination - while still barely working, one might say :) I spent most of the last decade in research and product orgs at Google and Meta, witnessing this progression up close, refining the skill to separate hype from reality, and yet I've been amazed every few months by new model capabilities and their potential to solve real problems facing us.

And while the 23 months since ChatGPT have felt like a whirlwind to everyone in the industry, I'm setting my sights on the next two decades as novel, durably useful applications will undoubtedly emerge after the giddy hype dies down and the foundational layers of the GenAI stack get solidified. It will be a long arduous slog - the Internet/ Cloud/ Mobile waves took decades to fully play out - and we've yet to see the Google/ Amazon/ Apple/ Facebook/ Salesforce/ Uber/ AirBnB level innovations of the GenAI wave materialize.

The beauty of the tech game is that what seems popular and successful today could become a footnote in history, like an AltaVista or Yahoo!. And billions of capital raised may pale against the ingenuity of small, scrappy, mission-driven teams. It's the infinite game and the possibilities are endless 😀

I'll be sharing more over the coming weeks/months - unfettered by any corporate PR policy - including what I’m building, my many learnings from Big Tech, my takes on the playing field today, and where I see defensible opportunities for startups. For now, here's a line from one of my favorite Hindi songs:

Subscribe now

This scenario is too familiar for any engineer that oversees production services, and it's costing companies in terms of both human and financial resources. I’m convinced that it is time to take a fresh look at how enterprises approach their IT operations management, by anchoring on the newer agentic capabilities of foundation models to reduce the workload on human engineers, and focus on making IT systems truly resilient. In this post, I’ll propose AgentOps1 as this new approach to IT Ops, and present a high level view of how this might play out based on my current understanding of the space.

From Reactive to Agentic: History of IT Ops

Imagine a world where engineers are supported by an AI agent that has already diagnosed the problem, gathered relevant data, and proposed solutions before they even open their laptop. That's the world AgentOps is creating.

The journey from siloed ops teams to AgentOps has been one of breaking down barriers, increasing collaboration, and leveraging technology to handle ever-growing complexity. Building on the foundations laid by DevOps and AIOps, AgentOps represents a leap in IT operations management. At the heart are AI agents – autonomous, intelligent systems capable of not just analyzing data but taking action and learning from outcomes.

Here's how AgentOps transforms each phase of your company's IT operations:

A key insight here, based on my own experience of building production systems for many years, is that every major/minor outage in the history of any product organization is a step towards increased maturity of the production system as well as the organization: new know-how is acquired by the team from each mistake, and in world-class product orgs, such mistakes typically stem out of missed details in intricately complex systems, which are usually beyond the capacity of even very smart people to have figured out a priori. AgentOps helps to institutionalize these learnings and make sure no hard-won knowledge is lost, so that the organization keeps getting better and more resilient with every mistake. In other words, AgentOps makes engineering orgs antifragile.

Why now

• The know-how around GenAI agents is crystallizing in the industry and it is becoming clearer what kinds of tasks will be addressable by agentic workflows:

  • broad, shallow agents (e.g. GPT store, Gemini Gems, Glean Apps, etc) that cater to a wide variety of consumer/enterprise needs and will be mostly built self-serve by end-users or companies,

  • deep, narrowly-focused agents that solve a specific, difficult problem for enterprises, mostly aligned towards either a function or a vertical, e.g. AgentOps targets the SRE function.

• Price/performance of foundation models has dropped significantly and it will continue to go down, making complex agentic loops operating over large amounts of context economical.

• As a higher % of production code switches to be AI-generated instead of human written, there is a naturally higher risk of complex outages of the kind which engineers would be under-equipped to handle alone without AI-assistance.

• Companies will be incentivized to divert a decent fraction of engineering headcount dollar savings obtained from AI-driven coding efficiencies towards better tooling for detecting, resolving and preventing issues, which should increase spend in this category.

The AgentOps Advantage: Agent-Driven Incident Resolution

At its core, AgentOps relies on advances in large foundation models, their emergent reasoning, planning and long context capabilities, and the agentic workflows that will be built around these models, to re-imagine the IT operations space from the ground up. In my previous post, RAGs to Agents, I described the PARMeSAN framework for thinking about agentic capabilities. AgentOps will succeed by hitting most of those capabilities:

1. Retrieval + Long Context: With their ability to process vast amounts of data, Agents can hook into all your production systems to pull in relevant pieces of context towards solving any issue. They don't just see the current issue; they understand the history, correlations across high cardinality data, and the potential ripple effects.

2. Planning and Reasoning: Gone are the days of blind troubleshooting. Agents will employ sophisticated reasoning methods combining LLMs and search algorithms like MCTS, to develop strategic plans for incident resolution, considering multiple scenarios and their potential outcomes.

3. Automated Actions: While human oversight remains crucial, Agents can automate many routine tasks, from log analysis to initial diagnostic steps, significantly reducing time-to-resolution.

4. Memory: Unlike human teams that may struggle with knowledge transfer, Agents maintain a perfect memory of past incidents, solutions, and best practices, ensuring that hard-won knowledge is never lost.

5. Async and Ambient: Even during normal operations, it is critical for the Agent to keep running checks behind the scenes and reasoning over any signals that seem weird, so that issues are caught on time.

Data, Data, Data

AgentOps' effectiveness stems from its ability to integrate and analyze data from a wide array of sources. From your codebase and company documents to system logs and metrics, AgentOps has a holistic view of your IT ecosystem. By analyzing historical and current discussions in email and Slack, AgentOps captures the human context often missing in purely technical analyses. Understanding user metrics and business KPIs allows AgentOps to prioritize issues based on their real-world impact. Finally, AgentOps stays updated on the latest vulnerabilities and best practices in the broader tech community, bringing external expertise to your internal challenges.

The Future of IT Ops is Agentic

The future of IT operations is here, and it's powered by GenAI Agents. Agents aren't just a tool; they're every engineer’s partner in creating a more resilient, efficient, and innovative IT ecosystem.

Well, we can certainly agree that there should be more economic value in “voodoo AI that does stuff” than “monkey jpegs”. So let’s not stand by and watch the dreamy potential of our agentic1 overlords get squandered by the agent bro2 hype brigade, lest we all end up in that worst hell for product builders: a scathing MKBHD review. Let’s bring order to the chaos, starting with a taxonomy that shall hopefully become the common ground upon which magnificent works shall be built. Think of it as a dusty old treasure map tracing an adventure from "glorified autocomplete" to "potentially terrifying super-intelligence" - with all the exciting pit stops in between.

Our framework, PARMeSAN, outlines seven levels of AI systems, each building upon the capabilities of its predecessor. This aligns with Andrew Ng's perspective that "there are different degrees to which systems can be agentic", offering a spectrum of capabilities rather than a binary classification. The goal here is clarity and comprehension over completeness. The primary audience is folks who understand the basics of LLMs and RAG, but get fuzzy around how agents actually work and what do they really do. This post will give you a good mental model to work off of and start asking more pointed questions to the next agent bro that comes along.

The secondary audience is folks who, like me, have been working on agents for a while and feel there’s too much chaos in this space. If this is you, none of this post will be new information per se, but ideally this helps structure your day-to-day discussions a little better.3 The framework is meant to spark conversation and not be a definitive answer by any means. So if you feel weird in your stomach about anything described here, or have a brilliant suggestion, please send that over to /dev/null… err, to me. Send that to ME!

Now let’s jump in!

PARMeSAN

The PARMeSAN framework consists of 7 capabilities that constitute an AI Agent: (P)lanning, (A)ctions, (R)etrieval, (Me)mory, (S)ynthesis, (A)sync and I(N)trospection. Alongside, there are 7 archetypes, each adding a new capability over its predecessor, which qualitatively changes the scope of system, by roughly an order of magnitude, building upwards from plain LLMs to agents.

This framework lets us arrive at a crisp definition of an agent: Agents are digital entities that make plans, take actions on other systems to retrieve and commit information, learn from memories of past interactions, synthesize outputs to meet the user's need, execute tasks asynchronously when required, and introspect over their own work to continuously optimize their processes and potentially discover new knowledge.

The framework does not necessarily impose opinion on how the agentic capabilities are implemented: they could be emergent from pre-training a large foundation model or they could be engineered as (thick) wrappers on top of said pre-trained model, or be somewhere in between. The important consideration is how the system behaves for an end user, and what product value can be expected from the system at different levels.

See the table for a summary:

Level 0: The Know-it-all is plain LLM and Level 1: The Bookworm is RAG, which serve as baselines with no agentic behavior. The breakout Gen AI products of the last couple years fall into these categories. These levels are well studied by now and properly understood by the industry (I hope!). Innovation continues to occur in these levels, e.g. Claude’s new Artifacts feature classifies as Level 0.

Level 2: The Hustler is where things start to get interesting: we add the ability to invoke external tools via API calls, which is a core capability for any agentic behavior. However the system is still primitive and cannot handle complex tasks just yet. Level 3: The Strategist and Level 4: The Prodigy layer on the important capabilities of planning and memory, which enable the system to deal with larger scoped tasks.

Level 5: The Maestro is where we introduce asynchronous execution, which is a critical piece for the system to be seen as agentic, since it enables execution of long running tasks without constant human supervision. Level 6: The Sage is the final level to round up our framework, by adding the ability to introspect on its own processes and come up with improvements. We can debate whether this final piece is necessary for a system to be called an agent or if it goes beyond our current expectations from them. But hey, work in AI long enough and you get used to the moving of goalposts, so we might as well keep this in as insurance.

Now let’s study each level in more detail.

Level 0: The Know-it-all (Plain LLM)

Core Capability: Synthesis (S) - The ability to generate coherent and contextually relevant responses to user input, based on pre-trained information.

Products like ChatGPT, Bard and Claude revolutionized the way we interact with AI, demonstrating the potential of large language models (LLMs) to engage in human-like conversations and assist with a wide range of tasks. These products were a massive innovation when they originally came out, yet they now seem quite quaint compared to all the advancements in the ~20 months since: literally all they did was to pass user input to an LLM and stream back the output!4

How to test for this archetype: The system’s performance is entirely dependent on its pre-training, with no ability to access external information or update its knowledge base, hence the archetype is named Know-it-all. The first products only supported text (and code), but more recently they are powered by multimodal models that support passing images, audio and video via input, and in some cases also generating those on the output side. We would still classify them under Level 0 if there is no interaction with the external world beyond the user provided inputs and the model’s outputs.

These Know-it-all products made advanced AI capabilities accessible to the general public, sparking widespread interest and adoption. Conversational interfaces became a preferred method for interacting with AI systems. They also raised important questions about AI bias, misinformation, and the need for responsible AI development. However, bounded by training data cutoff, these models are prone to hallucinations when asked about information beyond their training scope. This takes us to the next level.

Level 1: The Bookworm (RAG)

New Capability: Retrieval (R) - Combine a language model with a retrieval mechanism to fetch up-to-date information and let the model use that information in the synthesis of its response.

A simple-yet-clever trick (love those!) to solve for the knowledge cutoff problem of Know-it-alls is to dynamically fetch information that might be relevant to the query, and feed it to the LLM as additional context during inference. LLMs have shown to be quite effective at leveraging such in-context information to tailor the synthesized response to be more grounded in the retrieved facts. This is the essence of RAG, or Retrieval-Augmented Generation, which we call The Bookworm archetype.

Products like Gemini, Google SGE, Perplexity, ChatGPT w/ Browser, and others have significantly enhanced the utility of AI assistants by integrating real-time information retrieval to provide more factual responses. Enterprise products like Gemini Enterprise, Glean, Harvey, Hebbia and many others have done the same, except with fetching documents from private repositories (e.g. Google Drive) or paid sources (e.g. legal case transcripts).

How to test for this archetype: The system incorporates some form of retrieval during runtime, i.e. LLM inference. Typically this is either traditional keyword search that finds documents with words similar to the user’s input, or vector search, which converts documents and user input into numerical vectors, then finds matches based on their similarity in this mathematical space. Most production systems use a hybrid of the two approaches.

Retrieval has mainly three parts:

1. Query construction: Create a keyword search query from the user input, or map the input to a vector search query.

2. Search and ranking: Execute the search on the database and rank the resulting documents.

3. Context augmentation: Take the top-N documents and paste their contents to the input prompt during LLM inference. There is some art (“prompt poetry”) involved in effectively augmenting the prompt for maximal performance.

Bookworm systems have greatly reduced the problem of hallucinations in AI responses, increasing trust and reliability. The ability to incorporate user-specific documents has opened new possibilities for personalized AI assistance in various domains. However, they are limited to only generating outputs directed towards the user, and cannot control other systems. This takes us to the next level.

Level 2: The Hustler (Tool-Use)

New Capability: Actions (A) - The system can make API calls to control other systems, reasoning over their outputs and using that in its own synthesis.

While Bookworms can retrieve and incorporate information, Hustlers can actually do something about it. This archetype introduces the concept of "tool use," allowing the AI to interact with a variety of external systems and APIs to perform actions beyond text/media generation. Products like ChatGPT with Plugins, Gemini with Extensions (I worked on this!), Function Calling support in popular LLM APIs, and various GPTs and LangChain apps have expanded the practical applications: these systems can now perform tasks such as looking up flights and hotels, analyzing data, or even controlling smart home devices.

The key innovation (another simple-yet-clever trick) is to change the LLM’s role from a single-shot output synthesizer to a smart control loop that decides whether to invoke a tool, which tool to call, what parameters to pass to it, and how to process the tool’s response into the final system output. The entire prompt is redesigned to coax the LLM to make these higher level decisions, in addition to also consuming relevant context from retrieval5 and synthesizing the final output as before.

How to test for this archetype: The system allows for registering a number of tools (also referred to as “functions” - same thing) which can be invoked during the processing of a user request. Typically, the tool definitions (API specs) are shown upfront to the model as part of the prompt, in addition to the inputs and the retrieved context, so the model is aware of what tools are available for use. Then the model is instructed, via few-shot prompting or fine-tuning or both, to emit tool calls whenever appropriate.

A prompting approach that initially became quite popular is ReAct, short for Reason-and-Act, which forced to model to emit sequences of Thoughts (internal reasoning of what action it should take) and Actions (parameterized function calls that specify the tool invocation), which follows the chain-of-thought prompting mechanism to improve the model’s ability to pick the best actions. Other prompting approaches (Toolformer, CodeAct) have been explored as well.

The tool-use process typically involves:

1. Tool selection: Analyzing the user's request and determining which tool(s) might be needed. The available tools are provided in the prompt along with a specification of how to invoke them (i.e. parameters and their data types).

2. Action formulation: Constructing the appropriate API call or command for the selected tool. This is the step which requires most careful handling, because the model needs to properly encapsulate all the pieces of information so far into a correct API call.

3. Execution: Sending the command to the external tool and awaiting results. This requires necessary infrastructure to be in place to execute tools in a sandbox (to avoid security concerns from prompt injection, for instance) and deal with errors, retries, etc.

4. Result interpretation: Processing the tool's output and incorporating it into the overall response. Typically, the tool output is pasted verbatim into the context as part of the conversation history and the model is asked to decide how to proceed. But there could be a separate summarization step before passing the tool outputs to the model.

Hustler systems have significantly expanded the range of tasks AI can assist with, from simple calculations to complex data analysis and real-world interactions. The ability to interact with external tools has opened up the possibility of workflow automation, potentially reshaping various industries. However, in practice, they lack the ability to effectively plan sequences of actions for more complex tasks. We found that users very quickly hit the ceiling of what Hustler systems can do because even somewhat complex requests can lead the system astray as the probability of error multiplies with longer chains of tool calls. This limitation leads us to our next level.

Level 3: The Strategist (Planner / Multi-Step Reasoning)

New Capability: Planning (P) - The system can create and execute multi-step plans to achieve complex goals, by breaking down tasks into manageable steps.

Unlike the Hustler, which can use tools but lacks long-term strategy, the Strategist can break down complex tasks into a series of simpler steps, each potentially involving different tools or actions. Products like Perplexity Pro, DeepMind's Astra, and other emerging systems in this category are pushing the boundaries of AI problem-solving capabilities. These systems can tackle more intricate challenges that require a sequence of actions and logical reasoning.

The underlying innovation is yet another simple-but-clever trick (you should be used to them by now): let the LLM explicitly keep track of its plans by first having it verbalize the plan as a chain-of-thought, then tracking that plan as part of the prompt context in subsequent calls to the model, allowing it to make edits to the plan as needed. A good foundational paper to read on this is Plan-and-Solve prompting, which is also described well in this LangChain blog on Planning Agents.

If this sounds like a gross over-simplification, well it is. There’s a lot more careful design needed, for example, to deal with convoluted system states when things are not going according to plan, and setting up the right guardrails to let the effectively system recover from those. In addition, the standout products in this category will need a heavy lift on the UX layer, since all the additional complexity needs to be communicated effectively to the user, striking the right balance between keeping the user informed and making them feel overwhelmed.

How to test for this archetype: The system should demonstrate:

1. Task decomposition: Breaking down a complex goal into smaller, manageable subtasks. This requires advanced techniques that go beyond simple tool-use.

2. Sequential planning: Ordering subtasks logically to achieve the overall goal. This requires a robust orchestration framework to manage multi-step, graph-dependency workflows.

3. Adaptive execution: Adjusting the plan based on intermediate results or changing conditions. Needs mechanisms for intermediate result evaluation and plan adjustment.

4. Result synthesis: Combining the outcomes of all subtasks into a coherent final result.

5. Meta-cognition (baby steps): Reasoning about its own problem-solving process and explaining its strategy to the user.

Strategist systems enable AI to tackle more complex, multi-step problems, potentially revolutionizing fields like scientific research, strategic planning, and complex decision-making processes. However, they still lack the ability to learn and adapt from long-term interactions or to run tasks asynchronously, which brings us to the next levels of agent capabilities.

That’s enough tokens for today! I will finish the rest of the deep dive in Part 2 which will include more thoughts on Level 3, and all of Levels 4, 5 and 6. Subscribe to get it in your inbox as soon as its published.

A serendipitous free Apple TV+ subscription led me to Ted Lasso, and I was hooked faster than you can say "football is life." Three seasons of binging later, it was clear I had stumbled upon a goldmine of wisdom for tech leaders.

Why? Because Ted Lasso isn't a show about football (soccer). It's about building and leading a team in a high-pressure, high-stakes environment. Sound familiar? That's the daily grind for every tech leader, from startup founders to big tech execs to managers and tech leads.

In this post, I’ll delve into what this eccentric coach can teach us about leadership in the tech world.

The Premise: An Unlikely Leader in Hostile Territory

Imagine this: An American football coach, with zero soccer experience, is suddenly dropped into a struggling English Premier League team. The owner secretly wants him to fail. The team is skeptical. The fans are hostile (chants of “wanker” fill the stadium in the first match). The press is sharpening their knives. You get the picture.

The parallels to the tech world are striking. How often have we seen a new leader parachuted into a struggling company, or a founder entering an unfamiliar market, facing skepticism from all sides? I’m willing to bet that every Staff+ Engineer has to deal with having to turn around a messed up team at least a few times in their career.

But Ted Lasso is built different. He's relentlessly optimistic, unfailingly kind, and surprisingly perceptive. He has a folksy wisdom that initially comes across as naive but gradually reveals a deep emotional intelligence. Ted's secret weapons? Genuine curiosity, unwavering belief in people's potential, and a willingness to be vulnerable.

In the face of hostility, Ted responds with empathy. When confronted with skepticism, he doubles down on sincerity. Where others see limitations, Ted sees possibilities. It's an approach that initially baffles those around him but gradually transforms the entire organization. In an industry often characterized by cut-throat competition, "move fast and break things" mentalities, and the glorification of the "brilliant jerk," Ted Lasso offers a refreshingly different playbook.

If I were to simplify Ted Lasso’s entire approach into one sentence, it’d be: it’s all about the people.

Sounds trite, but that’s the core concept underlying his entire philosophy. Amidst all the excitement about new technologies, world-changing products and new-age engineering problems, we tend to forget that we’re still humans operating in a world populated with other humans. To achieve seemingly impossible feats and push the frontier, we must first value and understand the individuals behind the efforts. By building trust, fostering a positive environment, and caring about each person's well-being, we unlock extraordinary potential and drive true innovation.

With this in mind, I couldn’t think of a more fitting way to analyze Ted Lasso’s leadership approach than to focus on the individuals around Ted, their character journeys, and the impact of Ted’s leadership on each of them. Whether you're trying to turn around a struggling team, integrate a difficult but brilliant employee, or simply create a more positive and productive work culture, there's something to learn from the Lasso Way.

[Here begin SPOILERS: If you haven’t watched the show, I’d recommend stop here, watch all 3 seasons, and come back. If you need further motivation to watch this show, check out this scene:

Go on, I’ll wait here for you.]

The Lasso Playbook: Detailed Character Journeys

Let's dive deep into the transformative journeys of key characters under Ted's influence and extract the leadership gold that's particularly relevant for us in tech.

Roy Kent: The Power of Vulnerability and Reinvention

When we first meet Roy Kent, he's a fading star—bitter, closed-off, and resistant to change. Sound familiar? In tech, we often encounter brilliant minds who are set in their ways, resistant to new ideas or methodologies.

Ted's impact: Instead of sidelining Roy, Ted sees his potential as a leader. He encourages Roy to mentor younger players, pushing him out of his comfort zone. When Roy's playing career ends, Ted supports his transition to coaching, showing faith in Roy's ability to reinvent himself.

The leadership lesson: In tech, we need to recognize that our team members' roles can and should evolve. The brilliant coder of yesterday might be tomorrow's inspiring team lead. But this transition requires vulnerability—the willingness to admit what you don't know and to learn new skills.

Take Satya Nadella at Microsoft. When he became CEO, he had to transition from a technical role to a leadership position that required a completely different skill set. Like Roy, Nadella had to embrace vulnerability, admitting he didn't have all the answers and needed to learn. This openness to growth transformed Microsoft's culture and business.

The tech takeaway: Encourage your team members to embrace vulnerability and be open to reinvention. The next game-changing idea might come from the person willing to admit they don't have all the answers.

Jamie Tartt: Reforming the Brilliant Jerk

Jamie Tartt is the archetypal "brilliant jerk"—incredibly talented but utterly self-centered. Every tech company has at least one Jamie: the 10x engineer who writes amazing code but leaves a trail of interpersonal destruction in their wake.

Ted's impact: Instead of just tolerating Jamie's behavior for his talent, Ted challenges him. He benches Jamie when his selfish play hurts the team, sends him back to Manchester City to face consequences, and later welcomes him back with open arms when Jamie shows a willingness to change. Ted sees Jamie not for who he is, but for who he could be.

The leadership lesson: In tech, we often make the mistake of tolerating toxic behavior from top performers. But Ted shows us that with the right approach, even the most difficult team members can become collaborative assets.

This reminds me of the "No Asshole Rule" championed by Robert Sutton in Silicon Valley. Companies like Google have adopted policies that prioritize teamwork and collaboration over individual brilliance. Sometimes, the most valuable thing a leader can do is help a talented but difficult team member grow into a true team player.

The tech takeaway: Don't just manage your brilliant jerks—transform them. Set clear expectations, provide opportunities for growth, and create a culture where teamwork is valued as highly as individual contribution.

Sam Obisanya: Nurturing Potential and Ethical Leadership

Sam's journey from a shy, young player to a confident leader on and off the field is a masterclass in nurturing potential. But it's his ethical stand against the team's questionable sponsor that really resonates with the tech world.

Ted's impact: Ted creates an environment where Sam feels safe to grow, make mistakes, and find his voice. When Sam takes an ethical stand, Ted supports him unequivocally, showing that principles matter more than short-term gain.

The leadership lesson: In an industry often criticized for its "move fast and break things" mentality, Sam's story reminds us of the importance of ethical leadership and creating a safe space for growth.

This brings to mind leaders like Marc Benioff of Salesforce, who's been vocal about tech's social responsibilities. Like Ted did for Sam, Benioff creates an environment where employees feel empowered to speak up about ethical issues.

The tech takeaway: Create a culture where team members feel safe to grow, make mistakes, and stand up for what's right. The next big innovation might come from the quiet voice you've been nurturing all along.

Rebecca Welton: The Transformative Power of Authentic Leadership

Rebecca's evolution from a vengeful owner seeking to sabotage the team to a supportive leader is perhaps the most dramatic character arc in the show.

Ted's impact: Ted's unfailing kindness and authenticity gradually break down Rebecca's walls. He shows her that vulnerability isn't weakness, and that being true to oneself is the foundation of effective leadership.

The leadership lesson: In tech, we often see leaders who hide behind a facade of infallibility. Rebecca's story shows us that true leadership comes from authenticity and the willingness to be vulnerable.

This reminds me of Sheryl Sandberg's approach at Facebook (now Meta). By sharing her personal challenges in "Lean In," Sandberg didn't just write a book—she sparked a movement. Like Rebecca, she showed that vulnerability and openness can be powerful leadership tools.

The tech takeaway: Don't be afraid to show your human side. Authentic leadership builds trust, fosters open communication, and creates a more resilient organization.

Keeley Jones: Recognizing Talent in Unexpected Places

Keeley's transition from model to successful businesswoman highlights the importance of recognizing talent in unexpected places.

Ted's impact: Ted sees beyond Keeley's initial role and recognizes her emotional intelligence and marketing savvy. He encourages her to apply these skills professionally, leading to her career transition.

The leadership lesson: In tech, we often fall into the trap of hiring for a narrow set of skills or experiences. Keeley's journey reminds us of the importance of diverse perspectives in driving innovation.

This echoes the approach of leaders like Airbnb's Brian Chesky, who often hires from outside the tech industry to bring fresh perspectives to his team. The lesson? Sometimes the best ideas come from the least expected sources.

The tech takeaway: Look beyond traditional backgrounds and skill sets when building your team. The next big innovation might come from someone with a completely different perspective.

Nate Shelley: The Pitfalls of Unchecked Ambition and the Importance of Emotional Intelligence

Nate's journey from kit man to assistant coach to rival is a cautionary tale about the dangers of unchecked ambition and the importance of emotional intelligence in leadership.

Ted's impact: Ted recognizes Nate's tactical knowledge and elevates him to the coaching staff. However, Ted's positive reinforcement inadvertently feeds Nate's insecurity and ambition without addressing his underlying issues.

The leadership lesson: While it's important to recognize and promote talent, it's equally crucial to ensure that rising stars develop the emotional intelligence to handle their new roles.

This brings to mind the cautionary tales of leaders like Travis Kalanick at Uber. Like Nate, Kalanick's unchecked ambition led to a toxic culture that ultimately damaged the company.

The tech takeaway: Ambition needs to be balanced with emotional intelligence and empathy to create sustainable success. As you nurture rising talent, make sure you're also developing their leadership skills and emotional intelligence.

Dani Rojas: Channeling Passion and Maintaining Perspective

Dani Rojas bursts onto the scene with his infectious enthusiasm and his mantra "Football is life!" He's a bundle of talent and energy, reminiscent of those bright-eyed, bushy-tailed new grads who join tech companies ready to change the world with their code.

Ted's impact: Rather than tempering Dani's enthusiasm, Ted channels it. He creates an environment where Dani's passion can flourish while gently guiding him to be a team player. When Dani faces a crisis of confidence after accidentally killing the team mascot, Ted helps him regain his perspective and joy for the game.

The leadership lesson: In tech, we often encounter highly passionate individuals who are full of energy but might lack direction or struggle to work within a team. Dani's journey shows us the importance of nurturing passion while providing guidance and perspective.

This reminds me of how Marissa Mayer, in her early days at Google, was known for her boundless energy and enthusiasm for product design. Like Ted did with Dani, Google's leadership channeled Mayer's passion, leading to innovations like Google's famously sparse homepage.

The tech takeaway: Don't dampen the enthusiasm of your team members – channel it. Create an environment where passion can thrive, but be ready to provide guidance and perspective when challenges arise.

Trent Crimm: “I Can’t Help But Root For Him”

Trent's journey from cynical journalist to Ted Lasso convert is a powerful lesson in the importance of keeping an open mind.

Ted's impact: Through consistent authenticity and unexpected kindness, Ted gradually changes Trent's perspective. Ted shows Trent that there's more to success than just winning games.

The leadership lesson: In tech, we often become set in our ways, convinced that our way of doing things is the best. Trent's story reminds us of the value of remaining open to new ideas and approaches, even when they challenge our preconceptions.

This mirrors the journey of leaders like Jeff Bezos, who's famous for his "Day 1" philosophy at Amazon. Like Trent, Bezos emphasizes the importance of always being open to new ideas and approaches, no matter how successful you've become.

The tech takeaway: Stay curious and open-minded. The next big breakthrough might come from an idea or approach you initially dismissed.

Conclusion: Be Curious Not Judgmental

In the end, success in tech, like in soccer, isn't just about the product you ship or the code you write. It's about the team you build, the potential you nurture, and the culture you create. And in that respect, we could all learn a thing or two from Ted Lasso and the transformations he inspires:

1. Embrace vulnerability and support reinvention (Roy)

2. Transform, don't just tolerate, your difficult geniuses (Jamie)

3. Nurture potential and stand firm on ethics (Sam)

4. Lead with authenticity and openness (Rebecca)

5. Recognize talent in unexpected places (Keeley)

6. Balance ambition with emotional intelligence (Nate)

7. Channel passion while providing guidance and perspective (Dani)

8. Stay open to changing your perspective (Trent)

There's a treasure trove of leadership wisdom hidden in "Ted Lasso" that I've only begun to unpack here. This show is a masterpiece that rewards multiple viewings, each time revealing new insights about leadership, teamwork, and personal growth. So the next time you're looking for inspiration on how to lead your tech team, consider revisiting AFC Richmond.

Just don't forget the biscuits.

Subscribe now