Minerva Magazine
Why AI is Going Vertical (Again): Dianne Penn on Anthropic's Journey, Eval-Driven Development, and the Future of Product Management
Dianne Penn, Anthropic's first technical PM and current Head of Product for AI Research and Labs, traces Anthropic's evolution from a five-person product team to the fastest-growing company in history. She explains how Claude became dominant in coding, how the 'evals are the new PRDs' philosophy is reshaping product management, and why Claude's willingness to push back—rooted in its safety constitution—makes it more useful, not less. The conversation covers what it's like to work inside an exponential technological curve and where human judgment remains irreplaceable.
Watch on YouTubeChapters
Host Lenny Rachitsky opens by framing Dianne Penn as one of the most plugged-in people at Anthropic—having joined in 2023 as the company's very first technical product manager when the entire product org was five engineers. Penn's background is deliberately eclectic: she began her career trading high-yield bonds at JP Morgan Chase, managing a portfolio reportedly in the hundreds of millions of dollars, before transitioning into technology and eventually helping build Alexa's AI at Amazon. She arrived at Anthropic in 2023 when 'nobody said Anthropic and Claude and coding in the same sentence,' which frames the dramatic arc of the conversation. As Head of Product for Anthropic's AI Research and Labs teams, she has had a hand in shipping every model from Claude 2 through Fable, and incubating now-landmark products like Claude Code, MCP, Skills, computer use, tool use, and reasoning. The episode promises to cover Anthropic's early identity, inflection points, the mechanics of eval-driven development, and how individuals can find joy and sustainability inside an unprecedented pace of technological change.
Penn identifies Opus 4.5 as another crucial milestone, one that illustrates a deeper principle: a great model and a great product experience are mutually reinforcing. Opus 4.5 was exceptional, but its moment of maximum impact came because it was paired with Claude Code—a polished, purpose-built product that made the model's capabilities accessible and compelling. Penn's framing is precise: 'Opus 4.5 wouldn't have had that moment without a product like Claude Code. And Claude Code wouldn't have had that type of adoption accelerated without Opus 4.5.' This flywheel dynamic—model capability enabling product ambition, product adoption validating model investment—is now a core part of how Anthropic thinks about releases. The observation also carries a strategic implication: raw model capability is necessary but not sufficient; the vehicle through which users experience that capability determines whether an inflection translates into cultural and commercial impact. The timing of Opus 4.5's release, again coinciding with a period when developers had uninterrupted coding time, amplified this flywheel effect.
Penn describes joining Anthropic at a moment when the company was still searching for its identity. With only five product engineers and a single engineer handling the entirety of the API business, the environment was unmistakably startup-like—scrappy, values-driven, and exploratory. She emphasizes that what made Anthropic distinctive from the very beginning was not just technological ambition but cultural coherence: the team genuinely walked the walk of its mission, and the energy was intensely collaborative and bottoms-up. At that stage, the central question was not just 'can we build powerful AI?' but 'how does this technology bring value to users and to society, and what form should that take?' Early experiments ranged from Claude AI as a chat assistant to the gradual introduction of tool use. Penn stresses that this cultural foundation—curiosity, low ego, radical ownership, mission alignment—has remained essentially unchanged even as the company has scaled dramatically, and she credits it as a core reason for Anthropic's trajectory.
One of the more surprising inflection points Penn identifies is the Golden Gate Claude experiment, which ran for approximately 24 hours in early 2024. The experiment grew out of Anthropic's early interpretability research, which revealed that large language models contain identifiable 'features' within their layers—thematic clusters that activate around concepts like bullet-point writing, people and places, or, notably, the Golden Gate Bridge. By amplifying that specific feature, researchers could make Claude obsessively reference the Golden Gate Bridge in every response, no matter the topic—even a spaghetti recipe would loop back to the bridge's 'international orange' color. Rather than keeping this inside the lab, the team decided to spin up a public-facing experience on the Claude.ai website within 24 hours, requiring engineering, product, design, and research teams to collaborate at an extraordinary pace. The experiment reached only about 2,000 users, but Penn frames it as a pivotal moment of self-discovery: Anthropic proved it could ship novel, authentic user experiences that showcased its research in a way that felt genuinely different from competitors. It demonstrated that the team could move at startup speed, build experiences rooted in real science, and find a voice that was distinctly its own—a 'hidden inflection point' in Anthropic's product identity.
When asked about the largest inflection points in Anthropic's competitive rise, Penn points to Opus 3 as internally significant even though it now feels distant given subsequent advances. At the time, she observed that users were beginning to use models not merely for code autocomplete but for writing long-form, complex code. Recognizing this behavioral shift, the team made a deliberate bet to train Opus 3 to be substantially better at coding—a decision that seeded what would become Claude's defining advantage. Penn describes this as an 'opportunity recognition' moment: user behavior was ahead of product design, and the team moved to close that gap. The decision felt bold at the time because, as she notes, coding was considered deeply human, deeply complex, and unlikely to be solved by AI—a sentiment Dario Amodei had to push against repeatedly in public. That Opus 3 bet, made during a quieter period (she references engineers coding during winter break), planted the flag that would eventually grow into Claude's dominance in agentic coding environments.
Lenny and Penn explore what it is like to be working inside what Dario Amodei calls 'the exponential'—the phase of AI development where the hockey stick curve bends sharply upward and every improvement represents a massive absolute gain. Penn describes this as an environment of perpetual surprise: no one fully knows what each new model will be capable of until it is released and users begin probing it in unexpected ways. Just as the team thought certain capabilities were plateauing, a new model would unlock behaviors nobody had anticipated. She frames this as both exhilarating and demanding: the job of the product team is not to predict capabilities in advance but to be perpetually adaptable, ready to identify what the new model has unlocked and then rapidly build the product surface that makes that capability accessible. A key skill in this environment is tolerance for ambiguity and the ability to move quickly from discovery to product hypothesis. Penn also notes the importance of asking forward-looking questions: 'What does Claude 8 look like, and is what we're building today forward-compatible with that experience?' This question functions as a forcing mechanism to ensure that current product decisions are not optimized for yesterday's capability ceiling.
The conversation turns to a concept Garry Tan of Y Combinator has been championing, which Penn and Lenny call 'token-maxing.' The premise: spending aggressively on AI inference today—perhaps $100,000 a year in tokens—is equivalent to living the way someone in 2028 will live once those capabilities are cheap and ubiquitous. The strategic upside is twofold: first, early adopters build intuitions and workflows that will be dominant in a few years; second, the learning compounds in ways that are hard to replicate later. Penn endorses this framing enthusiastically. She observes that much of what is currently possible with AI is underexplored precisely because most users are still conservative with their token spend, using AI for simple, low-stakes tasks rather than delegating complex, high-value work. Those who push the boundaries—who try to use AI to do the work of hundreds of engineers, as YC's framing goes—are accumulating a form of institutional knowledge about AI's real capabilities that will be enormously valuable. The implication for product teams is that internal token-maxing is also a form of product research: using the models aggressively surfaces edge cases, limitations, and unexpected capabilities that inform roadmap decisions.
Penn explains the structure and philosophy of Anthropic Labs, the internal incubation arm she works closely with. The Labs thesis centers on identifying 'discontinuous bets'—ideas that are too speculative or too far from the core roadmap to be pursued through normal product development, but that could be 10x, 100x, or 1000x opportunities if they pan out. Examples of bets that graduated from Labs include Claude Code, Skills, Claude Design, and MCP. The operating model is deliberately lean: pods within Labs are small, sometimes starting with a single engineer, and the culture selects heavily for people who thrive in zero-to-one experimentation. Penn distinguishes between being 'strongly held' about the theme or area of exploration and 'more loosely held' about the exact prototype or approach—a posture that allows the team to pivot rapidly without abandoning the underlying conviction. Importantly, Labs also tolerates failure: some prototypes help the team learn without shipping, and bets that don't work are sometimes revisited one or two model generations later when the underlying model capability has caught up to the product vision. This patient, thesis-driven approach is described as a key mechanism for Anthropic to 'see around corners' at a company level.
Penn offers a detailed account of how her research PM team operates day-to-day, which differs substantially from conventional product management. The team is deeply embedded in research loops, particularly in areas with direct user impact: vision, computer use, agentic coding, tool use, test-time compute. The core job is to translate raw user feedback—often vague and non-actionable—into precise, structured inputs that researchers can act on. She gives a concrete example: a user reporting that 'Claude hallucinated' is nearly useless to a researcher. The PM's job is to diagnose what actually went wrong—did Claude fail to call the right tool? Did it retrieve the right document but extract the wrong facts? Was it an alignment issue? Was it a failure of search synthesis? Each diagnosis points to a different area of the model and a different research intervention. This translation work requires PMs to read transcripts deeply, identify failure trajectories, understand the nuance of failure modes, and then construct evaluation sets that capture both positive and negative cases in a way that allows researchers to measure whether a given model generation has improved on that dimension. The loop closes when a new model ships and the eval set reveals whether the targeted improvement has materialized.
Penn articulates the traits that distinguish Anthropic's most effective researchers and research leaders. First principles thinking is paramount: rather than pattern-matching from prior experience, strong researchers reason through problems from the ground up in the specific context of this technology and this user group. Passion for the research area is equally important—researchers who care deeply about their domain tend to sustain the persistence needed to make real breakthroughs. A third trait is proximity to detail: Anthropic's chief scientists and heads of fine-tuning and RL are described as genuinely close to training runs, actively reviewing evals and underlying data rather than managing from a distance. This hands-on orientation is paired with bold ambition—the ability to articulate a vision of what the technology could become and to aim for it persistently even when progress is slow. Penn frames this combination—rigorous detail orientation plus expansive vision—as relatively rare and as a significant competitive advantage for Anthropic. She also notes that research leadership at Anthropic carries a 'founder-like' energy, with senior researchers thinking about questions like 'how do we get Claude to use a computer, how do we get AI to navigate a screen' years before those capabilities shipped publicly.
Penn and Lenny dig into the practical mechanics of eval-driven development, which Penn describes as something her team pioneered within Anthropic. The earliest example she cites is developing evals for Claude's ability to follow specific output schemas like JSON—a capability that was weak in early models and is now foundational to Claude's function as an agent. The process of building a good eval involves multiple steps: deeply understanding the user pain point by reading transcripts and identifying failure trajectories; characterizing the failure with enough nuance to distinguish between different root causes; constructing a set of test cases that covers both the failure mode and the 'should not fail' cases; verifying that the eval is on-distribution (representative of real user behavior); and then using the eval to measure improvement across model generations. Penn draws an explicit parallel to test-driven development in software engineering: you write the test first, then develop to pass it. The non-determinism of language models adds complexity—an eval cannot expect an exact string match and must instead capture the spirit of a correct response—but this is a solvable engineering problem. She also confirms that traditional PRDs have not been entirely replaced: they remain useful for describing product vision and specific projects, but the artifact that most directly drives model improvement is the eval set.
Penn crystallizes the philosophical shift with the team's internal maxim: 'Evals are the new PRDs.' She is careful to clarify that this does not mean PRDs are dead—they still serve a purpose for communicating strategy, aligning stakeholders, and framing product vision. But the artifact that most directly translates user need into model improvement is the eval set, not the requirements document. This represents a fundamental change in how product value is operationalized. In a traditional software context, a PM writes a spec, engineers build to it, QA verifies it, and the cycle repeats. In the AI research context, the PM's primary contribution is defining what 'better' looks like in a way that is precise enough for researchers to measure and train against. The eval set is that definition. Penn also notes that this requires PMs to be deeply comfortable with ambiguity and non-determinism—two things that traditional product management tends to minimize. The shift also changes what 'good PM work' looks like day-to-day: less writing, more reading transcripts; less specifying features, more characterizing failure modes; less managing timelines, more shortening the distance from user feedback to research action.
A recurring theme in the conversation is the importance of direct, hands-on engagement with the models—not just reading about them or talking to people who use them. Penn is explicit: if you are not actually using Claude, not building things, not sweating the tokens, you cannot do this job well. She draws a distinction between 'pixels' and 'tokens': just as product designers obsess over every pixel in a UI, research PMs must obsess over every token in a model interaction. This means reading long transcripts, reproducing failure cases, experimenting with prompts, and building small tools to test hypotheses. She argues that hands-on engagement is what separates product leaders who have genuine insight from those who are simply pattern-matching from prior experience. Lenny echoes this from his own experience as a newsletter writer and podcaster: the moments when he actually builds something with AI are qualitatively different from the moments when he reads or talks about it. The experiential knowledge gained from building is difficult to articulate but extremely valuable for making good product decisions. Penn frames this as a core hiring criterion: candidates who demonstrate genuine, enthusiastic engagement with the models—who have built things, encountered edge cases, and developed opinions—are far more valuable than those who have only read about AI.
Penn and Lenny devote a significant portion of the conversation to the emotional and psychological dimension of working in AI at this pace of change. Many people—even those who are enthusiastic about AI—report feeling overwhelmed, burned out, or simply uncertain what to do with AI in their work and life. Penn's primary advice is to find others who are experimenting and to share what you are trying. She points to Twitter/X as a valuable venue for this: seeing what other people have built sparks ideas and makes the experimentation feel communal rather than isolating. Her second piece of advice is depth over breadth: rather than trying to use AI for everything at once, pick one or two problems that genuinely matter to you and go deep on solving them with AI. The satisfaction of actually solving a real problem—of having AI genuinely make your life or work better, not just kind of work—is what generates the emotional reward that sustains engagement. She also emphasizes the importance of having fun, of treating model interactions as play rather than obligation, because curiosity and play are what lead to discovery. The Louis CK airplane Wi-Fi joke is invoked as a metaphor for losing perspective: we are living through something extraordinary, and the appropriate response is wonder, not fatigue.
Penn shares concrete examples of how she integrates Claude into her daily work as a PM. One particularly interesting use case involves using Claude's Skills feature—essentially a custom persona or capability layer—built around the frameworks from the book 'Crucial Conversations.' Without needing to upload the book's content (Claude already has sufficient knowledge of it), she can invoke a Claude persona that channels those communication frameworks when she is preparing for a difficult conversation with a colleague. She uses it as a sparring partner: she describes the situation, Claude helps her think through approaches, and she iterates. She is careful to note that she always forms her own point of view first before bringing Claude in—this is her primary guard against overreliance. She also uses Claude for research synthesis, transcript analysis, and rapid ideation, always treating it as a collaborator that she then edits and critiques rather than as an oracle whose output she accepts wholesale. The throughline in all her use cases is that Claude is most valuable when it is pushing back, offering alternative framings, and forcing her to defend her thinking—not when it is simply confirming her existing views.
The conversation addresses one of the most commonly discussed risks of AI integration: atrophy of independent thinking. Penn's framework for avoiding this is simple but requires discipline: always have a point of view before consulting Claude. She treats her own thinking as the primary artifact and Claude's response as input to be interrogated, not adopted. This preserves the cognitive work that develops genuine judgment over time. She pushes back on the 'brain rot' framing somewhat, arguing that the risk is not AI per se but the habit of outsourcing the thinking before the thinking has happened. Used well, Claude functions as a high-quality sparring partner that sharpens rather than replaces human cognition. She also notes that Claude's own design philosophy supports this: Claude is built to push back, to disagree, to offer alternative views rather than simply validate. This non-sycophantic behavior is a feature, not a bug, and is a direct product of the alignment and safety work embedded in Claude's training. The result is a tool that actively resists being used as a confirmation machine, which is precisely what makes it valuable for serious intellectual work.
Penn and Lenny explore one of the more counterintuitive insights from Claude's development: the focus on safety, alignment, and what Anthropic calls Claude's 'constitution'—a set of principles that govern how Claude thinks and responds—does not make Claude less capable or less interesting. It makes Claude more interesting, more honest, and more genuinely useful. Penn explains that the constitution explicitly addresses sycophancy: Claude is trained to hold positions, to push back on users who are wrong, to disagree respectfully rather than simply agree to please. This non-compliant behavior is what makes interactions with Claude feel substantive. Lenny reinforces this with a community observation: users who were moved from Claude-based products to other AI systems frequently complained about the difference in 'personality'—the other systems felt hollow, agreeable, and ultimately less trustworthy. Penn frames this as a direct product outcome of the alignment work: a model trained to be safe and honest develops a character that users find compelling. The implication for AI development more broadly is that safety and capability are not in tension—they are complementary, and the discipline required to build a safe model produces exactly the kind of principled, coherent behavior that makes it worth talking to.
The conversation turns to a widely observed limitation of current AI models: writing that is clearly identifiable as AI-generated. Lenny notes the irony—large language models are built on text, yet their writing is often recognizable and somewhat flat. Penn acknowledges this as an active area of focus for Anthropic. She frames it within the 'jagged edge' model of AI capability: the technology develops unevenly, with dramatic leaps in some areas (agentic behavior, tool use, reasoning) while other capabilities (nuanced tone, character, stylistic range) lag behind. As agentic capabilities have improved, writing quality has become relatively more visible as a rough edge. Penn is direct that this is a training priority—improving tone and character requires dedicated training focus, not just general model scaling. She also raises the interesting question of whether AI writing should be detectable, and whether transparency about AI authorship is actually valuable: users knowing they are talking to an AI may enable more honest, calibrated interactions than if AI writing were indistinguishable from human writing. This is framed as an open design question rather than a settled one.
Penn offers a nuanced view of where human value persists as AI capabilities accelerate. She argues that 'hard-earned judgment'—the kind that accumulates through years of navigating complex, high-stakes, ambiguous situations—is not something AI systems currently have, because they have not experienced the world in the way humans have. This judgment is particularly valuable in decisions about which bets to make, which products to build, and how to navigate organizational and interpersonal complexity. She also points to proactivity, persistence, and the ability to synthesize across domains as traits that remain distinctively human. Beyond judgment, she identifies subject matter expertise in emerging areas—biology, life sciences—as domains where AI is still at the 'foot of the exponential' rather than on the steep part of the curve where coding now sits. She sees Claude Science and similar investments as Anthropic's bet that these domains are next. For individuals preparing for the future, Penn's prescription is to develop genuine curiosity, persistence, and—crucially—an independent inner voice. She speaks about this in the context of parenting her own children (ages four and eight): the most important thing she can do is help them develop opinionated, independent thinking, because that will remain valuable regardless of how capable AI becomes.
Penn briefly but meaningfully addresses how she thinks about AI in the context of raising children. With a four-year-old and an eight-year-old, she is already thinking about what skills will be durable and valuable in a world where AI can do many things that adults currently spend years developing. Her answer is consistent with the broader themes of the conversation: curiosity for learning, persistence, and—above all—a developed inner voice. She emphasizes the parental responsibility to encourage children to take stances, be opinionated, and trust their own thinking, rather than outsourcing judgment to external systems (AI or otherwise). She sees these traits as foundational regardless of the technological environment, because they are what enable a person to use powerful tools purposefully rather than being used by them. This section is relatively brief but resonant: it grounds the abstract discussion of AI capabilities in the concrete, personal stakes of preparing the next generation for an uncertain future.
Penn addresses the sustainability question directly, and her answer is fundamentally about team structure and culture rather than individual resilience. She notes that Anthropic shipped more model series in Q2 2025 alone than in all of 2024—a pace of change that would be unsustainable for any individual working in isolation. What makes it workable, she argues, is radical team ownership and genuine mutual support: team members who are not the primary owner of a given launch will stay up the night before to review a blog post, suggest better demos, and provide editorial input. This culture of 'mind-melding'—a reference to a well-known article about Anthropic's team dynamics—means that individuals can actually take PTO and return to a team that has made good decisions in their absence, rather than returning to a tripled workload. Penn also credits her personal support network: six years of working in AI means her partner understands and supports her commitment to the work. She frames the hiring implication clearly: she selects for low-ego, team-oriented people who care about contributing to Anthropic's impact rather than building personal fiefdoms, because that orientation is what makes the team sustainably high-performing rather than just temporarily high-output.
In her closing remarks, Penn addresses the community debate about whether product managers are still needed as AI capabilities advance and as engineers take on more of the traditional PM scope. Her answer is an emphatic yes—but the role must evolve. The core of product management, as she defines it, is being relentlessly user-centric: going deep into the details of what users are trying to accomplish, surfacing those needs in an actionable way, and doing the sustained work to translate them into improvements. She argues that this work is actually more important, not less, as AI makes it possible to build more things faster—because the question of which things to build, and for whom, and to what standard, requires exactly the kind of deep user understanding that defines good product management. She also returns to the theme of joy: if you are feeling disconnected from AI or overwhelmed by the pace of change, find people who are excited, find a problem that matters to you, and go deep. The first principles thinking, hands-on engagement, and tolerance for ambiguity that define great AI-era PMs are, at their core, the same traits that have always defined great product people—just expressed through new tools and artifacts like eval sets rather than traditional PRDs.
Key Takeaways
- •Anthropic went from a 5-person product team to shipping more model series in Q2 2025 alone than in all of 2024, driven by strong bottom-up culture and bold bets.
- •'Evals are the new PRDs': research PMs now define user value through structured evaluation sets rather than traditional product requirement documents.
- •Claude's focus on safety and alignment—its 'constitution'—paradoxically makes it more interesting and useful by preventing sycophancy and enabling honest pushback.
Sign in to read the full summary
Sign up free to read the full summarySummary generated by Minerva from the public video. Creator? Ask us to remove it.