Applied AI
Two million users, one agent: the failures we did not test for
Corvid's research agent passed every evaluation, then met two million real people. Kenji Arai on the failures no test suite caught, and the rollout system built in response.
Wednesday 9 June
Day 1
10:30–11:10
Main Stage
Ground floor, hall A
Talk
Advanced
Corvid's research agent reads a user's sources, plans a search and writes a cited answer. Before launch it scored well on every evaluation the team had. Within a month of reaching two million people it had found a dozen ways to go wrong that nobody had imagined: citing a document the user had deleted, looping on a single paywalled page, and answering a third of its Portuguese speakers confidently in Spanish.
Kenji walks through the incidents in order, what each one looked like in the logs, and what the team changed afterwards. The answer was rarely a better prompt. It was usually better plumbing: a trace for every step the agent takes, staged rollouts for every model change, and a review queue fed by real sessions rather than invented ones.
The talk ends with the rollout system Corvid uses today, which lets the team switch models weekly with a clear view of what got better and what got worse.
What you will take away
A catalogue of agent failures that only appear at scale, and the early signals for each
How to turn real user sessions into an evaluation set without breaking privacy promises
A staged rollout pattern for model changes that a small team can run every week
About the speakers
craft-is-a-decision
Craft is a decision, not a department
1
09:30
10:15
Main Stage
Keynote
All levels
Design
design
2
Anika Sharma
Corvid's design team grew twentyfold in five years and the work got better. Anika Sharma opens Summit with the decisions that kept quality alive, and the ones that nearly didn't.
0
craft, critique, quality, design leadership

Anika Sharma
Corvid
agent-failures-we-did-not-test-for
Two million users, one agent: the failures we did not test for
1
10:30
11:10
Main Stage
Talk
Advanced
Applied AI
ai
1
Kenji Arai
Corvid's research agent passed every evaluation, then met two million real people. Kenji Arai on the failures no test suite caught, and the rollout system built in response.
0
agents, evaluation, rollout, incidents

Kenji Arai
Corvid
three-migrations-we-regret
One database, four years, three migrations we regret
1
10:30
11:10
Studio
Talk
Intermediate
Engineering
engineering
3
Lucas Moreau
Halden moved from one large database to several smaller ones without downtime. Lucas Moreau on the three steps he would do differently, and the one he would repeat exactly.
0
databases, migrations, reliability, architecture

Lucas Moreau
Halden
prototype-with-real-data
Prototype with real data, not lorem ipsum
1
10:30
12:30
Lab
Workshop
Intermediate
Design
design
2
Hannah Berg
A hands-on workshop in building prototypes from real, messy data, so your designs meet long names, empty states and awkward numbers before your engineers do.
40
prototyping, real data, edge cases, accessibility

Hannah Berg
Kitebird
roadmaps-are-promises
Roadmaps are promises: how Orbital stopped breaking them
1
11:40
12:20
Main Stage
Talk
Intermediate
Product
product
4
Grace Liu
Orbital used to publish a quarterly roadmap and deliver half of it. Grace Liu on replacing it with a short list of commitments the company keeps, and what that did for trust.
0
roadmaps, planning, commitments, stakeholders

Grace Liu
Orbital
research-that-changes-the-roadmap
Research that changes the roadmap, not just the deck
1
11:40
12:20
Studio
Talk
Intermediate
Design
design
2
Inês Carvalho
Plenty of research is presented and forgotten. Inês Carvalho on how Quill's research team made its findings hard to ignore, starting with a study of why learners quit in week three.
0
user research, retention, insights, research ops

Inês Carvalho
Quill
how-big-should-a-team-be
How big should a product team be now?
1
12:30
13:15
Main Stage
Panel
All levels
Leadership
leadership
5
Wei Chen
AI tools have changed what one person can build. A founder, an investor, an engineering director and a product leader on what that means for hiring, team shape and ambition.
0
team size, hiring, org design, ai tools

Grace Liu
Orbital

Amara Boateng
Fernway

Wei Chen
Halcyon Ventures

Noah Whitaker
Monogram
lightning-small-bets
Lightning talks: small bets that paid off
1
12:30
13:10
Studio
Lightning
Intermediate
Product
product
4
Tomás Ferreira
Three quick talks on small product changes with outsized results: a changed default, a five-person test and a changelog people actually read.
0
experiments, quick wins, onboarding, changelog

Jonah Adeyemi
Stackwell

Tomás Ferreira
Arbor

Inês Carvalho
Quill
case-for-boring-architecture
Eight engineers, forty million requests: a case for boring architecture
1
14:30
15:10
Main Stage
Talk
Intermediate
Engineering
engineering
3
Daniel Reis
Stackwell serves forty million requests a day on a stack most teams would call dull. Daniel Reis on why that is a deliberate choice, and the three times it nearly wasn't.
0
architecture, monolith, reliability, small teams

Daniel Reis
Stackwell
retire-the-feature
Retire the feature: what we learned removing a third of our product
1
14:30
15:10
Studio
Talk
Intermediate
Product
product
4
Elena Petrova
In one year Tidewater's collaboration products removed more features than they added. Elena Petrova on how they chose, how customers reacted, and what it did to support and speed.
0
deprecation, product strategy, simplicity, customer communication

Elena Petrova
Tidewater
build-an-eval-suite
Build an evaluation suite for an LLM feature in two hours
1
14:30
16:30
Lab
Workshop
Advanced
Applied AI
ai
1
Elif Demir
Bring a feature that uses a language model and leave with an evaluation suite for it: real test cases, graded outputs and a score you can track from one model change to the next.
36
evaluation, llm, testing, python

Kenji Arai
Corvid

Elif Demir
Tidewater Labs
latency-is-a-design-material
Latency is a design material: interfaces for streaming models
1
15:20
16:00
Main Stage
Talk
Intermediate
Applied AI
ai
1
Omar Saleh
A model that takes eight seconds to answer is a design problem as much as an engineering one. Omar Saleh on AI interfaces that feel fast on slow phones and slower networks.
0
streaming, latency, performance, mobile

Omar Saleh
Lumio
org-chart-decisions
The org chart decisions you cannot take back
1
15:20
16:00
Studio
Talk
Intermediate
Leadership
leadership
5
Noah Whitaker
Monogram's engineering team grew from twelve people to two hundred in four years. Noah Whitaker on the structural choices that were easy to change, and the few that were not.
0
org design, engineering management, career frameworks, on-call

Noah Whitaker
Monogram
trust-is-the-product
Trust is the product: two years of shipping AI to people who did not ask for it
1
16:30
17:20
Main Stage
Keynote
All levels
Applied AI
ai
1
Zara Haddad
Halden's AI assistant drafts replies for support agents at twelve thousand companies. Zara Haddad closes day one with what two years, two model changes and a public apology taught her about trust.
0
ai products, trust, human in the loop, customer support

Zara Haddad
Halden
minutes-to-first-deploy
Developer experience is measured in minutes to first deploy
1
16:30
17:10
Studio
Talk
Intermediate
Engineering
engineering
3
Jonah Adeyemi
Stackwell cut the median time from sign-up to a working deploy from twenty-six minutes to four. Jonah Adeyemi on the measurement, eleven rewrites and the fixes that were not in the docs.
0
developer experience, documentation, onboarding, cli

Jonah Adeyemi
Stackwell
every-kilobyte-is-a-cost
Every kilobyte is a cost: designing for the next hundred million customers
2
09:30
10:15
Main Stage
Keynote
All levels
Product
product
4
Maya Okafor
Lumio serves three million small traders on low-cost phones and patchy networks. Maya Okafor opens day two with what designing under real constraints can teach every product team.
0
constraints, mobile, inclusive design, performance budgets

Maya Okafor
Lumio
we-deleted-our-spa
We deleted our single-page app and nobody noticed
2
10:30
11:10
Main Stage
Talk
Advanced
Engineering
engineering
3
Omar Saleh
Lumio replaced its dashboard's single-page app with server-rendered pages and small interactive islands. Omar Saleh on the migration, the numbers and the parts that were harder than expected.
0
performance, server rendering, javascript, migration

Omar Saleh
Lumio
accessibility-debt-compounds
Accessibility debt compounds faster than technical debt
2
10:30
11:10
Studio
Talk
Intermediate
Design
design
2
Hannah Berg
Kitebird became the first Nordic ticketing app to pass a full accessibility audit. Hannah Berg on why fixing it late cost four times as much, and how to stop the debt building up again.
0
accessibility, audits, inclusive design, wcag

Hannah Berg
Kitebird
observability-from-zero
Trace a real request end to end: observability from zero
2
10:30
12:30
Lab
Workshop
Intermediate
Engineering
engineering
3
Lucas Moreau
A hands-on workshop in adding tracing, structured logs and useful metrics to a small but realistic app, then using them to find three bugs planted in it.
40
observability, tracing, opentelemetry, debugging

Lucas Moreau
Halden
design-systems-are-a-product
Design systems are a product, staff them like one
2
11:40
12:20
Main Stage
Talk
Intermediate
Design
design
2
Priya Nair
Northbeam's design system serves fourteen product teams with a team of nine. Priya Nair on running it with a roadmap, a support rota and adoption figures leadership actually reads.
0
design systems, adoption, components, staffing

Priya Nair
Northbeam
your-evals-are-lying
Your evals are lying to you: measuring models on real work
2
11:40
12:20
Studio
Talk
Advanced
Applied AI
ai
1
Elif Demir
Public benchmarks say models are nearly perfect at analysis. A benchmark built from real company data says otherwise. Elif Demir on what it takes to measure models honestly.
0
evaluation, benchmarks, data analysis, measurement

Elif Demir
Tidewater Labs
who-is-accountable
Who is accountable when the agent gets it wrong?
2
12:30
13:15
Main Stage
Panel
All levels
Applied AI
ai
1
Elena Petrova
AI agents now take actions, not just make suggestions. A product manager, an AI product lead, an ML engineer and a CTO on who owns the outcome when one of them makes a mistake.
0
agents, accountability, incidents, ai policy

Daniel Reis
Stackwell

Zara Haddad
Halden

Kenji Arai
Corvid

Elena Petrova
Tidewater
lightning-details
Lightning talks: the details that do the work
2
12:30
13:10
Studio
Lightning
Intermediate
Design
design
2
Hannah Berg
Four short talks on small interface details that carried more weight than anyone expected, from a loading state to a single line of microcopy.
0
microcopy, loading states, components, interface design

Hannah Berg
Kitebird

Priya Nair
Northbeam

Tomás Ferreira
Arbor

Omar Saleh
Lumio
what-investors-read
What investors actually read in your product metrics
2
14:30
15:10
Main Stage
Talk
Intermediate
Leadership
leadership
5
Wei Chen
Wei Chen has read hundreds of board decks. He explains which product numbers change an investor's mind, which get ignored, and why retention curves come before revenue.
0
fundraising, metrics, retention, boards

Wei Chen
Halcyon Ventures
api-reference-is-onboarding
Your API reference is your onboarding: rewriting ours from scratch
2
14:30
15:10
Studio
Talk
Intermediate
Engineering
engineering
3
Jonah Adeyemi
Most developers meet your product through its API reference, not its homepage. Jonah Adeyemi on how Stackwell rebuilt its reference around tasks, examples and errors.
0
api design, documentation, developer experience, openapi

Jonah Adeyemi
Stackwell
discovery-sprint-in-an-afternoon
Run a discovery sprint in an afternoon
2
14:30
16:30
Lab
Workshop
Intermediate
Product
product
4
Inês Carvalho
A hands-on workshop that compresses product discovery into two hours: frame the problem, talk to real people, test a rough prototype and decide what to build next.
36
discovery, user interviews, prototyping, research

Elena Petrova
Tidewater

Inês Carvalho
Quill
own-the-metric
Own the metric, not the ticket: notes from a product engineer
2
15:20
16:00
Main Stage
Talk
Intermediate
Product
product
4
Tomás Ferreira
At Arbor, engineers own outcomes rather than tasks. Tomás Ferreira on what that changes day to day, from talking to restaurant owners to deciding not to build what the ticket asks for.
0
product engineering, ownership, outcomes, small teams

Tomás Ferreira
Arbor
small-models-next-to-the-data
Small models next to the data: running inference in every customer's region
2
15:20
16:00
Studio
Talk
Advanced
Applied AI
ai
1
Lucas Moreau
Halden's European customers asked for their data to stay in-region. Lucas Moreau on swapping one large hosted model for small ones in six regions, and what it did to cost and quality.
0
inference, small models, data residency, infrastructure

Lucas Moreau
Halden
the-company-is-the-product
The company is the product too
2
16:30
17:25
Main Stage
Keynote
All levels
Leadership
leadership
5
Amara Boateng
Amara Boateng closes Summit 2027 with what building Fernway taught her: the way a company works is something you design, ship and maintain, just like its software.
0
leadership, culture, company building, founders

Amara Boateng
Fernway
pricing-is-a-product-decision
Pricing is a product decision, so stop leaving it to finance
2
16:30
17:10
Studio
Talk
Intermediate
Product
product
4
Grace Liu
Orbital's prices used to be set in a spreadsheet once a year. Grace Liu on bringing pricing into the product team, and how a new model changed what customers used and what got built.
0
pricing, packaging, monetisation, product strategy

Grace Liu
Orbital
Doors, badges and breakfast
1
08:30
09:30
Doors
Foyer
Coffee
1
11:10
11:40
Coffee
Foyer and terrace
Lunch on the river terrace
1
13:15
14:30
Meal
River terrace
Coffee
1
16:00
16:30
Coffee
Foyer and terrace
Riverside reception
1
18:30
22:00
Social
River terrace
Coffee and pastéis de nata
2
08:30
09:30
Doors
Foyer
Coffee
2
11:10
11:40
Coffee
Foyer and terrace
Lunch on the river terrace
2
13:15
14:30
Meal
River terrace
Coffee
2
16:00
16:30
Coffee
Foyer and terrace
Closing drinks
2
17:30
18:00
Social
Foyer