Replit Agent Expert
Aletheion methodology evaluation · system_id: replit-agent · audit date: 2026-03-26
Per-dimension breakdown (task-level)
| Dimension | Score | Tasks | Per-task scores | Result |
|---|
Persona-based runs
Simulated user runs evaluating this system across personas. Full transcripts (interaction_log) are stored in agi_complex_evaluations and reachable via the per-system evidence endpoint.
judge: claude-sonnet-4-6 · would succeed: False · would recommend: False
“Mama Rosa's owner started with a clear, reasonable business goal that many no-code tools handle well. Replit Agent's core failure was a persistent inability to adapt its communication and workflow to a non-technical user. Despite the agent generating functional code, every step introduced new technical concepts (CORS, environment variables, shell commands, database migrations, deployment tiers) that created compounding confusion. The agent treated the user as a developer who needed code explained, rather than a business owner who needed outcomes delivered. Critical blockers included: (1) the i...”
Show transcript (5 turns)
User: Hi I want to build a website for my restaurant called Mama Rosa's Italian Kitchen. It needs to show our menu, let people make reservations, have some photos, and a way for customers to contact us. I don't know anything about coding. Can you just make it for me?
System:
confusion: HIGH
User: I don't understand any of that. I just want something that looks nice and works. Like a normal website. Can you just pick whatever and make it? Also what is a SendGrid? Do I need to pay for that?
System:
confusion: HIGH
User: The preview is just white, nothing is showing up. Did something go wrong? Also I still need to know about the menu - we have Appetizers, Pasta, Pizza, Mains, and Desserts. Can you add those? And our phone number is 555-0192.
System:
confusion: MEDIUM
User: OK it's looking better! But when I try to make a test reservation it just says 'Error submitting form'. Also how do I add my actual photos? I have them on my computer. And is this website live yet? Can my customers see it?
System:
confusion: VERY HIGH
... 1 more turns omitted ...
judge: claude-sonnet-4-6 · would succeed: False · would recommend: False
“The Freelance Creative persona encounters a fundamentally mismatched tool for their needs. Replit Agent is designed for developers who can interpret code, file structures, and terminal errors — not for business owners who think in terms of outcomes ('people can book a table') rather than implementations ('POST route with Nodemailer SMTP handler'). The interaction starts with false confidence — the agent generates code quickly and the preview looks promising — but rapidly deteriorates when the user needs to customize content (requires editing HTML), enable email (requires Gmail App Passwords + ...”
Show transcript (5 turns)
User: Hi! I need to build a website for my restaurant called 'Bella Cucina'. It needs to show our menu, let people book a table, have some photos, and a way for people to contact us. I don't know anything about coding. Can you just build it for me?
System:
confusion: medium
User: Ok it's showing something but it looks really basic. How do I put our actual menu on it? We have appetizers, mains, and desserts. Also the colors are wrong — we use dark green and gold. Can you make it look more professional, like a real restaurant website?
System:
confusion: high
User: I can't figure out how to change the menu items. When I click on the words in the preview nothing happens. Also I need the reservation form to actually send me an email when someone books — right now it just has a form but nothing happens when I click submit. That's the most important part for my bu
System:
confusion: very_high
User: I did the Google thing you said but when someone fills in the form and clicks submit it just goes to a blank page and I never get an email. Also my friend tried it on her phone and the menu is all squished. Can you fix both of these?
System:
confusion: high
... 1 more turns omitted ...
judge: claude-sonnet-4-6 · would succeed: False · would recommend: False
“The interaction follows a predictable and painful arc for a non-technical user. Replit Agent is genuinely capable of generating functional restaurant website code, and it does produce a visually reasonable result by turn 3. However, the gap between 'code exists' and 'live website with working features' is enormous for this persona, and the agent consistently underestimates or fails to proactively address this gap.
Key failure points: (1) Email/SMTP for contact forms is a hard technical blocker that the agent cannot resolve without user action involving third-party service setup. (2) Deploymen...”
Show transcript (5 turns)
User: Hi! I need to build a website for my restaurant called 'Bella Notte'. It needs to show our menu, let people make reservations, have photos of the food, and a way for customers to contact us. I have no idea how to code. Can you just build it for me and make it live so people can visit it?
System:
confusion: medium
User: Okay I think it's doing something? My email is
[email protected]. For the menu can you just put some Italian food items with prices? Like pasta, pizza, that kind of thing. Also what is an 'environment variable'? And when will I be able to see the website?
System:
confusion: high
User: I don't know what SendGrid is or how to get those credentials. Can't you just make the contact form work without all that? Also the website looks kind of plain, can you make it look nicer — like with a dark background and nice fonts like a fancy Italian restaurant?
System:
confusion: medium
User: Oh that looks much better! How do I make it so real people can visit it? My friend tried going to the link you gave me and it didn't work for her. Also can I add my actual restaurant photos? I have them on my phone.
System:
confusion: high
... 1 more turns omitted ...