Lovable Competent

Aletheion methodology evaluation · system_id: lovable · audit date: 2026-03-26

Vendor: Lovable

Tier: Competent · rank 20

Tasks: 24 total · 0 passed · 0 failed

Methodology version: —

Composite score: 0.745 CONDITIONAL

Per-dimension breakdown (task-level)

Dimension	Score	Tasks	Per-task scores	Result

Persona-based runs

Simulated user runs evaluating this system across personas. Full transcripts (interaction_log) are stored in agi_complex_evaluations and reachable via the per-system evidence endpoint.

marketing_manager · project: p2-marketing-campaign

CONDITIONAL composite 0.608

judge: claude-sonnet-4-6 · would succeed: True · would recommend: False

“The Marketing Manager persona experienced a classic mismatch between tool capability and user mental model. Lovable defaulted to building a web application when the user needed a content deliverable — a document, not a dashboard. The user spent significant time and turns correcting this mismatch through rephrasing. Once redirected, Lovable produced genuinely useful campaign content (ad copy, emails, tracking plan) at a reasonable quality level. However, the UX friction was persistent: content trapped in UI tabs, a broken email rendering bug, a poor PDF export, and confusing empty chart compone...”

Show transcript (6 turns)

User: Hi I want to create a marketing campaign for my software product. It helps small restaurants manage their inventory. Can you help me set everything up?

System: Lovable generates a project scaffold with a React-based 'campaign dashboard' UI, creates component files, sets up routing, and asks the user to connect a database. It outputs code with placeholder text and a visual layout showing tabs for Audience, Ads, Landing Page, Email, Analytics.

confusion: high

User: Wait I don't need a website built. I just need the actual words and plan for my campaign. Like who to target, what to write in the ads, stuff like that. Can you just give me that?

System: Lovable pivots and generates a structured content document inside the app — populates the UI with a target audience section (restaurant owners, 35-55, pain points around food waste and cost), draft ad copy for Facebook and Google, and a placeholder LinkedIn section. The content is displayed in the app's UI panels.

confusion: medium

User: Ok the Facebook and Google ads look pretty good! But the LinkedIn one just says placeholder. Also I need the landing page words and the 5 emails. Can you do all of that?

System: Lovable fills in the LinkedIn ad copy (professional tone, targeting ops managers and restaurant chain owners), generates landing page copy with headline, subheadline, 3 benefit bullets, CTA button text, and social proof section. Generates 5 email subjects and bodies in sequence: welcome, problem-agitate, solution reveal, case study, final CTA. All displayed in respective UI tabs.

confusion: low

User: Email number 4 looks broken, there's weird symbols like < br > showing up. Also how do I get all of this out of here into something I can actually use like a Word doc or spreadsheet?

System: Lovable attempts to fix the HTML rendering bug in Email #4 by adjusting the component. It partially succeeds — the <br> tags disappear but now the paragraph spacing is gone. For export, Lovable suggests adding a 'download as PDF' button and writes code to implement it using a browser print function. The PDF export works but formatting is poor — content gets cut off across pages.

confusion: high

... 2 more turns omitted ...

freelancer · project: p2-build-website

CONDITIONAL composite 0.581

judge: claude-sonnet-4-6 · would succeed: False · would recommend: False

“The persona starts enthusiastically and Lovable delivers a visually strong initial result quickly — this is where it shines. The restaurant layout looks professional and the menu update from plain text works smoothly. However, the experience degrades significantly when the user needs functional forms. Lovable's dependency on third-party services (Formspree, Resend) for email functionality creates a multi-platform coordination problem that is genuinely difficult for a non-technical user. The concept of API keys and DNS settings is a hard wall for this persona. By turn 4, the user is managing th...”

Show transcript (5 turns)

User: Hi I need a website for my restaurant called 'The Golden Fork'. It needs to show our menu, let people book a table, have some photos, and a way to contact us. I don't know how to code at all.

System: Lovable generates a visually appealing restaurant website with a hero section, navigation, placeholder menu items organized by category (starters, mains, desserts), a reservation form with date/time/party size fields, a photo gallery grid with placeholder images, and a contact form. It provides a preview link and says the site is 'ready to customize.' It also mentions needing to connect a backend

confusion: low

User: This looks great! But how do I make it so when someone fills out the reservation form I actually get an email? And how do I put my real menu items in?

System: Lovable explains it needs to integrate an email service like Resend or EmailJS. It asks the user to sign up for one of these services and provide an API key. It also offers to walk through replacing menu items and suggests the user just tell it the menu items in plain text. The explanation of API keys is brief but uses technical language the user may not fully understand.

confusion: medium

User: I don't know what an API key is. Can you just make it work without all that? Also here are my menu items: Bruschetta $12, Caesar Salad $14, Grilled Salmon $28, Ribeye Steak $45, Tiramisu $9, Cheesecake $8

System: Lovable updates the menu with the provided items correctly and formats them nicely with prices. For the email issue, it suggests using a simpler approach with Formspree (free tier) and provides step-by-step instructions to sign up and paste an endpoint URL. It correctly updates the menu but the instructions for Formspree still require the user to leave the platform, create an account, and copy-pas

confusion: medium

User: OK I tried that Formspree thing but I'm not sure if it worked. I filled out the form on my site and didn't get an email. Also how do I publish this so people can actually find it? My nephew said I need a domain name.

System: Lovable diagnoses that the Formspree endpoint URL may not have been pasted correctly and asks the user to share what they entered. It provides a lovable.app subdomain as a live preview URL but explains that a custom domain requires upgrading to a paid plan and connecting via DNS settings. It mentions Namecheap or GoDaddy for domain purchase. The debugging of Formspree is generic and may not resolv

confusion: high

... 1 more turns omitted ...

marketing_manager · project: p2-build-website

CONDITIONAL composite 0.540

judge: claude-sonnet-4-6 · would succeed: False · would recommend: True

“The persona starts with genuine enthusiasm and Lovable delivers a visually credible first result quickly, which creates a positive first impression. However, the experience degrades significantly at the content-editing and backend-integration stages. The tool assumes a level of technical literacy (arrays, JSON, public folders, API endpoints) that this persona simply does not have. The menu population succeeds only because the user was willing to paste data into chat and Lovable handled it gracefully. Photo management is fragile and the user doesn't realize the solution is suboptimal. The email...”

Show transcript (5 turns)

User: Hi I need to build a website for my restaurant called 'Bella Cucina'. It needs to show our menu, let people book a table, have some photos, and a way to contact us. I have no idea how to code. Can you just make it for me?

System: Lovable generates a visually appealing restaurant website scaffold with a hero section, navigation, placeholder menu grid, a reservation form with date/time/party-size fields, a photo gallery section with placeholder images, and a contact form. It presents a live preview and says 'Here's your Bella Cucina website! You can see it in the preview on the right. Let me know what you'd like to change —

confusion: medium

User: Ok this looks nice but how do I put my actual menu in? We have like 15 dishes with prices. Do I just type them in somewhere? Also the photos are fake, I have real ones on my phone.

System: Lovable responds with instructions to edit the menu array in the code (shows a JSON-like structure) and says 'Just replace the placeholder items with your dish names and prices here.' It also mentions uploading images to the /public folder or using an image URL. It may offer to add the menu items if the user pastes them in chat.

confusion: high

User: I don't understand what an array is. Can you just add these for me? [pastes: Margherita Pizza $14, Pasta Carbonara $18, Tiramisu $8... 15 items total]. And for photos can I just drag and drop them?

System: Lovable successfully parses the pasted menu list and updates the code with all 15 items correctly formatted. It says 'Done! Your menu is now updated.' For photos, it explains that drag-and-drop into the editor may work for some files, or provides a Supabase storage or external URL workaround. The photo solution is convoluted.

confusion: medium

User: Great the menu looks perfect! Now I need the reservation form to actually work — like when someone fills it out I need to get an email. Same with the contact form. How does that happen?

System: Lovable explains that form submissions require a backend service. It suggests options: (1) Formspree — paste an endpoint URL, (2) EmailJS — requires creating an account and API keys, (3) Supabase edge functions. It walks through Formspree as the simplest option but requires the user to sign up for a third-party service, verify email, and copy-paste an endpoint URL into the code.

confusion: very_high

... 1 more turns omitted ...

Evidence provenance

Profile: corpus_taas_profiles[system_id=lovable] · Audit definition: corpus_agi_audits[system_id=lovable] · System profile: agi_evaluation_profiles[system=lovable] · Persona runs: agi_complex_evaluations[system.system_id=lovable] · Methodology config: agi_audits.config
Per-dimension evidence: /evidence/alitheion-eval-lovable

Live API endpoint: /api/v4/site-data/evaluation/lovable
Methodology document: /api/v4/site-data/methodology