Bolt.new Competent

Aletheion methodology evaluation · system_id: bolt-new · audit date: 2026-03-26

50.4
Vendor: StackBlitz
Tier: Competent · rank 6
Tasks: 24 total · 0 passed · 0 failed
Methodology version: —
Composite score: 0.504 CONDITIONAL

Per-dimension breakdown (task-level)

DimensionScoreTasksPer-task scoresResult

Persona-based runs

Simulated user runs evaluating this system across personas. Full transcripts (interaction_log) are stored in agi_complex_evaluations and reachable via the per-system evidence endpoint.

marketing_manager · project: p2-marketing-campaign
CONDITIONAL composite 0.596
judge: claude-sonnet-4-6 · would succeed: False · would recommend: False
“The session illustrates a fundamental mismatch between bolt-new's default behavior (build a web app) and what a non-technical small business owner actually needs (a written plan/document). The AI consistently defaulted to code-first solutions even when the user explicitly asked for a document. Content quality when it finally emerged was decent — the ad copy was platform-appropriate after correction, the email sequence covered awareness through conversion, and the tracking plan was technically sound — but the delivery mechanism was wrong for this persona throughout. The user spent significant c...”
Show transcript (5 turns)
User: Hi I need help creating a marketing campaign for my small business. We sell organic skincare products and I want to run ads on Facebook, Google and LinkedIn and also need emails to send customers. Can you help me set this up?
System: bolt-new generates a React app scaffold with a campaign dashboard UI, including tabs for each platform, a form builder for ad copy, and an email template editor. It outputs JSX components, installs dependencies like react-router and tailwind, and shows a live preview of a dashboard. The terminal shows npm install running.
confusion: 3
User: Wait I don't need a website or anything like that. I just need like a document or plan I can follow. Like who should I target, what should the ads say, what emails to send. Can you just write that out for me?
System: bolt-new pivots and generates a markdown file (campaign-plan.md) inside the project, displaying it in the preview pane. It includes sections for target audience, ad copy for all three platforms, a landing page outline, 5 email templates, and a basic tracking plan. The content is reasonably good but the markdown renders oddly in the preview and there are some placeholder brackets like [YOUR PRODUCT
confusion: 2
User: Ok this looks better but some parts say YOUR PRODUCT NAME and YOUR WEBSITE - can you fill those in? My product is called GlowNature skincare and my website is glownature.com. Also the LinkedIn ad doesn't look right it sounds the same as Facebook
System: bolt-new updates the markdown file, replacing placeholders with GlowNature and glownature.com. It rewrites the LinkedIn ad copy to be more B2B/professional in tone (targeting estheticians, spa owners, wellness retailers as resellers). However, it also attempts to 'improve' the file by converting it into a React component that renders the plan as a styled page, breaking the simple document format t
confusion: 3
User: How do I save this or print it? I want to put it in a Word document or something. Also I don't see the tracking plan - like how do I know if the ads are working?
System: bolt-new adds a tracking plan section to the React component covering UTM parameters, Facebook Pixel, Google Analytics goals, and LinkedIn Insight Tag. It explains these concepts briefly inline. For saving, it suggests downloading the project as a zip or copying from the preview. It does NOT offer a simple export to PDF or plain text. The tracking plan mentions setting up Google Tag Manager which
confusion: 4
... 1 more turns omitted ...
freelancer · project: p2-marketing-campaign
CONDITIONAL composite 0.591
judge: claude-sonnet-4-6 · would succeed: True · would recommend: False
“The session illustrates bolt-new's core tension: it defaults to code-first outputs even when the task is purely content-driven. A non-technical freelance creative asking for marketing copy should never see 'npm install' in the first response. The system required two full turns just to get past the technical scaffolding and into actual business content. Once redirected to plain text, the quality of the marketing content itself was moderate-to-good — Facebook and Google ad copy were serviceable, the email sequence covered the basics, and the Squarespace tracking steps (turn 4) were genuinely hel...”
Show transcript (5 turns)
User: Hi I need to create a full marketing campaign for my handmade jewelry business. I sell to women 30-55 who like unique accessories. Can you help me build the whole thing - like ads for Facebook and Google and LinkedIn, plus a landing page and some emails to send people?
System: bolt-new generates a large scaffolded React/Next.js project with components for a landing page, placeholder ad copy sections, and an email template component. It outputs a file tree with 12+ files including package.json, tailwind config, and JSX components. It says 'Here's your marketing campaign app! Run npm install then npm run dev to see it.'
confusion: high
User: I don't know what npm is or how to run anything. I just need the actual words for the ads and emails, like in a document I can read and use. Can you just give me the text?
System: bolt-new pivots and generates a large markdown document with sections: Target Audience, Facebook Ad Copy (3 variants), Google Ad Copy (headlines + descriptions), LinkedIn Ad Copy, Landing Page copy, and 5 email templates. The content is reasonably good but LinkedIn copy is generic B2B-focused rather than appropriate for jewelry. The tracking plan section mentions Google Tag Manager, GA4 events, an
confusion: medium
User: This is helpful! But the LinkedIn stuff doesn't make sense for jewelry - why would I advertise jewelry on LinkedIn like it's a business product? Also the tracking plan talks about Tag Manager and pixels - I have no idea what any of that means. Can you fix the LinkedIn ads and explain the tracking in
System: bolt-new rewrites LinkedIn copy to target women professionals who want to express personal style at work - a reasonable fix. For tracking, it provides a simplified explanation but then immediately reverts to listing UTM parameters, conversion events, and suggests installing Facebook Pixel via 'adding a script tag to your website header.' It adds a note saying 'If you use Shopify or Squarespace, th
confusion: medium
User: OK the LinkedIn ads are better now. I use Squarespace for my website. Can you give me the tracking plan just for Squarespace? Like step by step what I actually click on to set it up?
System: bolt-new provides a step-by-step Squarespace tracking guide covering: connecting Google Analytics via Squarespace Settings > Advanced > External Services, adding Facebook Pixel ID in the same panel, and setting up basic conversion tracking. Steps are numbered and relatively clear. However, it also generates a new React component for 'tracking dashboard visualization' that was not asked for, clutte
confusion: low
... 1 more turns omitted ...
small_business_owner · project: p2-marketing-campaign
CONDITIONAL composite 0.578
judge: claude-sonnet-4-6 · would succeed: True · would recommend: False
“The small business owner eventually extracted most of the required campaign components but the journey was significantly friction-filled. bolt-new's default behavior of generating a full code project (React app, Node.js files, package.json) was immediately alienating for a tech-level-1 user who needed copyable text and simple instructions, not a software project. The first two turns were largely wasted on confusion and re-orientation. The system did respond reasonably well to plain-English correction requests, pivoting toward simpler HTML and text files, but it never fully shed its developer-c...”
Show transcript (6 turns)
User: Hi I need to create a marketing campaign for my bakery. We sell custom cakes and pastries. Can you help me make ads and stuff for Facebook and Google and maybe LinkedIn? Also I need emails to send customers.
System: bolt-new generates a large scaffolded project: creates a React app with multiple route pages, outputs JSX components for a 'landing page', dumps raw ad copy into code comments, and generates a Node.js email template file. The UI shows a code editor with multiple files open. No plain-English summary is provided upfront.
confusion: ?
User: Wait I don't understand any of this code stuff. I just need like the words for the ads and maybe a simple webpage. Can you just show me the text I need to copy and paste into Facebook?
System: bolt-new partially pivots: generates a plain HTML file with the ad copy written out in visible text sections, and adds a simple styled landing page in HTML/CSS. The Facebook ad copy appears in a <div> block. However it also still generates a package.json and suggests running 'npm install' to preview the landing page. The email sequence is now in a separate .txt file but the user has to navigate th
confusion: ?
User: Ok I can see some words now thank you. But where are the 5 emails? And what is this tracking plan thing I need? Also how do I actually put this webpage online so customers can see it?
System: bolt-new generates a new file 'emails.txt' with 5 email drafts (welcome, follow-up, promotion, re-engagement, loyalty). It adds a 'tracking-plan.md' markdown file listing Google Analytics events, Facebook Pixel setup, and UTM parameters. For deployment it suggests Netlify and provides CLI commands: 'netlify deploy --prod'. The tracking plan is technically correct but uses jargon like 'conversion e
confusion: ?
User: I don't know what netlify or UTM means. Can you explain the tracking plan in normal words? Like what do I actually DO to track if my ads are working? And the LinkedIn ad - I don't think you made one of those yet.
System: bolt-new rewrites the tracking-plan.md in simpler language, explaining UTM as 'special links that tell you where visitors came from' and Facebook Pixel as 'a small piece of code your web developer adds to your site.' It adds LinkedIn ad copy to a new file. However it still references 'your developer' for implementation steps, implicitly acknowledging the user can't do this alone. The LinkedIn ad c
confusion: ?
... 2 more turns omitted ...
Evidence provenance
Profile: corpus_taas_profiles[system_id=bolt-new] · Audit definition: corpus_agi_audits[system_id=bolt-new] · System profile: agi_evaluation_profiles[system=bolt-new] · Persona runs: agi_complex_evaluations[system.system_id=bolt-new] · Methodology config: agi_audits.config
Per-dimension evidence: /evidence/alitheion-eval-bolt-new

Live API endpoint: /api/v4/site-data/evaluation/bolt-new
Methodology document: /api/v4/site-data/methodology
Aletheion · Independent epistemic evaluation. We rate methodology, not conclusions. Page rendered 2026-09-09T01:32:23.004229Z.