Cursor Agent Competent
Aletheion methodology evaluation · system_id: cursor-agent · audit date: 2026-03-26
Per-dimension breakdown (task-level)
| Dimension | Score | Tasks | Per-task scores | Result |
|---|
Persona-based runs
Simulated user runs evaluating this system across personas. Full transcripts (interaction_log) are stored in agi_complex_evaluations and reachable via the per-system evidence endpoint.
judge: claude-sonnet-4-5-20250929 · would succeed: True · would recommend: True
“The small business owner would likely achieve a working weather app through this interaction, though with some frustration along the way. The cursor-agent performs well at generating initial code and responding to styling requests, showing good productivity gains over manual coding. However, the interaction reveals pain points typical for non-technical users: API key acquisition happens outside the tool, error messages require some technical literacy to interpret, and security best practices (environment variables) are mentioned without clear implementation guidance. The business owner experie...”
Show transcript (7 turns)
User: I need to build a simple weather app for my business website. Can you help me create something that shows the forecast?
System: I'll help you build a weather app! We'll use the OpenWeatherMap API. First, let me create an HTML file with a simple interface and JavaScript to fetch weather data.
confusion: LOW
User: That looks good but I don't have an API key. How do I get one?
System: You'll need to sign up at openweathermap.org/api. It's free for basic use. Once you have the key, replace 'YOUR_API_KEY' in the code with your actual key.
confusion: MEDIUM
User: I got the key but when I try it, nothing shows up. The page is blank.
System: Let me check the console for errors. It looks like there might be a CORS issue or the API call isn't completing. Let me add error handling and console logging to debug.
confusion: MEDIUM
User: I see 'Error fetching weather data' in red text now. What does that mean?
System: This usually means the API key isn't working or the city name format is wrong. Make sure you activated your API key (can take 10 minutes) and try entering 'London,UK' format.
confusion: MEDIUM
... 3 more turns omitted ...
judge: claude-sonnet-4-6 · would succeed: True · would recommend: True
“The college student successfully completed the task but required significantly more hand-holding than a well-designed tool should demand. cursor-agent produced functional, reasonably correct code on the first attempt, which is its strongest showing. However, it consistently failed to anticipate the user's environment and knowledge gaps: it didn't explain how to invoke the CLI tool, didn't proactively handle the matplotlib GUI issue (a very common problem for students on WSL/SSH), and didn't include edge case handling upfront. Each of these omissions cost an extra turn. The student experienced ...”
Show transcript (7 turns)
User: hey can you help me build a csv analyzer thing for the command line? it needs to show like mean median and count and also make a chart somehow. i have python installed
System:
confusion: low
User: ok i ran it and got ModuleNotFoundError: No module named 'pandas' lol what do i do
System:
confusion: low
User: ok now i ran python csv_analyzer.py and it says 'error: the following arguments are required: filename' how do i give it a file
System:
confusion: medium
User: can you make me a sample csv file to test with? like with some numbers
System:
confusion: low
... 3 more turns omitted ...
judge: claude-sonnet-4-6 · would succeed: True · would recommend: True
“The college student successfully built a working weather app but the journey had notable friction points. Cursor-agent was generally helpful and appropriately simplified explanations for a non-developer, but made several UX missteps: it didn't proactively warn about common pitfalls (pip vs pip3 on Windows, .env quote syntax), introduced concepts like python-dotenv without sufficient scaffolding, and the jump from simple hardcoded key to .env management was jarring for this skill level. The student spent roughly 15 of 47 minutes debugging preventable issues. The final product meets all success ...”
Show transcript (5 turns)
User: hey can you help me build a weather app that shows the forecast? I want to use a free weather API. I don't really know where to start lol
System:
confusion: low
User: ok I got the API key but where do I put it? also when I run it it says ModuleNotFoundError: No module named 'requests'
System:
confusion: medium
User: it works!! but the output looks really ugly, can you make it look nicer? also can I make it so I can type in any city instead of it being hardcoded
System:
confusion: low
User: what happens if someone types a city that doesn't exist? it just crashes with a weird error. also my friend said I shouldn't put my API key in the code, how do I fix that?
System:
confusion: high
... 1 more turns omitted ...