OpenAI Codex CLI Expert
Aletheion methodology evaluation · system_id: codex-cli · audit date: 2026-03-26
Per-dimension breakdown (task-level)
| Dimension | Score | Tasks | Per-task scores | Result |
|---|
Persona-based runs
Simulated user runs evaluating this system across personas. Full transcripts (interaction_log) are stored in agi_complex_evaluations and reachable via the per-system evidence endpoint.
judge: claude-sonnet-4-6 · would succeed: True · would recommend: False
“The college student eventually gets a working CSV analyzer, but the journey is bumpy. codex-cli generates functional code but consistently fails to anticipate the user's environment (missing dependencies, macOS python vs python3, blocking plt.show()). Each issue requires a follow-up turn to resolve, turning a potentially 3-turn task into 7 turns over ~35 minutes. The tool never proactively bundles dependencies into a requirements.txt or setup instructions, forcing the user to discover problems at runtime. The user's tech comfort level (2/5) means they can follow instructions but struggle when ...”
Show transcript (7 turns)
User: hey can you make a python script that reads a csv file and shows like the average and stuff for each column
System: codex-cli generates a ~40-line Python script using pandas: reads CSV via sys.argv[1], prints mean/median/count per numeric column using df.describe(). Outputs to terminal. No chart yet. Includes basic try/except for FileNotFoundError.
confusion: low
User: it says no module named pandas?? how do i fix that
System: codex-cli responds with 'run pip install pandas in your terminal'. Does not proactively offer a stdlib-only alternative. Does not check if user is in a venv or has pip configured.
confusion: medium
User: ok it works now but it doesnt show a chart, you said it would show a chart
System: codex-cli appends matplotlib bar chart code to the script. Uses df.mean().plot(kind='bar') and plt.show(). Adds 'pip install matplotlib' instruction. Chart pops up in a window.
confusion: medium
User: what happens if the csv is empty or has weird data, will it crash
System: codex-cli adds edge case handling: checks if DataFrame is empty and prints a message, wraps numeric conversion with errors='coerce' in pd.to_numeric, adds a check for all-NaN columns. Code grows to ~65 lines.
confusion: medium
... 3 more turns omitted ...
judge: claude-sonnet-4-6 · would succeed: True · would recommend: True
“The college student successfully built a working weather app but the journey was bumpy. codex-cli generated functional code quickly but consistently skipped setup prerequisites (pip install, Python path), assumed developer context the user didn't have, and missed important real-world concerns like .gitignore for API key safety. The user had to reprompt for nearly every practical concern — how to run it, dependency errors, security, error handling, UX improvements, and interactivity. Each individual response from codex-cli was mostly correct in isolation, but the system never proactively antici...”
Show transcript (7 turns)
User: hey can you make a weather app that shows me the forecast? I have an openweathermap api key
System: codex-cli generates a Python script using requests library, hardcodes a placeholder API_KEY variable, calls the OpenWeatherMap current weather endpoint, prints temperature and description. Outputs ~40 lines of code with no setup instructions.
confusion: ?
User: ok but where do I put my actual api key? and how do I run this
System: codex-cli explains to replace 'YOUR_API_KEY_HERE' with the actual key in the script, and says to run 'python weather.py' in terminal. Does not mention installing the requests library or creating a virtual environment.
confusion: ?
User: I got an error: ModuleNotFoundError: No module named 'requests'
System: codex-cli correctly tells user to run 'pip install requests' and retry. Response is short and accurate.
confusion: ?
User: nice it works! but the api key is just sitting in the file, my friend said thats bad. can you use a .env file or something
System: codex-cli rewrites the script to use python-dotenv, creates a sample .env file snippet, updates the code to load os.environ. Tells user to install python-dotenv. Does not explain that .env should be added to .gitignore.
confusion: ?
... 3 more turns omitted ...
judge: claude-sonnet-4-6 · would succeed: True · would recommend: False
“The college student managed to get a working real-time dashboard but the journey was rocky. The biggest friction points were environment setup (Node.js not installed, missing npm modules) which codex-cli handled reactively rather than proactively. The tool assumed terminal and Node familiarity that this persona doesn't have at tech level 2/5. Once the environment was sorted, the iterative feature requests (more stocks, dark theme, responsiveness) were handled reasonably well, though partial code snippets caused a regression that required a full-file re-request. The final product meets the core...”
Show transcript (7 turns)
User: hey can you make me a live dashboard that shows stock prices updating in real time? like something that looks cool with charts
System: codex-cli generates a project scaffold: creates index.html, a Node.js server (server.js) using Express + WebSocket (ws library), and a basic Chart.js frontend. It outputs ~120 lines of code across 3 files. The server sends mock stock data every second via WebSocket. Instructions say 'run npm install && node server.js'. No explanation of what npm is or how to install Node.
confusion: medium
User: ok i tried running it but i got an error that says 'node is not recognized as an internal or external command' what do i do
System: codex-cli responds with: 'You need to install Node.js. Download it from nodejs.org, run the installer, then restart your terminal.' It may also offer to rewrite the solution using a CDN-only approach with no backend. However it presents both options simultaneously without recommending one, leaving the user to decide.
confusion: high
User: ok i installed node now it says 'cannot find module ws' when i run server.js
System: codex-cli correctly identifies the issue and says 'run npm install ws express in your project folder'. It may also generate a package.json if one wasn't created initially. This is a straightforward fix but requires the user to understand what a 'project folder' means and navigate there in terminal.
confusion: medium
User: ok its running now! but the chart only shows one stock, can you add like 3 more stocks and make it look nicer with different colors and maybe a dark theme
System: codex-cli rewrites the frontend with 4 mock stock tickers (AAPL, GOOGL, TSLA, MSFT), adds distinct colors per series in Chart.js, and applies a dark CSS theme using CSS variables. The code is functional but the Chart.js config is somewhat verbose and not well-commented. The dark theme is basic (dark background, light text) but not polished. It overwrites the previous files entirely without diffing
confusion: low
... 3 more turns omitted ...