Hands-on smoke test · 2026-10-19 · AI Coding Agents & IDEs · harness smoke-0.1
Hands-on smoke test (install + headless flags, no credentials)
Installed each CLI from its public package registry on a clean Linux box (no accounts, no API keys), ran --version and --help, and tried one non-interactive prompt. All five installed and exposed a documented headless/non-interactive mode. Every prompt stopped at authentication, as expected, so nothing about output quality was measured.
Rank impact: none. No score or rank was changed by this log.
Method
- npm install (public npm registry): @anthropic-ai/claude-code, @openai/codex, @github/copilot, cline. uv tool install openhands (PyPI).
- Ran with a fresh HOME directory and no credentials in the environment.
- Captured --version and --help, and checked --help for the documented headless flag.
- Ran one prompt ('Reply with OK') in each tool's headless mode and saved the output verbatim (raw files below). Exit codes weren't captured, so we report the messages.
Results
| tool | version | installed | headless flag | headless run |
|---|---|---|---|---|
| Claude Code | 2.1.197 | true | -p / --print, --output-format json|stream-json | Stopped at auth: JSON result with is_error true, 'Not logged in · Please run /login' |
| OpenAI Codex | codex-cli 0.163.0 | true | codex exec (non-interactive), codex review | Stopped at auth: session started (codex exec), then '401 Unauthorized: Missing bearer or basic authentication' from api.openai.com after 5 reconnects |
| GitHub Copilot | GitHub Copilot CLI 1.0.98 | true | -p / --prompt (non-interactive) | Stopped at auth: 'Error: No authentication information found' (lists /login, COPILOT_GITHUB_TOKEN/GH_TOKEN/GITHUB_TOKEN, gh auth login) |
| Cline | 3.0.73 | true | --json, -y/--yolo, --auto-approve | Stopped at auth: JSON run_result finishReason error, 'Unauthorized ... re-authenticate your Cline account' |
| OpenHands | OpenHands CLI 1.16.0 (SDK v1.21.0) | true | --headless --json -t TASK | Stopped at setup: 'Headless mode requires existing settings. Please run: openhands to configure your settings' (no LLM configured) |
Not covered
- cursor (installer is a curl|bash script that then needs sign-in; not run)
- devin (CLI requires a Devin account)
Limitations
Confirms installability and the existence of a scriptable mode only. Says nothing about task success, speed, cost or code quality. No score or rank changed from this test.
Raw files
claude--version.txt
2.1.197 (Claude Code)
claude_run.txt
{"type":"result","subtype":"success","is_error":true,"api_error_status":null,"duration_ms":102,"duration_api_ms":0,"num_turns":1,"result":"Not logged in · Please run /login","stop_reason":"stop_sequence","session_id":"4d7a9e75-e2e3-4d74-9225-f76c565d156f","total_cost_usd":0,"usage":{"input_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":0,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"inference_geo":"","iterations":[],"speed":"standard"},"modelUsage":{},"permission_denials":[],"terminal_reason":"completed","fast_mode_state":"off","uuid":"434ced57-cd4f-4e0a-9587-1bda9a3d7a05"}
codex--version.txt
codex-cli 0.163.0
codex_run.txt
Reading additional input from stdin... OpenAI Codex v0.163.0 -------- workdir: ~ model: gpt-6.1-sol provider: openai approval: never sandbox: read-only reasoning effort: none reasoning summaries: none session id: 01a1527f-8944-73a2-ab2a-99bc60e4e531 -------- user Reply with OK 2026-10-19T04:50:52.467085Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://api.openai.com/v1/responses 2026-10-19T04:50:52.516425Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://api.openai.com/v1/responses 2026-10-19T04:50:52.755903Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://api.openai.com/v1/responses ERROR: Reconnecting... 2/5 2026-10-19T04:50:53.197787Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://api.openai.com/v1/responses ERROR: Reconnecting... 3/5 2026-10-19T04:50:54.103197Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://api.openai.com/v1/responses ERROR: Reconnecting... 4/5 2026-10-19T04:50:55.864824Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://api.openai.com/v1/responses ERROR: Reconnecting... 5/5 2026-10-19T04:50:59.047914Z ERROR codex_api::endpoint::responses_websocket: failed to connect to websocket: HTTP error: 401 Unauthorized, url: wss://api.openai.com/v1/responses warning: Falling back from WebSockets to HTTPS transport. unexpected status 401 Unauthorized: Missing bearer or basic authentication in header, url: wss://api.openai.com/v1/responses, cf-ray: a4cd2a3f0b8f6006-IAD ERROR: Reconnecting... 1/5 ERROR: Reconnecting... 2/5 ERROR: Reconnecting... 3/5 ERROR: Reconnecting... 4/5 ERROR: Reconnecting... 5/5 ERROR: unexpected status 401 Unauthorized: Missing bearer or basic authentication in header, url: https://api.openai.com/v1/responses, cf-ray: a4cd2a693bdedb5c-IAD, request id: req_b9602fa781f341b09886d0203dd875d3 ERROR: unexpected status 401 Unauthorized: Missing bearer or basic authentication in header, url: https://api.openai.com/v1/responses, cf-ray: a4cd2a693bdedb5c-IAD, request id: req_b9602fa781f341b09886d0203dd875d3
copilot--version.txt
GitHub Copilot CLI 1.0.98. Run 'copilot update' to check for updates.
copilot_run.txt
Error: No authentication information found. Copilot can be authenticated with GitHub using an OAuth Token or a Fine-Grained Personal Access Token. To authenticate, you can use any of the following methods: • Start 'copilot' and run the '/login' command • Set the COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN environment variable • Run 'gh auth login' to authenticate with the GitHub CLI
cline--version.txt
3.0.73
cline_run.txt
{"ts":"2026-10-19T04:50:41.276Z","type":"hook_event","hookEventName":"agent_start","agentId":"agent_bf51d949-5b24-4b91-a97a-0b93c0ce7a3e","taskId":"conv_1792385441269_wrufhoc","parentAgentId":null}
{"ts":"2026-10-19T04:50:41.278Z","type":"agent_event","event":{"type":"iteration_start","iteration":1}}
{"ts":"2026-10-19T04:50:41.360Z","type":"hook_event","hookEventName":"agent_error","agentId":"agent_bf51d949-5b24-4b91-a97a-0b93c0ce7a3e","taskId":"conv_1792385441269_wrufhoc","parentAgentId":null}
{"ts":"2026-10-19T04:50:41.362Z","type":"hook_event","hookEventName":"agent_start","agentId":"agent_bf51d949-5b24-4b91-a97a-0b93c0ce7a3e","taskId":"conv_1792385441269_wrufhoc","parentAgentId":null}
{"ts":"2026-10-19T04:50:41.363Z","type":"agent_event","event":{"type":"iteration_start","iteration":1}}
{"ts":"2026-10-19T04:50:41.405Z","type":"hook_event","hookEventName":"agent_error","agentId":"agent_bf51d949-5b24-4b91-a97a-0b93c0ce7a3e","taskId":"conv_1792385441269_wrufhoc","parentAgentId":null}
{"ts":"2026-10-19T04:50:41.405Z","type":"agent_event","event":{"type":"error","error":{"name":"Error","message":"Unauthorized: Please make sure you're using the latest version of Cline and re-authenticate your Cline account. (Request ID: JIHNpLRpxqbeVkozZgmcdjowxNXEskuI)","stack":"Error: Unauthorized: Please make sure you're using the latest version of Cline and re-authenticate your Cline account. (Request ID: JIHNpLRpxqbeVkozZgmcdjowxNXEskuI)\n at execute (/$bunfs/root/chunk-4qpecet5.js:307:7022)\n at processTicksAndRejections (native:7:39)"},"errorClass":"auth","recoverable":false,"iteration":1}}
{"ts":"2026-10-19T04:50:41.414Z","type":"run_result","finishReason":"error","iterations":1,"usage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0,"totalCost":0},"aggregateUsage":{"inputTokens":0,"outputTokens":0,"cacheReadTokens":0,"cacheWriteTokens":0,"totalCost":0},"durationMs":44,"text":"Unauthorized: Please make sure you're using the latest version of Cline and re-authenticate your Cline account. (Request ID: JIHNpLRpxqbeVkozZgmcdjowxNXEskuI)","model":{"id":"anthropic/claude-sonnet-5.5","provider":"cline","info":{"id":"anthropic/claude-sonnet-5.5","name":"Claude Sonnet 5.5","contextWindow":1000000,"maxInputTokens":1000000,"maxTokens":128000,"capabilities":["images","files","tools","reasoning","structured_output","prompt-cache"],"reasoningOptions":[{"type":"effort","values":["low","medium","high","xhigh","max"]}],"pricing":{"input":2,"output":10,"cacheRead":0.1,"cacheWrite":2.5},"releaseDate":"2026-09-28","family":"claude-sonnet","operation":"language"}}}
{"ts":"2026-10-19T04:50:41.414Z","type":"error","message":"Unauthorized: Please make sure you're using the latest version of Cline and re-authenticate your Cline account. (Request ID: JIHNpLRpxqbeVkozZgmcdjowxNXEskuI)"}
openhands--version.txt
~/uvtools/openhands/lib/python3.12/site-packages/openhands/sdk/llm/auth/openai.py:25: AuthlibDeprecationWarning: authlib.jose module is deprecated, please use joserfc instead. It will be compatible before version 2.0.0. from authlib.jose import JsonWebKey, jwt +----------------------------------------------------------------------+ | OpenHands SDK v1.21.0 | | | | Report a bug: github.com/OpenHands/software-agent-sdk/issues | | Get help: openhands.dev/joinslack | | Scale up: openhands.dev/product/sdk | | | | Set OPENHANDS_SUPPRESS_BANNER=1 to hide this message | +----------------------------------------------------------------------+ OpenHands CLI 1.16.0
openhands_run.txt
~/uvtools/openhands/lib/python3.12/site-packages/openhands/sdk/llm/auth/openai.py:25: AuthlibDeprecationWarning: authlib.jose module is deprecated, please use joserfc instead. It will be compatible before version 2.0.0. from authlib.jose import JsonWebKey, jwt +----------------------------------------------------------------------+ | OpenHands SDK v1.21.0 | | | | Report a bug: github.com/OpenHands/software-agent-sdk/issues | | Get help: openhands.dev/joinslack | | Scale up: openhands.dev/product/sdk | | | | Set OPENHANDS_SUPPRESS_BANNER=1 to hide this message | +----------------------------------------------------------------------+ Headless mode requires existing settings. Please run: openhands to configure your settings before using --headless. Goodbye! 👋 Conversation ID: b35d68e371cb4dfcb0da738ffd964d71 Hint: run openhands --resume b35d68e3-71cb-4dfc-b0da-738ffd964d71 to resume this conversation.