Issues

• Legacy /v1/completions providers (openai-api-completions, vllm-completions) drop cached_tokens and reasoning_tokens from usage

  UKGovernmentBEIS/inspect_ai#6480 · Filter by author gangjig opened on Nov 10, 2026
  1

• json_dataset() fails to load a JSON Lines file that contains a blank line

  UKGovernmentBEIS/inspect_ai#6474 · Filter by author KopalliAditya opened on Nov 10, 2026

• task_with(metrics=...) is ignored when computing results, and task_with(scorer=...) drops the task's metrics

  UKGovernmentBEIS/inspect_ai#6473 · Filter by author KopalliAditya opened on Nov 10, 2026

• Model inference inside a tool approver, reviewer or sentinel is exempt from token and turn limits but still charged to cost_limit

  UKGovernmentBEIS/inspect_ai#6472 · Filter by author reeve25 opened on Nov 10, 2026

• MCP tool results drop structuredContent, so scorers only see the text blocks

  UKGovernmentBEIS/inspect_ai#6468 · Filter by author SarthakB11 opened on Nov 10, 2026

• score_on_error counts one sample twice toward fail_on_error when its scorer also errors

  UKGovernmentBEIS/inspect_ai#6456 · Filter by author brandly opened on Nov 10, 2026

• Tavily web_search: configurable base URL, extra request headers and fast search depths

  UKGovernmentBEIS/inspect_ai#6451 · Filter by author Ar4l opened on Nov 10, 2026

• MCP tools: pydantic-model arguments reach the model with an empty schema ($ref/$defs dropped)

  UKGovernmentBEIS/inspect_ai#6450 · Filter by author zhouyi-xiaoxiao opened on Nov 10, 2026

• Chat completions: a reply that is only a <think> block keeps the raw tags in output.completion

  UKGovernmentBEIS/inspect_ai#6449 · Filter by author zhouyi-xiaoxiao opened on Nov 10, 2026

• Anthropic agent bridge turns non-Anthropic reasoning into <think> text that never gets parsed back

  UKGovernmentBEIS/inspect_ai#6448 · Filter by author roeizig opened on Nov 10, 2026

• Sharing lessons from CoSAI’s use of Inspect for code security evaluations

  UKGovernmentBEIS/inspect_ai#6447 · Filter by author thomas-bartlett opened on Nov 10, 2026

• Text containing an encoded <content-internal> tag is read back as provider state

  UKGovernmentBEIS/inspect_ai#6445 · Filter by author dragonstyle opened on Nov 10, 2026

• Anthropic inline text document conversion raises UnicodeEncodeError for lone surrogates

  UKGovernmentBEIS/inspect_ai#6442 · Filter by author haoq-dev opened on Nov 10, 2026
  1

• Agent bridge raw-response calls fail when a response contains a lone surrogate

  UKGovernmentBEIS/inspect_ai#6441 · Filter by author haoq-dev opened on Nov 10, 2026
  1

• at_least() reduces a dict key that is None on every epoch to 0, so aggregate(on_missing=...) never sees it

  UKGovernmentBEIS/inspect_ai#6440 · Filter by author chrikrah opened on Nov 10, 2026
  1

• mode, majority, max and at_least reducers treat a None dict score value as a vote (or 0.0) instead of as missing

  UKGovernmentBEIS/inspect_ai#6438 · Filter by author reeve25 opened on Nov 9, 2026

• Tool parameters typed as a mixed Literal or Enum reject valid calls

  UKGovernmentBEIS/inspect_ai#6435 · Filter by author aatif101 opened on Nov 9, 2026

• Bedrock cross-region inference profile ids (e.g. us.anthropic.claude-sonnet-4-6) get no model info

  UKGovernmentBEIS/inspect_ai#6433 · Filter by author zhouyi-xiaoxiao opened on Nov 9, 2026

• csv_dataset() ignores name= and names the dataset after the CSV's last column

  UKGovernmentBEIS/inspect_ai#6432 · Filter by author zhouyi-xiaoxiao opened on Nov 9, 2026
  1

• cost_limit setter hides the new ceiling from the control channel when the check trips

  UKGovernmentBEIS/inspect_ai#6420 · Filter by author Wilson0511 opened on Nov 9, 2026
  1

• eval() fails on a fresh install of 0.3.288: No module named 'acp.helpers' (agent-client-protocol 1.0.0)

  UKGovernmentBEIS/inspect_ai#6417 · Filter by author sk8ordie84 opened on Nov 9, 2026
  

• math() reads a base subscript like 11010_2 as a Python digit separator, so 11010_{2} scores INCORRECT against it

  UKGovernmentBEIS/inspect_ai#6414 · Filter by author zhouyi-xiaoxiao opened on Nov 9, 2026
  3

• ToolDef.as_tool() drops model_input, so a ToolDef-built tool plays back differently

  UKGovernmentBEIS/inspect_ai#6407 · Filter by author lxcxjxhx opened on Nov 8, 2026

• Agent bridge: input_file crashes with KeyError instead of the media policy error; tool-output files dropped

  UKGovernmentBEIS/inspect_ai#6405 · Filter by author nilayp-codes opened on Nov 8, 2026

• A sample without an id fails the eval with a duplicate id when its position is already used as another sample's id

  UKGovernmentBEIS/inspect_ai#6403 · Filter by author chaitanya0305 opened on Nov 8, 2026