[JosephTian876]

依赖：合并顺序为 #10202（原生视频输入模态）→ 本系列 K2（走 audio 槽的视频还原）→ 本 PR（模态门禁）。单独查看时会包含前两者；它们合并后，本 PR 自动只剩自己的 1 个提交。

fix(gemini): gate video injection on the declared modality

Background

ProviderGoogleGenAI delivers a video as an inline part as soon as it recognises one.
The first change in this series extended that recognition to a video that arrives
through the legacy audio slot (data:video/mp4;… or a saved .mp4), but the delivery
stayed unconditional, while every other route into the provider treats video as
opt-in through the provider's modalities list.

A Gemini model that cannot accept inline video therefore received one and failed the
whole request with an API error, instead of receiving [Video] text the way the agent
path does.

This change is a follow-up addition to that same media-family fix: it keeps the
routing the first change introduced and adds the missing capability check on the one
path that builds its user message inside the provider.

Changes

astrbot/core/provider/sources/gemini_source.py

resolve_video_part now decides before it resolves anything. Two additions do that.

The flag, read once at the top of assemble_context (gemini_source.py:1156-1163):

# Video input is strictly opt-in, the same rule the agent runner applies
# to `ProviderRequest.video_urls` before it assembles a message. This
# method builds the user message for `text_chat(prompt=...)` directly and
# never meets that gate, so a video routed here (including one that
# arrived through the audio slot) is degraded below instead of reaching a
# provider that never declared the modality and failing the whole request.
modalities = self.provider_config.get("modalities", None)
supports_video_input = isinstance(modalities, list) and "video" in modalities

The early return, at the head of resolve_video_part (gemini_source.py:1211-1235):

         async def resolve_video_part(video_ref: str) -> dict | None:
+            # A `[Video]` placeholder keeps the attachment visible in the message
+            # while sparing the model a payload it cannot take. The resolution is
+            # skipped as well: its result would be discarded, and the reference
+            # may be a multi-megabyte file that has to be read and encoded first.
+            if not supports_video_input:
+                return {"type": "text", "text": "[Video]"}
             try:
                 video_data = await resolve_media_ref_to_base64_data(
                     video_ref,

resolve_video_part is the only place on this path that produces a video block, and
all three video entries go through it: the audio slot (gemini_source.py:1186), a
VideoURLPart extra part (:1269), and the video_urls argument (:1292). Gating
there covers the legacy path completely and applies the same rule as the runner:
video has to be declared, so an empty or missing modalities list means "not
declared" and degrades.

The placeholder is decided before the media is read, so an unsupported reference costs
neither a base64 encode nor an ffmpeg run. [Video] is the same text the runner's gate
substitutes (tool_loop_agent_runner.py:388-390) and the same text the
unconfigured-provider sanitizer uses (modalities.py:97 and :165).

The comment above the check records why this duplication exists: the method is reached
through text_chat(prompt=…), which bypasses the runner, so the rule has to be applied
here as well.

Why

The legacy shape is Context.llm_generate, which calls the provider directly:

#	Step	Location
1	the SDK helper calls the provider with the media slots it exposes	astrbot/core/star/context.py:217
2	text_chat(prompt=…) assembles a user message	gemini_source.py:972 → :993-994
3	assemble_context resolves the audio slot	gemini_source.py:1146 → resolve_audio_part:1178
4	the reference is recognised as a video	media_utils.py:408, called at gemini_source.py:1185
5	a video_url block is produced	gemini_source.py:1210 → :1232-1235
6	the block is put into the request payload	gemini_source.py:1021
7	an inline video part is built from it	gemini_source.py:386 → process_video_url:417, dispatch :458

No modality check sits anywhere between step 5 and step 7:

• sanitize_contexts_by_modalities (astrbot/core/provider/modalities.py:66) has
  exactly two callers: tool_loop_agent_runner.py:693 and
  astrbot/core/agent/context/compressor.py:268. Both belong to the agent runner,
  which this path never enters.
• The runner's other gate, _assemble_request_context_for_provider
  (tool_loop_agent_runner.py:346-392), only filters request.video_urls before
  calling ProviderRequest.assemble_context (entities.py:303) — a different function
  from the provider's assemble_context. The runner also never passes prompt=
  (tool_loop_agent_runner.py:529-539), so the branch that produces the block is not
  reachable from it in the first place.
• gemini_source.py reads no input modality list. Every modalities occurrence there
  (:233, :243, :376, :728, :799) is a Gemini response modality
  (TEXT / IMAGE).
• text_chat retries an APIError by rotating API keys only (:1039), with no
  fallback provider and no context sanitization.

A video produced by the legacy path was therefore injected as-is under every provider
configuration. Applying the check at the single block-producing function is also what
keeps the change small: it needs no new parameter on assemble_context, no change to
the modality policy, and no change to the runner.

Tests

tests/unit/test_gemini_legacy_media_kind.py now has 9 tests and 21 cases (was 7 and
18):

• _provider() declares every modality by default, so the media-family tests keep
  testing the routing of a provider that opted into video;
• test_video_in_the_audio_slot_degrades_without_the_video_modality — two cases (an
  empty modalities list, and a list without video): the video is replaced by
  {"type": "text", "text": "[Video]"}, no video_url block is produced, and the
  media resolver is never awaited, which also proves the payload cannot be re-encoded
  to WAV on the way out;
• test_video_url_argument_degrades_without_the_video_modality — the separate
  video_urls channel follows the same rule.

python -m pytest tests/unit/test_gemini_legacy_media_kind.py \
                 tests/test_video_input_modality.py \
                 -q -n 4 -p no:cacheprovider
# 44 passed, 4 warnings (41 before this change)

Gemini and media regression surface — 169 passed:

python -m pytest tests/test_gemini_source.py tests/test_media_utils.py \
                 tests/test_agent_runner_media_resolver.py \
                 tests/test_model_image_preparation.py -q -n 4 -p no:cacheprovider

Full unit suite:

python -m pytest tests/unit -q -p no:cacheprovider
# 4 failed, 2193 passed, 23 skipped

2193 = 2190 + 3 new cases. The 4 failures are the parametrizations of
tests/unit/test_wakeup_pipeline.py::test_wakeup_uses_current_context_and_sends_final_reply
(state leakage between unit tests); the same 4 fail on the preceding change in this
series (4 failed, 2190 passed) and are unrelated to this change.

uvx --from ruff==0.15.22 ruff format --check .   # 546 files already formatted
uvx --from ruff==0.15.22 ruff check .            # All checks passed!

Compatibility

• Providers that declare video keep receiving an inline video on every entry
  point; nothing changes for them.
• Providers that do not declare video now get [Video] text instead of a request
  that fails at the API. This matches what the agent runner already does for
  ProviderRequest.video_urls and what the unconfigured-provider sanitizer already
  does for a replayed history.
• Opting in on this path uses the existing modalities checkbox list — the same
  switch the runner path has always required.
• Audio, images, and text are untouched. The check is only reachable from
  resolve_video_part. text_chat, text_chat_stream, and assemble_context keep
  their signatures, and no dashboard, i18n, or generated asset is touched.

Privacy and delivery

• Nothing is read for a model that cannot take the payload. The placeholder is
  returned before resolution, so an unsupported reference is never downloaded, read,
  or base64-encoded, and no temporary file is created for it.
• No new egress and no new logging. The change adds no request, no endpoint, and no
  log line; the [Video] placeholder carries no path.
• No credentials or user data in the diff. One production file and one test file;
  the tests use synthetic bytes and temporary files.
• Delivery. A single commit on top of the media-family fix in the same series, with
  no configuration change: an existing provider behaves exactly as before until an
  operator ticks video, and behaviour for audio and images is byte-for-byte
  unchanged.

Suggested merge order

This is the second half of one change: the first half makes the legacy path recognise
a video in the audio slot and deliver it, and this half makes that delivery respect the
declared modality. Reviewing them in order keeps each diff small and each step
independently testable.

1. the native video input modality change — adds VideoURLPart, the video modality,
   the video_urls transport, and process_video_url
2. the media-family fix for the legacy audio slot — recognises a video there and routes
   it to that channel
3. this change — gates the injection on the declared modality

Base — this pull request is stacked. Open it against the branch that carries the
media-family fix (fix/video-not-audio-in-legacy-path-k2), not against master. That
branch is reachable here, so the diff shows only the gate — 1 commit, 2 files, +71 / −3
— instead of repeating the previous change for review. With master as the base the
pull request would expose both commits and could not be reviewed on its own, and it
would not even apply before the media-family fix. Merge order: the native video input
modality change, then the media-family fix, then this change; retarget to master
after those two have landed.

If the series is landed as a single PR, this change is the second commit of the
media-family fix and needs no separate review cycle; if it is landed as separate PRs,
it has to follow the two above, because the function it guards does not exist before
them.

The OpenAI and Anthropic sources are unaffected by this change: their legacy
audio_urls branches still do not recognise video at all, and they remain a follow-up.

Summary by Sourcery

Support video attachments end to end while requiring explicit provider video capability and safely degrading unsupported inputs.

New Features:

• Add end-to-end video input support across provider requests, message content parts, media resolution, Gemini processing, and provider capability configuration.
• Expose video input capabilities and opt-in modality badges in the provider dashboard.

Bug Fixes:

• Preserve videos mistakenly delivered through legacy audio channels instead of converting them to audio and losing their visual content.
• Degrade unsupported video attachments to a [Video] placeholder rather than sending unsupported payloads or failing requests.
• Prevent transient video payloads from being persisted in conversation history.

Enhancements:

• Apply consistent video modality filtering to direct requests, fallback contexts, and historical message sanitization, while retaining existing behavior for other media types.
• Add video reference detection, provider handling, and regression coverage for routing, capability gating, history persistence, and attachment collection.

Tests:

• Add comprehensive tests for video request assembly, modality enforcement, Gemini conversion, media-family detection, transient history behavior, and attachment collection.

[sourcery-ai]

Sorry @JosephTian876, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 12 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 80.25478% with 31 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
astrbot/core/provider/entities.py	64.10%	14 Missing ⚠️
astrbot/core/provider/sources/gemini_source.py	74.35%	10 Missing ⚠️
astrbot/core/provider/modalities.py	92.30%	3 Missing ⚠️
astrbot/core/provider/sources/anthropic_source.py	33.33%	2 Missing ⚠️
astrbot/core/provider/sources/openai_source.py	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!