Repository navigation
Issues with the MS-Swift Pipeline Layout Feature When Used with MTP #9823
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
Swift’s Pipeline Layout is very similar to that of Megatron-LM/Bridge, but the following issues arise when MTP is used:
- Unlike Swift, Megatron requires you to enter as many “m”s in the pipeline layout as there are MTP layers. However, Swift only requires a single “m” regardless of the number of layers.
- In Swift, a VP stage containing MTP layers must include a transformer layer; otherwise (e.g.
E(|tt)*30,|mLfor Qwen 3.5 397B-A17B), the following error will occur:
[rank237]: Traceback (most recent call last):
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/cli/_megatron/sft.py", line 7, in <module>
[rank237]: megatron_sft_main()
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/megatron/pipelines/train/sft.py", line 97, in megatron_sft_main
[rank237]: return MegatronSft(args).main()
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/pipelines/base.py", line 52, in main
[rank237]: result = self.run()
[rank237]: ^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/megatron/pipelines/train/sft.py", line 70, in run
[rank237]: trainer = self.prepare_trainer()
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/megatron/pipelines/train/sft.py", line 37, in prepare_trainer
[rank237]: return MegatronTrainer(self.args, self.template)
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/megatron/trainers/base.py", line 69, in __init__
[rank237]: self.prepare_model()
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/megatron/trainers/base.py", line 186, in prepare_model
[rank237]: self.unwrapped_models = get_mcore_model(args, self.template.config)
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/swift/megatron/model/utils.py", line 85, in get_mcore_model
[rank237]: models = _get_mcore_model(config)
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/mcore_bridge/model/register.py", line 230, in get_mcore_model
[rank237]: model = loader.build_model(pre_process, post_process, vp_stage=i)
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/mcore_bridge/model/gpts/qwen3_next_gdn.py", line 139, in build_model
[rank237]: model = super().build_model(pre_process, post_process, vp_stage)
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/mcore_bridge/model/register.py", line 193, in build_model
[rank237]: mtp_block_spec = self.get_mtp_block_spec(transformer_layer_spec, vp_stage=vp_stage)
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/mcore_bridge/model/register.py", line 139, in get_mtp_block_spec
[rank237]: mtp_block_spec = get_gpt_mtp_block_spec(
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/megatron/core/models/gpt/gpt_layer_specs.py", line 742, in get_gpt_mtp_block_spec
[rank237]: return get_gpt_mtp_block_spec_for_backend(
[rank237]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank237]: File "/opt/venv-mbridge/lib/python3.12/site-packages/megatron/core/models/gpt/gpt_layer_specs.py", line 761, in get_gpt_mtp_block_spec_for_backend
[rank237]: transformer_layer_spec = spec.layer_specs[-1]
[rank237]: ~~~~~~~~~~~~~~~~^^^^
[rank237]: IndexError: list index out of range
Are these features or bugs?
How to Reproduce / 如何复现
- Model: Qwen 3.5 397B-A17B
--pipeline_model_parallel_layout 'E(|tt)*30,|mL'
Additional Information / 补充信息
N/A
Activity
taking-lying-flat commentedon Aug 3, 2026
https://github.com/modelscope/mcore-bridge/pull/155/changes。 可以在 mcore-bridge 这个开源仓库进行修复
github-actions commentedon Nov 2, 2026
This issue has been inactive for over 3 months and will be automatically closed in 7 days. If this issue is still relevant, please reply to this message.
HowardZorn commentedon Nov 4, 2026
This issue is ongoing and has not yet been resolved. However, it is more likely to belong to another repository (mcore-bridge).
hjh0119 commentedon Nov 6, 2026
sorry for the late reply, this issue should be fixed by modelscope/mcore-bridge#256