Skip to content

HybridEncoder position embedding is scrambled for non-square inputs #362

Open

Description

@ArgoHA

Hi. There is a bug with swapped H and W in position embeddings, so non-square inputs get scrambled positions:

build_2d_sincos_position_embedding(w, h) builds its grid as meshgrid(grid_w, grid_h, indexing="ij") (shape (w, h)), but the encoder flattens tokens row-major ((h, w)). For square maps this is only a transpose. For non-square maps the positions are wrong: on a 14×28 map, the token at row 0, column 20 is given row 1, column 6.

from src.zoo.dfine.hybrid_encoder import HybridEncoder

h, w = 14, 28
pe = HybridEncoder.build_2d_sincos_position_embedding(w, h, embed_dim=8)[0]
print((pe[20, [1, 5]].asin() * 100).round())  # sin(row/100), sin(col/100): expected [0, 20], got [1, 6]

Impact (measured on D-FINE-seg, a D-FINE derivative: S model, Cityscapes detection, val mAP50-95):

  • Square inputs: not affected.
  • Training and evaluating at the same non-square size: no change. The model learns either layout.
  • Square-trained weights, including the released ones, run at a non-square size: 0.281 with the bug vs 0.293 fixed at 448×896.

Possibly related: #111, #345. HuggingFace fixed the same bug: huggingface/transformers#41380.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions