ModernBertModel

ModernBertModel at 22 layers and hidden size 768. 2 published checkpoints share this shape.

Implementing it

How to read this sheet: method.

From granite-embedding-311m-multilingual-r2

x = layernorm(embed[ids], embeddings.norm, eps=1e-12)   # [T, 768], no position table

for i in 0 .. 21:
    glob = (i % 3 == 0)                   # `global_attn_every_n_layers: 3`
    h = x if i == 0 else layernorm(x, layers[i].attn_norm)
                                          # layer 0 has no attn_norm tensor
    q, k, v = split(h @ layers[i].attn.Wqkv.T, 3)
    q, k = rope(q, k, theta=150000 if glob else 160000)
    a = softmax(q @ k.T / sqrt(64), mask=padding) @ v    # bidirectional; a
                                          #   local layer also masks |m - n| > 64
                                          #   (query position m, key position n)
    x = x + a @ layers[i].attn.Wo.T
    h = layernorm(x, layers[i].mlp_norm)
    g, u = split(h @ layers[i].mlp.Wi.T, 2)   # [T, 1152] each; the first half
    x = x + (gelu(g) * u) @ layers[i].mlp.Wo.T   #   is the activated one

x = layernorm(x, final_norm)
# Every LayerNorm here has a weight and no bias (`norm_bias: false`); no
# projection has a bias.
emb = x[0]                                # the CLS row: sentence-transformers
                                          #   `1_Pooling`, `pooling_mode_cls_token`
emb = emb / norm(emb)                     # `2_Normalize`

Differs from the block above

The same arithmetic with these values changed. Everything not listed is as above.

Notes

The attention is bidirectional: the mask is built from the padding mask alone, with no causal term, and local layers add a window of local_attention // 2 = 64 positions either side (modeling_modernbert.py L929-L943). Layer i is global when i % global_attn_every_n_layers == 0, and local layers use local_rope_theta (L464-L471). A port that makes every layer global computes a different function once a sequence is longer than the window, not just a slower one.

Layer 0 has no attn_norm. transformers builds it as nn.Identity(), because the embedding norm has just run (L520-L523), so layer 0 carries five tensors and every other layer six. A loader that assumes uniform layers either fails to find the tensor or inserts a norm the model does not have.

The MLP's Wi output is split in half and the first half is activated: act(input) * gate (L241-L242).

position_embedding_type is not read. These configurations say absolute or relative_key_query, and modeling_modernbert.py at v4.57.1 never refers to the key: the model always rotates. The keys that decide the arithmetic are the two rope bases, global_attn_every_n_layers, local_attention, hidden_activation and norm_eps, and siblings differ in them.

The sentence vector is what each repository's sentence-transformers configuration builds: granite-embedding-311m-multilingual-r2; granite-embedding-english-r2. 1_Pooling/config.json selects the CLS row in all of them.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of ModernBertModel

Identical across all of them: heads 12.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• each of the 2 checkpoints of this shape: runs, recorded-forward — this checkpoint was loaded and completed a forward pass

Parts of the author's unpublished implementation that exist for this shape:

• an encoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture

Recorded pass: passes.