ModernBertModel

ModernBertModel at 22 layers and hidden size 768. 2 published checkpoints share this shape.

The pass (wrapper only)

The order of operations for one forward. ×N marks a position that fires once per layer.

One step. The model's forward was entered and nothing inside it was recorded, so this says the model ran and says nothing about the order of anything. It is not comparable to the fuller traces on other sheets. Recorded from a complete forward of this shape in the author's independent implementation — the structure is what that run emitted, not a reading of the configuration.

encodeIds

The same thing in canonical form:

0

Numbers are positions in the pass, not layer indices. Two passes count as the same structure when these strings match.

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

It is not the recorded pass above, which is captured from a real forward; it is what the published configuration says the arithmetic is. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

From granite-embedding-311m-multilingual-r2

This checkpoint only — its arithmetic differs from the others of this shape.

x = layernorm(embed[ids], emb_norm)       # [T, 768], no position table

for i in 0 .. 21:
    # Alternating attention: layer i is global when i % 3 == 0
    # (so layer 0 is), and windowed otherwise. The window compares
    # |i - j| <= 128 // 2, so `128` spans 64 either side.
    glob = (i % 3 == 0)
    # Layer 0 has no attn_norm — the embedding norm already ran, so
    # 21 layers carry six tensors and layer 0 carries five. A loader
    # that assumes uniform layers normalizes twice here.
    h = x if i == 0 else layernorm(x, layers[i].attn_norm)
    q, k, v = split(h @ layers[i].attn.Wqkv.T, 3)
    q, k = rope(q, k, theta=150000 if glob else 160000)
    a = softmax(q @ k.T / sqrt(64), mask=bidirectional, window=None if glob else 128) @ v
    # Not causal. Every position sees every other one — that is the whole
    # difference between this and the decoders above, and it is why these
    # models embed a sequence rather than continue it.
    x = x + a @ layers[i].attn.Wo.T
    h = layernorm(x, layers[i].mlp_norm)
    g, u = split(h @ layers[i].mlp.Wi.T, 2)
    x = x + (gelu(g) * u) @ layers[i].mlp.Wo.T

x = layernorm(x, final_norm)
emb = normalize(pool(x))                  # the sentence vector

From granite-embedding-english-r2

This checkpoint only — its arithmetic differs from the others of this shape.

x = layernorm(embed[ids], emb_norm)       # [T, 768], no position table

for i in 0 .. 21:
    # Alternating attention: layer i is global when i % 3 == 0
    # (so layer 0 is), and windowed otherwise. The window compares
    # |i - j| <= 128 // 2, so `128` spans 64 either side.
    glob = (i % 3 == 0)
    # Layer 0 has no attn_norm — the embedding norm already ran, so
    # 21 layers carry six tensors and layer 0 carries five. A loader
    # that assumes uniform layers normalizes twice here.
    h = x if i == 0 else layernorm(x, layers[i].attn_norm)
    q, k, v = split(h @ layers[i].attn.Wqkv.T, 3)
    q, k = rope(q, k, theta=80000 if glob else 10000)
    a = softmax(q @ k.T / sqrt(64), mask=bidirectional, window=None if glob else 128) @ v
    # Not causal. Every position sees every other one — that is the whole
    # difference between this and the decoders above, and it is why these
    # models embed a sequence rather than continue it.
    x = x + a @ layers[i].attn.Wo.T
    h = layernorm(x, layers[i].mlp_norm)
    g, u = split(h @ layers[i].mlp.Wi.T, 2)
    x = x + (gelu(g) * u) @ layers[i].mlp.Wo.T

x = layernorm(x, final_norm)
emb = normalize(pool(x))                  # the sentence vector

Notes that apply to more than one block

Stated once here rather than under each block above.

The attention here is bidirectional, which is the one difference that matters: there is no causal mask, so a position may read the whole sequence. Running this stack causally is fluent and is not the model's function.

local_attention: 128 with global_attn_every_n_layers: 3 is the whole reason this model reads long documents cheaply, and a port that makes every layer global is correct-looking and quadratically slower.

position_embedding_type on these checkpoints is inert and misleading. It says absolute on two of them and relative_key_query on the others, and this family ignores it entirely — it always rotates. The keys that decide anything are the two rope bases, global_attn_every_n_layers, local_attention and hidden_activation, and they differ between siblings in both directions, so none can be assumed from another.

Layer 0 carries five tensors where every other layer carries six, because the embedding norm has already run and its attn_norm is an identity. That is the trap the shapes announce and a loader is most likely to miss: assume uniform layers and it either fails to find the tensor or, worse, normalizes twice.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of ModernBertModel

Identical across all of them: heads 12.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

implemented, evidence grade recorded-forward, per the assessment, for each checkpoint of this shape: a converted copy of this checkpoint was loaded and completed a forward pass; the recorded pass on this page came from such a run. The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• an encoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture