RobertaModel

RobertaModel at 6 layers and hidden size 384. One published checkpoint has this shape.

Implementing it

How to read this sheet: method.

Generated from granite-embedding-30m-english.

x = word[ids] + position[pos] + token_type[0]   # [T, 384]
x = layernorm(x, embeddings.LayerNorm, eps=1e-12)
# pos = padding_idx + 1 + i for real tokens (`pad_token_id: 1`), and
#   padding_idx for padding: real tokens start at row 2 (see Notes).

for i in 0 .. 5:
    q = x @ layers[i].attention.self.query.T + layers[i].attention.self.query.bias
    k = x @ layers[i].attention.self.key.T   + layers[i].attention.self.key.bias
    v = x @ layers[i].attention.self.value.T + layers[i].attention.self.value.bias
    a = softmax(q @ k.T / sqrt(32), mask=padding) @ v     # bidirectional
    o = a @ layers[i].attention.output.dense.T + layers[i].attention.output.dense.bias
    x = layernorm(x + o, layers[i].attention.output.LayerNorm, eps=1e-12)   # post-norm
    h = gelu(x @ layers[i].intermediate.dense.T + layers[i].intermediate.dense.bias)   # [T, 1536]
    o = h @ layers[i].output.dense.T + layers[i].output.dense.bias
    x = layernorm(x + o, layers[i].output.LayerNorm, eps=1e-12)

emb = x[0]                                # the CLS row (`1_Pooling`)
emb = emb / norm(emb)                     # `2_Normalize`

Notes

Position ids start at padding_idx + 1 = 2. create_position_ids_from_input_ids numbers real tokens from padding_idx + 1 and gives padding the id padding_idx (modeling_roberta.py L1537-L1550). With pad_token_id: 1, row 0 of the 514-row position table is never read, row 1 only at padding positions, and a 512-token input reads rows 2 to 513, so the 514-row table (max_position_embeddings) holds 512 usable positions. A port that starts at row 0 looks everything up successfully and shifts every position by 2.

The attention is bidirectional: only padding is masked, so every position reads the whole sequence.

This lineage is post-norm: each sublayer's LayerNorm comes after the residual add, the opposite of every decoder on these pages.

The sentence vector is what the repository's sentence-transformers configuration builds: CLS pooling, then L2 normalisation (modules.json, 1_Pooling/config.json).

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of RobertaModel

Identical across all of them: heads 12, vocabulary 50,265, context 514.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-embedding-30m-english: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• an encoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture

Recorded pass: passes.