RobertaForSequenceClassification

RobertaForSequenceClassification at 4 layers and hidden size 576. One published checkpoint has this shape.

Implementing it

How to read this sheet: method.

No forward was recorded for this shape; the block is read from the published configuration.

Generated from granite-guardian-hap-38m.

x = word[ids] + position[pos] + token_type[0]   # [T, 576]
x = layernorm(x, embeddings.LayerNorm, eps=1e-12)
# pos = padding_idx + 1 + i for real tokens (`pad_token_id: 1`), and
#   padding_idx for padding: real tokens start at row 2 (see Notes).

for i in 0 .. 3:
    q = x @ layers[i].attention.self.query.T + layers[i].attention.self.query.bias
    k = x @ layers[i].attention.self.key.T   + layers[i].attention.self.key.bias
    v = x @ layers[i].attention.self.value.T + layers[i].attention.self.value.bias
    a = softmax(q @ k.T / sqrt(48), mask=padding) @ v     # bidirectional
    o = a @ layers[i].attention.output.dense.T + layers[i].attention.output.dense.bias
    x = layernorm(x + o, layers[i].attention.output.LayerNorm, eps=1e-12)   # post-norm
    h = gelu(x @ layers[i].intermediate.dense.T + layers[i].intermediate.dense.bias)   # [T, 768]
    o = h @ layers[i].output.dense.T + layers[i].output.dense.bias
    x = layernorm(x + o, layers[i].output.LayerNorm, eps=1e-12)

h = x[0]                                  # the <s> row, not a mean pool
h = tanh(h @ classifier.dense.T + classifier.dense.bias)
logits = h @ classifier.out_proj.T + classifier.out_proj.bias   # [576] -> 2

Notes

The head is out_proj(tanh(dense(h[0]))): four tensors, the first row rather than a mean, and a tanh between the two matrices (modeling_roberta.py L1426-L1445). out_proj has two rows in the published weights (model.safetensors); the configuration publishes no label names.

Position ids start at padding_idx + 1 = 2. create_position_ids_from_input_ids numbers real tokens from padding_idx + 1 and gives padding the id padding_idx (modeling_roberta.py L1537-L1550). With pad_token_id: 1, row 0 of the 514-row position table is never read, row 1 only at padding positions, and a 512-token input reads rows 2 to 513, so the 514-row table (max_position_embeddings) holds 512 usable positions. A port that starts at row 0 looks everything up successfully and shifts every position by 2.

The attention is bidirectional: only padding is masked, so every position reads the whole sequence.

This lineage is post-norm: each sublayer's LayerNorm comes after the residual add, the opposite of every decoder on these pages.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of RobertaForSequenceClassification

Identical across all of them: heads 12, vocabulary 50,265, context 514.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-guardian-hap-38m: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• an encoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture