RobertaForSequenceClassification

RobertaForSequenceClassification at 4 layers and hidden size 576. One published checkpoint has this shape.

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

No forward was recorded for this shape, so there is no pass above to compare it against — this block is what the published configuration says the arithmetic is, not a transcript of a run. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

Generated from granite-guardian-hap-38m.

x = layernorm(word[ids] + position[2..2+T] + token_type[0])   # [T, 576]
# Positions start at 2, not 0: RoBERTa offsets by pad_token_id + 1
# (`pad_token_id: 1`). Row 0 and row 1 of the table are never read.

for i in 0 .. 3:
    h = x
    q = x @ layers[i].attention.self.query.T
    k = x @ layers[i].attention.self.key.T
    v = x @ layers[i].attention.self.value.T
    a = softmax(q @ k.T / sqrt(48), mask=bidirectional) @ v
    # Not causal. Every position sees every other one — that is the whole
    # difference between this and the decoders above, and it is why these
    # models embed a sequence rather than continue it.
    x = layernorm(x + a @ layers[i].attention.output.dense.T,
                  layers[i].attention.output.LayerNorm)   # post-norm
    h = gelu(x @ layers[i].intermediate.dense.T)
    x = layernorm(x + h @ layers[i].output.dense.T, layers[i].output.LayerNorm)

h_cls = x[0]                              # the CLS row, not a mean pool
p     = tanh(h_cls @ classifier.dense.T + classifier.dense.bias)
score = p @ classifier.out_proj.T + classifier.out_proj.bias
# The tanh is not optional. Dropping it leaves a smooth, confidently
# wrong ordering rather than an error — nothing about the output says
# the head was mis-ported.

The position table is read from row 2, not row 0. RoBERTa derives position ids as pad_token_id + 1 + i (pad_token_id: 1 here), so rows 0..1 are never read and the table is 2 rows longer than the context to pay for it. Starting at row 0 is a lookup that succeeds and an embedding shifted by 2 positions everywhere.

The head is out_b + out_w · tanh(dense_w · h[CLS] + dense_b), not a single linear over a pooled row — four tensors, the CLS row rather than a mean, and a tanh between the two matrices. A reranker's entire output is the head's output, so this is also why the head stays f32 where the body is Q8: quantization noise here is not averaged over a residual stream, it lands in the number that decides the ranking.

The attention here is bidirectional, which is the one difference that matters: there is no causal mask, so a position may read the whole sequence. Running this stack causally is fluent and is not the model's function.

This lineage is post-norm — the norm comes after the residual add, not before it, which is the opposite of every decoder on these pages.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of RobertaForSequenceClassification

Identical across all of them: heads 12, vocabulary 50,265, context 514.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

implemented, evidence grade shape-pack, per the assessment: converted checkpoints of this shape run; this checkpoint's own weights are not among them. The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• an encoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture