GraniteForCausalLM

GraniteForCausalLM at 28 layers and hidden size 4096. 2 published checkpoints share this shape.

Implementing it

How to read this sheet: method.

Generated from granite-guardian-3.2-5b, which covers all 2 checkpoints of this shape — they agree on every field below.

x = embed[ids] * 12                    # [T, 4096]   <- embedding_multiplier

for i in 0 .. 27:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05)
    q = h @ layers[i].self_attn.q_proj.T       # [T, 32*128]
    k = h @ layers[i].self_attn.k_proj.T       # [T, 8*128]
    v = h @ layers[i].self_attn.v_proj.T       # [T, 8*128]
    q, k = rope(q, k, theta=10000000)
    k, v = repeat_kv(k, v, 4)            # 32 query heads share 8 key/value heads
    a = softmax(q @ k.T * 0.0078125, mask=causal) @ v      # <- attention_multiplier
    a = a @ layers[i].self_attn.o_proj.T
    x = x + a * 0.22        # <- residual_multiplier

    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g = silu(h @ layers[i].mlp.gate_proj.T)           # [T, 12800]
    u =       h @ layers[i].mlp.up_proj.T
    y = (g * u) @ layers[i].mlp.down_proj.T
    x = x + y * 0.22

x = rmsnorm(x, model.norm)
logits = (x @ embed.T) / 16        # tied to the input embedding; <- logits_scaling

head_dim is not published; 4096 / 32 = 128 is used.

Multiplier placement follows modeling_granite.py at transformers v4.57.1: embedding, attention scale, residual, logits.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

This is granite-guardian-3.2-5b's checkpoint. granite-guardian-3.2-5b-lora-harm-correction publishes the same configuration but ships none of these tensors: its only weight file is adapter_model.safetensors, rank-32 LoRA factors on self_attn.q_proj, k_proj and v_proj of the 28 layers (findings §10).

Other shapes of GraniteForCausalLM

Identical across all of them: KV heads 8.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-guardian-3.2-5b: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them
• granite-guardian-3.2-5b-lora-harm-correction: foldable, not packed, base-shape-pack — a stored checkpoint of the compatible base completed a forward pass; nothing was folded for this row

Parts of the author's unpublished implementation that exist for this shape:

• a decoder implemented from the published configuration
• a configuration parser that refuses what it cannot express
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture

Recorded pass: passes.