Recorded passes

Each pass below was recorded from one forward run of the author's unpublished implementation, not read from the configuration. The step names are that implementation's generic labels, the same across every class and not names from transformers or the checkpoint; the legend, the tree notation and the canonical form are explained in how to read an architecture sheet. Nothing on a sheet depends on its pass.

31 of the 55 shapes have one. A shape not listed here has no recorded pass, and its sheet is described from configuration and weight structure alone.

GPTBigCodeForCausalLM — gptbigcodeforcausallm-52l-6144d-undeclared

Sheet: gptbigcodeforcausallm-52l-6144d-undeclared.

Full forward. One forward: 16 steps, and layer fired 52 times.

forward
├─ embed          (token row plus learned position row, wte + wpe, in this model)
├─ layer ×N
│  ├─ rmsnorm_attn  (LayerNorm with bias, ln_1, in this model)
│  ├─ wq            (wq, wk, wv: slices of one fused c_attn in this model)
│  ├─ wk
│  ├─ wv
│  ├─ rope          (entered; no rotation in this model, positions come from wpe)
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo            (attn.c_proj, with bias)
│  ├─ rmsnorm_mlp   (LayerNorm with bias, ln_2, in this model)
│  ├─ w_up          (mlp.c_fc, with bias; no gate)
│  └─ w_down        (mlp.c_proj, with bias)
└─ head             (reads wte: tied)

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14) 15)

_{↑ top}

GPTBigCodeForCausalLM — gptbigcodeforcausallm-88l-6144d-undeclared

Sheet: gptbigcodeforcausallm-88l-6144d-undeclared.

Full forward. One forward: 16 steps, and layer fired 88 times.

forward
├─ embed          (token row plus learned position row, wte + wpe, in this model)
├─ layer ×N
│  ├─ rmsnorm_attn  (LayerNorm with bias, ln_1, in this model)
│  ├─ wq            (wq, wk, wv: slices of one fused c_attn in this model)
│  ├─ wk
│  ├─ wv
│  ├─ rope          (entered; no rotation in this model, positions come from wpe)
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo            (attn.c_proj, with bias)
│  ├─ rmsnorm_mlp   (LayerNorm with bias, ln_2, in this model)
│  ├─ w_up          (mlp.c_fc, with bias; no gate)
│  └─ w_down        (mlp.c_proj, with bias)
└─ head             (reads wte: tied)

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14) 15)

_{↑ top}

GraniteForCausalLM — graniteforcausallm-28l-4096d-undeclared

Sheet: graniteforcausallm-28l-4096d-undeclared.

Full forward. One forward: 17 steps, and layer fired 28 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteForCausalLM — graniteforcausallm-40l-2048d-undeclared

Sheet: graniteforcausallm-40l-2048d-undeclared.

Full forward. One forward: 17 steps, and layer fired 40 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteForCausalLM — graniteforcausallm-40l-2560d-undeclared

Sheet: graniteforcausallm-40l-2560d-undeclared.

Full forward. One forward: 17 steps, and layer fired 40 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteForCausalLM — graniteforcausallm-40l-4096d-undeclared

Sheet: graniteforcausallm-40l-4096d-undeclared.

Full forward. One forward: 17 steps, and layer fired 40 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteForCausalLM — graniteforcausallm-64l-4096d-undeclared

Sheet: graniteforcausallm-64l-4096d-undeclared.

Full forward. One forward: 17 steps, and layer fired 64 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteMoeForCausalLM — granitemoeforcausallm-24l-1024d-undeclared

Sheet: granitemoeforcausallm-24l-1024d-undeclared.

Full forward. One forward: 17 steps, and layer fired 24 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  └─ moe
│     ├─ router
│     └─ expert
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13(14 15)) 16)

_{↑ top}

GraniteMoeForCausalLM — granitemoeforcausallm-32l-1536d-undeclared

Sheet: granitemoeforcausallm-32l-1536d-undeclared.

Full forward. One forward: 17 steps, and layer fired 32 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  └─ moe
│     ├─ router
│     └─ expert
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13(14 15)) 16)

_{↑ top}

GraniteMoeHybridForCausalLM — granitemoehybridforcausallm-28l-1024d-attn

Sheet: granitemoehybridforcausallm-28l-1024d-attn.

Full forward. One forward: 17 steps, and layer fired 28 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteMoeHybridForCausalLM — granitemoehybridforcausallm-32l-768d-28m4a

Sheet: granitemoehybridforcausallm-32l-768d-28m4a.

Full forward. One forward: 18 steps, and layer fired 32 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ delta_mix
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  ├─ w_down
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope          (entered; no-op — position_embedding_type "nope")
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  └─ wo
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8) 2(3 9 10 11 12 13(14 15) 16 5 6 7 8) 2*N(3 4 5 6 7 8) 2(3 9 10 11 12 13(14 15) 16 5 6 7 8) 2*N(3 4 5 6 7 8) 2(3 9 10 11 12 13(14 15) 16 5 6 7 8) 2*N(3 4 5 6 7 8) 2(3 9 10 11 12 13(14 15) 16 5 6 7 8) 2*N(3 4 5 6 7 8) 17)

_{↑ top}

GraniteMoeHybridForCausalLM — granitemoehybridforcausallm-40l-1536d-36m4a

Sheet: granitemoehybridforcausallm-40l-1536d-36m4a.

Full forward. One forward: 21 steps, and layer fired 40 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ delta_mix
│  ├─ rmsnorm_mlp
│  ├─ moe
│  │  ├─ router
│  │  └─ expert
│  ├─ w_gate
│  ├─ w_up
│  ├─ w_down
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope          (entered; no-op — position_embedding_type "nope")
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  └─ wo
└─ head

Which checkpoint the pass comes from. The recording has a moe step with a router and an expert inside it, so it was made from one of the members with routed experts — granite-4.0-h-tiny, granite-4.0-h-tiny-base or one of the two tiny previews; the assessment does not record which. granite-4.0-h-1b and granite-4.0-h-1b-base publish num_local_experts: 0: they have no router and route nothing, so their forward has no router or expert step, only the always-on feed-forward written out in their block above.

Canonical form:

0(1 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 20)

_{↑ top}

GraniteMoeHybridForCausalLM — granitemoehybridforcausallm-40l-2048d-attn

Sheet: granitemoehybridforcausallm-40l-2048d-attn.

Full forward. One forward: 17 steps, and layer fired 40 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteMoeHybridForCausalLM — granitemoehybridforcausallm-40l-2560d-attn

Sheet: granitemoehybridforcausallm-40l-2560d-attn.

Full forward. One forward: 17 steps, and layer fired 40 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

GraniteMoeHybridForCausalLM — granitemoehybridforcausallm-40l-4096d-36m4a

Sheet: granitemoehybridforcausallm-40l-4096d-36m4a.

Full forward. One forward: 21 steps, and layer fired 40 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ delta_mix
│  ├─ rmsnorm_mlp
│  ├─ moe
│  │  ├─ router
│  │  └─ expert
│  ├─ w_gate
│  ├─ w_up
│  ├─ w_down
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope          (entered; no-op — position_embedding_type "nope")
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  └─ wo
└─ head

Canonical form:

0(1 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 2(3 12 13 14 15 16(17 18) 19 5 6(7 8) 9 10 11) 2*N(3 4 5 6(7 8) 9 10 11) 20)

_{↑ top}

GraniteMoeSWAForCausalLM — granitemoeswaforcausallm-28l-1280d-20s8a

Sheet: granitemoeswaforcausallm-28l-1280d-20s8a.

Full forward. One forward: 20 steps, and layer fired 28 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ moe
│  │  ├─ router
│  │  └─ expert
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13(14 15) 16 17 18) 19)

_{↑ top}

GraniteSpeechForConditionalGeneration — granitespeechforconditionalgeneration-40l-2048d-undeclared

Sheet: granitespeechforconditionalgeneration-40l-2048d-undeclared.

Block level. One complete forward of this shape. 7 steps, one level. Enough to name the steps, not enough to show what nests inside what.

encodeUnits
└─ block ×N
   ├─ ff1
   ├─ attn
   ├─ conv
   ├─ ff2
   └─ post_ln

The same thing in canonical form:

0(1*N(2 3 4 5 6))

_{↑ top}

GraniteSpeechForConditionalGeneration — granitespeechforconditionalgeneration-40l-4096d-undeclared

Sheet: granitespeechforconditionalgeneration-40l-4096d-undeclared.

Block level. One complete forward of this shape. 7 steps, one level. Enough to name the steps, not enough to show what nests inside what.

encodeUnits
└─ block ×N
   ├─ ff1
   ├─ attn
   ├─ conv
   ├─ ff2
   └─ post_ln

The same thing in canonical form:

0(1*N(2 3 4 5 6))

_{↑ top}

GraniteSpeechNarForASR — granitespeechnarforasr-40l-2048d-undeclared

Sheet: granitespeechnarforasr-40l-2048d-undeclared.

Block level. One complete forward of this shape. 6 steps, one level. Enough to name the steps, not enough to show what nests inside what.

block
├─ ff1
├─ attn
├─ conv
├─ ff2
└─ post_ln

The same thing in canonical form:

0(1 2 3 4 5)

_{↑ top}

GraniteSpeechPlusForConditionalGeneration — granitespeechplusforconditionalgeneration-40l-2048d-undeclared

Sheet: granitespeechplusforconditionalgeneration-40l-2048d-undeclared.

Block level. One complete forward of this shape. 7 steps, one level. Enough to name the steps, not enough to show what nests inside what.

encodeUnits
└─ block ×N
   ├─ ff1
   ├─ attn
   ├─ conv
   ├─ ff2
   └─ post_ln

The same thing in canonical form:

0(1*N(2 3 4 5 6))

_{↑ top}

GraniteSWAForCausalLM — graniteswaforcausallm-24l-2560d-17s7a

Sheet: graniteswaforcausallm-24l-2560d-17s7a.

Full forward. One forward: 17 steps, and layer fired 24 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

LlamaForCausalLM — llamaforcausallm-32l-2560d-undeclared

Sheet: llamaforcausallm-32l-2560d-undeclared.

Full forward. One forward: 17 steps, and layer fired 32 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

LlamaForCausalLM — llamaforcausallm-32l-4096d-undeclared

Sheet: llamaforcausallm-32l-4096d-undeclared.

Full forward. One forward: 17 steps, and layer fired 32 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

LlamaForCausalLM — llamaforcausallm-36l-4096d-undeclared

Sheet: llamaforcausallm-36l-4096d-undeclared.

Full forward. One forward: 17 steps, and layer fired 36 times.

forward
├─ embed
├─ layer ×N
│  ├─ rmsnorm_attn
│  ├─ wq
│  ├─ wk
│  ├─ wv
│  ├─ rope
│  ├─ attn
│  │  ├─ scores
│  │  └─ attn_mix
│  ├─ wo
│  ├─ rmsnorm_mlp
│  ├─ w_gate
│  ├─ w_up
│  └─ w_down
└─ head

Canonical form:

0(1 2*N(3 4 5 6 7 8(9 10) 11 12 13 14 15) 16)

_{↑ top}

ModernBertModel — modernbertmodel-12l-384d-undeclared

Sheet: modernbertmodel-12l-384d-undeclared.

Wrapper only. One forward: one step, the entry point only. It says the model ran and nothing about the order of anything inside it.

encodeIds

Canonical form:

0

_{↑ top}

ModernBertModel — modernbertmodel-22l-768d-undeclared

Sheet: modernbertmodel-22l-768d-undeclared.

Wrapper only. One forward: one step, the entry point only. It says the model ran and nothing about the order of anything inside it.

encodeIds

Canonical form:

0

_{↑ top}

RobertaForMaskedLM — robertaformaskedlm-6l-384d-undeclared

Sheet: robertaformaskedlm-6l-384d-undeclared.

Wrapper only. One forward: one step, the entry point only. It says the model ran and nothing about the order of anything inside it.

encodeManyIds

Canonical form:

1(2 3*N(4 5 6 7 8))

_{↑ top}

RobertaModel — robertamodel-12l-768d-undeclared

Sheet: robertamodel-12l-768d-undeclared.

Full forward. One forward: 8 steps.

encodeManyIds
├─ embed
└─ block ×N
   ├─ qkv
   ├─ attn
   ├─ attn_out
   ├─ ff_inter
   └─ ff_out

Canonical form:

0(1 2*N(3 4 5 6 7))

_{↑ top}

RobertaModel — robertamodel-6l-384d-undeclared

Sheet: robertamodel-6l-384d-undeclared.

Full forward. One forward: 8 steps.

encodeManyIds
├─ embed
└─ block ×N
   ├─ qkv
   ├─ attn
   ├─ attn_out
   ├─ ff_inter
   └─ ff_out

Canonical form:

0(1 2*N(3 4 5 6 7))

_{↑ top}

XLMRobertaModel — xlmrobertamodel-12l-768d-undeclared

Sheet: xlmrobertamodel-12l-768d-undeclared.

Full forward. One forward: 8 steps.

encodeManyIds
├─ embed
└─ block ×N
   ├─ qkv
   ├─ attn
   ├─ attn_out
   ├─ ff_inter
   └─ ff_out

Canonical form:

0(1 2*N(3 4 5 6 7))

_{↑ top}

XLMRobertaModel — xlmrobertamodel-6l-384d-undeclared

Sheet: xlmrobertamodel-6l-384d-undeclared.

Full forward. One forward: 8 steps.

encodeManyIds
├─ embed
└─ block ×N
   ├─ qkv
   ├─ attn
   ├─ attn_out
   ├─ ff_inter
   └─ ff_out

Canonical form:

0(1 2*N(3 4 5 6 7))

_{↑ top}