GraniteSpeechForConditionalGeneration

GraniteSpeechForConditionalGeneration at 40 layers and hidden size 2048. 3 published checkpoints share this shape.

The pass (block level)

The order of operations for one forward. ×N marks a position that fires once per layer.

7 steps, one level. Enough to name the steps, not enough to show what nests inside what. Recorded from a complete forward of this shape in the author's independent implementation — the structure is what that run emitted, not a reading of the configuration.

encodeUnits
└─ block ×N
   ├─ ff1
   ├─ attn
   ├─ conv
   ├─ ff2
   └─ post_ln

The same thing in canonical form:

0(1*N(2 3 4 5 6))

Numbers are positions in the pass, not layer indices. Two passes count as the same structure when these strings match.

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

It is not the recorded pass above, which is captured from a real forward; it is what the published configuration says the arithmetic is. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

From granite-speech-3.3-2b

This checkpoint only — its arithmetic differs from the others of this shape.

# Mel features in, tokens out. The encoder is a conformer; what sits on
# top of it is what distinguishes these four classes from each other.

x = input_linear(features)                # [T, 1024]
for i in 0 .. 15:                      # conformer block, five modules in this order
    x = x + 0.5 * ff1(prenorm(x))         # half-weight feed-forward
    # Blocked, not full: frames split into non-overlapping 200-frame
    # windows, attention is block-diagonal, no window reads another.
    for w in windows(x, 200):           # each window separately
        b = q @ k.T / sqrt(head_dim) + shaw_rel_pos(rel_pos_emb, w)
        w[:] = softmax(b) @ v             # Shaw relative positions:
                                          #   a learned bias per offset,
                                          #   added to the scores. No
                                          #   rope anywhere in this tower.
    x = x + to_out(attn_blocked(prenorm(x)))
    x = x + conv(x)                       # LayerNorm, pointwise, depthwise,
                                          #   BatchNorm, pointwise
    x = x + 0.5 * ff2(prenorm(x))         # the second half-weight FFN
    x = post_norm(x)
    if i == 8:                          # self-conditioning, once, mid-stack
        x = x + out_mid(softmax(out(x)))  # the CTC head's own opinion,
                                          # fed back into the residual

#   The conformer tower, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers         16   `num_layers`
#     width        1024   `hidden_dim`
#     heads           8   `num_heads`
#     head dim      128   `dim_head`
#     ffn mult        4   `feedforward_mult`
#     conv kernel    15   `conv_kernel_size`
#     output dim    256   `output_dim`

enc = x                                   # the final residual, only

q = learned_query                         # a fixed set of queries, reused per window [., 1024]
for each window of 15 encoder rows:        # windowed Q-Former, BLIP-2 block
    q = self_attn(q); q = cross_attn(q, window); q = ffn(q)   # post-norm
# The tail window is zero-padded to a full window, not truncated, so a
# clip that is not a multiple of the window still emits the full row
# count for its last window. Truncating loses the end of most clips.
rows = linear(q)                          # 3 rows per window, decoder-wide

#   The Q-Former projector, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers             2   `num_hidden_layers`
#     width           1024   `hidden_size`
#     heads             16   `num_attention_heads`
#     ffn width       4096   `intermediate_size`
#     activation      gelu   `hidden_act`
#     norm eps       1e-12   `layer_norm_eps`
#     vocab          30522   `vocab_size`
#     max positions   2048   `max_position_embeddings`

x = embed[ids]; x[at_the_audio_placeholder] = rows

# The decoder is an ordinary dense Granite, and its four multipliers are
# not optional. They are plain floats with no analogue in a Llama
# config, they apply at different points in the pass, and a port
# that drops them generates fluent text that is not this model's.

x = x * 12                                # embedding_multiplier, once, on the way in
for i in 0 .. 39:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05) # pre-norm: the norm feeds the
                                          #   block, the residual below
                                          #   carries the unnormalised x
    q, k, v = h @ layers[i].self_attn.q_proj.T, h @ layers[i].self_attn.k_proj.T, h @ layers[i].self_attn.v_proj.T # 32 q heads, 8 kv
    q, k = rope(q, k, theta=10000000)     # before the scores, not
                                          #   after. Rotating the output
                                          #   of attention is a model
                                          #   whose positions do nothing.
    s = q @ k.T * 0.015625                # attention_multiplier replaces
                                          #   1/sqrt(head_dim) — it is a
                                          #   substitute, not a factor
                                          #   applied beside it
    a = softmax(s, mask=causal)
    o = (a @ v) @ layers[i].self_attn.o_proj.T
    x = x + 0.22 * o                      # residual_multiplier on both
    h = rmsnorm(x, layers[i].post_attention_layernorm)
    m_ = (silu(h @ layers[i].mlp.gate_proj.T) * (h @ layers[i].mlp.up_proj.T)) @ layers[i].mlp.down_proj.T
    x = x + 0.22 * m_                     #   ...and on the MLP add too

x = rmsnorm(x, model.norm)                # pre-norm stacks normalise once
                                          #   more at the end. Without it
                                          #   the head reads a residual
                                          #   stream nothing scaled.
logits = x @ embed.T
                                          #   Tied to the input embedding
                                          #   (`tie_word_embeddings: true`)
logits = logits / 8                       # logits_scaling divides.
                                          #   Multiplying has the same
                                          #   shape and flattens or
                                          #   sharpens every sample.

From granite-4.0-1b-speech

Also covers granite-speech-4.1-2b.

# Mel features in, tokens out. The encoder is a conformer; what sits on
# top of it is what distinguishes these four classes from each other.

x = input_linear(features)                # [T, 1024]
for i in 0 .. 15:                      # conformer block, five modules in this order
    x = x + 0.5 * ff1(prenorm(x))         # half-weight feed-forward
    # Blocked, not full: frames split into non-overlapping 200-frame
    # windows, attention is block-diagonal, no window reads another.
    for w in windows(x, 200):           # each window separately
        b = q @ k.T / sqrt(head_dim) + shaw_rel_pos(rel_pos_emb, w)
        w[:] = softmax(b) @ v             # Shaw relative positions:
                                          #   a learned bias per offset,
                                          #   added to the scores. No
                                          #   rope anywhere in this tower.
    x = x + to_out(attn_blocked(prenorm(x)))
    x = x + conv(x)                       # LayerNorm, pointwise, depthwise,
                                          #   BatchNorm, pointwise
    x = x + 0.5 * ff2(prenorm(x))         # the second half-weight FFN
    x = post_norm(x)
    if i == 8:                          # self-conditioning, once, mid-stack
        x = x + out_mid(softmax(out(x)))  # the CTC head's own opinion,
                                          # fed back into the residual

#   The conformer tower, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers         16   `num_layers`
#     width        1024   `hidden_dim`
#     heads           8   `num_heads`
#     head dim      128   `dim_head`
#     ffn mult        4   `feedforward_mult`
#     conv kernel    15   `conv_kernel_size`
#     output dim    348   `output_dim`

enc = x                                   # the final residual, only

q = learned_query                         # a fixed set of queries, reused per window [., 1024]
for each window of 15 encoder rows:        # windowed Q-Former, BLIP-2 block
    q = self_attn(q); q = cross_attn(q, window); q = ffn(q)   # post-norm
# The tail window is zero-padded to a full window, not truncated, so a
# clip that is not a multiple of the window still emits the full row
# count for its last window. Truncating loses the end of most clips.
rows = linear(q)                          # 3 rows per window, decoder-wide

#   The Q-Former projector, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers             2   `num_hidden_layers`
#     width           1024   `hidden_size`
#     heads             16   `num_attention_heads`
#     ffn width       4096   `intermediate_size`
#     activation      gelu   `hidden_act`
#     norm eps       1e-12   `layer_norm_eps`
#     vocab          30522   `vocab_size`
#     max positions   2048   `max_position_embeddings`

x = embed[ids]; x[at_the_audio_placeholder] = rows

# The decoder is an ordinary dense Granite, and its four multipliers are
# not optional. They are plain floats with no analogue in a Llama
# config, they apply at different points in the pass, and a port
# that drops them generates fluent text that is not this model's.

x = x * 12                                # embedding_multiplier, once, on the way in
for i in 0 .. 39:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05) # pre-norm: the norm feeds the
                                          #   block, the residual below
                                          #   carries the unnormalised x
    q, k, v = h @ layers[i].self_attn.q_proj.T, h @ layers[i].self_attn.k_proj.T, h @ layers[i].self_attn.v_proj.T # 16 q heads, 4 kv
    q, k = rope(q, k, theta=10000)        # before the scores, not
                                          #   after. Rotating the output
                                          #   of attention is a model
                                          #   whose positions do nothing.
    s = q @ k.T * 0.0078125               # attention_multiplier replaces
                                          #   1/sqrt(head_dim) — it is a
                                          #   substitute, not a factor
                                          #   applied beside it
    a = softmax(s, mask=causal)
    o = (a @ v) @ layers[i].self_attn.o_proj.T
    x = x + 0.22 * o                      # residual_multiplier on both
    h = rmsnorm(x, layers[i].post_attention_layernorm)
    m_ = (silu(h @ layers[i].mlp.gate_proj.T) * (h @ layers[i].mlp.up_proj.T)) @ layers[i].mlp.down_proj.T
    x = x + 0.22 * m_                     #   ...and on the MLP add too

x = rmsnorm(x, model.norm)                # pre-norm stacks normalise once
                                          #   more at the end. Without it
                                          #   the head reads a residual
                                          #   stream nothing scaled.
logits = x @ lm_head.T
logits = logits / 8                       # logits_scaling divides.
                                          #   Multiplying has the same
                                          #   shape and flattens or
                                          #   sharpens every sample.

Notes that apply to more than one block

Stated once here rather than under each block above.

The conformer's five modules run in that order and the two feed-forwards are half-weighted — each contributes 0.5 * to the residual. That factor is structural, carries no tensor, and is invisible in a weight listing.

The tower never runs a full attention. Frames are split into non-overlapping context_size: 200-frame windows and attention is block-diagonal — no window reads another. Inside a window the positions are Shaw relative: attn.rel_pos_emb is a learned bias per offset, added to the scores. There is no rope in this tower at all, and a port that runs full attention with rope is more expensive, transcribes, and is not this model.

The self-conditioning detour fires once, after block 8 of 16: the mid-stack CTC head's distribution is projected back into the residual stream. A tower that skips it runs and transcribes worse.

The Q-Former's last window is zero-padded to a full window rather than truncated, so a clip whose frame count is not a multiple of the window still emits that window's full row count and the caller trims. Truncating instead loses the end of every clip that does not divide evenly, which is most of them.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of GraniteSpeechForConditionalGeneration

Identical across all of them: layers 40.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

The checkpoints of this shape do not all carry the same grade in the assessment. The implementation meant here and below is an unpublished independent inference implementation by the author.

• granite-speech-3.3-2b: implemented, class — the architecture class is implemented; nothing specific to this checkpoint was measured.
• granite-4.0-1b-speech, granite-speech-4.1-2b: implemented, shape-pack — converted checkpoints of this shape run; this checkpoint's own weights are not among them.

What stands behind the block above, beyond the published configuration it is read from:

• a conformer tower and projector implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture