Granite4VisionForConditionalGeneration

Granite4VisionForConditionalGeneration at 40 layers and hidden size 2560. One published checkpoint has this shape.

Layer map

One character per layer, input on the left.

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

No forward was recorded for this shape, so there is no pass above to compare it against — this block is what the published configuration says the arithmetic is, not a transcript of a run. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

Generated from granite-4.0-3b-vision.

# Eight connectors, not one, and eight injection points. Every other VLM
# here projects the tower's rows once and hands them to the decoder's
# embedding layer. This one does it eight times, at eight different
# depths of the tower, into eight different decoder layers.

#   layerwise (deepstack)  tower -19, -13, -7, -1  ->  decoder  9, 6, 3, 0
#   spatial                tower -1, four times    ->  decoder 12,15,18,21
#
# The destinations are 0, 3, 6, 9, 12, 15, 18, 21. Under the rule most
# engines have — feature k after layer k — they land in layers 0..7:
# right count, right shapes, right rows, wrong depth, and a fluent
# description of the picture.

grid = vision_tower(tile)                 # 384px at patch 16 -> 24x24
#   `downsample_rate: 4/8` — the query grid is the key grid
#   downsampled by the first factor and windowed by the second, which
#   is why this connector reads the picture twice at two resolutions.

for proj in eight_projectors:             # nine small problems each
    g = layernorm(grid_at(proj.tower_depth))
    windows = window(g, by=8)             # 3x3 = nine windows of 64 rows
    keys    = windows + image_positions   # learned [64, 1152], added to
                                          # the keys
    # The queries carry the picture. The same grid is downsampled 24->12,
    # windowed by 4 into the same 3x3, and each window's 16 rows are added
    # to the learned query [16, 1152]. So the Q-Former's input LayerNorm
    # cannot be folded the way granite-speech folds its own.
    q = learned_query + window(downsample(g, 24->12), by=4)
    out = qformer(q, keys)                # 16 rows per window
    rows[proj] = relayout(out, 12x12)     # 144 rows a tile, per projector

#   The SigLIP tower, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers                     27   `num_hidden_layers`
#     width                    1152   `hidden_size`
#     heads                      16   `num_attention_heads`
#     ffn width                4304   `intermediate_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`

# Eight additions and no scatter. There is no separate injection at the
# embedding: an image position gets its ordinary vocabulary row there,
# and all eight maps arrive as layer-input additions. A ninth write that
# replaces the embedding first has the right shapes, the right rows and
# one map applied twice.
x = embed[ids]                            # image tokens included — every
                                          # token is its own vocab row
#   The eight maps go in at the layer inputs below — before the layer
#   runs, not after it. The other convention is one layer off and reads
#   just as fluent.

# The decoder is an ordinary dense Granite, and its four multipliers are
# not optional. They are plain floats with no analogue in a Llama
# config, they apply at different points in the pass, and a port
# that drops them generates fluent text that is not this model's.
#
# Its `mamba_*` keys are inert. This config publishes the whole
#   block (d_state 256, 128 heads, chunk 256)
#   and then lists every layer as attention. `layer_types` decides;
#   the presence of the keys does not. A loader that reads them as a
#   stack builds a model this checkpoint has no weights for.

x = x * 12                                # embedding_multiplier, once, on the way in
for i in 0 .. 39:
    if i in {0, 3, 6, 9, 12, 15, 18, 21}: # at the layer's input, before it runs
        x[image_positions] += rows[projector_for(i)]
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05) # pre-norm: the norm feeds the
                                          #   block, the residual below
                                          #   carries the unnormalised x
    q, k, v = h @ layers[i].self_attn.q_proj.T, h @ layers[i].self_attn.k_proj.T, h @ layers[i].self_attn.v_proj.T # 40 q heads, 8 kv
    q, k = rope(q, k, theta=10000000)     # before the scores, not
                                          #   after. Rotating the output
                                          #   of attention is a model
                                          #   whose positions do nothing.
    s = q @ k.T * 0.015625                # attention_multiplier replaces
                                          #   1/sqrt(head_dim) — it is a
                                          #   substitute, not a factor
                                          #   applied beside it
    a = softmax(s, mask=causal)
    o = (a @ v) @ layers[i].self_attn.o_proj.T
    x = x + 0.22 * o                      # residual_multiplier on both
    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g, u = split(h @ layers[i].shared_mlp.input_linear.T, 2)   # [T, 8192] each
    m_ = (silu(g) * u) @ layers[i].shared_mlp.output_linear.T
    x = x + 0.22 * m_                     #   ...and on the MLP add too

x = rmsnorm(x, model.norm)                # pre-norm stacks normalise once
                                          #   more at the end. Without it
                                          #   the head reads a residual
                                          #   stream nothing scaled.
logits = x @ embed.T
                                          #   Tied to the input embedding
                                          #   (`tie_word_embeddings: true`)
logits = logits / 10                      # logits_scaling divides.
                                          #   Multiplying has the same
                                          #   shape and flattens or
                                          #   sharpens every sample.

The decoder half is 40 layers at width 2,560; whatever the connector emits must be that wide, and nothing else in the configuration ties the two halves together.

The placeholder count has to be known before the tower runs, because the prompt is tokenized with one placeholder per row the tower will produce. Getting it wrong shifts every position after the image.

The eight destinations are not consecutive. Each map needs its own destination layer recorded, because the obvious rule — feature k after layer k — puts every row at the wrong depth with every shape still correct.

There are exactly eight writes into the residual stream and none of them is at the embedding. Every other VLM here scatters the projector's rows over the placeholder positions before the decoder runs; this one does not. The image positions carry their own vocabulary rows and the maps are added to eight layers' inputs. A port that keeps the familiar scatter and then adds all eight applies one map twice — same shapes, same row count, a different model.

The queries are not constants. They carry a downsampled copy of the same grid the keys come from, which is why this Q-Former's input LayerNorm cannot be folded into the weights.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Other shapes of Granite4VisionForConditionalGeneration

Identical across all of them: layers 40, width 2,560, heads 40, KV heads 8, FFN width 8,192, context 131,072.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

implemented, evidence grade shape-pack, per the assessment: converted checkpoints of this shape run; this checkpoint's own weights are not among them. The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• a windowed multi-depth connector
• a vision tower implemented from the published configuration
• a decoder implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture