LlavaNextForConditionalGeneration

LlavaNextForConditionalGeneration at 40 layers and hidden size 2048. 4 published checkpoints share this shape.

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

No forward was recorded for this shape, so there is no pass above to compare it against — this block is what the published configuration says the arithmetic is, not a transcript of a run. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

From granite-vision-3.1-2b-preview

Also covers granite-vision-3.2-2b.

# One projection, one injection, at the embedding layer. This is the
# simple case, and it is worth saying so because the Granite 4 connector
# next door is not.

# Anyres tiling, and the order of the rows is load-bearing. The image
# becomes a base tile plus a grid of crops; the grid's tiles are woven
# back into one raster — row-major across the grid, not tile after tile
# — and unpadded before the rows are counted. So the prompt's row count
# is not tiles x 729, and a port that concatenates the tiles in order
# produces the right number of rows in the wrong arrangement.
tiles = base_tile + anyres_grid(image)    # e.g. 1x2 for a square image
rows = vision_tower(tiles)                # SigLIP, 729 rows a tile
rows = unpad(weave(rows, grid))           # row-major across the grid,
                                          #   then the padding dropped
rows = gelu_mlp(rows)                     # multi_modal_projector: two
                                          # Linears with a GELU between

#   The SigLIP tower, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers       27   `num_hidden_layers`
#     width      1152   `hidden_size`
#     heads        16   `num_attention_heads`
#     ffn width  4304   `intermediate_size`

ids  = tokenize(prompt)                   # one <image> per tower row
x    = embed[ids]
x[at_the_placeholder] = rows              # scatter: the rows take the
                                          # placeholder's positions, so
                                          # the text around them keeps its
                                          # own

# The decoder is an ordinary dense Granite, and its four multipliers are
# not optional. They are plain floats with no analogue in a Llama
# config, they apply at different points in the pass, and a port
# that drops them generates fluent text that is not this model's.

x = x * 12                                # embedding_multiplier, once, on the way in
for i in 0 .. 39:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05) # pre-norm: the norm feeds the
                                          #   block, the residual below
                                          #   carries the unnormalised x
    q, k, v = h @ layers[i].self_attn.q_proj.T, h @ layers[i].self_attn.k_proj.T, h @ layers[i].self_attn.v_proj.T # 32 q heads, 8 kv
    q, k = rope(q, k, theta=300000)       # before the scores, not
                                          #   after. Rotating the output
                                          #   of attention is a model
                                          #   whose positions do nothing.
    s = q @ k.T * 0.015625                # attention_multiplier replaces
                                          #   1/sqrt(head_dim) — it is a
                                          #   substitute, not a factor
                                          #   applied beside it
    a = softmax(s, mask=causal)
    o = (a @ v) @ layers[i].self_attn.o_proj.T
    x = x + 0.22 * o                      # residual_multiplier on both
    h = rmsnorm(x, layers[i].post_attention_layernorm)
    m_ = (silu(h @ layers[i].mlp.gate_proj.T) * (h @ layers[i].mlp.up_proj.T)) @ layers[i].mlp.down_proj.T
    x = x + 0.22 * m_                     #   ...and on the MLP add too

x = rmsnorm(x, model.norm)                # pre-norm stacks normalise once
                                          #   more at the end. Without it
                                          #   the head reads a residual
                                          #   stream nothing scaled.
logits = x @ embed.T
                                          #   Tied to the input embedding
                                          #   (`tie_word_embeddings: true`)
logits = logits / 8                       # logits_scaling divides.
                                          #   Multiplying has the same
                                          #   shape and flattens or
                                          #   sharpens every sample.

From granite-vision-3.3-2b

Also covers granite-vision-3.3-2b-chart2csv-preview.

# One projection, one injection, at the embedding layer. This is the
# simple case, and it is worth saying so because the Granite 4 connector
# next door is not.

# Anyres tiling, and the order of the rows is load-bearing. The image
# becomes a base tile plus a grid of crops; the grid's tiles are woven
# back into one raster — row-major across the grid, not tile after tile
# — and unpadded before the rows are counted. So the prompt's row count
# is not tiles x 729, and a port that concatenates the tiles in order
# produces the right number of rows in the wrong arrangement.
tiles = base_tile + anyres_grid(image)    # e.g. 1x2 for a square image
rows = vision_tower(tiles)                # SigLIP, 729 rows a tile
rows = unpad(weave(rows, grid))           # row-major across the grid,
                                          #   then the padding dropped
rows = gelu_mlp(rows)                     # multi_modal_projector: two
                                          # Linears with a GELU between

#   The SigLIP tower, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers                     27   `num_hidden_layers`
#     width                    1152   `hidden_size`
#     heads                      16   `num_attention_heads`
#     ffn width                4304   `intermediate_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`

ids  = tokenize(prompt)                   # one <image> per tower row
x    = embed[ids]
x[at_the_placeholder] = rows              # scatter: the rows take the
                                          # placeholder's positions, so
                                          # the text around them keeps its
                                          # own

# The decoder is an ordinary dense Granite, and its four multipliers are
# not optional. They are plain floats with no analogue in a Llama
# config, they apply at different points in the pass, and a port
# that drops them generates fluent text that is not this model's.

x = x * 12                                # embedding_multiplier, once, on the way in
for i in 0 .. 39:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05) # pre-norm: the norm feeds the
                                          #   block, the residual below
                                          #   carries the unnormalised x
    q, k, v = h @ layers[i].self_attn.q_proj.T, h @ layers[i].self_attn.k_proj.T, h @ layers[i].self_attn.v_proj.T # 32 q heads, 8 kv
    q, k = rope(q, k, theta=300000)       # before the scores, not
                                          #   after. Rotating the output
                                          #   of attention is a model
                                          #   whose positions do nothing.
    s = q @ k.T * 0.015625                # attention_multiplier replaces
                                          #   1/sqrt(head_dim) — it is a
                                          #   substitute, not a factor
                                          #   applied beside it
    a = softmax(s, mask=causal)
    o = (a @ v) @ layers[i].self_attn.o_proj.T
    x = x + 0.22 * o                      # residual_multiplier on both
    h = rmsnorm(x, layers[i].post_attention_layernorm)
    m_ = (silu(h @ layers[i].mlp.gate_proj.T) * (h @ layers[i].mlp.up_proj.T)) @ layers[i].mlp.down_proj.T
    x = x + 0.22 * m_                     #   ...and on the MLP add too

x = rmsnorm(x, model.norm)                # pre-norm stacks normalise once
                                          #   more at the end. Without it
                                          #   the head reads a residual
                                          #   stream nothing scaled.
logits = x @ embed.T
                                          #   Tied to the input embedding
                                          #   (`tie_word_embeddings: true`)
logits = logits / 8                       # logits_scaling divides.
                                          #   Multiplying has the same
                                          #   shape and flattens or
                                          #   sharpens every sample.

Notes that apply to more than one block

Stated once here rather than under each block above.

The decoder half is 40 layers at width 2,048; whatever the connector emits must be that wide, and nothing else in the configuration ties the two halves together.

The placeholder count has to be known before the tower runs, because the prompt is tokenized with one placeholder per row the tower will produce. Getting it wrong shifts every position after the image.

The tiling is not optional and the weaving is the part that bites. The crops are re-arranged into a single raster in grid row-major order and then unpadded, so the tower's rows do not arrive as tile after tile and the prompt's row count is not a multiple of 729. Feeding the tiles in tile order gives the same count with the picture scrambled, and the model describes it fluently.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

How this was checked

implemented, evidence grade shape-pack, per the assessment, for each checkpoint of this shape: converted checkpoints of this shape run; this checkpoint's own weights are not among them. The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• a vision tower implemented from the published configuration
• a tower-to-decoder connector
• the image path that assembles tiles and injects their rows
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture