LlavaNextForConditionalGeneration

LlavaNextForConditionalGeneration at 40 layers and hidden size 2048. 4 published checkpoints share this shape.

Implementing it

How to read this sheet: method.

No forward was recorded for this shape; the block is read from the published configuration and from LlavaNextForConditionalGeneration at transformers v4.57.1 (modelling, processor).

From granite-vision-3.3-2b

Also covers granite-vision-3.3-2b-chart2csv-preview.

# The image path: anyres tiles, four tower depths, one projector,
# then packing, then one scatter into the embedding.
tiles = [base_tile] + anyres_grid(image)  # 384x384 tiles; the grid is the
                                          #   `image_grid_pinpoints` entry
                                          #   that best fits the image
hs = vision_tower(tiles).hidden_states
f  = concat(hs[-24], hs[-20], hs[-12], hs[-1])   # [729, 4*1152] a tile
                                          # `vision_feature_layer`: four
                                          #   depths side by side. 729 =
                                          #   (384 // 14)^2, all kept
                                          #   (`vision_feature_select_strategy:
                                          #   full`: SigLIP has no CLS row)
r  = linear_2(gelu(linear_1(f)))          # multi_modal_projector:
                                          #   4608 -> 2048 -> 2048, with biases
grid = weave(r[1:], grid_shape)           # one raster, row-major across the
                                          #   whole grid, not tile after tile
grid = unpad(grid, image_size)            # crop the letterbox padding
grid = append_to_each_raster_row(grid, image_newline)   # learned [2048]
rows = concat(r[0], flatten(grid))        # the base tile's 729 rows first

#   The SigLIP tower, as published:
#     layers                     27   `num_hidden_layers`
#     width                    1152   `hidden_size`
#     heads                      16   `num_attention_heads`
#     ffn width                4304   `intermediate_size`
#     patch / image          14 / 384 `patch_size`, `image_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`

ids = tokenize(prompt)                    # the processor has already expanded
                                          #   each <image> to len(rows) copies
x   = embed[ids]
x[ids == 49155] = rows                    # masked scatter over the
                                          #   placeholders (`image_token_index`)

x = x * 12                                # <- embedding_multiplier
for i in 0 .. 39:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05)
    q = h @ layers[i].self_attn.q_proj.T       # [T, 32*64]
    k = h @ layers[i].self_attn.k_proj.T       # [T, 8*64]
    v = h @ layers[i].self_attn.v_proj.T       # [T, 8*64]
    q, k = rope(q, k, theta=300000)
    k, v = repeat_kv(k, v, 4)            # 32 query heads share 8 key/value heads
    a = softmax(q @ k.T * 0.015625, mask=causal) @ v      # <- attention_multiplier
    a = a @ layers[i].self_attn.o_proj.T
    x = x + a * 0.22        # <- residual_multiplier

    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g = silu(h @ layers[i].mlp.gate_proj.T)           # [T, 8192]
    u =       h @ layers[i].mlp.up_proj.T
    y = (g * u) @ layers[i].mlp.down_proj.T
    x = x + y * 0.22

x = rmsnorm(x, model.norm)
logits = (x @ embed.T) / 8        # tied to the input embedding; <- logits_scaling

Differs from the block above

The same arithmetic with these values changed. Everything not listed is as above.

Notes

head_dim is not published; 2048 / 32 = 64 is used.

Output head: the configuration says tied, the checkpoint ships a separate one. granite-vision-3.3-2b-chart2csv-preview sets tie_word_embeddings: true, which is what the block follows, but its checkpoint also ships a separate language_model.lm_head.weight [49156, 2048], which LlavaNext renames to lm_head on load (L540). A standard transformers load reads that tensor and then, because the configuration asks for tying, replaces it with the input embedding (modeling_utils.py L2994, L5065), so the logits it computes are the ones written here. The published headers carry shapes, not values, so whether the shipped head equals embed is not known; if it does not, a loader that uses the shipped lm_head.weight computes different logits (findings §10).

The decoder's four multipliers are placed as in modeling_granite.py at transformers v4.57.1: embedding, attention scale, residual, logits.

The image path, line by line, at transformers v4.57.1. The four tower depths are concatenated on the feature axis (L404-L408), which is why linear_1 reads 4 × 1,152 = 4,608 (L204-L207). The projector runs before packing (L413). Packing weaves the grid tiles into one raster, unpads it, appends image_newline to every raster row and puts the base tile first (L306-L339); image_newline is a learned parameter of the model (L263). The rows then replace the placeholder embeddings by masked_scatter (L505).

The row count is fixed by the processor, before the tower runs. It expands each <image> into base rows + unpadded rows + one newline per raster row (processing_llava_next.py L166-L169, L186-L226). So the count is not a multiple of 729, and a port that concatenates tiles in order gets neither the count nor the arrangement right.

Tower defaults, where a configuration omits them: SiglipVisionConfig L176-L177; projector defaults: LlavaNextConfig L93, L99.

The decoder half is 40 layers at width 2,048; the projector's output width has to match it.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• each of the 4 checkpoints of this shape: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• a vision tower implemented from the published configuration
• a tower-to-decoder connector
• the image path that assembles tiles and injects their rows
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture