Idefics3ForConditionalGeneration

Idefics3ForConditionalGeneration at 30 layers and hidden size 576. One published checkpoint has this shape.

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

No forward was recorded for this shape, so there is no pass above to compare it against — this block is what the published configuration says the arithmetic is, not a transcript of a run. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

Generated from granite-docling-258M.

# Pixel shuffle first, then one projection. The shuffle is how this
# family pays for a high-resolution grid: r^2 neighbouring patches are
# folded into one row, so the row count falls by r^2 and the width grows
# by it, before anything is projected.

rows = vision_tower(patches)              # a tiled grid plus a base tile,
                                          #   1024 patches of 768 per tile
rows = pixel_shuffle(rows, r=4)           # [1024, 768] -> [64, 12288]
                                          #   r is 4 here, not 2: sixteen
                                          #   neighbouring patches fold
                                          #   into one row
rows = projector(rows)                    # one Linear, 12,288 -> 576

#   The SigLIP tower, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers                     12   `num_hidden_layers`
#     width                     768   `hidden_size`
#     heads                      12   `num_attention_heads`
#     ffn width                3072   `intermediate_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`

ids  = tokenize(prompt)                   # tiles are separated by learned
                                          # row/column marker tokens
x    = embed[ids]
x[at_the_placeholder] = rows

# The decoder is a plain causal decoder (`model_type: llama`) — it carries none of
# Granite's four multipliers, and that absence is the fact worth
# knowing: a reader coming from the other pages on this site will
# be looking for them. Its rope base is its own, too.

for i in 0 .. 29:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05) # pre-norm: the norm feeds the
                                          #   block, the residual below
                                          #   carries the unnormalised x
    q, k, v = h @ layers[i].self_attn.q_proj.T, h @ layers[i].self_attn.k_proj.T, h @ layers[i].self_attn.v_proj.T # 9 q heads, 3 kv
    q, k = rope(q, k, theta=100000)       # before the scores, not
                                          #   after. Rotating the output
                                          #   of attention is a model
                                          #   whose positions do nothing.
    s = q @ k.T / sqrt(head_dim)
    a = softmax(s, mask=causal)
    o = (a @ v) @ layers[i].self_attn.o_proj.T
    x = x + o
    h = rmsnorm(x, layers[i].post_attention_layernorm)
    m_ = (silu(h @ layers[i].mlp.gate_proj.T) * (h @ layers[i].mlp.up_proj.T)) @ layers[i].mlp.down_proj.T
    x = x + m_

x = rmsnorm(x, model.norm)                # pre-norm stacks normalise once
                                          #   more at the end. Without it
                                          #   the head reads a residual
                                          #   stream nothing scaled.
logits = x @ embed.T
                                          #   Tied to the input embedding
                                          #   (`tie_word_embeddings: true`)

The decoder half is 30 layers at width 576; whatever the connector emits must be that wide, and nothing else in the configuration ties the two halves together.

The placeholder count has to be known before the tower runs, because the prompt is tokenized with one placeholder per row the tower will produce. Getting it wrong shifts every position after the image.

The shuffle happens before the projector, so the projector's input width is d * r^2 and not the tower's width. Projecting first and shuffling after has the same output shape and different weights.

r is 4 on this checkpoint, which the configuration does not state anywhere — graniteDoclingVisionEmbed hardcodes it. Each tile is 1,024 patches at width 768, which the shuffle turns into 64 rows at 12,288, and the projector reads that width. r = 2 is the value a reader carries over from SmolVLM and it produces 256 rows at 3,072: a shape the projector refuses, which is the good case.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

How this was checked

implemented, evidence grade shape-pack, per the assessment: converted checkpoints of this shape run; this checkpoint's own weights are not among them. The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• a vision tower implemented from the published configuration
• the image path that assembles tiles and injects their rows
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture