Idefics3ForConditionalGeneration

Idefics3ForConditionalGeneration at 30 layers and hidden size 576. One published checkpoint has this shape.

Implementing it

How to read this sheet: method.

No forward was recorded for this shape; the block is read from the published configuration and from Idefics3ForConditionalGeneration at transformers v4.57.1 (modelling, processor).

Generated from granite-docling-258M.

# Pixel shuffle first, then one projection. r^2 neighbouring patches are
# folded into one row, so the row count falls by r^2 and the width grows
# by it, before anything is projected.

tiles = split(image, 512) + [global_image]   # 512x512 tiles, then the whole
                                          #   image resized to one tile
rows = vision_tower(tiles)                # 512 / 16 = 32: [1024, 768] a tile
rows = pixel_shuffle(rows, r=4)           # [1024, 768] -> [64, 12288]
                                          #   (`scale_factor: 4`): sixteen
                                          #   neighbouring patches fold into
                                          #   one row
rows = rows @ modality_projection.proj.T  # one Linear, 12,288 -> 576, no bias

#   The vision tower (`model_type: idefics3_vision`), as published:
#     layers                     12   `num_hidden_layers`
#     width                     768   `hidden_size`
#     heads                      12   `num_attention_heads`
#     ffn width                3072   `intermediate_size`
#     patch / image          16 / 512 `patch_size`, `image_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`

ids = tokenize(prompt)                    # per tile: <fake_token_around_image>
                                          #   <row_i_col_j> then 64 <image>;
                                          #   the global image last, after
                                          #   <global-img>
x   = embed[ids]
x[ids == 100270] = rows                   # masked scatter (`image_token_id`)

# The decoder is a plain causal decoder (`model_type: llama`): none of
# Granite's four multipliers, and its own rope base.
for i in 0 .. 29:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05)
    q = h @ layers[i].self_attn.q_proj.T       # [T, 9*64]
    k = h @ layers[i].self_attn.k_proj.T       # [T, 3*64]
    v = h @ layers[i].self_attn.v_proj.T       # [T, 3*64]
    q, k = rope(q, k, theta=100000)
    k, v = repeat_kv(k, v, 3)            # 9 query heads share 3 key/value heads
    a = softmax(q @ k.T / sqrt(64), mask=causal) @ v
    x = x + a @ layers[i].self_attn.o_proj.T

    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g = silu(h @ layers[i].mlp.gate_proj.T)           # [T, 1536]
    u =       h @ layers[i].mlp.up_proj.T
    x = x + (g * u) @ layers[i].mlp.down_proj.T

x = rmsnorm(x, model.norm)
logits = x @ embed.T                      # tied to the input embedding

Notes

head_dim: 64 is published in the text configuration.

Where each step is in transformers v4.57.1. The connector is a pixel shuffle followed by modality_projection, a single nn.Linear of width vision hidden × scale_factor² with no bias (modeling_idefics3.py L274-L282, L390-L410); its rows replace the placeholder embeddings by masked_scatter (L606). The processor writes one <row_i_col_j> marker and image_seq_len placeholders per tile, then the global image, which the image processor resizes to one tile (image_processing_idefics3.py L445-L450, processing_idefics3.py L46-L62); the markers are ordinary vocabulary tokens. This checkpoint's processor sets image_seq_len: 64 (processor_config.json) and splits images into tiles of at most 512 on the longest edge (preprocessor_config.json), matching (512 / 16)² / 4² = 64.

The decoder half is 30 layers at width 576; the projector's output width has to match it.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-docling-258M: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• a vision tower implemented from the published configuration
• the image path that assembles tiles and injects their rows
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture