GraniteVisionEmb

GraniteVisionEmb at 40 layers and hidden size 2048. One published checkpoint has this shape.

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

No forward was recorded for this shape, so there is no pass above to compare it against — this block is what the published configuration says the arithmetic is, not a transcript of a run. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

Generated from granite-vision-3.3-2b-embedding.

# This is granite-vision-3.3-2b with one extra linear and no lm_head.
# The tower, the connector and the decoder are the 3.x path unchanged —
# all of the new arithmetic is the last four lines.

rows = gelu_mlp(vision_tower(patches))    # the 3.x connector, unchanged

#   The SigLIP tower, as published. Every line here is a tensor shape a
#   port allocates, and none of it is stated anywhere else on this page.
#     layers                     27   `num_hidden_layers`
#     width                    1152   `hidden_size`
#     heads                      16   `num_attention_heads`
#     ffn width                4304   `intermediate_size`
#     activation  gelu_pytorch_tanh   `hidden_act`
#     norm eps                1e-06   `layer_norm_eps`
x    = embed[ids]
x[at_the_placeholder] = rows

# The decoder is an ordinary dense Granite, and its four multipliers are
# not optional. They are plain floats with no analogue in a Llama
# config, they apply at different points in the pass, and a port
# that drops them generates fluent text that is not this model's.

x = embed[ids]
x = x * 12                                # embedding_multiplier, once, on the way in
for i in 0 .. 39:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05) # pre-norm: the norm feeds the
                                          #   block, the residual below
                                          #   carries the unnormalised x
    q, k, v = h @ layers[i].self_attn.q_proj.T, h @ layers[i].self_attn.k_proj.T, h @ layers[i].self_attn.v_proj.T
    q, k = rope(q, k, theta=300000)       # before the scores, not
                                          #   after. Rotating the output
                                          #   of attention is a model
                                          #   whose positions do nothing.
    s = q @ k.T * 0.015625                # attention_multiplier replaces
                                          #   1/sqrt(head_dim) — it is a
                                          #   substitute, not a factor
                                          #   applied beside it
    a = softmax(s, mask=causal)
    o = (a @ v) @ layers[i].self_attn.o_proj.T
    x = x + 0.22 * o                      # residual_multiplier on both
    h = rmsnorm(x, layers[i].post_attention_layernorm)
    m_ = (silu(h @ layers[i].mlp.gate_proj.T) * (h @ layers[i].mlp.up_proj.T)) @ layers[i].mlp.down_proj.T
    x = x + 0.22 * m_                     #   ...and on the MLP add too

x = rmsnorm(x, model.norm)                # pre-norm stacks normalise once
                                          #   more at the end. Without it
                                          #   the head reads a residual
                                          #   stream nothing scaled.
#   ...and this checkpoint stops here: no lm_head, nothing
#   generated. What reads `x` is below.
#   (`logits_scaling: 8` is published and unreachable:
#    there is no head for it to scale.)

# The retrieval projection is here, after the decoder, and only here.
# It reads the last hidden state — it is not the connector, and a sheet
# that projects to 128 before the decoder would be scattering rows too
# narrow for a 2048-wide stack to accept.
e = x @ custom_text_proj.T + bias         # [2048] -> 128
e = e / ||e||                             # L2, per row

# A document keeps 729 rows and drops the rest, text included.
doc = sort(topk(image_positions, k=729))  # the last 729 image rows. 729
                                          # is the SigLIP 27x27 grid and a
                                          # literal in the modeling code,
                                          # not a config field.

# Late interaction, not one dot product.
score = sum over query rows i of  max over doc rows j of  (q_i . d_j)

The decoder half is 40 layers at width 2,048; whatever the connector emits must be that wide, and nothing else in the configuration ties the two halves together.

The placeholder count has to be known before the tower runs, because the prompt is tokenized with one placeholder per row the tower will produce. Getting it wrong shifts every position after the image.

The projection is after the decoder, not before it. custom_text_proj reads the final hidden state. Putting a 128-wide projection where the connector goes would emit rows too narrow to scatter into the decoder at all — the geometry on this page contradicts it.

There is no lm_head and nothing is generated. A query scores a document by MaxSim over rows.

Three things here are silent when wrong, none a shape error: which 729 rows are kept (the last image positions, which happen to be the base tile only because base_image_feature_location is last), that text rows are dropped entirely, and that the score is a sum of per-query maxima. Each returns documents, in a different order.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

How this was checked

implemented, evidence grade shape-pack, per the assessment: converted checkpoints of this shape run; this checkpoint's own weights are not among them. The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• a retrieval model implemented from the published configuration
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture