GraniteSwitchForCausalLM

GraniteSwitchForCausalLM at 41 layers and hidden size 4096. One published checkpoint has this shape.

Layer map

One character per layer, input on the left.

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Implementing it

One forward, written out, per checkpoint whose arithmetic actually differs. A shape groups checkpoints by class, depth, width and layer stack — none of which decides an activation, a rope base or a routing rule — so where the members of this shape disagree there is a block each, and each names the checkpoint it was generated from.

No forward was recorded for this shape, so there is no pass above to compare it against — this block is what the published configuration says the arithmetic is, not a transcript of a run. Where the configuration does not say, the line says that instead of guessing.

The names are the ones the published checkpoint uses where a name is published, and canonical otherwise. A checkpoint loader may store them differently — fusing a gate/up pair into one matrix, or splitting one published projection in two — and those are that loader's names, not the model's. Everything spelled here is what the download contains.

Generated from granite-switch-4.1-8b-preview.

# This config publishes a full `mamba_*` block and uses none of it.
#   d_state 256, 128 heads, 1 group(s), conv width 4,
#   chunk 256 — and `layer_types` lists every layer as
#   attention. Nothing in `GraniteSwitchForCausalLM` names a state-space
#   layer; the keys are template leftovers, the layer table says dense,
#   and the layer table is the fact. A loader that reads the keys as a
#   stack builds a model this checkpoint has no weights for.

# Tensor names in this block are illustrative: the config's own
# `lora_target_modules` name these modules qkv_proj (q/k/v fused),
# o_proj, shared_input_linear and shared_output_linear (the shared MLP).

x = embed[ids] * 12                    # [T, 4096]   <- embedding_multiplier

for i in 0 .. 40:
    h = rmsnorm(x, layers[i].input_layernorm, eps=1e-05)
    q = h @ layers[i].self_attn.q_proj.T       # [T, 32*128]
    k = h @ layers[i].self_attn.k_proj.T       # [T, 8*128]
    v = h @ layers[i].self_attn.v_proj.T       # [T, 8*128]
    q, k = rope(q, k, theta=10000000)
    k, v = repeat_kv(k, v, 4)            # 32 query heads share 8 key/value heads
    a = softmax(q @ k.T * 0.0078125, mask=causal) @ v      # <- attention_multiplier
    a = a @ layers[i].self_attn.o_proj.T
    x = x + a * 0.22        # <- residual_multiplier

    h = rmsnorm(x, layers[i].post_attention_layernorm)
    g = silu(h @ layers[i].mlp.gate_proj.T)           # [T, 12800]
    u =       h @ layers[i].mlp.up_proj.T
    y = (g * u) @ layers[i].mlp.down_proj.T
    x = x + y * 0.22

# Adapter switching — what this class is, and it happens inside the
# block above rather than after it. 12 LoRA adapters ship in the
# checkpoint; a control token in the prompt selects which one is live.
#   W_eff = W + (B_a @ A_a)   for the selected adapter a, rank <= 32
#   applied to: o_proj, qkv_proj, shared_input_linear, shared_output_linear
#   the control token's embedding is scaled by 15 so the
#   choice survives the depth of the stack.

x = rmsnorm(x, model.norm)
logits = (x @ embed.T) / 16        # tied to the input embedding; <- logits_scaling

head_dim is not published; 4096 / 32 = 128 is used.

The Granite multipliers are the part with no Llama analogue: embedding_multiplier, attention_multiplier, residual_multiplier, logits_scaling. They are ordinary floats and a port that ignores them still produces fluent text, which is what makes them worth printing where they act.

num_local_experts: 0 on a class whose tensors are named for a mixture of experts. The MLP is the shared expert alone; nothing is routed.

Geometry

Other shapes of GraniteSwitchForCausalLM

Identical across all of them: stack attention, KV heads 8, active —¹, vocabulary 100,364, context 131,072.

¹ The config sets num_experts_per_tok: 2 with num_local_experts: 0 — a template leftover on a model with no experts (findings §2); nothing is routed.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

Why this is refused

refused by design, evidence grade refusal-path-tested, per the assessment: the author's implementation deliberately does not express this configuration.

This configuration is refused rather than run, and the refusal is a tested path: an unpublished independent inference implementation by the author rejects it by name at load, with its own tests asserting the refusal. Nothing about it is approximated.

Checkpoints with this architecture