Architecture classes

The class name a model declares in its configuration is the most reliable label it carries: two checkpoints that share it run through the same code path, whatever their names suggest. It does not make them the same function — the configuration still decides the layer mix, the activation, the rope base, the multipliers and more, and findings §1 shows a class whose members do not even share a layer type. A base model, an instruction-tuned model and a safety classifier of one generation are routinely one class.

The class name links to its sheets — one per distinct shape, with the layer map, the recorded order of operations, the geometry and the weight stack. A class usually covers several shapes, and the sheet index shows exactly what separates them.

What actually differs between them

Dense. Every layer is attention followed by a feed-forward block, and the whole parameter count runs for every token. GraniteForCausalLM (36 models) and the Llama-class models use a gated SiLU block; GraniteForCausalLM adds the four Granite multipliers, which 3.x and 4.1 publish as values such as 12, 0.22 and 16. The 4.2 models set the embedding, residual and logits values to 1 and keep only the attention scale. Seven models that declare the hybrid class list every layer as attention and are dense in fact (findings §1).

Sparse (mixture of experts). The 3.0 and 3.1 a400m models route each token to 8 of 32 experts (1.3–1.4B parameters in total), and the a800m models to 8 of 40 (3.3–3.4B); their configurations name no shared expert. The 4.0 hybrid h-tiny routes to 6 of 64 (6.9B; the preview release 6 of 62) and h-small to 10 of 72 (32.2B), each alongside a shared expert that runs for every token. granite-swash-3b-a600m routes to 4 of 48, with a shared expert. In every case most of the published total is idle for any one token.

Hybrid. The granite-4.0-h-* models, and the two granite-4.0-tiny previews despite their names, are 36 Mamba state-space layers and 4 attention layers in 40 (at layers 5, 15, 25 and 35), or 28 and 4 in 32 for h-350m, whose attention layers sit at 10, 13, 17 and 27. A state-space layer caches a fixed-size state instead of keys and values — for h-small, 128 heads of 64 × 128 per layer, plus a convolution state that is fixed-size too: the last 4 inputs (mamba_d_conv) across 8,448 channels, which is mamba_expand × width + 2 × mamba_n_groups × mamba_d_state (cache layout at transformers v4.57.1) — so only the 4 attention layers keep a cache that grows with context. Those attention layers use no positions at all: position_embedding_type is nope (v4.57.1).

Sliding-window attention. Two models. granite-swash-2b has 17 layers that attend over a 128-position window and 7 that attend over everything; granite-swash-3b-a600m has 20 and 8, and is also a mixture of experts. In both, full attention is at layer 0, layer 3 and every fourth layer after it (7, 11, 15 …), and the trained context is 8,192. Both checkpoints publish a per-head self_attn.sinks tensor on every layer that the configuration does not mention.

Adapter switching. The three granite-switch-4.1 previews carry 12 LoRA adapters of rank 16 or 32 inside the checkpoint, on the fused query/key/value projection, the attention output and the shared feed-forward block (lora_target_modules). They are activated by control tokens in the chat template, and one extra attention layer detects those tokens and tells every decoder layer which adapter to apply (model card). Their configurations set num_hidden_layers to 41, 41 and 65, but one of those slots is the weightless switch, so the checkpoints hold 40, 40 and 64 decoder layers, the depths of the 4.1 models of the same widths (modeling_granite_switch.py L198–L199).

Code, two generations. The 20B and 34B code models are GPTBigCodeForCausalLM: 52 or 88 layers of width 6,144, 48 query heads sharing one key/value head (multi_query: true), LayerNorm, learned absolute positions with 8,192 rows, and an ungated GELU block (transformers v4.57.1: LayerNorm, positions, feed-forward). The 3B and 8B code models are LlamaForCausalLM: rotary positions, RMSNorm and a gated block, with attention_bias and mlp_bias both set, which no GraniteForCausalLM model does.

Vision–language. Every Granite vision model except Docling pairs a 27-layer SigLIP image tower of width 1,152 with a 40-layer Granite language model: width 2,048 in the four LlavaNextForConditionalGeneration models and the GraniteVisionEmb retriever, width 2,560 in the two Granite4VisionForConditionalGeneration models. Of those two, granite-4.0-3b-vision wraps a hybrid-class text configuration that lists every layer as attention, and granite-vision-4.1-4b a plain granite one. granite-docling-258M is smaller throughout: a 12-layer tower of width 768 and a 30-layer, 576-wide Llama-type text model.

Speech. Every speech model starts from a 16-layer encoder of width 1,024 (10 layers in granite-speech-3.2-8b); what differs is what sits on top. GraniteSpeechForConditionalGeneration (5 models) passes a conformer encoder's output through a 2-layer Q-Former projector into a 40-layer Granite language model that generates the transcript; GraniteSpeechPlusForConditionalGeneration has the same projector type and language model and adds cat_hidden_layers: [3] to its encoder. GraniteSpeechNarForASR takes encoder layers 4, 8, 12 and the last (encoder_layer_indices), drafts a transcript with a CTC head over the 100,352-token language vocabulary, and has its 40-layer decoder edit that draft in one pass instead of generating token by token (model card). GraniteSpeech5ForCTC has no language model: the encoder emits one of 16,384 tokens per frame and decodes greedily (model card).