GPTBigCodeForCausalLM

GPTBigCodeForCausalLM at 52 layers and hidden size 6144. 5 published checkpoints share this shape.

Implementing it

How to read this sheet: method.

From granite-20b-code-base-8k

Also covers granite-20b-code-base-r1.1, granite-20b-code-instruct-8k, granite-20b-code-instruct-r1.1.

x = wte[ids] + wpe[0..T]                 # [T, 6144], learned absolute positions

for i in 0 .. 51:
    h = layernorm(x, layers[i].ln_1, eps=1e-05)
    qkv = h @ layers[i].attn.c_attn.T + layers[i].attn.c_attn.bias
    q, k, v = split(qkv, [48*128, 128, 128])
    # Multi-query: 48 query heads, one key/value head — and one
    # fused projection. The configuration says so with
    # `multi_query: true` and publishes no `num_key_value_heads`.
    a = softmax(q @ k.T / sqrt(128), mask=causal) @ v
    x = x + a @ layers[i].attn.c_proj.T + layers[i].attn.c_proj.bias

    h = layernorm(x, layers[i].ln_2)
    y = gelu(h @ layers[i].mlp.c_fc.T + layers[i].mlp.c_fc.bias)   # [T, 24576]
    x = x + y @ layers[i].mlp.c_proj.T + layers[i].mlp.c_proj.bias

x = layernorm(x, ln_f)
logits = x @ wte.T                        # tied to the input embedding

Differs from the block above

The same arithmetic with this value changed. Everything not listed is as above.

Notes

This is the older code lineage and it differs from the Granite classes at every point a reader might carry an assumption across: LayerNorm, not RMSNorm; learned absolute positions, not rotary; a GELU MLP with no gate, not SwiGLU; one fused c_attn rather than three projections; and a bias on every projection, where the Granite classes carry none. Each is in modeling_gpt_bigcode.py at transformers v4.57.1: LayerNorm, position table, fused c_attn, its q/k/v split, tied head.

wpe has 8,192 rows (n_positions), which is the hard context bound: there is no rotary base to extend.

Geometry

Weight structure

The tensors one element of the repeating stack holds, by the names the published checkpoint uses.

Outside the stack: transformer.wte.weight (the token embedding, which the output also reads), transformer.wpe.weight (the learned position table), and the final norm transformer.ln_f.weight, transformer.ln_f.bias.

Other shapes of GPTBigCodeForCausalLM

Identical across all of them: width 6,144, heads 48, KV heads 1, FFN width 24,576, vocabulary 49,152, context 8,192.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-20b-code-base-8k, granite-20b-code-base-r1.1, granite-20b-code-instruct-r1.1, granite-20b-functioncalling: runs, recorded-forward — this checkpoint was loaded and completed a forward pass
• granite-20b-code-instruct-8k: runs, shape-pack — stored checkpoints of this shape run; this one's own is not among them

Parts of the author's unpublished implementation that exist for this shape:

• a decoder implemented from the published configuration
• a configuration parser that refuses what it cannot express
• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Checkpoints with this architecture

Recorded pass: passes.