PatchTSTFMForPrediction

PatchTSTFMForPrediction at 30 layers and hidden size 1024. One published checkpoint has this shape.

Implementing it

How to read this sheet: method.

No forward was recorded for this shape, so the block is read from the published configuration and from IBM's modelling code in granite-tsfm v0.3.9, not transcribed from a run.

Generated from granite-timeseries-patchtst-fm-r2.

# There is no separate horizon: the forecast occupies the tail of the
# fixed 8,192-point window, marked by a mask the model reads.

fl = max(64, ul, 16 * max(pretrain_mask_cont, 2))   # = 128; ul = -1 in this path,
                                                    #   pretrain_mask_cont: 8
x  = concat(last 8064 observed points,          # 8192 - 128
            [nanmean(series)] * 128)            # placeholder tail
mask = [0]*8064 + [1]*128                       # 128 masked, not 64
# (a shorter history is left-padded with the mean and marked as padding)

# Scaler: statistics over the unmasked points only, divisor = their count.
mean, std = stats(x, where=~mask, divisor=count(~mask))
std = std if std > 1e-5 else 1                  # replaced by 1, not epsilon'd
z = asinh((x - mean) / std)
z = where(mask, 0, z)                           # masked points read zero

p = unfold(z, size=16, step=8)                  # 1023 patches
m = unfold(mask, size=16, step=8)
h = residual_block(concat(p, ~m))               # 32 -> 1024
h = h + pos_embed[0 .. 1022]                    # learned, additive

for i in 0 .. 29:                               # conformer, pre-norm
    k = conv_kernel_size[i]                     # 5,5,3,3 repeating
    h = h + 0.5 * mlp(layernorm(h))             # 1024 -> 2048 -> 1024
    h = h + attn(layernorm(h))                  # 16 heads, qkv_bias: false
    h = h + gelu(depthwise_conv(layernorm(h), k))   # over patches
    h = h + 0.5 * mlp(layernorm(h))
# mlp activation: tanh-approximate GELU; conv activation: exact GELU.
# LayerNorm eps 1e-6 throughout. No norm after the conv or the block.
h = layernorm(h)                                # `prehead_norm: true`

# residual_block(x) = layer2(sigmoid(layer1(x))) + residual(x)
rows = residual_block(h)                        # 1024 -> 16 * 100 per patch
q = rows[0] + cumsum(softplus(rows[1:100]) / 99)    # 99 levels, cannot cross

# Fold the overlapping patches back onto the time axis, in normalised
# space, with a Hamming window scaled to mean 1.
w = 0.54 - 0.46*cos(2*pi*i/15) for i in 0..15;  w /= mean(w)
out[t] = sum(q_patch * w) / sum(w)              # over patches covering t

out = sinh(out) * std + mean                    # every level, then ...
out = out[-128:][:64]                           # ... the first 64 of the tail
forecast = trapz(out, levels) + out[0]*0.01 + out[98]*(1 - 0.99)

Sources for the block, all at granite-tsfm v0.3.9: the 128-point window and the appended placeholder tail are PatchTSTFMForPrediction.forward (with ul set at L570) and forecast_single_step_fast; the masked statistics are RevIN._get_statistics; patching, the mask channel and the positional embedding are forward and decode; the block is ConformerBlock.forward with its conv module and MLP; the head is decode with ResidualBlock; the fold is _make_patch_window and _overlap_add, called before the inverse scaler; the point forecast is the trapezoid.

This revision differs from r1 in more than the block type. Against patchtst-fm-r1, the catalogue's configuration keys change as follows: depth 20 → 30, n_patch 512 → 1,023, and patch_stride: 8, patch_loss_windowing: "hamming", block_type: "conformer" and prehead_norm: true are added. The published r2 configuration also sets per-layer conv_kernel_size and qkv_bias: false, where r1's leaves both at the code defaults (block_type defaults to "transformer", qkv_bias to true). Same class, same width, same context — a different function; Generations describes the change.

The forecast sits inside the context, not after it. The wrapper keeps the last 8,064 observed points, appends 128 placeholder positions set to the series mean, and marks them masked; the backbone always sees exactly 8,192 points. The window is max(prediction_length, ul, d_patch * max(pretrain_mask_cont, 2)) = 128, where ul is fixed at -1 on this path, and the caller keeps the first 64 of it. Masking only 64 would change both the scaler's statistics and what the model is asked to fill.

The scaler is not affine. asinh on the way in and sinh on the way out; a linear inverse is wrong by a factor that grows with the value. A std at or below 1e-5 is replaced by 1 rather than having an epsilon added, so a flat series gets scale 1.

The masked statistics divide by the unmasked count. With a mask, RevIN computes the mean and the variance over the unmasked points and divides by their number; the no-mask branch calls x.std, which is torch's N-1 default. The prediction path always passes a mask. At 8,064 unmasked points the two divisors differ by about 0.012% in variance and 0.006% in std — small, but systematic.

The head emits 100 rows and 99 levels. Row 0 is a base, not a quantile; each of rows 1..99 adds softplus(...)/99 to a running total. The median is level 0.50, index 49 of the 99 levels; reading the 100 rows as levels puts it one off.

Fold, then sinh, then integrate. The overlap-add runs on the normalised quantiles; the inverse scaler is applied to every level after the fold; the trapezoid integrates the denormalised levels. Because sinh is not linear, sinh(integrate(...)) is a different number, and the gap grows with the value.

Overlapping patches are folded, not concatenated. At stride 8 and patch 16 every interior point is covered by two patches (the first and last 8 points by one); the reference accumulates value * w and w and divides, with w a Hamming window scaled to mean 1.

Geometry

Other shapes of PatchTSTFMForPrediction

Identical across all of them: width 1,024, heads 16.

Only the columns that differ are shown; a cell with several values means the checkpoints of that shape disagree.

How this was checked

Status and evidence grade in the assessment (see what the grades mean):

• granite-timeseries-patchtst-fm-r2: runs, class — the architecture class is implemented; nothing checkpoint-specific was measured

The implementation meant here and below is an unpublished independent inference implementation by the author.

What stands behind the block above, beyond the published configuration it is read from:

• a checkpoint loader, which is where published tensor names are read
• an independent reference implementation of this architecture in Python, driving the published modelling code — an executable statement of what the model should compute, written against the publication rather than against any one implementation of it

Where an independent implementation and the published configuration disagree, the configuration is what this page reports and the disagreement is what it says.

Checkpoints with this architecture