AbstractPhil/alephllm-mini-beatrix-training

alephllm — Mini-Beatrix training runs

Live training ground for AlephLLM: signed-address (aleph) language models. Code, presets, trainer, and the full test array live in the source repo — github.com/AbstractEyes/alephllm (pip install git+https://github.com/AbstractEyes/alephllm, package geolip.alephllm). This repo holds what training produces, one prefix per craft:

<craft>/manifest.json                      what is trained, what is planned,
                                           tokens run, phase statuses, ckpt index
<craft>/checkpoints/step_XXXXXXXX.safetensors    bf16 weights
<craft>/checkpoints/fp8/step_XXXXXXXX.safetensors  fp8-e4m3 shipping variant
                                           (inference testing only — never train from these)
<craft>/resume/latest.pt                   full resume state: fp32 model, Muon+Adam
                                           optimizer states, data-stream position, RNG
<craft>/runs/                              TensorBoard event files (rendered in the
                                           Training metrics tab)

Mission log

• mini-beatrix-2s-control — COMPLETE (2026-09-03). The softmax twin of mini-beatrix-2s: the identical preset with the hub list empty (16-head causal SDPA in every one of the 20 blocks; 257.4M vs the craft's 237.1M), the identical recipe and diet, 16.101B tokens / 61,422 steps, one craft per arm, under a pre-registered protocol whose headline is the endpoint validation. The endpoint (anneal_mix @61,422, same step, same eval): splat 1.1097 vs sdpa 2.8846 bpb, +1.775 sdpa deficit. The crossover holds at 11 of 11 same-gauge paired rows: the twin led the open (0.3B: sdpa 2.106 vs splat 2.363) and trailed from the fineweb boundary on (5.3B: splat 1.1172 vs sdpa 1.1763, +.059; s0 +.253; s1 +.137; s2 +.176; s3 +.300; s4 +.333; s5 +.926; s6 +1.926; s7 +1.932; anneal_nochat +1.812; anneal_mix +1.775). The s8 pair (1.0737 vs 3.0810) is struck: the two rows were measured on different gauges (the craft's on the pre-0.8.5 synthetic fallback, the control's on the fineweb-edu holdout). Every other row — every curriculum stage and both anneals — is on the one fineweb-edu holdout, so the control's slide is a single comparable series: 1.18 (fineweb) → 1.57 (s4) → 2.18 (s5) → 3.19 (s6) → 3.12 (s7) → 2.93 → 2.88. The control is a DESTABILIZED model at its endpoint, not a trailing one. Its pre-clip gradient norm (train/grad_norm, clip 1.0) first crossed the clip at step 17,600 — inside the fineweb phase, on the RTX PRO 6000, while it still led — then escalated stage by stage: medians 0.29 on fineweb → 6.5 (s0) → 2.4 → 12 → 13 → 24.5 → 126 (s5; nine steps above 1,000, max 11,877) → 471 (s6; max 41,070) → 162 → 145 → 167 → 215 (anneal_mix), clipped on 813 of 824 logged curriculum and anneal steps, while the craft's raw norm never reached the clip (0 of 824; max 0.62). The anneal is a DIET change (1B of prose, then 1B with the chat frame), not a learning-rate decay: train/lr_scale read 1.000 first to last on both crafts. Bank dependence at the end: control toggle_bank_off 6.11 vs the craft's 3.31; the control's born-null aleph head stayed buried at every row. Grade: an INSTANCE at one craft per arm; the protocol completed unaltered. The claim that SDPA is not an adequate representative of this model's attention requirement is AbstractPhil's (2026-09-02); its audit (2026-09-04) minted the specific form — under this recipe (Muon 2e-2 / Adam 3e-4 flat, clip 1.0, no QK-norm, no weight decay, no LR decay) the softmax twin is a destabilized control and the endpoint gap measures stability, not capacity — and holds the general form as a CANDIDATE behind five confounds: the recipe (the documented softmax logit-growth instability, fixed in the literature by a normalization or a decay), one seed with 20.3M extra parameters in the losing arm, the shape (mini-beatrix-1, thirteen softmax blocks and three hubs, trained clean under the identical configuration: post-warmup grad-norm max 0.88, no clipped step), the aleph-addressed expert banks present in both crafts (the control's dispatch entropy collapsed before its gradients did), and the learned position table on decay-free Muon. Provenance: session caps at 22,854 and 53,792 (reports/v2/session_cap_step*.json) — the first resume landed on a rare H100 (s0–s8 rows), a third session from the 56,000 checkpoint ran the two anneal rows on another card; the escalation began before any resume. Fifteen reports under mini-beatrix-2s-control/reports/v2/ pair with the craft's by stage and step; the paired chart is article_assets_ft2/01_control_crossover.png.

• mini-beatrix-2s — COMPLETE (2026-08-31). 16.101B tokens / 61,422 steps on one card: warmup + fineweb 5B + curriculum s0-s8 + two-phase anneal (distribution shift without the chat frame, then with it). Final val 1.1097 bpb (run best, set in the chat phase); zero loss spikes; 17 boundary/session reports under mini-beatrix-2s/reports/v2/. Shipped automodel: mini-beatrix-2s. Notable ops in the record: the first BOUNDARY-WRITE head revival (reports/v2/revival_step24860.json) and the archived pre-specials run0 (mini-beatrix-2s/archive/run0-nospecials/).

• mini-beatrix-1 — COMPLETE. The 3-hub hybrid rung; automodel.

The mission ladder

craft	d / L / ctx	params	tokenizer	role
mini-beatrix-0	512 / 12 / 1024	37.6M	byte-trigram	gate craft — its first toggle evals are the anchored-bank-under-AR screen
mini-beatrix-1	768 / 16 / 2048	112.5M	byte-trigram	first Colab mission
mini-beatrix-2	1024 / 20 / 2048	249.1M	byte-trigram	second mission
beatrix-voyager	1536 / 24 / 4096	775.3M	BPE (gpt2)	flagship, gated on the mini verdicts

Each craft has a *-control twin (identical minus the aleph attention layers) — the running architecture control. For the 2s rung the twin is mini-beatrix-2s-control (the same preset with hub_layers: [], so 16-head causal SDPA in every block; 257.4M vs the craft's 237.1M), run to the same 16.101B-token endpoint under a pre-registered protocol — see the mission log; its paired boundary reports sit under mini-beatrix-2s-control/reports/v2/. Training is resume-first: sessions are manually triggered on Colab (by default the RTX 6000 Pro, 96GB, bf16), each session pulls manifest.json + resume/latest.pt and continues where the last one stopped.

The architecture in one paragraph (the mini-beatrix-1 era)

The 2s rung differs: mini-beatrix-2s carries a CausalSplatHUB in every one of its 20 blocks and no softmax attention; its control twin carries 16-head SDPA in every block and no hubs (hub_layers: [] in its manifest). See the mission log and the model card of mini-beatrix-2s.

Trigram byte embedding (dedicated pad row) → pre-norm stack of standard causal SDPA plus three CausalSplatHUB layers (causal linear attention through a 2K-half-axis signed address, exact chunked scan) → per-layer anchored FFN bank (always-on trunk + 3 dispatched experts, expert outputs zero-init so the dispatch is born contributing exactly zero, gates σ(−3), no balance machinery) → dual head whose aleph read enters at γ=0 and must earn its way in by gradient. Muon on transport weights + pure Adam elsewhere; flat LR; bf16 autocast over fp32 masters; fp8 is a shipping format, never a training format.

Reading the instruments

TensorBoard carries the full born-in gauge suite: per-layer hidden-state effective rank, consumed-address erank per hub layer, coefficient-of- variation load analysis per bank, sign census, gate/γ trajectories, anchor drift, denominator health, structural collapse flags (anchor merging, dispatch-entropy collapse, erank floor, loss spikes), canary recall (clean-protocol in-context binding probes), and the toggle ledger — bpb deltas with each aleph mechanism switched off, the causal record of what the addresses actually contribute.

Related record: the attention-side measurement campaign lives at aleph-splat-0; the encoder-side anchored-bank record at alephlm-0.