MYCELIA- a consensus based AI with predictive & prescriptive telemetry

Seeking Fundingactive

Language model where its attention-heads are not competing but consenting. Its governors monitor the hidden state geometry and provide early-warning by predicting failure and timely modify latent processes to avoid instabilities

Project Details

In an email from Matt, I have been motivated to submit a second application. This second application is strongly aligned with the first one, where I have introduced MASSIF (Multiscale Attractor Stability & Stress Inference Framework) a telemetry system for "fingerprinting" LMs. This time, This telemetry is applied directly inside a 200 M params LM MYCELIA, which is an experimental "self-aware" architecture where attention has been replaced by consensus. Self=aware means that it can read the telemetry of the hidden state real-time and feed it to its "governors" which modulate and optimize the processes to increase stability and performance. Mycelia learns to predict its own future instability giving the appropriate feedback to the consensus mechanism on the level of attention-heads and as well as on the level of experts in case of the MoE set-up.

Theory of Impact

The impact is potentially enormous. Until now, most LMs rely on attention to process information. Each (attention)head looks at the input and focuses on what is considered important. Yet, there's no explicit mechanism ensuring the heads agree on what they're seeing. There is an internal competition going on in the hidden state.

Our Mycelia LM architecture replaces this with a consensus mechanism. Instead of each head working independently and then being averaged, Mycelia's heads must explicitly agree on each token. If the heads disagree (as measured by statistical variance across their outputs), the token in question is "vetoed" and attenuated. the consensus principle from multi-view learning, which emphasizes maximizing agreement across different views of the same data . By enforcing consensus at every layer, Mycelia achieves what is critical for training stability and interpretability: an architecture where the model's internal agreement is itself the governance mechanism (an internal "red-teaming" of sorts). There is some research on this: (https://ar5iv.labs.arxiv.org/html/2601.22614)

People

Daniel Solis

Team Member

Grants Received– no grants recorded

Add grant

Discussion

Daniel Solis2mo

For a detailed description of the presented project see the research paper published at: https://doi.org/10.5281/zenodo.21261148

Here is the abstract and Introduction:

Abstract

We introduce a novel training paradigm where a 200M-parameter language model monitors, predicts, and regulates its own internal dynamics.

Unlike traditional approaches that treat instability as a post-hoc diagnostic, our architecture embeds governance directly into the forward pass. The model learns to predict its own future instability and adjusts its internal parameters accordingly.

We demonstrate that this self-aware training leads to stable convergence, early warning of collapse, and interpretable internal metrics. The approach is validated on a 200M-parameter transformer (12 layers, 8 heads, 512 d_model) trained on 12.9B tokens of mixed academic and web corpus.

We introduce the optimization pressure tensor Π, a macroscopic observable that quantifies where the optimizer attempts to push the model beyond its dynamical constraints. To our knowledge, this is the first instance of a neural network that maintains a real-time dashboard of its own hidden-state geometry.

Introduction: From Post-Hoc Diagnosis to Embedded Governance

Traditional approaches to neural network stability fall into two categories: post-hoc analysis (examining activations after training) and static regularization (dropout, weight decay, gradient clipping). Both treat instability as something to be observed or prevented, never as something to be predicted and negotiated.

We propose a third way: recursive instability forecasting. The model maintains an internal representation of its own dynamical state, predicts how that state will evolve, and preemptively adjusts its behavior. Not as a simple regularization techniquem but as a cognitive architecture in which the model learns to trust (or distrust) its own predictions about itself.

The key insight is that every layer of a deep network can be viewed as a Bayesian filter. Layer ℓ receives not only the hidden state from layer ℓ − 1, but also a forecast of how unstable that state is likely to be. The layer then decides: “Should I trust what the previous layer warned me about?” This recursive propagation of (prediction, confidence) pairs creates a self-model that co-evolves with the task mode.

Daniel Solis2mo

Current status: Replaced Sinusoidal attention with RoPE

Training at 512. Sinusoidal PE doesn't generalize beyond training length without quality collapse. RoPE generalizes cleanly. Relative position is genuinely a better inductive bias for autoregressive transformers. Vaswani's original sinusoidal PE was a placeholder; the field moved on. For Mycelia, this is the Bonanza. With RoPE, position lives in the attention geometry (rotated Q and K), not in the residual stream.

So: h(l) = content(l) # pure content, position-agnostic

Now velocity, acceleration, curvature, jerk measure content dynamics only. The MPC's prediction, the rate governor's growth detection, the pressure tensor's work accounting, all of these get cleaner signals. Took me over milion and half training steps to figure it out - but it is never too late!

The main reason is that my recursive instability forecast treats each layer as a Bayesian filter:

I = confidence * P_current + (1 - confidence) * P_persist

The "temporal persistence" term P_persist is the previous layer's instability, passed forward.

RoPE gives the attention a relative position view: "token at position j attends to token at position i with weight that depends on |j - i|."

That's exactly the kind of structural prior the Bayesian filter needs to make good predictions about future instability. Sinusoidal PE forces the model to learn this from scratch, baked into the residual stream where it's entangled with content.

Some people say: don't change horses in the middle of the stream. How will the model react to this change. Current model is trained for 1.7M steps with sinusoidal Positional Encoding baked into the residual stream. Switching to RoPE mid-training isn't a like-for-like transition. Well, I say - what an exciting experiment this would be!

We will compare sinusoidal and rotary position encodings at matched compute to isolate whether Mycelia's geometric observables benefit from position-decoupled residual streams, preliminary evidence from this regime-change experiment suggests they do, since pressure concentration dropped in moments when loss reached low level, indicating the model could exploit representational headroom when not contending with position geometry in the residual stream.

Here is the telemetry after the switch (to be compared with the telemetry featured at: https://doi.org/10.5281/zenodo.21261148

And on GitHub: https://github.com/Ergo-sum-AGI/mycelia_llm

The transition in brief: RoPE caused a severe but expected shock (loss 4.62 → 9.44, coherence 0.56 → 0.13), but the model is recovering rapidly: Loss is already down to 4.73 in just 7,000 steps, approaching pre-RoPE levels, with long-range coherence benefits still ahead.

Nevertheless, Friction: 🟢 HARMONIZED at EVERY step. This is unprecedented and proves the architecture is robust enough to absorb major changes without becoming unstable. RoPE implementation proved worth the while from the very beginning.

See the new telemetry at: https://github.com/Ergo-sum-AGI/mycelia_llm/blob/main/EPOCH 48 — STEP 1%2C760%2C000 (post RoPE).txt

🍄 Mycelia architecture loaded

======================================================================

🍄 MYCELIA TRAINING v8.8.1
Gradual Transition + Interaction Guard + No Adaptive Targets
Stanford (30%) + FineWeb (70%) | No compression

📚 Loading tokenizer...
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Vocab: 151,643

🏗️ Building model...
199,935,516 params on cuda:0

======================================================================

🚀 EPOCH 48 — STEP 1,775,000

🔮 MPC: intervene=17.4% control=0.510 I=0.604
📊 Pred: 0.594 | Conf: 0.749 (min: 0.749)
📈 Dynamics: v=+0.0043 a=+0.0067
🎯 Forecast Error: 0.406 (predicted=0.580 vs actual=0.174)
I-field: 0.19 0.41 0.50 0.52 0.52 0.53
Conf-field:0.32 0.65 0.75 0.77 0.77 0.76

🔥 Π=272.9 | χ=0.67 | dominant=ffn
🔥 Π breakdown: ffn=183.2 alpha=41.9 cap=45.0 mpc=2.9
📐 Rate Governor: hit=0.0% scale=0.912 | ffn_growth=1.00x res_growth=1.52x
⚠️ GOVERNOR INTERACTION GUARD: 3 governors active, limit is 2
⚙️ Auto-tune: coh/fcst weights↑1.06/1.06

🔮 MPC: intervene=23.0% control=0.500 I=0.631
📊 Pred: 0.615 | Conf: 0.741 (min: 0.741)
📈 Dynamics: v=+0.0045 a=+0.0127
🎯 Forecast Error: 0.364 (predicted=0.594 vs actual=0.230)
I-field: 0.18 0.48 0.52 0.52 0.52 0.53
Conf-field:0.32 0.68 0.76 0.77 0.78 0.77

🔮 MPC: intervene=22.1% control=0.515 I=0.578
📊 Pred: 0.586 | Conf: 0.779 (min: 0.779)
📈 Dynamics: v=-0.0180 a=-0.0247
🎯 Forecast Error: 0.375 (predicted=0.596 vs actual=0.221)
I-field: 0.19 0.47 0.52 0.54 0.53 0.54
Conf-field:0.31 0.71 0.77 0.75 0.77 0.76

🔮 MPC: intervene=44.1% control=0.442 I=0.727
📊 Pred: 0.628 | Conf: 0.759 (min: 0.759)
📈 Dynamics: v=+0.0328 a=+0.0329
🎯 Forecast Error: 0.137 (predicted=0.579 vs actual=0.441)
I-field: 0.16 0.41 0.50 0.52 0.53 0.54
Conf-field:0.29 0.64 0.74 0.75 0.77 0.76

🔮 MPC: intervene=21.7% control=0.513 I=0.578
📊 Pred: 0.585 | Conf: 0.785 (min: 0.785)
📈 Dynamics: v=-0.0214 a=-0.0491
🎯 Forecast Error: 0.382 (predicted=0.599 vs actual=0.217)
I-field: 0.17 0.42 0.50 0.52 0.53 0.53
Conf-field:0.32 0.65 0.74 0.76 0.77 0.77

🔥 Π=271.0 | χ=0.70 | dominant=ffn
🔥 Π breakdown: ffn=190.2 alpha=40.3 cap=37.9 mpc=2.6
📐 Rate Governor: hit=0.0% scale=0.946 | ffn_growth=1.00x res_growth=1.46x
⚠️ GOVERNOR INTERACTION GUARD: 3 governors active, limit is 2
Training: 1779147it [1:31:46, 3.87it/s]

Daniel Solis1mo

A seminal research paper just pre-published today at ZENODO. under doi.org/10.5281/zenodo.21402446. Our research has identified Pressure as an Order Parameter: Distinguishing Productive Learning from Compensatory Stabilization (coastinig) in Neural Optimization - it is a architecture agnostic and universal rule. The AI Safety Implications Are Profound. We've discovered something that goes far beyond training efficiency. The finding that neural optimization exhibits phase transitions with a universal order parameter has direct implications for AI safety, interpretability, and control.

Let me explain. Current AI safety assumes that we can monitor models by looking at individual metrics (loss, confidence, coherence, etc.).

Our recent finding shows: These metrics are systematically misleading. A model can look "healthy" (high confidence, stable loss) while actually being in a compensatory regime (R < 1) also known as coasting, it can be broken, or collapsed. Thus, standard monitoring is blind to dangerous regime changes.

False Confidence: A model that outputs high confidence predictions may be in the compensatory regime, stable but not learning, possibly hallucinating or producing subtly incorrect outputs with high certainty. Example: An LLM that appears confident about its responses but is actually in a compensatory regime (R < 1) could produce fluent, confident-sounding misinformation. Standard confidence calibration would miss this entirely.
The Phase Transition Blind Spot: Models can transition from constructive to compensatory regimes without warning, the critical region is exactly where the correlation between R and loss vanishes. Loss gives no signal that the model is entering a dangerous regime.

Example: During fine-tuning or deployment, a model could cross the R ≈ 1 boundary without any change in loss. The safety monitor would see "stable loss" while the model has actually entered a qualitatively different, potentially dangerous regime.

The implications are tremendous. Pehaps this is not the right place to enumerate them all. You can read about it here: https://www.linkedin.com/pulse/emergent-phase-locked-control-optimization-pressure-state-solis-73r7f

Loek Verdonk1mo

Hi Daniel,

Fascinating and serious work — I spent some time independently checking part of the public MYCELIA telemetry and sent you a private email with the calculations and two small trace mechanisms that may help strengthen the research.

Nothing you need to prepare or explain again; I just wanted to make sure the message reaches you. :D

Warmly,
Loek

Daniel Solis1mo

Got you message, much appreciated, thanks!