oa::ml::nn::EmpyrealmConfig
Layer schedule and widths for the hybrid Empyrealm stack. attention_every interleaves one causal self-attention block after every attention_every - 1 Mamba-3 blocks, at positions k * attention_every - 1; zero produces a pure Mamba-3 stack. Every block owns a pre-normalized residual mixer plus a dense or routed-expert feed- forward tail, and the stack ends in a final RMSNorm.
Declaration
pub struct EmpyrealmConfig { pub layers: usize, pub attention_every: usize, pub attention_heads: usize, pub sequence_length: usize, pub ffn_hidden: usize, pub moe_experts: usize, pub moe_experts_per_token: usize, pub epsilon: f32, pub mamba: Mamba3Config, }Public Rust declaration.
Public Methods
Associated Functions
Public Data Members
layersusizefieldattention_everyusizefieldattention_headsusizefieldsequence_lengthusizefieldmoe_expertsusizefieldmoe_experts_per_tokenusizefieldepsilonf32fieldmambaMamba3ConfigfieldPublic Method Documentation
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.