oa::ml::nn::CausalLinearAttention
Multi-head linear attention over packed [B*S, D] values. Uses the Katharopoulos (2020) ELU+1 kernel rather than softmax. Causal heads up to 64 use fused forward and recomputed prefix/suffix adjoints. Saved projections/context scale as O(BHSd), with bounded O(d²) scan state per head. Larger heads retain the compositional path, whose training tape saves O(BHSd²) prefixes. Bidirectional mode uses the global KV composition. Standard attention stores S² probabilities; Flash recomputes them and does not store a global S² score matrix. Parameter layout Four biased projections (q_proj, k_proj, v_proj, out_proj) with the same names and seed stride as MultiHeadAttention, so checkpoints are transferable between the two modules.
Declaration
pub struct CausalLinearAttention { /* private fields */ }Public Rust declaration.
Public Methods
Associated Functions
Public Method Documentation
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
Construct four biased projections with causal visibility. Errors Returns an error unless dimensions are positive, model_width is divisible by num_heads, or child construction/registration fails.
Construct four biased projections with causal visibility. Errors Returns an error unless dimensions are positive, model_width is divisible by num_heads, or child construction/registration fails.
Construct four biased projections with causal visibility. Errors Returns an error unless dimensions are positive, model_width is divisible by num_heads, or child construction/registration fails.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.
No public source comment is attached to this declaration.