C++ tutorial

Character GRU language model

The current OA all-position next-character tutorial: one shared corpus, a 27-symbol vocabulary, implicit autograd, validation, generation, and checkpoint round-trip.

OaMatrixOaGradientTape300 steps · batch 64

Contract	Value
Input and target	[64, 16] UInt8; target is input shifted by one token
Model	Embedding(27→32) → GRU(32→64) → Linear(64→27)
Objective	Cross entropy at every sequence position
Validation	Loss decreases; accuracy >50%; save/load accuracy within 0.5 points

Model

This excerpt follows Tutorial/Ml/TutorialNlpCharModels.h; the full source also initializes weights and owns registered submodules.

Tutorialnlpcharmodels.h

class OaCharGruLM : public OaModule {
public:
    OaCharGruLM() {
        Embed_ = OaMakeSharedPtr<OaEmbedding>(kCharVocabSize, kDModel);
        Gru_ = OaMakeSharedPtr<OaGru>(kDModel, kHiddenDim, 1);
        Head_ = OaMakeSharedPtr<OaLinear>(kHiddenDim, kCharVocabSize);
        RegisterModule("embed", Embed_);
        RegisterModule("gru", Gru_);
        RegisterModule("head", Head_);
        for (auto* p : AllParameterPtrs()) p->Data.SetRequiresGrad(true);
    }

    OaMatrix Forward(const OaMatrix& tokens) override {
        const OaI32 b = static_cast<OaI32>(tokens.Size(0));
        const OaI32 s = static_cast<OaI32>(tokens.Size(1));
        auto embedded = Embed_->Forward(tokens).Reshape(OaMatrixShape{b, s, kDModel});
        auto hidden = Gru_->Forward(embedded).Reshape(OaMatrixShape{b * s, kHiddenDim});
        return Head_->Forward(hidden);
    }
};

Training loop

The forward graph is recorded once per step and OaGradientTape walks the registered gradient rules. There is no parallel hand-written backward tutorial hidden behind this page.

Tutorialnlpcharmodels.h

NlpAllPositionSampler sampler(NlpCorpus(), kBatch, NlpCharEncode);
auto model = OaMakeSharedPtr<OaCharGruLM>();
auto optimizer = OaMakeUniquePtr<OaAdamW>(model->AllParameterPtrs(), 0.01F);

OaMatrix x, y;
while (not training.Loop.IsDone()) {
    sampler.NextBatch(x, y);
    optimizer->ZeroGrad();
    OaGradientTape tape;
    auto logits = model->Forward(x);
    auto targets = y.Reshape(OaMatrixShape{y.NumElements()});
    auto loss = OaFnLoss::CrossEntropy(logits, targets);
    tape.Backward(loss);
    training.Loop.Next(loss);
}

Same task across models

RNN, GRU, Transformer, MoE, and Mamba variants share the sampler, dimensions, corpus, validation, and generation contract.

Language pair

Switch to Python to open the direct nanobind port of this same character GRU workload.

Checkpoint verification

The test reloads weights and AdamW state, then re-runs accuracy before passing.

No frozen benchmark prose

Measured results live in the versioned NLP suite document; this page documents the executable contract.

Build and run

Terminal

cmake --build Build/Release --target TutorialNlpCharGruAg -j
./Bin/Release/Tutorial/Ml/TutorialNlpCharGruAg

View current source