test-model-thing

A small proof-of-concept language model (not an LLM) that aims to incorporate latent-space prediction, internal state using recurrent trace units, and byte-by-b

What is test-model-thing?

A small proof-of-concept language model (not an LLM) that aims to incorporate latent-space prediction, internal state using recurrent trace units, and byte-by-byte output, built with MLX.

Overview of test-model-thing

What it is

Test-Model-Thing (TMT) is a small proof-of-concept language model (not an LLM) built with MLX. It aims to incorporate latent-space prediction, internal state using recurrent trace units (RTUs), byte-by-byte input/output, continuous data streaming, and test-time training. The model is designed to run on Apple Silicon devices and has been tested on Linux via community ports. It is not a finished work. The architecture was designed between July and August 2026 by a solo high school developer with pair programming assistance from Gemini. The model has 4.5 million parameters (dim=512, layers=16) and was trained for about 12 hours on a small Wikipedia dataset (simplewiki-20260801-pages-articles.xml.bz2). Results show the model can close quotes and brackets but tends to misspell characters due to byte-level output. Model weights are not provided in the repository because of file size limitations, but users can train their own models.

Key features

  • Latent-space prediction
  • Internal state with recurrent trace units (RTUs)
  • Byte-level input and output
  • Continuous data streaming
  • Test-time training
  • Built with MLX for Apple Silicon and Linux compatibility
  • Supports training, chat, frozen mode, and benchmark modes

Use cases

  • Experimental research in language model architectures
  • Proof-of-concept exploration of byte-level and recurrent mechanisms
  • Educational projects for understanding model training and evaluation
  • Forking and extending the codebase for further experimentation

Getting started

To train your own model, initialize a virtual environment and install dependencies. On Mac: pip install mlx. On Linux with GPU: pip install mlx[cuda]. On Linux with CPU: pip install mlx[cpu]. Then run the following commands:bashpython main.py trainpython main.py chat# only use crossentropy losspython main.py chat --ce-only# does not save to diskpython main.py chat --no-save# does not modify weightspython main.py chat --frozen# does not save to disk or modify weightspython main.py chat --no-save --frozen# benchmark with CoLApython benchmark.pyYou will need to configure your own dataset for dataset mode. Chat mode can run without modification if you have weights already. The program saves weights periodically and on ^C.

License and pricing

This project is licensed under the MIT open source license. It is free and open source.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/jrz97619761test-model-thing.svg)](https://yunodb.com/tools/jrz97619761test-model-thing?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/jrz97619761test-model-thing?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/jrz97619761test-model-thing.svg" alt="Featured on YunoDB" height="20"></a>