Instructions to use Gabriel/diffu-moss-above with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use Gabriel/diffu-moss-above with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("Gabriel/diffu-moss-above", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
diffu-moss-above
A full fine-tune of the DiT of MOSS-Image-Base that draws whole synthetic archive pages: Swedish handwritten records of the 16th to 19th centuries, in the look and pen of a sample page. It is a drawing model of diffu, which makes training pages for page-level handwriting recognition: a page layout (a PAGE/ALTO file, or the lines' boxes as JSON), the text to write in it and a sample page go in; a page image and its exact ALTO and PAGE labels come out.
The model reads a MOSS Layout prompt, one text item per line with its exact text and its box, and a sample page given as extra image tokens. Only the DiT is in this repository. The text encoder, tokenizer, VAE and scheduler are MOSS-Image-Base's, unchanged.
How it differs from diffu-moss-r6b:
- The sample page lies above the drawn page. MOSS-Image lays a reference image over the drawn one, each of its
tokens at the position of the drawn token under it. This model was trained with the sample's tokens placed just
above the drawn page instead, where the lines before the page's first would lie (
reference_place: aboveinconfig.json). diffu'sMossTransformerreads that setting and places the sample accordingly. Code that places the sample over the page (MOSS-Image's own pipeline, r6b's settings, the SGLang plugin) gives this model inputs it was never trained on: use diffu at commitb8e07deor later. - It was trained to take the hand from the sample. Each window of a page was paired with the other window of the same page as its sample (the same scribe on the same sheet, for certain), so the pen can only come from the sample. r6b's samples came from other pages of the volume, and its hand did not follow the sample.
- 2 MP pages (r6b: about 1 MP), three passes over the data with the learning rate wound down at the end.
Files
| File | What |
|---|---|
diffusion_pytorch_model.safetensors |
the DiT's weights, bfloat16 (10.2 GB, the same names and shapes as Base's DiT): the exponential moving average (decay 0.999) at the last step, 10,257 |
config.json |
diffu_moss.transformer.MossTransformer (diffu's own implementation of MOSS-Image's DiT), with reference_place: above |
export.json |
takes_reference: true, sample_size: legible: the model takes a sample page, sized at the drawn page's writing scale |
Use
With diffu:
hf download OpenMOSS-Team/MOSS-Image-Base --revision 85975793d8b3d432f1b6e27f352abaa4c2a4c0d3 --local-dir base
hf download Gabriel/diffu-moss-above --local-dir dit
git clone https://github.com/Borg93/diffu2 && cd diffu2 && uv sync --all-packages
uv run diffu-moss-init --base ../base --transformer ../dit --out ../moss-diffu
uv run diffu-generate --model ../moss-diffu --templates templates/0105.xml --text-from text.txt \
--reference samples/court-sample.jpg --reference-caption sample --whole-page --page-megapixels 2 --out pages
The sample page is an ordinary scan (JPEG or PNG) of a page whose paper, ink and hand the drawn page should have.
Run files for many pages work as for r6b (see its card and diffu's docs/INFERENCE.md).
Drawn in the checks below: 2 MP, 50 steps, guidance 3, shift 5, block 22 skipped in the unconditional pass (MOSS-Image's own recipe), no reference guidance. Reference guidance (a third pass toward the sample), which r6b recommends, has not been measured with this model yet.
Training
- Data: diffu's Swedish archival handwriting ground truth at 2 MP, held-out volumes excluded: 54,700 training pairs, windows of whole lines each with the other window of its own page as its sample, and whole pages in their own prompt forms. Court, chancery, church, police, mining, military and travel records, 16th to 19th century.
- Recipe: MOSS-Image's own full fine-tune from Base at revision
85975793: Muon on the joint blocks' matrices and AdamW elsewhere, learning rate 5e-5 after warmup, wound down linearly to 0 over the last 1,709 steps, gradients clipped at 1, float32 weights under bfloat16 autocast, an exponential moving average of the weights at 0.999; text encoder and VAE frozen. Samples grouped by size into rounds (size blocks of 32). - Scale: 16 samples a step over 8 A100 64 GB (two nodes of the Leonardo supercomputer, CINECA; FSDP2), 10,257 steps: 164,112 samples, three passes, each pair seen three times. Trained 13-17 Nov 2026 as a chain of 30-minute jobs, each resuming from the last checkpoint with its optimizer state and data order (every step trained exactly once).
- Training loss (mean over 1,000 steps): 0.616 (steps 1-1,000), 0.601 (3,001-4,000), 0.598 (6,001-7,000), 0.590 (the last 257).
Evaluation
Judged by eye beside the real scans, on held-out pages from volumes the model never saw in training.
Over training, six held-out pages each drawn with a sample from its own page, the same noise at every checkpoint: at step 1,000 the model writes its own heavy, near-printed hand whatever the sample; from step 2,000 it writes in the sample's pen (stroke weight, slant, ink colour); steps 3,000 to 10,257 differ only in detail. Layout, line breaks and paper follow the page at every step.
The swap test (final weights): each of the six pages drawn again with the samples of two pages from other volumes (other scribes), the same noise as with its own sample:
| Check | Pages of 12 |
|---|---|
| The page takes the sample's look (paper and ink colour, grey or colour scan) | nearly all |
| The page takes the sample's pen (stroke weight, slant, how regular the writing is) | about 8 |
| The page takes the sample's letter forms as well | about 3 |
Letter size and line spacing stay with the layout, as they should: the line boxes set them.
Limitations
- The hand follows the sample's pen more than its letter forms. In most pages the drawn hand moves toward the sample's but keeps much of a generic hand for the period.
- The placement is fixed: the sample has to be placed above the page (see above). Older diffu code and other MOSS-Image pipelines place it over the page.
- New layouts (the template test: 18 templates of letters, certificates, forms, lists, tables and registers, each with sample pages from two unseen volumes, 30 pages at 2 MP, beside r6b's drawings of the same at 1 MP): 24 drawn, the layouts and their text followed as r6b follows them, the paper and scan (colour or grey) taken from the sample more often than r6b takes them. The hand moves toward the sample's pen weight but stays a tidy hand of the period on most pages: a sample in a rough 17th-century hand does not make a 19th-century certificate rough. One page drew table rules the template does not have, and one copied a page number like the sample's. The other 6 are dense registers whose prompts are longer than the text encoder's 4,096 tokens: such a page has to be drawn in parts.
- Every line has to be checked: diffu reads each drawn line back and marks a line it cannot confirm as unclear in the labels.
- Trained on Swedish archival handwriting; printed forms and other kinds of pages are untested.
Licence
Apache-2.0, as MOSS-Image-Base.
- Downloads last month
- 5
Model tree for Gabriel/diffu-moss-above
Base model
OpenMOSS-Team/MOSS-Image-Base