RRSD: Learning from Relative Rewards via On-Policy Self-Distillation of Flow Models

Official Stable Diffusion 3.5 Medium (SD3.5-M) multi-reward LoRA checkpoints from Learning from Relative Rewards via On-Policy Self-Distillation of Flow Models.

Relative Reward Self-Distillation (RRSD) is a gradient-free on-policy post-training framework for diffusion and flow-matching generators. At selected intermediate denoising states, RRSD branches the trajectory with antithetic Gaussian perturbations, completes both rollouts through a shared deterministic ODE solver, cancels the shared rollout midpoint, and shifts the current clean prediction along a bounded relative-reward endpoint displacement to form a detached MSE target.


Available Checkpoints

Both checkpoints are rank-32 (lora_alpha=64) PEFT LoRA adapters trained for 100 outer rounds (115.2k primary trajectories) on stabilityai/stable-diffusion-3.5-medium under the shared multi-reward objective (PickScore + HPSv2 + GenEval + OCR) without classifier-free guidance (guidance_scale=1.0).

Subfolder Variant States / Rollout Terminal Evals H100 GPU-h LoRA Rank / Alpha Format
RRSD/ RRSD (Full) 9 2.19M 375 32 / 64 PEFT safetensors
RRSD-Lite/ RRSD-Lite 4 1.04M 185 32 / 64 PEFT safetensors

Benchmark Results (SD3.5-M Multi-Reward Post-Training)

Held-out results reported in Tables 1 and 2 of the paper (512Γ—512, 40 deterministic Euler ODE steps, guidance_scale=1.0):

Preference, Compositional GenEval, and Text Rendering (Table 1)

Model / Method Traj. (k) PickScore ↑ CLIPScore ↑ HPSv2 ↑ ImageReward ↑ GenEval Micro ↑ GenEval Macro ↑ GenEval Strict ↑ GenEval Soft ↑ OCR Acc. ↑
SD3.5-M (w/o CFG) / 20.51 0.237 0.204 -0.58 0.2857 0.2972 0.1894 0.2314 0.1246
+ FlowGRPO 115.2 22.30 0.267 0.290 0.81 0.5344 0.5498 0.4227 0.5763 0.5724
+ DiffusionNFT 115.2 22.99 0.271 0.316 1.17 0.6623 0.6761 0.5416 0.6730 0.8584
+ Self-OPD 115.2 23.45 0.270 0.333 1.21 0.6533 0.6671 0.5118 0.6415 0.7916
RRSD-Lite 115.2 23.55 0.276 0.342 1.37 0.7242 0.7357 0.6252 0.7569 0.8764
RRSD 115.2 23.74 0.281 0.346 1.42 0.7468 0.7592 0.6609 0.7703 0.8923

Compute Efficiency and Qwen-Image-Bench (Table 2)

Method Eval. (M) GPU-h Quality ↑ Aesthetics ↑ Alignment ↑ Fidelity ↑ Creativity ↑ Overall ↑
FlowGRPO 0.12 54 42.99 40.06 31.29 37.19 19.51 35.50
DiffusionNFT 0.58 126 42.69 41.39 31.92 37.17 20.68 36.02
Self-OPD 2.19 398 44.26 40.03 31.97 37.29 21.95 36.33
RRSD-Lite 1.04 185 45.31 41.68 34.10 38.22 23.25 37.83
RRSD 2.19 375 45.68 42.43 34.61 38.68 23.75 38.32

Quickstart Usage (diffusers + peft)

import torch
from diffusers import StableDiffusion3Pipeline
from peft import PeftModel

base_model = "stabilityai/stable-diffusion-3.5-medium"
adapter_variant = "RRSD"  # Choose "RRSD" or "RRSD-Lite"

pipe = StableDiffusion3Pipeline.from_pretrained(
    base_model,
    torch_dtype=torch.float16,
)
pipe.transformer = PeftModel.from_pretrained(
    pipe.transformer,
    "WeiChow/RRSD",
    subfolder=adapter_variant,
    is_trainable=False,
)
pipe.transformer.eval()
pipe = pipe.to("cuda")

prompt = 'A vintage brass compass resting on an old sea map with a banner reading "NORTHBOUND"'
generator = torch.Generator(device="cuda").manual_seed(42)

image = pipe(
    prompt=prompt,
    height=512,
    width=512,
    num_inference_steps=40,
    guidance_scale=1.0,
    generator=generator,
).images[0]

image.save("rrsd_sample.png")

License

The LoRA adapter weights and accompanying configuration files in this repository are released under the Apache-2.0 License. Usage of the base model stabilityai/stable-diffusion-3.5-medium remains subject to the upstream Stability AI Community License.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for WeiChow/RRSD

Adapter
(123)
this model