YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Standard-W4

Configuration Value
Block allocation 18 sequential blocks
Outer d / m / f 1408 / 704 / 704
Branch d / m / f โ€”
Outer Q / K / V heads 22 / 11 / 11
Branch Q / K / V heads โ€”
Head dimension 64
MoE 64 experts, top 8
Context 4,096 tokens

Megatron-native model-only safetensors checkpoint.

Downloads last month
40
Safetensors
Model size
2B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including gaotang/Standard-W4