AccentCL: Robust Accent Classification with Incremental Expansion

Accepted at IEEE Spoken Language Technology Workshop (SLT) 2026

📄 Paper: arXiv:2610.07426
💻 Code: GitHub – PSI-TAMU/AccentCL

Overview

AccentCL is a class-incremental learning framework for English accent classification, designed to address class imbalance, cross-corpus domain shift, and the incremental addition of new accent categories.

The framework leverages multi-layer representations from a frozen Whisper-Large-v3 encoder and incorporates imbalance-aware learning, domain mean alignment, and replay-based continual learning.

Available Checkpoints

Checkpoint	Description
accentcl_base.pt	Initial 5-class accent classifier
accentcl_spanish.pt	6-class classifier (+ Spanish-accented English)
accentcl_spanish_chinese.pt	7-class classifier (+ Chinese-accented English)

We recommend using accentcl_spanish_chinese.pt for inference across all seven supported accent categories.

Supported Accent Labels

The pretrained checkpoints support the following accent categories. Class IDs correspond to the output indices of the model's classification logits and remain consistent across incremental learning stages.

Class ID	Accent	Label	5-Class	6-Class	7-Class
0	North American	north_american	✓	✓	✓
1	British Isles	british_isles	✓	✓	✓
2	Australasian	australasian	✓	✓	✓
3	South Asian	south_asian	✓	✓	✓
4	Southeast Asian	southeast_asian	✓	✓	✓
5	Spanish-accented English	spanish	—	✓	✓
6	Chinese-accented English	chinese	—	—	✓

For example, a predicted class index of 5 corresponds to Spanish-accented English, while an index of 6 corresponds to Chinese-accented English in the 7-class checkpoint.

Usage

Install the dependencies and clone the official GitHub repository.

import torch
from huggingface_hub import hf_hub_download
from src.accentcl.models.accentcl import WhisperCLAccentModel

repo_id = "Morris88826/accentcl"

# Download model configuration
hf_hub_download(repo_id, "config.json")

# Download the selected checkpoint
checkpoint = hf_hub_download(
    repo_id,
    "accentcl_spanish_chinese.pt"
)

device = "cuda" if torch.cuda.is_available() else "cpu"

model = WhisperCLAccentModel.from_checkpoint(
    checkpoint, device=device
)
model.eval()

For audio preprocessing and complete inference examples, see the GitHub repository.

Evaluation Results

We evaluate AccentCL on regional English accent classification and class-incremental learning, following the experimental protocol described in our SLT 2026 paper.

1. Regional Accent Classification

We compare AccentCL against CommonAccent and Voxlect using a shared five-class regional accent label space. We report accuracy (Acc), balanced accuracy (BAcc), and macro-F1 across the full evaluation set (All) and out-of-domain (OOD) evaluation sets.

Method	All Acc ↑	All BAcc ↑	All Macro-F1 ↑	OOD Acc ↑	OOD BAcc ↑	OOD Macro-F1 ↑
CommonAccent	56.0	48.3	48.6	79.0	53.6	58.2
Voxlect	64.8	65.8	68.1	83.8	77.5	82.6
AccentCL (ours)	76.0	77.1	76.9	89.7	79.6	83.0

All values are percentages (%). OOD results are evaluated on Speech Accent Archive and IDEA, which are excluded from the training sources. Baseline predictions are mapped to the shared five-class label space, with predictions outside that label space counted as errors.

Compared to Voxlect (Whisper-Large-v3), AccentCL achieves +11.3 percentage points in balanced accuracy and +8.8 percentage points in macro-F1 on the full evaluation set.

2. Class-Incremental Accent Learning

AccentCL incrementally expands its accent label space from five to seven classes:

• 5 → 6: Add Spanish-accented English.
• 6 → 7: Add Chinese-accented English.

We report the F1 score on the newly introduced accent class (New F1) and balanced accuracy on previously learned classes (Old BAcc).

Incremental Step	Method	New F1 ↑	Old BAcc ↑
5 → 6 (+ Spanish)	Replay only	81.4	76.2
	Replay + retention	82.8	77.1
	Replay + margin	82.2	76.1
	AccentCL (ours)	83.3	77.3
6 → 7 (+ Chinese)	Replay only	41.9	75.1
	Replay + retention	48.5	75.9
	Replay + margin	59.9	77.1
	AccentCL (ours)	61.8	77.6

All values are percentages (%). Old BAcc is calculated over the accent classes learned before each incremental step.

AccentCL achieves 83.3% F1 when adding Spanish-accented English and 61.8% F1 when subsequently adding Chinese-accented English, while maintaining balanced accuracy on previously learned classes.

The continual-learning experiments use approximately 10% of the original training data as replay memory, avoiding the need to retrain on the full previous dataset.

For complete experimental settings, forgetting metrics, and additional analyses, please refer to the paper.

Citation

If you use AccentCL in your research, please cite:

@misc{tseng2026accentclrobustaccentclassification,
  title={AccentCL: Robust Accent Classification with Incremental Expansion},
  author={Mu-Ruei Tseng and Waris Quamer and Ghady Nasrallah and Ricardo Gutierrez-Osuna},
  year={2026},
  eprint={2610.07426},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.07426}
}

Publication status: Accepted at IEEE SLT 2026. The arXiv citation is provided until the official conference proceedings are available.