Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignment (ICCV 2025)

Project page | Paper | arXiv | 中译版 | Jittor

This repository contains the official Pytorch implementation of training & evaluation code and the trained models for Offset Learning & OffSeg.

Offset Learning —— An efficient plug-and-play semantic segmentation paradigm that replaces existing per-pixel classification paradigm to boost performance with negligible parameters.

Overview of the Offset Learning framework for semantic segmentation.

Abstract

Offset Learning is a new semantic segmentation paradigm that efficiently learns feature offsets and class offsets to dynamically refine both spatial features and class representations, addressing the inherent misalignment problem in per-pixel classification. Based on this paradigm, we design OffSeg, an efficient segmentation network that delivers consistent accuracy improvements on multiple benchmarks. Notably, the Offset Learning paradigm is plug-and-play, allowing it to directly replace other segmentation paradigms in existing models to achieve performance gains with only negligible parameter overhead.

Features

• Offset Learning: Learns feature offsets and class offsets to dynamically refine spatial features and class representations.
• Plug-and-play: Compatible with existing segmentation frameworks like SegFormer, SegNeXt, and Mask2Former.
• Lightweight & Efficient: Achieves consistent accuracy gains on multiple benchmarks with negligible parameter overhead.
• Proven Effectiveness: Validated across diverse models and datasets, showing strong improvements especially in lightweight settings.

News

• 2025.09.06: The Chinese version has been updated for Chinese readers.
• 2025.08.13: Add tutorial on how to apply the Offset Learning paradigm to your own models.
• 2025.08.12: The full training & evaluation code and the trained models are released.
• 2025.06.26: Our paper is accepted to ICCV 2025!

TODO

• Release the full training & evaluation code and model weights.
• Tutorial on how to apply the Offset Learning paradigm to your own models.
• Release the jittor version for jittor users.
• Release the Python library for easier installation via pip install.
• Explore the generalization ability of Offset Learning on tasks beyond semantic segmentation.

Get Started

Installation

conda create -n offseg python=3.9 -y
conda activate offseg

# Install PyTorch (CUDA 11.8 example)
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia

# install mmcv using mim
pip install -U openmim
mim install mmengine
mim install mmcv==2.0.0
mim install mmdet

pip install ftfy transformers==4.28.0

# Install OffSeg
pip install -e .

Data Preparation

For data preparation, please refer to the guidelines in mmsegmentation. It is recommended to symlink the dataset root to OffSeg/data.

For convenience, the recommended folder structure is as follows:

OffSeg
├── data
│   ├── ade
│   │   ├── ADEChallengeData2016
│   │   │   ├── annotations
│   │   │   │   ├── training
│   │   │   │   ├── validation
│   │   │   ├── images
│   │   │   │   ├── training
│   │   │   │   ├── validation
│   ├── cityscapes
│   │   ├── leftImg8bit
│   │   │   ├── train
│   │   │   ├── val
│   │   ├── gtFine
│   │   │   ├── train
│   │   │   ├── val
│   ├── coco_stuff164k
│   │   ├── images
│   │   │   ├── train2017
│   │   │   ├── val2017
│   │   ├── annotations
│   │   │   ├── train2017
│   │   │   ├── val2017
│   ├── VOCdevkit
│   │   ├── VOC2010
│   │   │   ├── JPEGImages
│   │   │   ├── SegmentationClassContext
│   │   │   ├── ImageSets
│   │   │   │   ├── SegmentationContext
│   │   │   │   │   ├── train.txt
│   │   │   │   │   ├── val.txt
│   │   │   ├── trainval_merged.json

Checkpoints

The trained models can be downloaded at:

Model	GoogleDrive	OneDrive	BaiduNetdisk
OffSeg	GoogleDrive	OneDrive	BaiduNetdisk
SegFormer w/ Offset Learning	GoogleDrive	OneDrive	BaiduNetdisk
SegNeXt w/ Offset Learning	GoogleDrive	OneDrive	BaiduNetdisk
Mask2Former w/ Offset Learning	GoogleDrive	OneDrive	BaiduNetdisk

Evaluation

Single GPU Evaluation

# OffSeg-T on ADE20K
python tools/test.py local_configs/offseg/Tiny/offseg-t_ade20k_160k-512x512.py /path/to/checkpoint.pth

# SegFormer-B0 with Offset Learning on COCO-Stuff
python tools/test.py local_configs/segformer_offset_learning/B0/segformer_mit-b0_offset_learning_8xb2-80k_stuff164k-512x512.py /path/to/checkpoint.pth

# SegNeXt-T with Offset Learning on Pascal Context
python tools/test.py local_configs/segnext_offset_learning/Tiny/segnext_mscan-t_offset_learning_80k_pascal-context-59_480x480.py /path/to/checkpoint.pth

Multi-GPU Evaluation

# Example with 8 GPUs
bash tools/dist_test.sh local_configs/offseg/Tiny/offseg-t_ade20k_160k-512x512.py /path/to/checkpoint.pth 8

Evaluation with Visualization

# Show results during evaluation
python tools/test.py local_configs/offseg/Tiny/offseg-t_ade20k_160k-512x512.py /path/to/checkpoint.pth --show

# Save visualization results
python tools/test.py local_configs/offseg/Tiny/offseg-t_ade20k_160k-512x512.py /path/to/checkpoint.pth --show-dir ./vis_results

Training

Single GPU Training

# Train OffSeg-B on ADE20K
python tools/train.py local_configs/offseg/Base/offseg-b_ade20k_160k-512x512.py

# Train with custom work directory
python tools/train.py local_configs/offseg/Base/offseg-b_ade20k_160k-512x512.py --work-dir ./work_dirs/offseg-b_ade20k_160k-512x512

# Resume training from checkpoint
python tools/train.py local_configs/offseg/Base/offseg-b_ade20k_160k-512x512.py --resume

Multi-GPU Training

# Example with 8 GPUs
bash tools/dist_train.sh local_configs/offseg/Base/offseg-b_ade20k_160k-512x512.py 8

Visualization

Image Demo

# Single image inference
python demo/image_demo.py demo/demo.png local_configs/offseg/Tiny/offseg-t_ade20k_160k-512x512.py /path/to/checkpoint.pth --out-dir ./demo_results

Citation

If you find this work useful for your research, please cite our paper:

@article{zhang2025revisiting,
  title={Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignment},
  author={Zhang, Shi-Chen and Li, Yunheng and Wu Yu-Huan and Hou, Qibin and Cheng, Ming-Ming},
  journal={arXiv preprint arXiv:2508.08811},
  year={2025}
}

Acknowledgment

This project is built upon MMSegmentation. We thank the MMSegmentation team for their open-source contribution. We also thank the following open-source projects for their inspiring work: SegFormer, SegNeXt, Mask2Former, FreqFusion, EfficientFormerV2.

License

The code is limited to non-commercial, academic, or research purposes only. For commercial use, please contact the authors for licensing.

Contact

For questions and issues, please contact:

• Email: [zhangshichen@mail.nankai.edu.cn]
• Issues: GitHub Issues