MorphPT is a domain-adapted vision model for cell-type classification from DAPI-stained nuclei. Each cell is represented by paired multi-scale crops — a 2.5× view (fine nuclear morphology) and a 10× view (broader tissue context) — encoded by a shared DINOv3 ViT-B backbone with LoRA adaptation and fused by a learnable ScaleGate module. A coarse router predicts one of six broad morphology-informed groups, and group experts resolve fine cell types via soft routing.
- 📦 Pretrained weights (Hugging Face): https://huggingface.co/jilab/MorphPT
- 🗂️ CellImageNet dataset: huggingface.co/datasets/jilab/CellImageNet
- 🤖 MorphAgent (NL controller): github.com/ZBox1005/MorphAgent
- 📄 Paper: citation forthcoming
- 🔖 Archived release used in the manuscript: https://doi.org/10.5281/zenodo.21031925 (all versions: https://doi.org/10.5281/zenodo.20941949)
Code lives here; model weights are hosted on Hugging Face (the checkpoints are too large for git). See Pretrained weights.
git clone https://github.com/AnitaCao/MorphPT.git
cd MorphPT
pip install -r requirements.txt
export PYTHONPATH=$PWDPython ≥3.10 and a CUDA GPU are recommended.
Download from Hugging Face:
from huggingface_hub import snapshot_download
ckpt = snapshot_download("jilab/MorphPT") # router/ + expert_<Group>/ at the rootLayout ($CKPT = the snapshot root):
router/best.pt router/coarse_to_id.json
expert_Cancer/best.pt expert_Cancer/class_to_idx.json
expert_Lymphoid/best.pt expert_Lymphoid/class_to_idx.json
expert_Neuroglial/best.pt expert_Neuroglial/class_to_idx.json
expert_Tissue_Vascular/best.pt expert_Tissue_Vascular/class_to_idx.json
A worked, runnable vignette (with 10 bundled real example cells) lives in
examples/. Predict a single cell from its two crops:
python examples/predict_cell.py --weights_dir $CKPT \
--img_2p5x examples/real_cells/<cell>_2p5x.png \
--img_10x examples/real_cells/<cell>_10x.pngOr programmatically (downloads the weights from the Hub):
from morphpt import MorphPTPredictor
model = MorphPTPredictor.from_pretrained("jilab/MorphPT")
print(model.predict_one("cell_2p5x.png", "cell_10x.png"))See examples/README.md for batch prediction, the data
format, and a fine-tuning walkthrough.
CellImageNet is a large-scale single-cell image database (~10M cells, 31 cell types, across diverse species/tissues/conditions) built from public 10x Genomics Xenium data. Each cell has two paired DAPI crops — a 2.5× view (fine nuclear morphology) and a 10× view (broader tissue context) — and a cell-type label. Available at huggingface.co/datasets/jilab/CellImageNet.
MorphPT consumes the crops directly; you only supply the two views per cell. The only input you build is a small manifest listing, per cell, the two crop paths and the label:
img_path_2p5x,img_path_10x,label
/data/crops/cell0_2p5x.png,/data/crops/cell0_10x.png,T cells
/data/crops/cell1_2p5x.png,/data/crops/cell1_10x.png,Neurons- A
.csvor.parquetmanifest both work. The model reads the PNG crops at runtime (resized to 224×224, ImageNet-normalized). - Only
img_path_2p5x,img_path_10x, and the label column are required.cell_id,tissue, andx/ycentroids are optional metadata (auto-filled if absent). For router training usecoarse_labelas the label column. - This manifest is the universal input for
examples/predict_cell.py,trainer/train_*.py(--train_dir), andscripts/eval_moe_e2e.py.
Optional, not required:
data/prepare_multiview_parquet.pyreproduces our CellImageNet manifest from the Xenium metadata. If you already have your own paired crops, skip it and write the 3-column manifest above directly.data/preprocess_memmap.pybuilds a memmap tensor cache for faster GPU throughput during large-scale training (pass--cache_dir). Training and evaluation read the PNG crops directly without it.
Cropping/segmentation (raw slide → per-cell 2.5×/10× crops) is out of scope: MorphPT assumes the paired crops already exist.
morphpt/ pip package: MorphPTPredictor (from_pretrained inference API)
models/ backbone, LoRA, ScaleGate, MultiViewClassifier
data/ dataset (parquet + crops), preprocessing / caching
trainer/ router / expert / single-view training (DDP)
scripts/ end-to-end evaluation and analysis
examples/ minimal runnable vignette (inference + fine-tuning)
visiumHD/ downstream task: MorphPT router features -> spatial expression regression
splits/ class maps (coarse/fine) and the train/val/test split manifest
configs/ configuration
utils/ shared helpers
Router (six-way coarse stage):
python trainer/train_router.py \
--train_dir <manifest.parquet> --class_map <coarse_to_id.json> \
--label_col coarse_label --fuse gate \
--lora_rank 16 --lora_alpha 32 --batch_size 128 --epochs 35 \
--lr_head 3e-4 --lr_lora 1e-4 --cons_weight 0.10 \
--out_dir <out>Group expert (initialized from the router checkpoint):
python trainer/train_expert.py \
--train_dir <manifest.parquet> --class_map <expert_class_to_idx.json> \
--label_col label --expert_name Cancer \
--router_ckpt <router/best.pt> --fuse gate \
--batch_size 128 --epochs 30 --lr_head 3e-4 --lr_lora 3e-5 \
--out_dir <out>--train_dir takes a manifest (.csv/.parquet) or a directory of parquet
shards and reads the crops directly; add --cache_dir <memmap> only if you want
the optional GPU-throughput cache. Multi-GPU runs use torchrun/SLURM (rank,
local rank, world size are read from the environment).
python scripts/eval_moe_e2e.py \
--test_parquet <test.parquet> \
--router_ckpt $CKPT/router/best.pt \
--expert_dir $CKPT \
--out_dir <out>Reports macro-F1, per-class metrics, confusion matrices, and confidence scores for soft / top-1 / top-K / oracle routing.
visiumHD/ uses MorphPT's router features to predict spatial gene
expression, demonstrating transfer of the morphology-adapted representation. See
that directory for its scripts.
- All scripts take paths via CLI flags or environment variables — no site-specific
paths are baked in. The downstream
visiumHD/scripts read a repo/data root fromMORPHPT_ROOT(defaults to the repo root); the VisiumHD dataset/trainer readVISIUM_ROOT,VISIUM_MORPHPT_DATA, andVISIUM_CACHE. Set these to point at your own data. CellImageNet(the training corpus) is released separately (link above).- Cropping/segmentation is out of scope; MorphPT operates on paired 2.5×/10× crops you provide.
MorphAgent is a natural-language controller that orchestrates cell segmentation, MorphPT-based cell-type classification, and tumor–immune spatial analysis through an interactive web interface — github.com/ZBox1005/MorphAgent (MIT License). MorphAgent trains no models of its own: it delegates cell-type prediction to MorphPT and performs planning with a general-purpose LLM accessed through a provider API (e.g. OpenAI GPT-4o or Anthropic Claude), so no model weights are distributed with it. A versioned archival snapshot will be deposited on Zenodo upon publication.
@article{cao2026visual,
title = {A visual foundation model for cell classification},
author = {Cao, Ting and Zhuang, Haotian and Zhang, Boxuan and
Pang, Zhiping P. and Tang, Ruixiang and Liu, Dongfang and
Ji, Zhicheng},
year = {2026}
}The code in this repository is released under LICENSE. The
pretrained weights (hosted on Hugging Face)
embed Meta's DINOv3 backbone and are distributed under the DINOv3 License —
see the model card and LICENSE.md in the weights repository.