A↔F 3D Vision–Language
MULTI-REPRESENTATION · UNIFIED 3D UNDERSTANDING

From Alignment to Fusion
in 3D Vision-Language

Align complementary representations.
Preserve geometry. Understand the scene.

Xueqi Qiu1,*Xingyu Miao1,*Jingjing Deng2Haoran Duan3,†Yang Long1,†Ling Shao4
1 Durham University2 University of Bristol3 Tsinghua University4 UCAS-Terminus AI Lab, University of Chinese Academy of Sciences

* Equal contribution   † Corresponding authors

↓   Manuscript (PDF)Explore the framework ↗
Comparison of task-specific and unified approaches with our Triple Alignment and Orthogonal Fusion framework, and a radar chart across eight benchmarks.
One framework, complementary representations, four task families. Select any figure to view it at full resolution.
3Scene representations
4Task families
8Evaluation datasets
+10.6 ptsNr3D accuracy vs. PQ3D
01 / ABSTRACT

Better correspondence.
Controlled adaptation.

Unified 3D vision-language systems must combine complementary geometry, scale, and appearance cues while supporting tasks from instance segmentation to language-guided reasoning. Existing methods often process point clouds, voxel grids, and multi-view images independently; directly combining these heterogeneous representations may leave substantial feature discrepancy unresolved, while subsequent unconstrained adaptation may distort their internal geometry.

We propose an align-then-fuse framework that first applies triple pairwise cosine alignment to establish segment-level correspondence across the three representations and then retrieves task-conditioned features with a prompt-guided query decoder. Before fusion, representation-specific query features are transformed by learnable mappings constrained to the special orthogonal group. These mappings preserve inner products and Euclidean distances within each representation, permitting controlled representation-specific re-parameterisation without arbitrarily distorting its internal geometry. The transformed features are subsequently combined through Adaptive Fusion under downstream task supervision.

Experiments cover eight datasets for instance segmentation, visual grounding, question answering, and dense captioning. Compared with PQ3D, the model improves average precision by 3.2 points on ScanNet200 and grounding accuracy by 2.9, 10.6, 4.6, and 4.1 points on ScanRefer, Nr3D, Sr3D, and Multi3DRefer, respectively, while also improving performance on ScanQA, SQA3D, and Scan2Cap. Ablations further support the complementary roles of alignment and orthogonal re-parameterisation and the effectiveness of Adaptive Fusion.

02 / METHOD

Align first. Then fuse.

Separate cross-representation correspondence from geometry-preserving adaptation.

01

Triple Alignment

Pairwise cosine objectives align spatially matched scene segments across point-cloud, voxel, and multi-view image features.

ℒalign = ∑(a,b) wab ℒab
02

Orthogonal adaptation

Learnable representation-specific mappings in SO(D) adapt the latent basis while preserving inner products and Euclidean distances.

RᵀR = I   ·   det(R) = 1
03

Adaptive Fusion

Learned weights combine transformed features. Prompt-conditioned queries support segmentation, grounding, answering, and captioning.

Geometry + structure + appearance
03 / RESULTS

Consistent gains across eight datasets.

Selected metrics reported in the manuscript. All comparisons below use PQ3D as the baseline; higher is better.

Performance comparison with PQ3D
Task / DatasetMetricPQ3DOursGain
Instance segmentationScanNet200AP27.030.2+3.2
Visual groundingScanReferAvg. accuracy @ IoU 0.551.254.1+2.9
Nr3DAvg. accuracy66.777.3+10.6
Sr3DAvg. accuracy79.784.3+4.6
Multi3DReferAvg. @ IoU 0.550.154.2+4.1
Question answeringScanQAEM@1 · test with object26.128.2+2.1
SQA3DAvg. exact match47.148.7+1.6
Dense captioningScan2CapCIDEr @ IoU 0.580.381.2+0.9

Gains are absolute metric differences. Nr3D and Sr3D use ground-truth object masks. Metrics differ across datasets and should not be compared directly.

Ablation: why use both alignment and orthogonal fusion?

On ScanNet200, combining Triple Alignment (TA) and Orthogonal Fusion (OF) improves all three reported instance segmentation metrics.

VariantAPAP₅₀AP₂₅
Baseline27.038.946.3
TA only30.041.147.2
OF only29.841.347.6
TA + OF30.241.648.8
04 / QUALITATIVE RESULTS

From object perception to scene reasoning.

Original examples from the paper compare our predictions with PQ3D and ground truth.

Qualitative comparisons for language-conditioned instance segmentation and visual grounding.
Instance segmentation & visual grounding. Category, functional, and relational prompts guide object understanding.
Qualitative comparisons for question answering and dense captioning, including object identity and spatial relations.
Question answering & dense captioning. Reasoning over object identity, attributes, and spatial context.
05 / SCOPE

Evaluation scope & limitations

The framework relies on pretrained, representation-specific encoders. Evaluation covers eight predominantly indoor-scene datasets; robustness to outdoor scenes, missing representations, and sensor corruption remains untested.

Relative to PQ3D, parameters increase from 247.88M to 254.95M and computation from 25.81 to 26.94 GFLOPs. Reported forward-pass time increases from 0.1340 s to 0.1648 s.

06 / CITATION

Cite this work

Manuscript citation; publication venue, year, and DOI are not specified in the supplied source.

@misc{qiu_alignment_fusion,
  title  = {From Alignment to Fusion in 3D Vision-Language},
  author = {Qiu, Xueqi and Miao, Xingyu and Deng, Jingjing and
            Duan, Haoran and Long, Yang and Shao, Ling},
  note   = {Manuscript},
  url    = {https://nora202.github.io/Alignment-Fusion/}
}