UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web

Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, Yuxuan Liang†

• First-ever LLM-enhanced framework that integrates the knowledge of textual modality into urban imagery profiling.

The International World Wide Web Conference 2024, Singapore (WWW’24)

Oral Presentation

GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understanding

Yibo Yan, Joey Lee†

• A pipeline integrating linguistic and geospatial information, showcasing the advantages of an LLM-assisted workflow over conventional methods in geo-reasoning tasks.

33rd ACM International Conference on Information and Knowledge Management, Idaho, USA (CIKM’24)

Best Short Paper Award

ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection

Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, Aoxiao Zhong, Kun Wang, Hui Xiong, Philip S. Yu, Xuming Hu†, Qingsong Wen†

• First benchmark designed to assess MLLMs’ complex reasoning capabilities in multimodal error detection.

Workshop on Reasoning and Planning for LLMs, 13th International Conference on Learning Representations, Singapore (ICLR Workshop’25)

A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges

Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, Xuming Hu†

• First comprehensive analysis of mutlimodal mathematical reasoning in the era of MLLMs.

Under Review

MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection

Yibo Yan, Shen Wang, Jiahao Huo, Philip S. Yu, Xuming Hu†, Qingsong Wen†

• A novel Mixture-of-Math-Agent framework designed specifically to address mutlimodal error detection.

Under Review

Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning

Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu†, Philip S. Yu, Carla Gomes, Bart Selman, Qingsong Wen†

• This position paper argues that MLLMs can significantly advance scientific reasoning across disciplines such as mathematics, physics, chemistry, and biology.

Under Review

Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook

Xingchen Zou*, Yibo Yan*, Xixuan Hao, Yuehong Hu, Haomin Wen, Erdong Liu, Junbo Zhang, Yong Li, Tianrui Li, Yu Zheng, Yuxuan Liang†

• First comprehensive survey that systematically reviews studies on deep learning-based multimodal and multi-source data fusion models in urban computing.

Information Fusion Journal (IF=15)

MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models

Jiahao Huo*, Yibo Yan*, Xu Zheng, Yuanhuiyi Lyu, Xin Zou, Zhihua Wei, Xuming Hu†

• We propose to reformulate the task of multimodal MU in the era of MLLMs, and develop a novel geometry-constrained gradient descent method MMUnlearner.

Under Review

EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models

Jiamin Su*, Yibo Yan*, Fangteng Fu, Han Zhang, Jingheng Ye, Xiang Liu, Jiahao Huo, Huiyu Zhou, Xuming Hu†

• First multimodal benchmark to evaluate Automated Essay Scoring (AES) capabilities across lexical-, sentence-, and discourse-level traits.

Under Review

Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, Xuming Hu†

• A causal inference framework that applies structural causal modeling to MLLMs, treating modality priors as a confounder between attention mechanisms and output.

13th International Conference on Learning Representations, Singapore (ICLR’25)

MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model

Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, Xuming Hu†

• Investigation of the distribution of domain-specific neurons and the mechanism of how MLLMs process features from diverse domains.

Conference on Empirical Methods in Natural Language Processing 2024, Florida, USA (EMNLP’24)

Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models

Kening Zheng*, Junkai Chen*, Yibo Yan, Xin Zou, Xuming Hu†

• A comprehensive benchmark specifically targeting relation hallucinations, consisting of over 20k samples derived from real-world scenarios.

Under Review

Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

Xin Zou*, Yizhou Wang*, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, Xuming Hu†

• A novel hallucination mitigation paradigm that without the need for external knowledge retrieval or additional fine-tuning.

Under Review

UrbanVLP: A Multi-Granularity Vision-Language Pre-Trained Model for Urban Indicator Prediction

Xixuan Hao*, Wei Chen*, Yibo Yan, Siru Zhong, Kun Wang, Qingsong Wen, Yuxuan Liang†

• First urban region representation learning framework that explores multi-granularity cross-modal alignment.

39th Annual AAAI Conference on Artificial Intelligence, Philadelphia, USA (AAAI’25)

UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation

Siru Zhong, Xixuan Hao, Yibo Yan, Ying Zhang, Yangqiu Song, Yuxuan Liang†

• First cross-domain framework that integrates the power of LMM and SAM into satellite image-text retrieval.

32nd ACM Multimedia Conference, Melbourne, Australia (ACM MM’24)