Skip to main navigation Skip to search Skip to main content

Are a Thousand Words Better Than a Single Picture? Beyond Images - A Framework for Multi-modal Knowledge Graph Dataset Enrichment

  • Pengyu Zhang
  • , Klim Zaporojets
  • , Jie Liu
  • , Jia Hong Huang
  • , Paul Groth

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images, an automatic data-centric enrichment pipeline with optional human auditing. This pipeline operates in three stages: (1) large-scale retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG datasets and multiple baseline models, we observe consistent gains (up to +7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements (+201.35% MRR and +333.33% Hits@1). Additionally, we release a lightweight Text–Image Consistency Check Interface for optional targeted audits, improving description quality and dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code, datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images.

Original languageEnglish
Title of host publicationThe Semantic Web - 23rd European Semantic Web Conference, ESWC 2026, Proceedings
EditorsMaribel Acosta, Marieke van Erp, Sebastian Rudolph, Olaf Hartig, Blerina Spahiu, Anisa Rula, Daniel Garijo, Francesco Osborne
PublisherSpringer Science and Business Media Deutschland GmbH
Pages82-101
Number of pages20
ISBN (Print)9783032251558
DOIs
StatePublished - 2026
Event23rd European Semantic Web Conference, ESWC 2026 - Dubrovnik, Croatia
Duration: May 10 2026May 14 2026

Publication series

NameLecture Notes in Computer Science
Volume16549 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference23rd European Semantic Web Conference, ESWC 2026
Country/TerritoryCroatia
CityDubrovnik
Period05/10/2605/14/26

Keywords

  • Dataset Enrichment
  • Entity Representation
  • Image-to-Text
  • Link Prediction
  • Multi-modal Knowledge Graphs

Fingerprint

Dive into the research topics of 'Are a Thousand Words Better Than a Single Picture? Beyond Images - A Framework for Multi-modal Knowledge Graph Dataset Enrichment'. Together they form a unique fingerprint.

Cite this