TY - GEN
T1 - Evaluating Class Membership Relations in Knowledge Graphs Using Large Language Models
AU - Allen, Bradley P.
AU - Groth, Paul T.
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
PY - 2025
Y1 - 2025
N2 - A backbone of knowledge graphs are their class membership relations, which assign entities to a given class. As part of the knowledge engineering process, we propose a new method for evaluating the quality of these relations by processing descriptions of a given entity and class using a zero-shot chain-of-thought classifier that uses a natural language intensional definition of a class. We evaluate the method using two publicly available knowledge graphs, Wikidata and CaLiGraph, and 7 large language models. Using the gpt-4-0125-preview large language model, the method’s classification performance achieves a macro-averaged F1-score of 0.830 on data from Wikidata and 0.893 on data from CaLiGraph. Moreover, a manual analysis of the classification errors shows that 40.9% of errors were due to the knowledge graphs, with 16.0% due to missing relations and 24.9% due to incorrectly asserted relations. These results show how large language models can assist knowledge engineers in the process of knowledge graph refinement. The code and data are available on Github (https://github.com/bradleypallen/evaluating-kg-class-memberships-using-llms).
AB - A backbone of knowledge graphs are their class membership relations, which assign entities to a given class. As part of the knowledge engineering process, we propose a new method for evaluating the quality of these relations by processing descriptions of a given entity and class using a zero-shot chain-of-thought classifier that uses a natural language intensional definition of a class. We evaluate the method using two publicly available knowledge graphs, Wikidata and CaLiGraph, and 7 large language models. Using the gpt-4-0125-preview large language model, the method’s classification performance achieves a macro-averaged F1-score of 0.830 on data from Wikidata and 0.893 on data from CaLiGraph. Moreover, a manual analysis of the classification errors shows that 40.9% of errors were due to the knowledge graphs, with 16.0% due to missing relations and 24.9% due to incorrectly asserted relations. These results show how large language models can assist knowledge engineers in the process of knowledge graph refinement. The code and data are available on Github (https://github.com/bradleypallen/evaluating-kg-class-memberships-using-llms).
KW - Knowledge engineering
KW - knowledge graph refinement
KW - large language models
KW - natural language generation
UR - http://www.scopus.com/inward/record.url?scp=85218446264&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-78952-6_2
DO - 10.1007/978-3-031-78952-6_2
M3 - Contribución a la conferencia
AN - SCOPUS:85218446264
SN - 9783031789519
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 14
EP - 24
BT - The Semantic Web
A2 - Meroño Peñuela, Albert
A2 - Corcho, Oscar
A2 - Groth, Paul
A2 - Simperl, Elena
A2 - Tamma, Valentina
A2 - Nuzzolese, Andrea Giovanni
A2 - Poveda-Villalón, Maria
A2 - Sabou, Marta
A2 - Presutti, Valentina
A2 - Celino, Irene
A2 - Revenko, Artem
A2 - Raad, Joe
A2 - Sartini, Bruno
A2 - Lisena, Pasquale
PB - Springer Science and Business Media Deutschland GmbH
T2 - European Semantic Web Conference, ESWC 2024
Y2 - 26 May 2024 through 30 May 2024
ER -