Parameter-Efficient Sparse Retrievers and Rerankers Using Adapters

Vaishali Pal, Carlos Lassance, Hervé Déjean, Stéphane Clinchant

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Parameter-Efficient transfer learning with Adapters have been studied in Natural Language Processing (NLP) as an alternative to full fine-tuning. Adapters are memory-efficient and scale well with downstream tasks by training small bottle-neck layers added between transformer layers while keeping the large pretrained language model (PLMs) frozen. In spite of showing promising results in NLP, these methods are under-explored in Information Retrieval. While previous studies have only experimented with dense retriever or in a cross lingual retrieval scenario, in this paper we aim to complete the picture on the use of adapters in IR. First, we study adapters for SPLADE, a sparse retriever, for which adapters not only retain the efficiency and effectiveness otherwise achieved by finetuning, but are memory-efficient and orders of magnitude lighter to train. We observe that Adapters-SPLADE not only optimizes just 2 but outperforms fully fine-tuned counterpart and existing parameter-efficient dense IR models on IR benchmark datasets. Secondly, we address domain adaptation of neural retrieval thanks to adapters on cross-domain BEIR datasets and TripClick. Finally, we also consider knowledge sharing between rerankers and first stage rankers. Overall, our study complete the examination of adapters for neural IR. (The code can be found at: https://github.com/naver/splade/tree/adapter-splade.)
Original languageEnglish
Title of host publicationAdvances in Information Retrieval
EditorsJaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, Annalina Caputo
Place of PublicationCham
PublisherSpringer Nature
Pages16-31
Number of pages16
ISBN (Print)978-3-031-28238-6
DOIs
StatePublished - 2023
Externally publishedYes

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Nature Switzerland

Keywords

  • Adapters
  • Information Retrieval
  • Sparse neural retriever

Fingerprint

Dive into the research topics of 'Parameter-Efficient Sparse Retrievers and Rerankers Using Adapters'. Together they form a unique fingerprint.
  • DL: ICAI Discovery Lab

    van Harmelen, F., De Rijke, M., Siebert, M., Hoekstra, R., Tsatsaronis, G., Groth, P., Cochez, M., Pernisch, R., Alivanistos, D., Mansoury, M., van Hoof, H., Pal, V., Pijnenburg, T., Mitra, P., Bey, T. & de Waard, A.

    10/1/1903/31/25

    Project: Research

Cite this