medium10 pts · first answer countsmodel-servinglatencyretrieval

Cross-encoder reranker is slow

A cross-encoder reranker scores every one of 1,000 retrieved passages per query and adds 2 seconds of latency. What is the usual fix?

A.Rerank all 10,000 passages for accuracyB.Replace the first-stage retriever with the rerankerC.Run the reranker on the CPUD.Rerank only the top 20 to 100 search results

Discussion 0

Join free to post in the discussion.

Cross-encoder reranker is slow

Discussion 0

Related questions