Support for pg_bm25

General
1

For retrieval tasks in our RAG pipeline or even just generic search, it is useful to do re-rank with full-text. We are currently using pg_trgm, but trigram matching is not sufficient. The performance is not viable for re-ranking. To that end, we are hoping to get support for pg_bm25 if at all possible.

I was going to ask about pgroonga as well, but it seems like that thread was already closed.

2

Hey @skeptrune,

I’ll share your request with the team about adding support for pg_bm25 :smile:

2 Likes
3

Hey, have you heard anything back from them as of yet?

4

Hey @skeptrune, apologies for the delayed response. Unfortunately, this extension is not compatible with our storage system, so we won’t be able to support it

5

Hi everyone,

Maker of pg_bm25 here. We use Postgres IndexAM, but storage data in a more optimized way than the regular Postgres storage. It is compatible with any pure Postgres solution – does this mean Neon is not fully Postgres-compatible?

6

Neon separates compute and storage. It reconstruct pages from WAL at page server and send them to compute. Any data stored by extension in file system can not be persisted by Neon. Only extensions which access data using Postgres buffer manager are supported.