7 comments

  • gopalv 3 hours ago
    > Instead of joining the tables completely and sorting at the end, a Rank Join streams its inputs in descending score order (sorted access, or SA), building candidate result-sets as it goes.

    Your algorithm feels like a variant of the WAND algorithm the Snowflake Top-K pruning paper [1] pulls from.

    You can go distributed with this approach if you have a "gossip" protocol between workers to distribute the heap boundaries periodically.

    Before the AI age, a "papers we love" group was a great thing within any company.

    Now, Claude or Codex will know the SOTA literature around a topic like this and "uniformly distribute" the past into every single implementation of a similar idea. The future isn't uniformly distributed, but the past is getting the treatment & that's good.

    [1] - https://arxiv.org/html/2504.11540v1#S5

    • stuhood 2 hours ago
      Yes! Search engines like Tantivy have long employed WAND to score across terms. The difference though is that WAND is not joining terms in descending order by score: rather, by doc-id.

      This algorithm is closer to Fagin's Threshold Algorithm (also mentioned in your link): the Threshold Algorithm was the first iteration of a rank join.

  • anon373839 2 hours ago
    ParadeDB is the best! I really like the Django integration (https://github.com/paradedb/django-paradedb). You get proper BM25 scoring, fuzzy matching, hybrid search, metadata filtering while still just using QuerySets. And you're keeping everything in Postgres, so there isn't any need to manage or sync a separate DB/search engine.
  • amluto 1 hour ago
    This is neat, but I’m a bit confused by the query that motivates the whole article. If I search for ‘ocaml’, I want to match users’ about_me (the users.about_me ||| 'ocaml' part), but I do really want this to match on the basis of each (completely irrelevant) post and/or comment that gets picked up by the join? I think I would rather match the user, possibly with some user scoring by the overall user score independent of my search term.

    I feel like I would probably execute three separate queries and merge the results myself. And the user query wouldn’t join against posts at all. Doing this would end up implicitly doing something like the optimization in the article.

  • jamesgresql 3 hours ago
    Any questions please go for it, ParadeDB team is watching.
  • dan_thinks 4 hours ago
    This looks exciting for some searches (top k), but the headline doesn't really fit. Yes - we still have to denormalize. Just not for some of the use cases paradedb targets. Impressive work.
  • xiaoliangliang 1 hour ago
    [flagged]
  • lakhya 2 hours ago
    [flagged]