Skip to content

[BUG] ivf_rabitq needs seralize & deserilize steps before the index is searchable #2537

Description

@tfeher

Describe the bug

If I run ivf_rabitq::build() followed by ivf_rabitq::search(), it will not return correct neighbors. In contrast, other indices work correctly: an index reutrned by build() is searchable (assuming it fits memory).

To get correct results from the ivf_rabitq index we need to save to file and load again. This step is included in cuvs-bench as well as the unit tests.

Steps/Code to reproduce bug
In the following example if save_load is true, then recall would be correct, otherwise 0.

    cuvs::neighbors::ivf_rabitq::index_params index_params;
    index_params.n_lists      = n_lists;
    index_params.bits_per_dim = bits_per_dim;

    auto index = cuvs::neighbors::ivf_rabitq::build(
      res, index_params, raft::make_const_mdspan(d_dataset.view()));

    std::string save_path = "ivf_rabitq_minimal.ibin";
    if (save_load) {
      cuvs::neighbors::ivf_rabitq::serialize(res, save_path, index);
      cuvs::neighbors::ivf_rabitq::deserialize(res, save_path, &index);
    }

    cuvs::neighbors::ivf_rabitq::search_params search_params;
    search_params.n_probes = n_probes;
    cuvs::neighbors::ivf_rabitq::search(res,
                                        search_params,
                                        index,
                                        raft::make_const_mdspan(d_queries.view()),
                                        d_neighbors.view(),
                                        d_distances.view());

Expected behavior
It is expected that we can search right after build, without saving/loading the dataset.

Additional context

Full example code to illustrate the bug
https://gist.github.com/tfeher/77f9e956f4bc66747fb5be0996961c54

Usage

./RABITQ_MINIMAL 10000 128 64 8 8 rabitq 0
./RABITQ_MINIMAL 10000 128 64 8 8 rabitq 1

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions