[WIP] ANNs for clustering at training time - #1938
Open
HowardHuang1 wants to merge 19 commits into
Open
Conversation
…es to find inflection point
… applied to accelerate extend()
aamijar
added
non-breaking
Member
|
/ok to test d26e916 |
…nment for clarity
…sed by upstream changes. Add 2 helper functions to recover use_ann_for_extend flag after serialize/deserialize
…rrors Extend_NearestCentroidLookup_Speedup for clarity
…up_Speedup and BM_IVFPQ_Extend_NearestCentroidLookup_Speedup. Also rename use_ann_for_fit to use_ann_for_build_fit to clear up confusion because ANN for nearest cluster assignment can occur in 2 places within build(): (1) in fit and (2) post-fit
…stfit boolean and test cases. Since extend and postfit both use ann on fixed centroids, we factored out common code. Renamed functions in kmeans_balanced.cuh for clarity: assign_nearest_centroid_cagra now more accurately mirrors assign_nearest_centroid_cagra_with_index_reuse.
…evice_matrix_views
…weeps The old Args() list varied N and K independently with no consistent ratio (2x-312x points-per-cluster across entries), confounding the two and making the brute-force-vs-CAGRA crossover point ill-defined. Replace it with two controlled sweeps over the same K range (1K-1M clusters): RegisterConstantRatioSweep (N = 5 * K, isolates K's effect at a fixed points-per-cluster ratio) and RegisterFixedNVaryKSweep (N held at 2,000,000, isolates K's effect at a fixed dataset size). Add plot_cluster_assignment_bench.py to run the benchmark and plot brute force vs CAGRA time against K, with automatic crossover detection and a --side-by-side mode to compare both sweeps at once.
…terval sweeps Extends the cluster-assignment benchmark with additional controlled sweeps to fully characterize the brute-force-vs-CAGRA crossover: - RegisterFixedKVaryNSweep: holds K fixed, varies N, to check whether growing N alone (independent of K) can flip the crossover. - RegisterGridSweep: coarse (N, K) grid for a 2D crossover boundary. - BM_ClusterAssignment_CAGRA_SearchOnly / RegisterCagraSearchOnlySweep: builds the CAGRA index once outside the timed loop, isolating steady-state search cost from one-time graph-build cost. - RegisterDimSensitivitySweep: checks whether the crossover K shifts across realistic embedding dimensions (dim was previously untested outside 128). - BM_ClusterAssignment_CAGRA_AmortizedInterval / RegisterCagraAmortizedIntervalSweep: empirically measures "build once, search `interval` times back-to-back" as a single timed unit, instead of projecting it from separately-measured build/search numbers, to see how ann_rebuild_interval actually amortizes build cost. Add plot_all_cluster_assignment_benchmarks.py, producing all 8 analysis panels (the two original sweeps, the three new ones, a speedup-ratio view, a measured build-vs-search breakdown, and assignment quality vs ann_rebuild_interval from the existing CUVS_FIT_ANN_REBUILD_TUNING_BENCH) in one figure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.