Search Quality Evaluator (searcheval)

searcheval is a command-line tool for benchmarking, evaluating, and tuning search relevance for pkgsite. It supports comparing standard PostgreSQL full-text keyword search (ts_rank) against Hybrid Vector Search combining keyword scoring with Vertex AI text embeddings via Reciprocal Rank Fusion (RRF).

Key Concepts & Evaluation Metrics

• Reciprocal Rank Fusion (RRF): This is an informatioan retrieval aggregation technique that enables us to combine rankings from distinct search algorithms - in our system this is text search and vector search. For example, text search outputs scores with arbitrary scales (e.g. 0.15, 0.24) but vector search outputs similarity on a scale from (-1.0 and 1.0). RRF ignores the raw score, but just looks at the rank order. See Cormack et al. (2009).
• Mean Reciprocal Rank (MRR): A statistic evaluating ranking quality against target ground-truth packages. For a single query, the Reciprocal Rank (RR) is $\frac{1}{\text{rank}}$ of the first expected target package (or $0.0$ if outside top-$N$). MRR@5 is the average of Reciprocal Ranks across all golden queries.

Prerequisites & Database Setup

1. Start Cloud SQL Proxy

Choose one of the following options to launch the database proxy on port 5431:

• Option A: Connect to Main Dev Database (dev-discovery-db)

  ./private/devtools/run_cloud_sql_proxy.sh dev
  

• Option B: Connect to Custom / Cloned Database Instance (dev-discovery-db-clone)

  cloud-sql-proxy --port 5431 go-discovery:us-central1:dev-discovery-db-clone
  

2. Export Required Environment Variables

Set your GCP project, Cloud SQL password, and connection parameters:

export GOOGLE_CLOUD_PROJECT=go-discovery
export GO_DISCOVERY_DATABASE_PASSWORD=<password>
export GO_DISCOVERY_DATABASE_PORT=5431
export GO_DISCOVERY_DATABASE_NAME=dev-discovery-db

1. Golden Query Evaluation (eval)

Runs search quality benchmarks against a curated set of ground-truth queries (golden_queries.json) and reports Mean Reciprocal Rank (MRR@5) metrics. Each cell displays the target package's 1-based Rank and its corresponding Reciprocal Rank score (1/rank), which is averaged in the summary row.

go run ./devtools/cmd/searcheval eval

Output Example:

Query                     | Target Package                      | Keyword     | Vector      | Hybrid
----------------------------------------------------------------------------------------------------
http router               | github.com/go-chi/chi/v5            | #2 (0.500)  | #6 (0.167)  | #2 (0.500)
structured logger         | go.uber.org/zap                     | >5 (0.000)  | #4 (0.250)  | >5 (0.000)
web framework             | github.com/gin-gonic/gin            | #1 (1.000)  | #3 (0.333)  | #1 (1.000)
command line parser       | github.com/spf13/cobra              | >5 (0.000)  | #2 (0.500)  | #4 (0.250)
orm                       | gorm.io/gorm                        | #2 (0.500)  | #1 (1.000)  | #1 (1.000)
----------------------------------------------------------------------------------------------------
MEAN RECIPROCAL RANK (MRR) |                                     | 0.479       | 0.449       | 0.562

2. Bulk Production Query Log Evaluation (log-eval)

Evaluates performance and zero-result recovery rates across unlabelled, real-world production query logs (e.g. extracted via queries_for_compare.sh or log exports). This will enable one to benchmark against real user search traffic.

# Optional: Pull fresh query logs from Cloud Logging for the last 24h
./private/devtools/cmd/search/queries_for_compare.sh 24h > fresh_24h.queries

# Run comparative log evaluation
go run ./devtools/cmd/searcheval -log-queries=fresh_24h.queries log-eval

Metrics Measured:

• Keyword Zero-Result Queries: Count and percentage of production queries returning 0 results with standard keyword search.
• Hybrid Zero-Result Queries: Count and percentage of queries returning 0 results with hybrid search.
• Zero-Result Recoveries: Queries where keyword search returned 0 results, but hybrid search successfully surfaced $\ge 1$ result via vector search.
• Average Result Overlap: Mean top-5 result set overlap between standard keyword search and hybrid search across queries.

3. Interactive Query Debugger (search)

Runs a single search query string and displays top 10 search results side-by-side across Keyword, Pure Vector, and Hybrid RRF modes:

go run ./devtools/cmd/searcheval search "structured logger"

4. Embedding Backfill (embed)

Scans PostgreSQL for active packages (imported_by_count >= 1 AND embedding IS NULL), generates 256-dimensional text embeddings via Vertex AI (text-embedding-004), and updates search_documents.embedding in the connected database:

go run ./devtools/cmd/searcheval embed

5. Local Database Seeding (seed)

[!CAUTION] Do NOT run searcheval seed while connected to Cloud SQL database proxies (dev, staging, prod). seed writes module data into PostgreSQL and is intended ONLY for fresh local Postgres instances (localhost:5432). searcheval automatically blocks seed when connected to non-standard ports unless GO_DISCOVERY_ALLOW_SEED=true is set.

Populates a fresh local PostgreSQL database with popular Go modules (e.g. chi, gin, zap, pgx, cobra, viper, gorm, uuid, grpc, testify, zerolog, bun, badger, bbolt, etc.) fetched directly from proxy.golang.org:

# Seed default set of 100 popular benchmark modules
go run ./devtools/cmd/searcheval seed

# Or seed specific custom modules
go run ./devtools/cmd/searcheval seed github.com/go-chi/chi/v5@v5.0.12 github.com/gin-gonic/gin@v1.9.1

Search & Scoring Parameter Flags

You can experiment with search scoring parameters (such as feature weights and result limits) dynamically without recompiling code:

Example using custom weights:

go run ./devtools/cmd/searcheval -vector-weight=1.2 -popularity-weight=0.8 eval