Rerank search results with Jev: a typed relevance rubric you can test

What this guide shows: request formats and application code follow the TypeSafe documentation checked on 2026-09-25. The tickets, labels and thresholds are synthetic examples of business rules. This page reports no model outputs, accuracy, speed or cost comparisons; measure on your own labeled data before relying on a result.

1. Split the work: retrieval, then reranking

Fast search (keyword search or BM25, embeddings, filters) compares the query with the whole corpus and returns a shortlist. Reranking compares the query with each shortlisted candidate and reorders only that shortlist. TypeSafe's re-ranking cookbook and Elastic's semantic reranking documentation both describe this two-stage structure. Keep permissions, freshness, deduplication and business filters in the retrieval stage; a reranker should never see results the user is not allowed to open.

Reranking cannot recover a document that retrieval never returned. If the right answer is often missing from the shortlist, fix retrieval first.

2. Write the rubric

These are the four levels from the lab recipe. The right-hand column shows synthetic candidates for the query "How can I use Jev from Python?"; they are not model judgments.

The instructions say what to judge: relevance to this query, not the general quality of the source. If you also need quality, freshness or authority, ask for them as separate questions and weight them in code (TypeSafe's composite scoring pattern).

3. One request per query and candidate

Open the relevance-score recipe in the request lab and it loads this request.

{
  "model": "jev-1.13.0",
  "state": {
    "query": "How can I use Jev from Python?",
    "result": "The official TypeSafe Python SDK supports synchronous and asynchronous System One requests."
  },
  "questions": {
    "decision": {
      "type": "score",
      "instructions": "How directly does this result address the query? Judge relevance, not the source's general quality.",
      "criteria": [
        "Unrelated to the query",
        "Related topic, but not an answer",
        "Useful context or a partial answer",
        "Directly addresses the query"
      ]
    }
  }
}

curl --fail-with-body --max-time 20 https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JEV_ATLAS_REQUEST'
{
  "model": "jev-1.13.0",
  "state": {
    "query": "How can I use Jev from Python?",
    "result": "The official TypeSafe Python SDK supports synchronous and asynchronous System One requests."
  },
  "questions": {
    "decision": {
      "type": "score",
      "instructions": "How directly does this result address the query? Judge relevance, not the source's general quality.",
      "criteria": [
        "Unrelated to the query",
        "Related topic, but not an answer",
        "Useful context or a partial answer",
        "Directly addresses the query"
      ]
    }
  }
}
JEV_ATLAS_REQUEST

Send one request per candidate, so no candidate's text can influence another candidate's score. That is the structure TypeSafe's re-ranking cookbook uses, although the cookbook asks a yes/no Noul for each pair instead of a Score. Both are reasonable designs; your labeled set can tell you which ordering serves your users better. Each request is billed on its input tokens. Estimate with the lab's cost calculator ($0.042 per million input tokens on TypeSafe's models page, checked 2026-09-25).

4. Sort in code, with explicit tie and no-answer rules

// Node 18+ or any runtime with fetch. NO_ANSWER_BELOW, TIE_BAND and CONCURRENCY are
// example values to calibrate on your labeled queries, not measured results.
const ENDPOINT = 'https://api.typesafe.ai/v1/systemone';
const LEVELS = [
  'Unrelated to the query',
  'Related topic, but not an answer',
  'Useful context or a partial answer',
  'Directly addresses the query',
];
const NO_ANSWER_BELOW = 2;
const TIE_BAND = 0.1;
const CONCURRENCY = 4;

async function scoreCandidate(query, text, apiKey) {
  const response = await fetch(ENDPOINT, {
    method: 'POST',
    headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'jev-1.13.0',
      state: { query, result: text },
      questions: {
        relevance: {
          type: 'score',
          instructions: "How directly does this result address the query? Judge relevance, not the source's general quality.",
          criteria: LEVELS,
        },
      },
    }),
    signal: AbortSignal.timeout(20_000),
  });
  if (!response.ok) throw new Error(`typesafe_http_${response.status}`);
  return (await response.json()).answers.relevance;
}

// candidates: [{ id, text }] in the order your retrieval step returned them.
export async function rerank(query, candidates, apiKey) {
  const scored = [];
  for (let start = 0; start < candidates.length; start += CONCURRENCY) {
    const batch = candidates.slice(start, start + CONCURRENCY);
    const answers = await Promise.all(batch.map(item => scoreCandidate(query, item.text, apiKey)));
    answers.forEach((answer, offset) => scored.push({
      ...batch[offset],
      retrievalRank: start + offset,
      score: answer.score,
      direct: answer.probabilities[String(LEVELS.length - 1)],
      confidence: answer.confidence,
    }));
  }
  const best = Math.max(-1, ...scored.map(item => item.score));
  // Nothing reaches "useful context": say so and keep the retrieval order.
  if (best < NO_ANSWER_BELOW) return { noDirectAnswer: true, results: candidates };
  const band = item => Math.round(item.score / TIE_BAND);
  scored.sort((a, b) => band(b) - band(a) || b.direct - a.direct || a.retrievalRank - b.retrievalRank);
  return { noDirectAnswer: false, results: scored };
}

A Score answer's score is the probability-weighted level and can land between levels. TypeSafe's Jev 1.13 limitations page says to use it for threshold checks rather than to reconstruct exact quantities. So the code treats small differences as ties (TIE_BAND) and breaks ties by the probability of the top level, then by the original retrieval rank.

When no candidate reaches "useful context", say so instead of promoting a weak match: keep the retrieval order, show a "no direct answer" notice and suggest refining the query. For candidates with low confidence (a spread-out distribution), keep the retrieval rank, or send high-stakes lists to review.

5. Build a small labeled test set

Before rollout, collect real queries (with permission), take the shortlist your retrieval returns, and label each candidate from 0 to 3 with the rubric. Use two labelers if you can, and resolve disagreements by editing the rubric. Freeze the labels before running Jev. The table below is a synthetic example for the query "How can I use Jev from Python?".

// labels: { [candidateId]: 0 | 1 | 2 | 3 }, written before any model run.
// Orders are arrays of candidate IDs; retrieval order is the baseline.
export function compareOrders(labels, retrievalOrder, rerankedOrder) {
  const summary = order => ({
    topIsDirect: labels[order[0]] === 3,
    unrelatedInTopThree: order.slice(0, 3).filter(id => labels[id] === 0).length,
    firstDirectPosition: order.findIndex(id => labels[id] === 3) + 1 || null,
  });
  return { retrieval: summary(retrievalOrder), reranked: summary(rerankedOrder) };
}

Useful measures on your own data include the share of queries whose top result is labeled 3, the number of level-0 results in the top three, and whether reranking moved labeled-3 candidates up or down compared with retrieval. Keep the retrieval-only order as your baseline. This page reports no results; the numbers only mean something for your corpus and your queries.

Projects that use this pattern

• Hev Reranker: Reorder search candidates with one Jev relevance judgment per document.
• Psearch: Search and follow web evidence with Jev-ranked pages and links.
• Sift: Reorder Google results using Jev relevance and promotional-content judgments.

These entries were source-checked from their READMEs on 2026-09-18. Jev Atlas has not run, benchmarked or security-audited them.

Sources

• TypeSafe: Score
• TypeSafe: Re-ranking cookbook
• TypeSafe: Classifying RAG passages
• TypeSafe: Composite scoring
• TypeSafe: Confidence
• TypeSafe: Jev 1.13 jaggedness
• TypeSafe: Models
• Elastic: Semantic reranking

TypeSafe pages checked 2026-09-25. Models, limits and prices can change; check the provider before integrating.

Try it

Open relevance-score in the lab · Write better decision questions