Appointment Retrieval Under Failure: A 4-Layer Recovery Design for Healthcare Assistants

Short answer

Short answer: treat retrieval failure as a typed, auditable state, then fall back through four layers—fresh local index, last-known-good snapshot, constrained schedule service, and a human handoff—before the assistant can suggest an appointment. In healthcare, a plausible answer with stale availability is worse than a clear refusal.

I frame this around an appointment assistant because the failure is easy to miss. A patient asks for an opening next Tuesday; the retriever returns an old clinic policy, the generator fills in a confident time, and nobody notices until the patient arrives. The architecture has to make freshness and recovery visible to the caller, not leave them as prompt instructions.

Start with the constraint: freshness beats recall

An appointment knowledge base mixes data with very different clocks. A provider directory may change weekly. A cancellation policy can change today. Slot availability can change between two HTTP requests. Putting all of it into one vector index and assigning one time-to-live creates a false sense of consistency.

Separate the corpus into freshness classes. Store an effective_at timestamp, an expires_at policy, and a source revision for every chunk. Retrieval should filter on those fields before ranking semantic similarity. A high similarity score cannot rescue an expired chunk.

The practical unit is a citation-ready chunk, not an arbitrary token window. Keep the clinic, department, timezone, appointment type, and source revision attached to the text. When a chunk is replaced, mark the old revision inactive; do not silently blend both versions into one embedding set. This makes an answer reproducible during a compliance review.

A small amount of denormalized metadata pays for itself here. It lets the system answer “I found no current slot” instead of “there are no slots,” which are different claims.

How should a healthcare appointment assistant recover retrieval failures?

Use an explicit state machine. A request starts in FRESH_SEARCH, moves to SNAPSHOT_SEARCH when the index is unavailable or freshness checks fail, and then to STRUCTURED_LOOKUP for facts that belong in a transactional schedule system. HUMAN_REVIEW is a valid terminal state, not an exception path.

The generator must receive the state and evidence age as input. If the best evidence is a snapshot, the response should say that availability needs confirmation and avoid inventing a time. If no evidence survives policy checks, return a concise handoff message with a correlation ID. Do not ask the model to “try harder”; that turns an infrastructure error into a content error.

Here is a compact Python sketch. The interfaces are intentionally generic so the recovery contract can sit behind any search engine or database.

from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass
class Evidence:
    text: str
    source_revision: str
    observed_at: datetime
    freshness: str  # fresh, snapshot, structured

def recover(query, now, index, snapshot, schedule):
    fresh = index.search(query, at=now, require_current=True)
    if fresh:
        return {"state": "FRESH_SEARCH", "evidence": fresh}

    saved = snapshot.search(query, at=now)
    if saved and all(item.observed_at <= now for item in saved):
        return {"state": "SNAPSHOT_SEARCH", "evidence": saved}

    structured = schedule.lookup(query)
    if structured:
        return {"state": "STRUCTURED_LOOKUP", "evidence": structured}

    return {"state": "HUMAN_REVIEW", "evidence": [], "reason": "no_current_evidence"}

Each transition should emit a metric and a structured log event. Include the request class, source revision, evidence age, and final state, but keep patient identifiers out of general logs. A separate access-controlled trace can hold the minimum data needed for investigation.

Failure modes and rollout checks

An empty result is not the same as a timeout. An empty result after a successful, current index query can mean “no matching appointment.” A timeout says nothing about availability. Give them different error codes and user-facing language.

This distinction matters in a concrete sequence: a patient asks for a Tuesday visit, the index times out, a retry starts, and the schedule service responds just after the conversational deadline. If the controller collapses those events into “no slots,” the assistant emits a false negative; if it treats the timeout as permission to use an old chunk, it emits a false positive. Recording the state at each boundary lets the handoff explain exactly what still needs confirmation.

Keep this visible.

Stop.

Index lag is another category. During re-indexing, dual-read the active and candidate indexes and compare revision coverage. Promote the candidate only after it contains every required source revision. If the candidate is incomplete, the last-known-good snapshot remains the recovery source.

Embedding or reranking drift can lower recall without producing an obvious error. Keep a fixed evaluation set of appointment intents: reschedule, new patient, language preference, location, and provider type. Run it against every model or chunking change. Track recall at a citation threshold, not just answer fluency. The original retrieval-augmented generation paper describes the basic pattern of combining a parametric generator with non-parametric memory; production recovery adds the operational states around that pattern.

In practice, this test set should include awkward combinations that expose routing mistakes: a new patient seeking a pediatrician who speaks Spanish at a clinic across town, a reschedule request that quotes an obsolete department name, and a question that mixes insurance with a time window. Keep the expected evidence IDs and allowed response class beside each case. When a chunking change moves a sentence across a boundary, the test should tell you whether the citation disappeared, whether a stale revision slipped through, or whether the controller correctly escalated. That detail is more useful than a single aggregate recall number because it points to the layer that needs repair.

Policy changes need a stronger guard. If a source revision carries a legal or clinical effective date in the future, exclude it until that date. If two current revisions conflict, surface the conflict for review. Never resolve a policy contradiction by selecting the chunk with the higher similarity score.

Compare recovery contracts, not vendor feature lists

Teams often compare hosted search, a relational database with full-text indexes, and a self-managed vector store as if one choice determines reliability. It does not. The deciding question is which system can expose revision, freshness, timeout, and deletion semantics to your recovery controller.

A hosted index may reduce operational work but still require your own snapshot export and revision ledger. A relational store makes transactional schedule reads straightforward, yet semantic recall may need a separate index. A self-managed vector service gives tuning control and places backup, capacity, and restore testing on your team. These are engineering trade-offs, not rankings.

The catch is that this layered design is not suitable when the assistant is expected to book appointments without a source of truth that supports atomic holds. In that case, keep the conversational layer read-only and use the scheduling system’s booking workflow, with confirmation, for the final action. Stick with a simpler keyword or structured query path when the corpus is small and fields are predictable; adding embeddings then adds failure surface without improving the decision.

Roll out recovery in small, observable steps

Start by replaying sanitized historical intents through a failure-injection harness. Inject index timeouts, stale revisions, partial snapshot restores, and schedule-service latency. Assert the state, evidence age, and wording class for each case. A test that checks only HTTP 200 misses the dangerous failure: a fluent answer with the wrong date.

Next, ship shadow retrieval. The new controller records what it would have returned while the existing assistant serves traffic. Compare citation coverage, fallback rates, and handoff volume by clinic and appointment type. Your mileage may vary across departments; measure before setting one global freshness window.

For production, set budgets for each layer and a total response deadline. A five-second retry loop can turn a transient search outage into a poor patient experience. Cap retries, add jitter, and fail closed on authorization or deletion signals. Preserve a correlation ID across the retriever, generator, and handoff queue so an operator can reconstruct the decision.

Review the metrics weekly: percentage of answers from fresh evidence, snapshot age distribution, structured-lookup success, human-review rate, and conflicting-revision count. I’m not sure any single threshold works for every clinic, but a rising snapshot age or conflict count is an actionable signal long before a complaint arrives.

The design goal is modest: every answer is either grounded in current evidence, clearly labeled as needing confirmation, or handed to a person. That contract survives index swaps, model updates, and changing chunking strategies.

References

• https://arxiv.org/abs/2005.11401
• https://www.hhs.gov/hipaa/for-professionals/security/laws-regulations/index.html
• https://www.rfc-editor.org/rfc/rfc9110