2.2 Knowledge Source Integration, Synchronization & Freshness

2.2 Knowledge Source Integration, Synchronization & Freshness

Data integration and freshness controls

The official blueprint extends beyond a single S3 demonstration. A production design may connect document-management systems, internal wikis, databases, or custom pipelines. Use a canonical document identifier and version so an update replaces the intended record. Attach useful metadata such as tenant, jurisdiction, effective date, document type, and access group. Keep the fields small and consistent enough for the vector store's filter rules.

Define a freshness service-level objective. A policy assistant might require updates within minutes, while a historical archive may tolerate a nightly sync. Monitor ingestion-job duration, failed files, stale source versions, vector counts, and retrieval checks against newly added canary documents. If a connector cannot represent source permissions, do not claim permission-aware retrieval; enforce the missing control in the application or choose a different isolation design.

Evaluation and failure isolation

RAG quality has at least two independent stages. Retrieval evaluation asks whether the right evidence was found and ranked. Generation evaluation asks whether the answer is correct, complete, faithful, safe, and properly cited. A low-quality answer with missing evidence calls for chunking, embedding, filter, query, or reranking changes. A low-faithfulness answer with good evidence calls for prompt, model, Guardrails, or response-validation changes.

Operational failures also differ. A failed ingestion job is not fixed by increasing generation tokens. A throttled model call is not fixed by rebuilding the vector index. Instrument the stages separately and include source version, query transformation, retrieved document IDs, model target, latency, and outcome metrics without logging prohibited sensitive content.

For exam scenarios, choose Retrieve when custom processing must occur between retrieval and generation. Choose RetrieveAndGenerate when managed orchestration and citations satisfy the requirements. In both cases, treat synchronization, authorization, and evaluation as explicit responsibilities.

Synchronization design patterns

Full synchronization is simple but expensive for large sources. Incremental synchronization processes changed records, but only if the connector and source metadata reliably identify changes and deletions. Event-triggered scheduling can reduce delay, yet bursts should be coalesced because a separate ingestion job for every uploaded file can exceed quotas or serialize inefficiently. A scheduled reconciliation remains useful to repair missed events.

Use a source manifest containing canonical ID, version, checksum, effective date, deletion state, and expected metadata. After ingestion, compare manifest counts with indexed counts and query canary records from each important source class. A job status of complete does not prove that a malformed or unsupported document produced useful chunks.

Deletion semantics deserve a test. Remove a canary document, synchronize, and confirm that neither its text nor prior metadata remains retrievable. For regulated retention, deletion from the search index may differ from deletion of the audit source; document both lifecycles.

Connector credentials belong in Secrets Manager or the service's supported secret integration, scoped to the required source. Rotate them, monitor failed authentication, and avoid copying a broad source-system token into Lambda environment variables. If source permissions cannot be mapped safely, isolate corpora or enforce an application retrieval service that owns authorization.

Conflict, retry, and recovery

Prevent overlapping ingestion jobs from racing on the same source when the service or connector does not support that pattern. Use a work queue or lock keyed by knowledge base and data source, coalesce bursts, and retry only transient job failures. Record the source watermark that each completed job represents.

If an embedding model or chunking policy changes, incremental sync is insufficient because existing records were produced under a different transformation. Build a parallel index or perform a governed full reingestion, evaluate both, and cut over with rollback. The same rule applies when a metadata schema changes in a way that affects filters.

Freshness also includes effective dates. A newly uploaded policy may not yet be effective, and an older policy may remain valid for historical questions. Store effective-from and effective-to metadata and make the application choose the appropriate time context rather than always preferring the newest file.

Service-level reporting

Report freshness by source and document class rather than one global average. A fast S3 feed can hide a failing enterprise connector. Publish the oldest unprocessed change, failed-record count, last successful watermark, and canary result, then route each breach to the source owner who can correct it.

Define who owns a source-side schema change. A connector that still authenticates can ingest empty or misclassified content after an upstream change, so canary retrieval and record-quality checks must accompany availability monitoring.

Freshness control matrix