7.4 Vector Stores, Indexes & LLM Data Processing

7.4 Vector Stores, Indexes & LLM Data Processing

Vector workloads add a new data type and retrieval pattern to the data engineer's toolkit. The pipeline still needs ingestion, quality, security, lineage, monitoring, and lifecycle controls; a foundation model does not remove those responsibilities.

Embeddings and similarity

An embedding model converts text, images, or other supported content into a numeric vector with a fixed number of dimensions. Vectors that represent semantically related content tend to be close under the metric used by the model and index, such as cosine similarity, inner product, or Euclidean distance.

Model, dimension, and metric must agree with the index. Changing the embedding model can change both vector meaning and dimension, so store an embedding-version field and rebuild or isolate the index during migration. Never mix incompatible vectors in one field and expect meaningful distances.

Exact nearest-neighbor search compares a query with every vector and becomes expensive at scale. Approximate nearest-neighbor (ANN) indexes trade a small amount of recall for much lower latency.

HNSW and IVF

Hierarchical Navigable Small World (HNSW) builds a multilayer proximity graph. Queries enter at a sparse upper layer and navigate toward nearby vectors at denser layers. HNSW usually provides strong low-latency recall but consumes memory and takes time to build. Construction and search breadth parameters trade resources for recall.

Inverted File (IVF) clustering assigns vectors to coarse lists. A query first selects nearby cluster centroids and searches a configured number of lists. Searching more lists improves recall and increases work. IVF needs representative training and can perform poorly when cluster distribution shifts.

Do not claim one index is always better. Benchmark recall at the required latency with production-like vectors and filters.

AWS store selection

Amazon OpenSearch Service combines text search, filtering, aggregations, and vector search. It fits hybrid keyword-plus-semantic retrieval and log/search platforms.

Amazon Aurora PostgreSQL with pgvector fits applications that need vectors near relational rows and transactional metadata. Relational filtering, joins, and operational consistency can outweigh the scale advantages of a dedicated search engine.

Amazon MemoryDB can serve fast key/value and supported vector use cases when in-memory latency and Redis-compatible access are central. Other stores supported by Amazon Bedrock Knowledge Bases can be appropriate. Choose from vector count, update rate, metadata filters, latency, durability, regional availability, and operational ownership.

Bedrock Knowledge Bases ingestion

For unstructured sources, a typical Knowledge Bases flow is:

1. Connect a supported source such as S3.
2. Parse documents and split them into chunks.
3. Use an embedding model to vectorize chunks.
4. Write vectors plus source metadata to a vector store.
5. At query time, embed the question, retrieve similar chunks, and provide them as context to a model.

Chunking affects retrieval quality. Chunks that are too small lose context; chunks that are too large combine unrelated topics and consume the model's context window. Preserve document ID, version, page or section, access classification, and chunk offsets so a response can be traced back to authorized source material.

Structured sources can use managed natural-language-to-SQL paths rather than vectorizing every row. The data engineer must still restrict schemas, validate generated queries, cap cost, and prevent a request from reading unauthorized tables.

LLMs as data processors

Version 1.1 explicitly includes integrating large language models for data processing. Useful cases include classification, metadata enrichment, extraction from irregular text, and summarization. LLM output is probabilistic. Do not silently replace deterministic validation with a model response.

A production pattern writes model output to a candidate layer with:

• source object and checksum;
• model, prompt, parser, and guardrail versions;
• invocation time and token or cost metrics;
• confidence or validation result;
• human-review status where impact is high.

Validate required fields with deterministic rules, constrain output to a schema, quarantine parsing failures, and sample accepted outputs for drift. Remove or mask sensitive data before invocation when policy requires it, and ensure the selected service and Region meet residency rules.

Retrieval security and freshness

The nearest chunk is not necessarily authorized. Apply tenant and classification filters at retrieval time, and ensure the caller cannot manipulate filters to cross boundaries. When a source document is deleted or access is revoked, remove or tombstone its chunks and verify the vector store no longer returns them.

Track ingestion lag from source version to searchable vector version. A successful model response based on a stale policy document is a data freshness failure, even if the prose sounds convincing.

Vectorization Lifecycle

Vectorization is a versioned data transformation, not a one-time storage setting. Record the embedding model, model version, output dimension, normalization choice, source chunk, and preprocessing version with each vector. A new model can change both meaning and dimension, so mixing old and new embeddings in one index can make distance scores incomparable or invalid. Re-embed into a parallel index, validate retrieval quality and latency, then shift consumers with a rollback path.