11.4 Unstructured Data, Embeddings, and Retrieval-Augmented Generation

11.4 Unstructured Data, Embeddings, and Retrieval-Augmented Generation

Sub-section 4.2 of the v4.2 exam guide has exactly two bullets. The first is feature engineering and model training with BigQuery ML, covered earlier in this chapter. The second is new and catches most candidates unprepared: "Preparing unstructured data for embeddings and retrieval-augmented generation (RAG)."

Read the verb. The blueprint does not ask you to build a chatbot; it asks you to prepare the data that a retrieval system depends on. That is a data engineering job with a familiar shape — parse, chunk, transform, index, keep fresh, govern — and the exam tests it as such.

---

The Five Stages of a RAG Data Pipeline

Cloud Storage objects
      |
  (1) Parse      ->  object table + ML.PROCESS_DOCUMENT / ML.TRANSCRIBE / ML.ANNOTATE_IMAGE
      |
  (2) Chunk      ->  split into retrievable passages + carry source metadata
      |
  (3) Embed      ->  ML.GENERATE_EMBEDDING  ->  ARRAY<FLOAT64> column
      |
  (4) Index      ->  CREATE VECTOR INDEX (IVF or TreeAH)
      |
  (5) Retrieve   ->  VECTOR_SEARCH / AI.SEARCH  ->  ground ML.GENERATE_TEXT
                     with the retrieved passages

Stage 1: Parse

Unstructured source files live in Cloud Storage. BigQuery reaches them through an object table — a table of references to the underlying objects — and the extraction functions read from there:

• ML.PROCESS_DOCUMENT extracts structured fields and text from PDFs and scanned documents through Document AI.
• ML.TRANSCRIBE converts audio to text through Speech-to-Text.
• ML.ANNOTATE_IMAGE returns Cloud Vision labels, objects, and detections.

Google documents this whole flow as achievable inside BigQuery: parse documents, run vector search over the content, and generate summarized answers to natural-language questions with Gemini models.

Stage 2: Chunk

A document is almost never the right retrieval unit. A 90-page policy manual embedded as one vector produces a single, blurry point in embedding space that matches everything and answers nothing. Chunking splits the text into passages small enough to be semantically specific and small enough to fit in the generation model's context alongside several siblings.

Three engineering decisions matter:

That last row is the one with real exam consequences. Metadata columns stored alongside the embedding are what you filter on in stage 5 — without them you cannot restrict a search to one tenant, one language, or documents newer than a given date.

Stage 3: Embed

ML.GENERATE_EMBEDDING produces the vector column. It operates on structured text in ordinary columns and, through ObjectRef values, on text, image, and video inputs. The output is an array of floats stored alongside the chunk.

The data-engineering obligations here are unglamorous and heavily testable:

• Embeddings and the model that produced them are coupled. Different embedding models produce vectors of different dimensionality and different geometry. Switching models means re-embedding the entire corpus, not embedding new rows with the new model and leaving old rows alone — a mixed-model table returns meaningless distances.
• Freshness is a pipeline problem. When a source document is revised, its chunks must be re-chunked and re-embedded. An incremental pipeline keyed on object generation or update time is the standard pattern.
• De-duplicate before embedding. Embedding the same boilerplate footer 40,000 times wastes both the embedding call and the retrieval slots it will later occupy.

Stage 4: Index

Without an index, VECTOR_SEARCH performs a brute-force scan and returns exact results. With a vector index it uses approximate nearest neighbor search, which is far faster at the cost of some recall.

BigQuery offers two index types:

Supported distance types are EUCLIDEAN (the default), COSINE, and DOT_PRODUCT. Index training always uses Euclidean distance internally, but the distance used at search time can differ, and a distance_type passed to VECTOR_SEARCH overrides the index's default.

Three index behaviors that make excellent exam questions:

1. A table smaller than 10 MB will not have its vector index populated. If an indexed table shrinks below 10 MB, the index is temporarily disabled and searches report BASE_TABLE_TOO_SMALL. The search still works — it falls back to brute force.
2. Newly added rows are never missed. There is a delay between inserting rows and their appearance in the index, but VECTOR_SEARCH and AI.SEARCH search the index for indexed records and brute-force the not-yet-indexed ones.
3. Stored columns are ignored when security is applied. The STORING clause caches metadata columns in the index for efficient pre-filtering, but stored columns are not used if the table has a row-level access policy or if the column carries a policy tag.

Stage 5: Retrieve and Ground

VECTOR_SEARCH returns the nearest chunks; AI.SEARCH provides semantic or hybrid search against a string, which is convenient when autonomous embedding generation is enabled on the table. The retrieved passages are then passed as context into ML.GENERATE_TEXT or AI.GENERATE so the model answers from your corpus rather than from its training data.

Pre-filter versus post-filter is the retrieval-quality decision: a pre-filter narrows the candidate set before the nearest-neighbor search, so you still get top_k results from within the filtered population; a post-filter applies the condition to results the search already returned, which can leave you with far fewer than top_k rows — or none. When a scenario complains that a tenant-scoped search returns too few results, the answer is to pre-filter.

One cost note worth memorizing: VECTOR_SEARCH and AI.SEARCH queries are not accelerated by BI Engine, so an in-memory reservation will not rescue a slow semantic search dashboard.

---

BigQuery Vector Search vs. Vertex AI Vector Search

The framing that resolves these: BigQuery vector search is analytics-grade retrieval colocated with your data; Vertex AI Vector Search is serving-grade retrieval for an application in the request path.

---

Governance for Retrieval Corpora

• De-identify before you embed. An embedding derived from text containing regulated identifiers is a derivative of that data and travels with the same obligations. Run Cloud DLP de-identification during chunking, not after indexing.
• Isolate tenants with a pre-filter, not with trust. A multi-tenant corpus in one table must carry a tenant column and pre-filter on it; relying on the generation prompt to "only use tenant A's documents" is not a control.
• Keep policy tags on the source columns. Remember that policy-tagged columns are excluded from index stored columns, so plan pre-filtering around unclassified metadata such as tenant ID and document date.
• Align regions. The dataset, the Cloud resource connection, and the model endpoint must be regionally compatible; an EU corpus embedded through a US endpoint is a residency finding.

---

Exam Traps and Antipatterns Summary