1.9 Scoring a Model with Features from a Feature Table
Key Takeaways
- `fe.score_batch(model_uri, df)` scores a DataFrame that contains only lookup keys — the packaged `FeatureLookup` graph retrieves the feature values automatically.
- The scoring DataFrame must supply every lookup key, plus the timestamp column when the model was trained with a point-in-time lookup.
- `score_batch` returns the input columns, the retrieved features, and a `prediction` column, and it accepts alias URIs such as `models:/catalog.schema.model@champion`.
- Automatic retrieval only works for models logged with `fe.log_model`; a model logged with `mlflow.sklearn.log_model` has no lookup metadata and must be given fully assembled features.
- For real-time serving, the same packaged lookups resolve against a published online store, which is why the offline table must be published before the endpoint can use it.
1.9 Scoring a Model with Features from a Feature Table
The payoff of logging a model with fe.log_model() arrives at scoring time. Because
the artifact stores the FeatureLookup specifications used during training, the
inference caller no longer has to know which feature tables exist, which columns to
select, or how to join them. It supplies keys; Databricks supplies features.
Scoring request Packaged model Feature tables
+---------------------+ +----------------------+ +----------------------+
| customer_id | | FeatureLookup graph | | customer_demographics|
| transaction_ts | ---> | + trained estimator | <--- | customer_temporal |
+---------------------+ +----------------------+ +----------------------+
|
v
customer_id, transaction_ts, <features...>, prediction
Batch Scoring with fe.score_batch
from databricks.feature_engineering import FeatureEngineeringClient
fe = FeatureEngineeringClient()
# The scoring DataFrame carries ONLY keys (and the timestamp, if the model
# was trained with a point-in-time lookup).
scoring_df = spark.table("prod_catalog.gold.customers_to_score").select(
"customer_id", "transaction_timestamp"
)
predictions_df = fe.score_batch(
model_uri="models:/prod_catalog.ml_models.fraud_detection_gbm@champion",
df=scoring_df,
result_type="double",
)
display(predictions_df.select("customer_id", "prediction"))
What the output contains
score_batch returns a Spark DataFrame with three groups of columns:
- every column of the input DataFrame,
- the feature columns retrieved from each looked-up table, and
- a
predictioncolumn holding the model output.
Because the retrieved features come back in the result, the output doubles as an audit record of exactly which feature values produced each prediction.
Parameters worth knowing
| Parameter | Purpose |
|---|---|
model_uri | models:/catalog.schema.model@alias, models:/catalog.schema.model/3, or a runs:/ URI |
df | Spark DataFrame containing the lookup keys (and timestamp key when required) |
result_type | Spark type of the prediction column — "double" for probabilities or regression, "string" for class labels |
env_manager | "local" by default; "virtualenv" or "conda" rebuilds the logged environment when dependency fidelity matters |
params | Optional dictionary of inference parameters forwarded to the model's predict call |
use_spark_native_join | False by default; when True, the feature joins execute as native Spark joins |
What the Scoring DataFrame Must Contain
This is where exam scenarios put the trap.
- Every lookup key. A composite key of
["user_id", "device_id"]means both columns must be present. A missing key column fails the lookup outright. - The timestamp column, when the model used one. If training used
timestamp_lookup_key="transaction_timestamp", scoring must supply that column so the as-of join can pick the correct historical feature row. Supplying the current time instead of the event time silently changes which feature values are retrieved. - Every source key the model scores on. Beyond the join keys,
dfmust carry any column the model consumes that is not itself looked up. Both requirements are recorded in thefeature_spec.yamlartifact packaged with the model. - No column named
prediction. That name is reserved for the model output, and the documented contract forbids it in the input. Any other extra column is allowed and is passed straight through to the result.
Supplied Columns Override the Feature Table
You normally pass keys only — but if you do include a feature column, it wins. The
official score_batch contract states that packaged features are retrieved "unless
present in df", and that "if a feature is included in df, the provided feature
values will be used rather than those stored in feature tables."
That is the reverse of what most candidates assume, and it cuts both ways:
- Deliberately, it is how what-if analysis is run. Score the champion model with a
promo_discount_pctcolumn you set by hand and the model evaluates the hypothetical instead of the stored value — no retraining and no shadow feature table. - Accidentally, it is a silent correctness bug. A scoring DataFrame built with
SELECT *off an upstream table may happen to carry a stale column whose name collides with a governed feature. The lookup is skipped for that column, the model scores on the stale copy, and nothing in the output flags it.
The defensive habit is to project the scoring DataFrame down to exactly the keys before
calling score_batch, which is why the example above uses an explicit .select()
rather than passing the source table straight through.
Real-Time Serving with Packaged Lookups
A model logged with fe.log_model can be deployed to a Databricks Model Serving
endpoint and retain automatic lookup. The request body then carries only the keys:
{ "dataframe_records": [{ "customer_id": "cust_9921" }] }
The endpoint resolves the features from the published online store rather than the offline Delta table, because a REST call cannot wait for a Spark job. That is the practical reason the offline table must be published before the endpoint is created — covered in Section 1.10.
Failure Modes
| Symptom | Cause | Fix |
|---|---|---|
score_batch reports missing input columns the caller never trained on | Model was logged with mlflow.<flavor>.log_model, so no lookup metadata exists | Retrain and log with fe.log_model(training_set=...) |
| Predictions differ between the training notebook and batch scoring | Features were joined manually during training | Rebuild the training set with fe.create_training_set so the same join is packaged |
| Endpoint returns feature-lookup errors while batch scoring works | Feature table has not been published to an online store | Publish the table, then update the endpoint |
| Historical backfill scores look impossibly good | Scoring passed the current timestamp instead of each row's event timestamp | Pass the true event timestamp column |
A fraud model was logged with fe.log_model(..., training_set=training_set) and registered in Unity Catalog. Which DataFrame should be passed to fe.score_batch for daily batch scoring?
An engineer trains a model on a Unity Catalog training set but logs it with mlflow.sklearn.log_model(). What happens when a colleague later calls fe.score_batch() with a keys-only DataFrame?
A model was trained with FeatureLookup(..., timestamp_lookup_key='event_time'). During a historical backfill, the scoring job passes current_timestamp() as event_time for every row. What is the consequence?
A scoring DataFrame is built with SELECT * from an upstream table and happens to include a column named credit_utilisation, which is also a governed feature the model looks up from a feature table. What does fe.score_batch do?