Hybrid & neural recommenders

Why one model is never enough

Every recommendation technique has a structural weakness:

• Collaborative filtering (CF) knows nothing about item content. It cannot recommend a brand-new item until enough users have rated it (the item cold-start problem), and it is blind to signals like genre, cast, or description.
• Content-based filtering treats every user as an island. It can only recommend items similar to ones a user already liked, and it ignores the rich signal that millions of other users provide.
• Matrix factorization is a powerful CF variant, but its latent factors still require sufficient observed ratings to stabilize and cannot incorporate side information (user demographics, item metadata, context) without modification.

The natural response is to hybridize — combine methods so each covers the other’s blind spots — and to go neural, replacing hand-crafted similarity with learned representations that can absorb all available signals.

Hybrid recommenders

A hybrid recommender combines two or more recommendation models or signal types. The four main hybridization strategies are:

1. Weighted blend

The simplest approach: compute scores from multiple models and take a weighted average.

score(u, i) = α · score_CF(u, i)  +  (1 − α) · score_CB(u, i)

The weight α can be a fixed hyperparameter tuned on a held-out set, or it can itself be a learned function of context (time of day, session length, device). This is effective when both component models are well-calibrated.

2. Switching by context

Use CF when you have enough interaction data, fall back to content-based when you do not. A user who just signed up triggers content-based recommendations; a returning user with 50+ interactions switches to CF. The switching rule can be a simple threshold or a classifier trained on session features.

3. Feature combination

Inject content features directly into the CF model. In matrix factorization terms this means augmenting the user or item embedding with hand-crafted features before training. The SVD++ variant from the Netflix Prize is an early example: it adds a term for the set of items a user has rated, regardless of the rating value, as an implicit signal vector.

4. Stacking / cascade

Use one model to generate a shortlist of candidates, then a second (more expensive) model to re-rank them. This is not just a hybrid strategy — it is the dominant production pattern in large-scale systems, described in detail below.

Neural recommenders

Starting around 2016, the field shifted from hand-designed similarity functions toward end-to-end learned models. The core shift: replace explicit dot products between hand-crafted feature vectors with embeddings — dense, low-dimensional representations of users and items learned from raw interaction data (and optionally from content).

Embeddings

An embedding is a vector of real numbers, typically 32 to 512 dimensions, that is learned jointly with the rest of a model. Two users with similar taste should end up close in embedding space. Unlike latent factors from plain matrix factorization, neural embeddings can be conditioned on context, updated incrementally, and composed with other learned modules.

Neural Collaborative Filtering (NCF)

In classic matrix factorization the predicted score is a dot product:

score(u, i) = embedding_user(u) · embedding_item(i)

Neural Collaborative Filtering (He et al., 2017) replaces that dot product with a small multi-layer perceptron (MLP):

score(u, i) = MLP( [embedding_user(u) ; embedding_item(i)] )

The square bracket denotes concatenation. The MLP can learn interaction patterns that a simple dot product cannot — crossed feature interactions, non-linear taste patterns, and so on. In practice the NeuMF variant combines both: a dot-product path (generalised matrix factorisation) and an MLP path, concatenating their outputs before a final sigmoid.

The two-tower architecture

The most influential neural architecture for large-scale retrieval is the two-tower model (also called a dual-encoder). The name describes its structure exactly: a user tower and an item tower that independently encode their respective inputs into a shared embedding space. The predicted relevance score is the dot product of the two output embeddings.

The key engineering insight: because the two towers are independent of each other (no cross-tower interaction during encoding), you can pre-compute embeddings for every item offline and store them in a vector index. At query time you only need to run the user tower (fast, one forward pass), then find the closest item embeddings using approximate nearest neighbor (ANN) search (e.g., HNSW or FAISS). This enables sub-millisecond retrieval over hundreds of millions of items — something that is impossible with models that require user-item pairs as joint input.

The two-tower model is trained with a contrastive loss: push the user embedding close to items the user engaged with and away from randomly sampled negative items.

Sequence and session models

Users’ tastes shift within a session. If someone has been watching cooking videos for 20 minutes, the next recommendation should reflect that context, not just their six-month history. Sequential recommenders model the ordered list of recent interactions as a sequence.

The dominant architecture is a transformer applied to the sequence of recent item embeddings. The attention mechanism lets the model focus on the most contextually relevant past items when predicting the next one. BERT4Rec and SASRec are widely-cited examples; production variants at Amazon and Pinterest use similar ideas.

The industrial two-stage funnel

Large-scale systems universally decompose recommendation into two stages:

Stage 1 — Candidate generation (retrieval). A cheap model selects a few hundred to a few thousand items from a corpus of millions. Speed is everything here; quality is secondary. The two-tower model with ANN search is the canonical approach. Matrix factorization with pre-computed item vectors also works and is simpler to maintain.

Stage 2 — Ranking. A much more expensive model (a deep neural network with hundreds of features, including cross-user-item interaction features, freshness signals, diversity penalties, and business rules) scores only the retrieved candidates and produces the final ranked list.

This funnel appears in Google’s YouTube recommendations (described in the 2016 paper by Covington, Adams, and Sargin), Pinterest’s Pinnability system, Spotify’s session personalization, and virtually every other large-scale recommendation deployment. The funnel is often extended to three stages (retrieval → pre-ranking → ranking → re-ranking/business rules) as systems grow.

Embedding dot-product score in the browser

The playground below shows how a user embedding and an item embedding produce a relevance score via dot product — the same operation that powers two-tower retrieval.

A note on LLM-era recommenders

Large language models have introduced two new patterns:

1. LLMs as feature encoders. Pass item descriptions through a pre-trained language model to get rich text embeddings, then use those as the item tower’s input instead of a learned ID embedding. This dramatically helps cold-start for new items that have metadata but no interaction history.

2. LLMs as rankers or explainers. A small language model can re-rank candidates by reasoning over user history described in natural language, or generate an explanation of why an item is recommended. This is experimental and expensive at scale, but it points toward more transparent, contextually aware systems.

Neither pattern obsoletes the two-stage funnel — the retrieval bottleneck is a compute constraint, not a modeling constraint.