Agent Personalized Benchmark

Introduction

LLM-powered agents struggle with personalization, which requires resolving vanilla intents, inferring user-specific preferences, and finding useroptimal solutions from competing alternatives. Existing benchmarks largely fail to assess these challenges under vague goals and noisy interaction histories. We propose personalized product search (PPS) as a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from holistic action logs, featuring rich histories, vanilla intents, and hard candidate sets. We benchmark eight state-of-the-art LLMs with agentic workflows. LLMs excel on user-refined explicit queries, but struggle on initial vague queries that require latent intent and preference discovery. Fine-grained analysis attribute this gap to ineffective use of historical interactions for intent and preference inference. We further find that a simple history-conditioned refinement of vague queries yields consistent gains, emphasizing the role of effective history utilization in personalization.

Leaderboard

Select architecture in the first row, then switch Intent Query or Refined Query in the second row to compare HR@1 and HR@5.

#	Model	Model Param	Hist. Len.	Date	HR@1					HR@5
#	Model	Model Param	Hist. Len.	Date	Personal Care	Clothing	Food & Beverages	Other	Avg.	Personal Care	Clothing	Food & Beverages	Other	Avg.
1	UniSAR	0.1B	240	2024-04-15	--	--	--	--	25.9	--	--	--	--	59.2
2	GPT-5.5	--	60	2026-04-23	--	--	--	--	25.4	--	--	--	--	78.1
3	GPT-5.5-Think	--	60	2026-04-23	--	--	--	--	25.0	--	--	--	--	78.2
4	GPT-4o	--	60	2024-11-20	--	--	--	--	24.9	--	--	--	--	75.7
5	DeepSeek-R1	671B	60	2025-05-28	--	--	--	--	24.2	--	--	--	--	74.7
6	Gemini-3-Pro	-	60	2025-11-18	--	--	--	--	24.1	--	--	--	--	76.7
7	GPT-4.1	--	60	2025-04-14	--	--	--	--	23.9	--	--	--	--	73.3
8	Qwen3.5	397B	60	2026-02-16	--	--	--	--	22.9	--	--	--	--	72.4

#	Model	Model Param	Hist. Len.	Date	HR@1					HR@5
#	Model	Model Param	Hist. Len.	Date	Personal Care	Clothing	Food & Beverages	Other	Avg.	Personal Care	Clothing	Food & Beverages	Other	Avg.
1	GPT-5.5-Think	-	60	2026-04-23	--	--	--	--	37.3	--	--	--	--	84.4
2	GPT-5.5	-	60	2026-04-23	--	--	--	--	37.2	--	--	--	--	84.2
3	GPT-4o	-	60	2024-11-20	--	--	--	--	37.0	--	--	--	--	82.8
4	DeepSeek-R1	671B	60	2025-05-28	--	--	--	--	36.2	--	--	--	--	80.7
5	Qwen3.5	397B	60	2026-02-16	--	--	--	--	36.1	--	--	--	--	81.9
6	Gemini-3-Pro	-	60	2025-11-18	--	--	--	--	35.1	--	--	--	--	83.2
7	GPT-4.1	-	60	2025-04-14	--	--	--	--	34.5	--	--	--	--	78.8
8	UniSAR	0.1B	60	2024-04-15	--	--	--	--	29.2	--	--	--	--	64.5

#	Architecture	Backbone	Hist. Len.	Date	HR@1					HR@5
#	Architecture	Backbone	Hist. Len.	Date	Personal Care	Clothing	Food & Beverages	Other	Avg.	Personal Care	Clothing	Food & Beverages	Other	Avg.
1	VQRA-GPT5.2-Think	GPT-5.2-Think	60	2025-12-11	--	--	--	--	26.9	--	--	--	--	78.2
2	ReAct-GPT5.5-Think	GPT-5.5-Think	60	2026-04-23	--	--	--	--	26.5	--	--	--	--	78.1
3	ReAct-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	26.3	--	--	--	--	75.8
4	VQRA-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	26.3	--	--	--	--	75.7
5	VQRA-DeepSeek-R1	DeepSeek-R1	60	2025-05-28	--	--	--	--	26.2	--	--	--	--	76.3
6	Mem0-Gemini-3	Gemini-3-Pro	60	2025-11-18	--	--	--	--	26.1	--	--	--	--	78.5
7	Mem0-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	25.9	--	--	--	--	76.5
8	UniSAR	--	240	2024-04-15	--	--	--	--	25.9	--	--	--	--	59.2
9	ReAct-GPT4.1	GPT-4.1	60	2025-04-14	--	--	--	--	25.8	--	--	--	--	77.9
10	Mem0-GPT4.1	GPT-4.1	60	2025-04-14	--	--	--	--	25.8	--	--	--	--	77.1
11	Mem0-Qwen3.5	Qwen3.5-397B	60	2026-02-16	--	--	--	--	25.1	--	--	--	--	74.9
12	MemAgent-GPT4.1	GPT-4.1	60	2025-4-14	--	--	--	--	25.0	--	--	--	--	76.1
13	VQRA-GPT4.1	GPT-4.1	60	2025-4-14	--	--	--	--	24.9	--	--	--	--	75.7
14	ReAcT-Gemini-3	Gemini-3-Pro	60	2025-11-18	--	--	--	--	24.8	--	--	--	--	76.9
15	ReAcT-DeepSeek-R1	DeepSeek-R1	60	2025-05-28	--	--	--	--	24.8	--	--	--	--	73.9
16	ReAct-Qwen3.5	Qwen3.5-397B	60	2026-02-16	--	--	--	--	24.4	--	--	--	--	71.3
17	MemAgent-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	24.2	--	--	--	--	75.4
18	Mem0-GPT5.5	GPT-5.5	60	2026-04-23	--	--	--	--	24.1	--	--	--	--	76.9
19	MemAgent-GPT5.5	GPT-5.5	60	2026-04-23	--	--	--	--	24.0	--	--	--	--	77.1
20	MemAgent-Gemini-3	Gemini-3-Pro	60	2025-11-18	--	--	--	--	23.8	--	--	--	--	76.5
21	DeerFlow-GPT5.2-Think	GPT-5.2-Think	60	2025-12-11	--	--	--	--	23.5	--	--	--	--	73.8
22	DeerFlow-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	23.2	--	--	--	--	71.5
23	DeerFlow-DeepSeek-R1	DeepSeek-R1	60	2025-05-28	--	--	--	--	22.9	--	--	--	--	70.9
24	DeerFlow-GPT4.1	GPT-4.1	60	2025-04-14	--	--	--	--	22.2	--	--	--	--	72.3
25	MemAgent-Qwen3.5	Qwen3.5-397B	60	2026-02-16	--	--	--	--	22.5	--	--	--	--	70.5

#	Architecture	Backbone	Hist. Len.	Date	HR@1					HR@5
#	Architecture	Backbone	Hist. Len.	Date	Personal Care	Clothing	Food & Beverages	Other	Avg.	Personal Care	Clothing	Food & Beverages	Other	Avg.
1	ReAct-GPT5.5	GPT-5.5	60	2026-04-23	--	--	--	--	43.2	--	--	--	--	84.8
2	ReAct-GPT5.5-Think	GPT-5.5	60	2026-04-23	--	--	--	--	43.0	--	--	--	--	84.5
3	ReAct-GPT4.1	GPT-4.1	60	2025-04-14	--	--	--	--	40.4	--	--	--	--	83.2
4	MemAgent-GPT4.1	GPT-4.1	60	2025-04-14	--	--	--	--	38.6	--	--	--	--	82.8
5	Mem0-GPT4.1	GPT-4.1	60	2025-04-14	--	--	--	--	38.2	--	--	--	--	82.9
6	Mem0-GPT5.5	GPT-5.5	60	2026-04-23	--	--	--	--	38.1	--	--	--	--	82.1
7	MemAgent-GPT5.5	GPT-5.5	60	2026-04-23	--	--	--	--	37.9	--	--	--	--	82.5
8	ReAct-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	37.8	--	--	--	--	84.2
9	Mem0-Qwen3.5	Qwen3.5-397B	60	2026-02-16	--	--	--	--	37.9	--	--	--	--	82.1
10	ReAct-Qwen3.5	Qwen3.5-397B	60	2026-02-16	--	--	--	--	37.8	--	--	--	--	82.1
11	Mem0-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	37.5	--	--	--	--	83.9
12	MemAgent-GPT4o	GPT-4o	60	2024-11-20	--	--	--	--	37.3	--	--	--	--	80.2
13	ReAcT-Gemini-3	Gemini-3-Pro	60	2025-11-18	--	--	--	--	37.1	--	--	--	--	83.5
14	ReAcT-DeepSeek-R1	DeepSeek-R1	60	2025-05-28	--	--	--	--	36.0	--	--	--	--	81.7
15	Mem0-Gemini-3	Gemini-3-Pro	60	2025-11-18	--	--	--	--	34.3	--	--	--	--	81.5
16	MemAgent-Gemini-3	Gemini-3-Pro	60	2025-11-18	--	--	--	--	34.2	--	--	--	--	82.0
17	MemAgent-Qwen3.5	Qwen3.5-397B	60	2026-02-16	--	--	--	--	34.2	--	--	--	--	79.9
18	UniSAR	--	240	2024-04-15	--	--	--	--	29.2	--	--	--	--	64.5

Benchmark

Benchmark Collection

All data are newly collected from user data on a leading content platform with shopping functionality, not from any existing dataset.

Data Examples

Construction Pipeline of APeB: from behavioral logs to benchmark components: curated heterogeneous interactions, non-trivial shopping session discovery (search refinement, extensive browsing, and intent-order alignment), and intent/refined query candidate pair identification.

Data Statistics

Benchmark Comparison

Citation

@misc{yang2026apebbenchmarkingpersonalizationability,
      title={APeB: Benchmarking Personalization Ability of Large Language Model Agents}, 
      author={Garry Yang and Zizhe Chen and Xinru Chen and Yongqiang Chen and Jianxiang Wang and Deyu Zou and Linyi Ding and Jialiang Wu and Yunzhong He and Yu Gong and James Cheng and Huaixiao Tou},
      year={2026},
      eprint={2607.03162},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.03162}, 
}

This website is adapted from https://nerfies.github.io/ under Creative Commons Attribution-ShareAlike 4.0 International License.

APeB: Benchmarking Personalization Ability of Large Language Model Agents

A benchmark for agentic personalization under vague goals and noisy histories