ScaleToT: Generalizing Structured LLM Reasoning for Billion-Scale Low-Activity User Modeling

2026-06-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

ScaleToT is a novel framework designed to generalize structured Large Language Model (LLM) reasoning for billion-scale low-activity user modeling, addressing the challenge of sparse interaction histories. It learns complex user-state inference from a small, LLM-processed data subset and extends this understanding to a wider user population. The system enhances reasoning reliability by constructing typed user-state chains through a bounded entropy-guided Tree-of-Thought (ToT) refinement procedure. These teacher-curated chains then train a student model on static profiles using supervised fine-tuning (SFT) and Outcome-Driven Segment-Aware Implicit Reward Policy Optimization (OSIPO). Finally, the student's reasoning representations are transferred to a lightweight profile encoder, enabling shared reasoning signals for remaining users without direct LLM inference. In a billion-scale advertising deployment, ScaleToT increased LT30 by 6.738% in a randomized online A/B test, while its offline reasoning covered only 7.32% of the potential population, drastically cutting compute costs compared to full-population LLM reasoning.

Key takeaway

For Machine Learning Engineers building user models for billion-scale low-activity populations, ScaleToT offers a viable path to apply LLM reasoning without prohibitive costs. If you face challenges with sparse user data or expensive LLM inference, consider adopting a teacher-student distillation approach. This method allows you to achieve significant performance gains, like a 6.738% LT30 increase, while drastically reducing compute requirements by avoiding full-population LLM deployment.

Key insights

ScaleToT generalizes LLM reasoning for sparse user data by training a student model from LLM-generated chains, then transferring it to a lightweight encoder.

Principles

• Structured LLM reasoning can be distilled.
• Sparse data can be augmented via teacher models.
• Costly LLM inference can be avoided at scale.

Method

ScaleToT constructs typed user-state chains with entropy-guided ToT, trains a student model via SFT and OSIPO on static profiles, then transfers reasoning to a lightweight profile encoder.

In practice

• Improve LTV prediction for low-activity users.
• Reduce LLM inference costs in advertising.
• Model user states from static profiles.

Topics

• ScaleToT
• User Modeling
• Large Language Models
• Tree-of-Thought
• Model Distillation
• Lifetime Value Prediction

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space — DLCMs improve LLM efficiency by shifting computation from tokens to a compressed, concept-based reasoning space.
• I Built a Language Model From Scratch on My MacBook — No GPU, No Cloud, No Excuses — Building a small LLM from scratch reveals the core mechanics of all GPT-style models, demystifying their operation.
• Context-CoT: Forcing LLMs to Actually Think (No ICL) — LLMs struggle to genuinely learn and reason from novel, complex knowledge provided solely in the prompt, often defaulting to pre-trained biases.
• Researchers automated LLM reasoning strategy design and cut token usage by 69.5% — AutoTTS automates LLM test-time scaling strategy design, cutting token usage by 69.5% and improving accuracy.
• Generalization in LLM Problem Solving: The Case of the Shortest Path — LLMs generalize spatially but fail at length scaling in shortest-path problems due to recursive instability.
• LLM Watch Weekly: When Scale Isn't Enough — Scaling alone cannot overcome data biases or fundamental reasoning gaps in LLMs and VLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.