Hanging out: An observational study

2026-02-16 · Source: Statistical Modeling, Causal Inference, and Social Science · Field: Science & Research — Social Sciences & Behavioral Studies, Research Methodology & Innovation · Depth: Intermediate, short

Summary

Researchers Paolo Parigi and Bruno Abrahao are analyzing complex data from a two-phase online investment game involving approximately 9,000 Airbnb users, designed to measure trust. Phase 1 participants were reinvited six weeks later for Phase 2, with about 5,000 returning. Between phases, Airbnb tracked platform usage, revealing that roughly 3,500 users traveled. A subset of travelers reported a "hangout" experience, treated as a quasi-random treatment. Initial analysis using a difference-in-differences model on travelers suggested hangouts increased sensitivity to reviews and homophily. However, this excluded non-travelers who returned for Phase 2, raising concerns about data exclusion and modeling nested hierarchical treatments and attrition. The researchers clarified that "hangout" experiences reinforced reaction to homophily (stronger preference for similar profiles) and sensitivity to reputation (more trusting of higher-reputation profiles).

Key takeaway

For AI Scientists analyzing complex behavioral data with multiple treatment levels and attrition, you should recognize that a single "best way" to analyze data does not exist. Instead, consider performing both descriptive analyses and observational studies. Prioritize adjusting for pre-treatment variables like demographics and past behavior, and frame specific questions as treatment interactions, visualizing these estimates rather than focusing on isolated effects, to gain comprehensive insights.

Key insights

There is no single "best way" to analyze complex data; multiple analytical approaches can yield diverse insights.

Principles

• Multiple insights from one dataset
• Descriptive analysis precedes causal
• Adjust for pre-treatment variables

Method

Frame specific questions as treatment interactions and display estimates graphically, rather than isolating a few interactions.

In practice

• Perform descriptive comparisons first
• Adjust for demographics and geography
• Graphically display treatment interactions

Topics

• Causal Inference
• Observational Studies
• Trust Measurement
• Homophily & Reputation
• Difference-in-Differences

Best for: AI Scientist, Research Scientist, Data Scientist

Related on AIssential

• # Why Most Data Science Doesn’t Answer the Question You’re Asking — Distinguishing correlation from causation is fundamental for making effective, intervention-based decisions in data science.
• The social sciences need tools for the 21st century | Letters — Social science needs new tools and methods to improve replicability and advance understanding of human behavior.
• Instrumented data for causal scientific machine learning — Instrumented data integrates mechanistic models and uncertainty into each datum for causal scientific machine learning.
• Difference-Making without Making a Difference — A new analysis challenges established classifications of actual causation, arguing their distinctions are invalid.
• Do you agree with Judea that learning from data is not everything? [D] — Purely data-driven machine learning is mathematically limited in inferring causation from correlation.
• é — Causal Inference Methods Every Data Scientist Should Know — Causal inference distinguishes correlation from causation, enabling robust decision-making in complex systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Statistical Modeling, Causal Inference, and Social Science.