Preferred but Less Safe? Human Preference–Safety Conflict in Paired AI Comparison Data

Human preference data are a primary input to the alignment of large language models. A preference judgement records which of two responses a person considers better overall; it is not by itself a measurement of safety. We test how closely the two correspond in paired comparison data that carry both an overall human preference and an independent safety label for each response. We analyse PKU-SafeRLHF v1 (82,067 comparison pairs, 39,990 unique prompts, 14.64% safety-discordant) as the primary dataset, and PKU-SafeRLHF-30K (29,861 pairs) as a separate replication release that is never pooled with it. Inference uses cluster-robust standard errors by prompt and Benjamini-Hochberg control across the 19 harm categories. Where the two responses differ in safety, the recorded preference selects the safer response in 83.28% of pairs and the less-safe response in 16.72% (95% CI 16.07 to 17.40). Conflict varies across harm categories from 9.36% to 21.66%, but every category association is small (largest Cramér’s V = 0.0571). Response position is a first-order confound: conflict is 20.12% when the safe response is listed first and 14.24% when it is listed second. In the replication dataset the relationship reverses, with a conflict rate of 61.06% (risk difference +44.33 percentage points). Human preference and response safety overlap substantially but are not interchangeable, and the size and direction of the relationship depend on dataset construction and response presentation. Preference-based alignment should treat safety as a separately measured objective rather than assume that an overall preference encodes it.