Reference-Based Metric Analysis for Evaluating Spanish Text Simplification

Reliable evaluation remains a bottleneck for Spanish text simplification. We examine how five reference-based automatic metrics Bilingual Evaluation Understudy (BLEU), RecallOriented Understudy for Gisting Evaluation (ROUGE), Metric for Evaluation of Translation with Explicit Ordering (METEOR), System output Against References and the Input sentence (SARI), and Bidirectional Encoder Representations from Transformers Score (BERTScore)) respond to diverse, linguistically valid edits in human simplifications. Using the FEINA-test corpus (financial education texts simplified for visually impaired readers, with four human simplifications per segment and attribute annotations), we conduct a two-stage analysis. First, we compute each metric per segment: for BLEU, ROUGE, METEOR, and BERTScore, each simplification is compared to the complex source and the scores are averaged across annotators; for SARI, we rotate the four simplifications as hypotheses and use the remaining ones as references. Second, we introduce the Attribute Diversity Index (ADI), defined as the number of distinct linguistic attributes modified in the references for each segment, and assess metric sensitivity via Pearson correlation with ADI. All metrics show a negative association with edit diversity; BERTScore and ROUGE are the most sensitive, while SARI is comparatively more tolerant. METEOR and BERTScore yield higher mean scores overall, yet they also decline as diversity increases. These findings provide empirical evidence that commonly used reference-based metrics can penalize valid transformations in Spanish, particularly when multiple edit types are present, and suggest the potential for combining overlap-based metrics with SARI in accessibilityfocused evaluations.