This study addresses the challenge of evaluating creativity in Russian-language Large Language Models (LLMs) by adapting a specialized benchmark. Traditional NLP metrics fail to capture qualitative aspects like coherence and originality, especially in Russian. The research modifies SimulBench for Russian by adjusting prompts, cultural references, and evaluation criteria. A diverse set of Russian LLMs was tested using adapted metrics focusing on creativity, coherence, and diversity. Google's Gemini Flash was used as an automated evaluator. The study provides a foundation for improving the assessment and generation of creative AI-driven content in Russian.

Абдурахимов М.А. (науч. рук. Ходорченко М.А.) Development of a creativity benchmark for russian language when assessing the quality of language models // Сборник тезисов докладов конгресса молодых ученых. Электронное издание. – СПб: Университет ИТМО, [2025]. URL: https://kmu.itmo.ru/digests/article/14593