Login
Search
Search
0 Dates
2026
2025
2024
2023
2022
2021
2020
2019
2018
0 Events
CPC 2018
CPC 2019
Curso de Atualização em Medicina Cardiovascular 2019
Reunião Anual Conjunta dos Grupos de Estudo de Cirurgia Cardíaca, Doenças Valvulares e Ecocardiografia da SPC
CPC 2020
CPC 2021
CPC 2022
CPC 2023
CPC 2024
CPC 2025
CPC 2026
0 Topics
A. Basics
B. Imaging
C. Arrhythmias and Device Therapy
D. Heart Failure
E. Coronary Artery Disease, Acute Coronary Syndromes, Acute Cardiac Care
F. Valvular, Myocardial, Pericardial, Pulmonary, Congenital Heart Disease
G. Aortic Disease, Peripheral Vascular Disease, Stroke
H. Interventional Cardiology and Cardiovascular Surgery
I. Hypertension
J. Preventive Cardiology
K. Cardiovascular Disease In Special Populations
L. Cardiovascular Pharmacology
M. Cardiovascular Nursing
N. E-Cardiology / Digital Health, Public Health, Health Economics, Research Methodology
O. Basic Science
P. Other
0 Themes
01. History of Cardiology
02. Clinical Skills
03. Imaging
04. Arrhythmias, General
05. Atrial Fibrillation
06. Supraventricular Tachycardia (non-AF)
07. Syncope and Bradycardia
08. Ventricular Arrhythmias and Sudden Cardiac Death (SCD)
09. Device Therapy
10. Chronic Heart Failure
11. Acute Heart Failure
12. Coronary Artery Disease (Chronic)
13. Acute Coronary Syndromes
14. Acute Cardiac Care
15. Valvular Heart Disease
16. Infective Endocarditis
17. Myocardial Disease
18. Pericardial Disease
19. Tumors of the Heart
20. Congenital Heart Disease and Pediatric Cardiology
21. Pulmonary Circulation, Pulmonary Embolism, Right Heart Failure
22. Aortic Disease
23. Peripheral Vascular and Cerebrovascular Disease
24. Stroke
25. Interventional Cardiology
26. Cardiovascular Surgery
27. Hypertension
28. Risk Factors and Prevention
29. Rehabilitation and Sports Cardiology
30. Cardiovascular Disease in Special Populations
31. Pharmacology and Pharmacotherapy
32. Cardiovascular Nursing
33. e-Cardiology / Digital Health
34. Public Health and Health Economics
35. Research Methodology
36. Basic Science
37. Miscellanea
0 Resources
Abstract
Slides
Vídeo
Report
CLEAR FILTERS
Benchmarking Large Language Models for Cardiovascular Risk Stratification: A Bilingual Vignette-Based Study
Session:
Sessão de Comunicações Orais 09 – Inteligência Artificial e tomada de decisão no risco cardiovascular e nos sistemas de saúde
Speaker:
José Ferreira Santos
Congress:
CPC 2026
Topic:
N. E-Cardiology / Digital Health, Public Health, Health Economics, Research Methodology
Theme:
33. e-Cardiology / Digital Health
Subtheme:
33.4 Digital Health
Session Type:
Comunicações Orais
FP Number:
---
Authors:
Jose Ferreira Santos; Regina de Brito Duarte; Ines Mota; Rita Carvalheira Santos; Jose Maria Moreira; Joana Campos; Bernardo Neves; Ricardo Ladeiras-Lopes; Francisca Leite; Helder Dores
Abstract
<p style="text-align:start"><span style="font-size:medium"><span style="font-family:Aptos,sans-serif"><span style="color:#000000"><strong>Background:</strong> Cardiovascular risk assessment is critical to prevention, yet implementation remains inconsistent. Clinicians often rely on unaided judgment, leading to systematic risk underestimation. Large language models (LLMs) show promise as decision support tools but require rigorous validation against standard of care like SCORE2 before clinical deployment</span></span></span></p> <p style="text-align:start"> </p> <p style="text-align:start"><span style="font-size:medium"><span style="font-family:Aptos,sans-serif"><span style="color:#000000"><strong>Objectives: </strong>To benchmark 11 contemporary LLMs in (i) extracting cardiovascular risk factors from clinical text and (ii) generating accurate, ESC-aligned risk classifications, compared against expert-adjudicated reference standards.</span></span></span></p> <p style="text-align:start"> </p> <p style="text-align:start"><span style="font-size:medium"><span style="font-family:Aptos,sans-serif"><span style="color:#000000"><strong>Methods:</strong> We evaluated 30 systematically developed bilingual (Portuguese/English) outpatient vignettes using 11 LLMs (Claude Opus 4.1, Claude Sonnet 4.5, DeepSeek V3, Gemini 2.0 Flash, Gemini 2.5 Pro, GPT-4.1, GPT-4o, GPT-5, GPT-5 Nano, Grok-3, Llama 3.3 70B) in zero-shot setting. Performance was compared against gold-standard adjudication by senior cardiologists using 2021 ESC guidelines. Primary outcomes: risk-factor extraction accuracy (micro-F1) and three-class risk classification agreement (quadratic-weighted Cohen's kappa, κw). Secondary outcomes: numeric SCORE2 agreement (MAE) and guideline exception identification accuracy.</span></span></span></p> <p style="text-align:start"> </p> <p style="text-align:start"><span style="font-size:medium"><span style="font-family:Aptos,sans-serif"><span style="color:#000000"><strong>Results: </strong>All models achieved excellent traditional risk factor extraction (micro-F1: 0.97–0.99), but lower performance for risk modifiers (0.58–0.82). Risk classification agreement was moderate (Figure); GPT family models performed best, with GPT-4o achieving highest agreement (κw=0.69, 95%CI: 0.44–0.84). Critically, 10/11 models systematically underestimated risk categories. Numeric SCORE2 calculation was poor; all but one model showed clinically significant MAE >5 percentage points. Most models correctly identified guideline exceptions (>95% accuracy). No significant Portuguese-English performance differences were found.</span></span></span></p> <p style="text-align:start"> </p> <p style="text-align:start"><span style="font-size:12pt"><span style="color:#000000"><span style="font-family:Aptos,sans-serif"><strong>Conclusions:</strong> LLMs excel at structured data extraction and guideline eligibility screening with robust bilingual capabilities. However, inconsistent risk stratification, systematic underestimation bias, and poor numeric accuracy preclude autonomous clinical use. While promising as support tools, further refinement in clinical reasoning and mathematical computation is necessary for safe integration into cardiovascular prevention workflows.</span></span></span></p>
Our mission: To reduce the burden of cardiovascular disease
Visit our site