We are developing a transparent validation framework for ambient AI scribe applications — AI systems that automatically capture and summarize clinical conversations to reduce administrative burden for healthcare professionals. Although these tools have great potential to transform healthcare, research shows that only 2% of AI projects make it to the bedside. Closing this gap requires responsible and trustworthy validation of developed AI models. Our framework evaluates both performance (e.g., accuracy, comprehensiveness) and ethical dimensions (e.g., fairness, impact on the patient–clinician relationship) of ambient AI scribe applications. Using techniques such as sentiment analysis and LLM-as-a-judge, and powered by SAS technology, we compute these metrics and present them in a clear, intuitive dashboard. The dashboard gives evaluators an at-a-glance overview of strengths, risks, and trade-offs, making AI evaluation both transparent and actionable. To further enhance usability, we are also exploring the integration of an explanatory LLM that provides contextual insights and interprets the results displayed. | 
