Artificial intelligence (AI) is rapidly entering clinical workflows. Andrews J et al. recently evaluated the safety of Large Language Models (LLMs) for surgical prophylaxis. This research highlights critical aspects of AI antimicrobial prophylaxis safety prior to clinical implementation. While LLMs show immense potential, they require rigorous simulation-based testing. Specifically, researchers must assess these models within specific institutional frameworks to prevent patient harm.

The study utilized twenty simulated surgical scenarios to test a prompt-conditioned LLM. The model generated recommendations for drug choice, dosing, and timing based on public guidelines. Consequently, local clinicians evaluated the outputs for accuracy and potential harm. Results indicated that overall guideline concordance was high. For example, the model achieved perfect accuracy in dosing. However, performance in other areas was less consistent.

Enhancing AI Antimicrobial Prophylaxis Safety

The evaluation revealed clinically significant risks in 10% of the simulated scenarios. These errors included the omission of required anaerobic coverage. Additionally, the system failed to recommend necessary redosing during prolonged procedures. Such errors pose a direct threat to patient safety and antimicrobial stewardship. Therefore, clinician oversight remains indispensable when using digital decision-support tools. Because accuracy for timing was only 70%, human verification is essential to maintain standards. Furthermore, citation accuracy was low at 45%, making it difficult for users to verify sources quickly.

In conclusion, this study demonstrates the feasibility of building guideline-based AI systems. Nevertheless, significant stewardship-relevant risks currently preclude their use without further validation. Hospitals should focus on constructing institutionally governed systems. These must undergo strict safety evaluations to ensure they align with local protocols and safety standards.

Frequently Asked Questions

What were the main risks identified in the AI system?

The primary risks included missing anaerobic coverage and failing to suggest redosing for long surgeries. These errors could lead to increased surgical site infections.

Is the AI system ready for clinical use?

No, the system is not ready for direct clinical use. The study found a 10% error rate in significant prescribing risks, necessitating further validation and refinement.

Disclaimer: This content is for informational and educational purposes only. It does not constitute medical advice or a professional relationship. Always consult a qualified healthcare provider for diagnosis and treatment. Refer to the latest local and national guidelines for clinical practice.

References

Andrews J et al. Pre-implementation safety evaluation of an AI decision-support system for surgical antimicrobial prophylaxis. Intern Med J. 2026 Jun 10. doi: 10.1111/imj.70470. PMID: 42268651.

Garnier M, et al. Guidelines on Antibiotic Prophylaxis in Surgery 2024. PubMed PMID: 41628822.

Ong JC et al. Large language model as clinical decision support system augments medication safety in 16 clinical specialties. NPJ Digit Med. 2025. PMC11421721.