Evaluating AI Performance in Obstetrics

Artificial intelligence is rapidly becoming a first point of information for expectant parents. A recent study evaluated AI in pregnancy queries, comparing ChatGPT-3.5, Gemini, and ChatGPT-4.0. Consequently, seventy-five obstetrics and gynecology specialists assessed the accuracy and reliability of these AI models. Their findings provide a critical roadmap for clinicians who must navigate the rise of digital health advice in their daily practices.

Superior Performance of ChatGPT-4.0 in AI Pregnancy Queries

Research indicates that ChatGPT-4.0 outperformed other models across almost all domains. Specifically, it achieved a median accuracy score of 4.35 on a 5-point Likert scale. Furthermore, it excelled in patient-friendliness, making it a potentially useful tool for general obstetric inquiries. In contrast, ChatGPT-3.5 consistently received the lowest scores from the specialists. While Gemini performed well in comprehensibility, it did not reach the overall consistency of GPT-4.0. Therefore, the choice of model significantly impacts the quality of medical information provided to patients.

Inter-Rater Consistency and Clinical Implications

Interestingly, the study found high inter-rater consistency among the evaluating specialists. This suggests that the strengths and weaknesses of these AI models are clearly identifiable by medical professionals. Despite the high scores for GPT-4.0, researchers emphasize that AI should supplement rather than replace professional consultation. Specialists must remain vigilant as patients increasingly rely on these tools. Ultimately, guiding patients toward validated digital resources is now a necessary part of modern prenatal care.

Frequently Asked Questions

Which AI model is most accurate for pregnancy questions?

Based on the latest study, ChatGPT-4.0 demonstrated the highest accuracy and patient-friendliness. While Gemini showed good comprehensibility, GPT-4.0 was more reliable overall.

Can AI replace a consultation with an OB/GYN?

No, AI cannot replace a specialist. While models like GPT-4.0 provide accurate general information, they lack the clinical judgment and personalized care required for safe pregnancy management.

What should doctors tell patients about AI in pregnancy queries?

Doctors should acknowledge the use of AI in pregnancy queries but remind patients that AI-generated advice requires professional verification. Encouraging patients to discuss AI findings during appointments ensures safety.

Disclaimer: This content is for informational and educational purposes only. It does not constitute professional medical advice, diagnosis, or treatment. Always seek the advice of your physician or other qualified healthcare provider with any questions you may have regarding a medical condition. Refer to the latest local and national guidelines for clinical practice.

References

• Keyif B et al. Evaluation of AI language models in answering pregnancy-related questions assessed by obstetrics specialists. Sci Rep. 2026 Feb 16. doi: 10.1038/s41598-026-40609-0. PMID: 41699404.


• Wan C et al. (2023) ChatGPT: An Evaluation of AI-Generated Responses to Commonly Asked Pregnancy Questions. Open Journal of Obstetrics and Gynecology, 13, 1528-1546. doi: 10.4236/ojog.2023.139129.


• Lee P, et al. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. N Engl J Med. 2023;388(13):1233-1239. doi: 10.1056/NEJMsr2214184.