Can ChatGPT support patient–physician communication regarding spine disorders?

Abbreviations

• AI – artificial intelligence
• GPT – generative pre-trained transformer
• LLM – large language model
• p – p-value (probability value)
• Q1–Q7 – survey questions 1 to 7
• SD – standard deviation

Introduction

In recent years, large language models (LLMs), such as OpenAI’s ChatGPT, have rapidly gained attention across a wide range of domains, including healthcare. These tools are capable of generating natural language responses to complex queries, offering instant access to information, explanations, and recommendations. Given the increasing demand for accessible medical knowledge and the growing reliance on digital solutions, the integration of AI-based systems in health communication presents both promising opportunities and serious challenges [1-2].

Although ChatGPT is not a certified medical tool, its use among the general public and health professionals alike has become increasingly common. Patients may turn to such models for preliminary information, second opinions, or to clarify medical terminology [3-4].

Previous research has highlighted the potential of AI chatbots to support healthcare by improving communication, enhancing patient education, and reducing barriers to access. However, concerns remain regarding the factual reliability of AI-generated information, the ethical implications of unsupervised use, and the risk of misinterpretation by users without medical training. These concerns underscore the need for empirical evaluation of AI tools in real-world health scenarios, especially from the perspective of both healthcare providers and patients [5].

The present study aims to fill this gap by systematically assessing how ChatGPT’s responses to medical questions are perceived in terms of satisfaction, clarity, helpfulness, and clinical accuracy. The study further examined whether these perceptions differ across respondent groups (patients vs. physicians, specialists vs. residents) and explored the influence of demographic factors (e.g. age, sex and medical history) on the evaluation of AI-generated responses.

Material and methods

Patients and physicians were asked the same questions in surveys regarding low back pain, lumbar disc disease, and degenerative disease of the lumbar spine. All responses assessed in this study were generated using the GPT-4 version of the ChatGPT (OpenAI, San Francisco, USA) model, obtained via the official website.

Inclusion and exclusion criteria

Patients were eligible for inclusion if they were ≥ 18 years of age or older, had a history of low back pain and/or degenerative lumbar spine disease, were able to read and understand the questionnaire written in Polish language, provided informed consent to participate in the study, and completed all survey questions. Patients were excluded if they were < 18 years of age, submitted incomplete questionnaires, declined participation or withdrew consent during the study, had cognitive impairment or any other condition (somatic or psychiatric) that limited their ability to understand and interpret the survey content.

Each participant provided informed, written consent to participate in this study. The survey was completely anonymous and did not contain any personal or sensitive information. Ethical approval for this questionnaire-based study was obtained from the Bioethics Committee of the Collegium Medicum, Nicolaus Copernicus University in Bydgoszcz (KEWL No. 35/2026). The study was conducted in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments.

The survey

A survey was conducted on paper and electronically using the Forms software (Google, Mountain View, CA, USA). The 7 questions used in this study were developed based on clinical experience and observations of the questions most frequently asked by patients during outpatient visits related to back pain and degenerative diseases of the lumbar spine. The final set of questions was reviewed and selected by neurosurgery specialists to ensure its high substantive value and relevance to daily clinical practice. The 7 questions addressed the causes, symptoms, treatment options, indications for surgery, potential complications, and management of worsening symptoms in lumbar spine disorders. The complete questionnaire and ChatGPT-generated responses are provided in the Supplementary Material.

Survey questions

Q1. What are the causes of back pain?
Q2. What symptoms require medical consultation and
how urgently should a doctor be consulted?
Q3. What are the types of lumbar spine surgery?
Q4. What are the complications of lumbar spine surgery?
Q5. What are the treatment options for lower back pain?
Q6. What are the indications for lumbar spine surgery?
Q7. My back pain is getting worse. What should I do?

Statistical analyses

Statistical analyses were performed using IBM SPSS Statistics software (Armonk, NY, USA). Continuous variables were described using means, standard deviations and medians. Normality of distribution was assessed using the Shapiro-Wilk test. Comparisons between groups were performed using the Mann-Whitney U test. Associations between categorical variables were analyzed using the chi-square test. Statistical significance was set at p < 0.05.

Results

Patients

The patient group consisted of 50 individuals (68% female and 32% male). The largest age group included patients aged 46-60 (30%), followed by those aged 18-30 (38%), 31-45 (22%), and the smallest group were individuals over 60 years old (10%). Majority of the patients (62%) had higher education, whereas 22% had secondary education and 16% had vocational education. A total of 14% of patients (n = 7) reported having undergone spine surgery.

Physicians

This study included 20 physicians, of whom 15 were neurosurgery specialists and 5 were in neurosurgery residency training. The physicians represented a more homogeneous group in terms of education, but varied in their levels of clinical experience.

Overall evaluation of ChatGPT responses

Statistical analysis showed that ChatGPT’s responses were generally evaluated positively by all study participants. The highest average scores were recorded for clarity (mean: 4.15; SD: 0.91) and helpfulness defined as the usefulness of the response in addressing the question (mean: 4.05; SD: 0.76) as shown in Table 1. All variables significantly deviated from a normal distribution (Shapiro–Wilk test, p < 0.01), though the distributions were approximately symmetric, with slight left skewness and kurtosis values indicating moderate flattening (Table 1).

Table 1. Average ratings of ChatGPT responses by all participants (n = 70)

I. Patients vs. physicians

Statistically significant differences were found between patients and physicians. Patients rated ChatGPT responses higher than physicians, particularly in terms of clarity (Table 2 and Figure 1).

Table 2. Comparison of ratings: patients vs. physicians (Mann–Whitney U Test)

Figure 1. Comparison of median ChatGPT response ratings by patients and physicians for selected questions

II. Specialists vs. residents

Within the physician group, a trend emerged showing that residents gave higher ratings to ChatGPT responses. The most notable difference concerned Q4 – helpfulness (p = 0.020) (Table 3).

Table 3. Physicians’ evaluation of ChatGPT responses, by level of training

Influence of age, sex and medical history

Participants over 45 years old were more likely to report that ChatGPT suggested next steps, and gave higher ratings for satisfaction and helpfulness (p < 0.05 for multiple variables, e.g. Q2, Q3, Q5, Q6) (Table 4). Most variables showed no statistically significant differences. However, women rated Q4 – satisfaction higher than men (p = 0.037). Participants who had undergone spinal surgery were more likely to state that ChatGPT suggested further action (Q6, p = 0.025) and were more willing to continue consultations with neurosurgeon.

Table 4. Age differences in satisfaction ratings

Findings from Chi-Square Tests of Independence

Chi-square analyses revealed no significant overall differences in declarations of willingness to continue consultations or in perceptions of whether ChatGPT suggested further action, except for the following:

Q2 – Sex: women more frequently reported that ChatGPT indicated further action (p = 0.039).

Q3 – Sex: men more frequently than women declared that ChatGPT partially suggested which procedure is indicated for the particular diagnosis (p = 0.004).

Discussion

Overall, our findings indicate a positive reception of ChatGPT’s responses, particularly in terms of clarity and helpfulness. Notably, patients rated its responses significantly higher than physicians, particularly regarding claritiy (p < 0.01) and helpfulness (p < 0.05). This suggests that ChatGPT may serve as an effective educational and informational tool for individuals without formal medical training. However, physicians evaluated the factual accuracy of the responses as moderately high, highlighting the need for ongoing validation of LLM outputs against current medical standards.

A significant finding is that neurosurgery residents in training tended to assess the responses as more helpful compared to specialists. This may reflect a greater openness to emerging technologies among younger clinicians. Furthermore, among patients, older age and medical history (e.g. past spinal surgery) were associated with higher ratings of response utility and more frequent reports that ChatGPT suggested appropriate next steps [6].

While no statistically significant differences were found between the studied groups regarding whether ChatGPT suggested further medical action, trends emerged indicating the patients’ greater acceptance of that LLM. Importantly, most variables did not show sex-based differences, although isolated findings (e.g., for Q2 and Q3) suggest that individual characteristics may still influence perception.

The rapid development of LLMs has opened new possibilities for patient education and health communication. These models have significant potential to provide rapid and accessible medical information to large numbers of patients. A review of recent literature reveals both promising applications and important limitations of ChatGPT, particularly in the context of spine-related disorders [7-9]. Several studies have evaluated the accuracy, readability, and clinical relevance of ChatGPT’s responses to commonly asked patient questions. For example, ChatGPT-3.5 provided generally correct but overly technical responses to questions regarding pediatric scoliosis [10]. Similar results were found in relation to cervical spine surgery and lumbar disc herniation: the LLM demonstrated moderate accuracy (up to 70%) but struggled to adjust the complexity of its language to the average patient’s reading level [11]. This highlights a significant gap between technical accuracy and practical accessibility [12].

Interestingly, attempts to simplify answers to a sixthgrade reading level resulted in only modest improvements in clarity, suggesting limitations in ChatGPT’s ability to effectively adapt communication for populations with lower health literacy. This is particularly important, as effective patient education requires not only factual accuracy but also clarity, empathy, and personalization [13].

In clinical simulations, ChatGPT showed inconsistent performance in generating differential diagnoses and recommending treatment. In a comparative study involving spine surgeons, the model misinterpreted traumatic cases and proposed management strategies that were judged clinically inappropriate by the participating specialists in more than 50% of the evaluated scenarios [12]. These findings indicate that while ChatGPT may serve as a supportive educational tool, it certainly cannot replace clinical judgment by a medical professional. On the other hand, the model showed utility in research-related tasks, such as generating topics for systematic reviews or summarizing clinical guidelines, particularly regarding low back pain and spine surgery. This suggests a complementary role for LLMs in medical education and academic support [13-14].

Additionally, other studies have shown that ChatGPT-4 can provide reliable and understandable responses to patient questions related to spinal cord stimulation, highlighting the potential of more advanced versions of LLMs [15]. However, even these models require further clinical validation and patient-specific adaptation [16]. Future research should explore real-world interactions between patients and LLMs, evaluating outcomes such as comprehension, trust, behavioral impact, and emotional support. New models (e.g. GPT4o) offer enhanced multimodal and conversational abilities, which warrant continuous, systematic evaluation regarding their potential in healthcare contexts [17].

Limitations and clinical implications This study has several important limitations that should be considered when interpreting its results. First, the physician sample size was limited (n = 20), with only 5 of those physicians in residency training. This sample structure may impact the representativeness of the data and limit the generalizability of the results to the broader healthcare professional population. Furthermore, participants were recruited voluntarily, which carries the risk of self-selection bias: individuals more interested in new technologies or using digital tools may have been overrepresented in the sample, potentially inflating the overall assessment of the usefulness and understandability of ChatGPT responses. Another limitation is that this study was perceptual: participants rated responses based on subjective impressions (clarity, helpfulness, satisfaction), but there was no independent assessment of the responses’ compliance with current clinical guidelines or standards of medical practice. Furthermore, the study did not compare the responses generated by ChatGPT with other popular information sources, e.g. search engines, Wikipedia, traditional medical websites or advice provided by physicians. Finally, a single version (GPT-4) of the ChatGPT LLM was used in this study. These models are subject to dynamic changes, updates, and optimization, so responses generated in the future may differ from those evaluated in this study. The variability of the LLMs over time is a challenge to the repeatability and comparability of results across studies and time [18-19].

Conclusions ChatGPT responses were generally well-rated by both physicians and patients, particularly for clarity and helpfulness. Patients provided significantly higher ratings than physicians, highlighting ChatGPT’s potential as an educational support tool in patient–physician interactions. Physicians in training were more receptive to ChatGPT’s outputs than experienced specialists. Age and prior medical experiences (e.g. past surgery) positively influenced perceptions of satisfaction and usefulness of the chatbot responses. Medical accuracy of ChatGPT’s responses requires further verification, as physicians’ evaluations were moderately positive but not conclusive.

Funding

No external funding was received for this research.

Conflict of interest

The authors declare no conflicts of interest. Data availability statement The datasets generated and analyzed during the current study are not publicly available due to patient privacy and ethical restrictions but are available from the corresponding author on reasonable request.