Abstract
This review provides an overview of the application of artificial intelligence (AI) in radiation therapy (RT) from a radiation oncologist’s perspective. Over the years, advances in diagnostic imaging have significantly improved the efficiency and effectiveness of radiotherapy. The introduction of AI has further optimized the segmentation of tumors and organs at risk, thereby saving considerable time for radiation oncologists. AI has also been utilized in treatment planning and optimization, reducing the planning time from several days to minutes or even seconds. Knowledge-based treatment planning and deep learning techniques have been employed to produce treatment plans comparable to those generated by humans. Additionally, AI has potential applications in quality control and assurance of treatment plans, optimization of image-guided RT and monitoring of mobile tumors during treatment. Prognostic evaluation and prediction using AI have been increasingly explored, with radiomics being a prominent area of research. The future of AI in radiation oncology offers the potential to establish treatment standardization by minimizing inter-observer differences in segmentation and improving dose adequacy evaluation. RT standardization through AI may have global implications, providing world-standard treatment even in resource-limited settings. However, there are challenges in accumulating big data, including patient background information and correlating treatment plans with disease outcomes. Although challenges remain, ongoing research and the integration of AI technology hold promise for further advancements in radiation oncology.
INTRODUCTION
Radiotherapy has a history of improvement along with the advances in diagnostic imaging. With the advent of computed tomography (CT), the ability to depict tumors not as shadows but as 3D structures has advanced radiotherapy from 2D to 3D [1]. Furthermore, the diagnosis of tumor spread and boundaries was made by contrasting preoperative images with surgical pathology [2–5]. As the boundary between tumors and normal organs has become clearer and with improved computational power, intensity-modulated radiotherapy (IMRT) technology has enabled the reduction of the radiation dose to normal organs, while delivering a high dose to the entire tumor, even for more complex tumor shapes. The improved spatial positioning accuracy of images and the capability to capture tumor motion during treatment have made it possible to further lower the dose to normal organs while administering a very high dose to the tumor, thus enabling stereotactic radiotherapy, even at metastatic sites if feasible [6–10]. However, as the treatment plan becomes more precise, the standardization of contouring becomes more critical. Although contouring atlases have been created in various countries to standardize contouring [11–19], treatment plans are subject to the preferences and styles of planners [20]. Therefore, there are several problems with standardizing the segmentation and treatment plans. Several attempts have been made to reduce the time required for treatment planning while promoting standardization by incorporating artificial intelligence (AI)-based automation [21–24]. Over the past 5 years, numerous studies on AI-adapted radiation therapy (RT) have been published. RT consists of three crucial steps: preparation, delivery and evaluation. If we want to apply AI in these three steps, they all start with ‘segmentation’. After appropriate segmentation, we can proceed further with planning, optimization and online adaptive radiotherapy and then evaluate and predict the outcome (Fig. 1).
Number of publications on AI in RT since 2010.
This review summarizes the use of AI in RT, focusing on the clinician’s perspective rather than on the technical aspects of AI development. First, we summarize how auto-segmentation has progressed, followed by the current trends in the use of AI for planning, optimization and prognostic evaluation and prediction, and our expectations that AI will benefit both patients and medical staff.
For the literature review, we searched the PubMed database through 30 June 2023 for studies related to radiotherapy evaluation using AI. As a basic policy, we extracted reports from the most recent 5 years. For each section, ‘radiotherapy’ and ‘artificial intelligence’ were used as keywords, with ‘segmentation’, ‘quality assurance’, ‘optimization’, ‘planning’, ‘adaptation’ and ‘radiomics and/or prognosis’ added. The citations and references of the retrieved studies were used as additional sources of information for this narrative review and were manually searched.
SEGMENTATION AND DEFORMABLE MEDICAL IMAGE REGISTRATION
One of the most important steps in the preparation, delivery and evaluation of radiotherapy, as well as the time-consuming tasks in radiotherapy planning, is the segmentation of the target and organs at risk (OARs). Since the advent of CT and its use in treatment planning, radiation oncologists have spent a lot of time contouring targets and OARs. In the past, auto-segmentation was used with intensity thresholds; however, this method was inadequate because it could only automatically segment the lungs, intracranial and spinal canal. Subsequently, atlas-based segmentation methods were introduced [25–27] wherein a mono- or multi-atlas is used, and segmentation is performed through an installed atlas using the deformable medical image registration (DIR) technique.
Segmentation of OAR
DIR has been widely used and validated in commercial and open-source applications, and various algorithms are currently in use. The most common DIR method is intensity-based DIR, which enables the segmentation of organs with similar intensities, such as the liver and kidneys. To improve the accuracy of multi-atlas-based auto-segmentation, it is recommended that more atlases be read to select those that are similar to the target image. Therefore, one approach to improve the accuracy of multi-atlas-based auto-segmentation is to have the system select atlases that are similar to the target case by reading more data. Schipaanboord et al. reported that auto-contouring performance of a level corresponding to clinical quality could be consistently expected with a database of 5000 atlases, assuming perfect atlas selection [28].
The number of patients newly treated with radiotherapy in Japan in 2019 was reported to be 237 000, and 3D conformal RT using CT, or an even more precise treatment, is currently being implemented [29]. If contour data from all patients in Japan could be centrally collected, it would be possible to create a highly accurate contour atlas; however, this method may be more time-consuming.
DIRs are most effective when used to change plans for patients. Ideally, DIRs in the same patient should be perfectly matched, but this is not always possible due to organ motion and deformation. In particular, organs such as the bladder and intestines are highly deformed, both intra- and inter-operatively. This deformation necessitates adaptation during treatment, but it also makes automatic contouring challenging. This could be best described with prostate cancer, a common cancer with a high incidence rate, which has been the focus of extensive research in recent years [30–35]. During prostate cancer radiotherapy, it is difficult for therapists to control the rectum, which is located on the dorsal side of the prostate. Maintaining the same bladder volume and rectal condition at the time of treatment planning remains a persistent challenge for those involved in radiotherapy. However, with the advent of adaptive radiotherapy, it may be possible to auto-contour the rectum, which changes daily, rather than adjusting the bowel and bladder to the planned CT and to instantaneously change the irradiation plan [36–39]. Takayama et al. compared the accuracy of DIR for the prostate, rectum, bladder and seminal vesicles between intensity-based and hybrid-based DIR using Dice and a shift of emphasis. They reported that the accuracy of DIR for the prostate was 0.84 ± 0.05 and that for the rectum was 0.75 ± 0.05, relatively high agreement rates, even with intensity-based DIR [30]. The same report showed that using the hybrid-based method with an anatomically constrained deformation algorithm resulted in a dice similarity coefficient (DSC) of 0.9 or higher for all organs. However, the hybrid-based method requires contour creation for both DIR images, making it unsuitable for creating deformation plans during irradiation, although it is very useful for the dosimetry assessment of plans irradiated at different body positions. In recent years, there have been several reports on segmentation using deep learning (DL) methods. Highly applied methods include the encoder–decoder-type convolutional neural network (CNN) and 2D U-net or 3D U-net [40–42]. Xiao et al. described the usefulness of new 2D and 3D automatic segmentation models based on Refine Net for the clinical target volume (CTV) and OARs for postoperative cervical cancer based on CT. Their generated RefineNetPlus3D demonstrated good performance with a DSC of 0.97, 0.95, 0.91, 0.98 and 0.98 for the bladder, small intestine, rectum, and right and left femoral heads, respectively. Furthermore, the average manual CTV and OAR contouring time for one patient with cervical cancer patient was 90–120 min, and the mean computation time of RefineNetPlus3D for these OARs was 6.6 s [43, 44]. These results show great potential for the development of adaptive radiotherapy. However, the major limitation of this report is that the patient underwent postoperative irradiation for cervical cancer: that is, there was no gross tumor volume (GTV), and the bowel bag, space potentially occupied by the small and/or large bowel at any time during the treatment or at the time of imaging [45], was contoured, not the intestine itself. They described the difficulty in achieving a good DSC for the rectum owing to its small volume and unclear outline. Because the gastrointestinal tract is constantly moving, the ultimate goal is to transform the dose distribution according to movement while monitoring during irradiation; however, there are still issues to be solved. A recent report by Liao et al. also demonstrated the successful segmentation of 16 OARs in the abdomen using the DL technique (3D U-Net was used as the baseline model). They reported perfect contouring of the liver, kidneys and spleen. The most common achievement of their algorithm was its robustness. Their results were acquired from heterogeneous CT scans and patients, whereas most previous studies have used more homogeneous data. However, they failed to achieve satisfactory results in the duodenum (DSC < 0.7) [46]. Although auto-segmentation of OAR does have room for improvement, it can be inferred with a reasonable degree of confidence that OAR segmentation is nearing completion.
The attainment of targets is promising; however, more research is needed to determine its full potential. As described previously, postoperative adjuvant radiotherapy is typically applied to the CTV, where bone structures and other indices are targeted. A typical example is postoperative radiotherapy for cervical or head-and-neck cancers [47–49]. The advent of automated segmentation and planning techniques for these domains is imminent, and their successful implementation is anticipated in the near future given the trajectory of advancements.
Segmentation of the tumor
The segmentation of GTVs themselves has also been studied. Many reports have been on the auto-segmentation of GTVs [50–55]. Although most of these were small internal cohort studies, one interesting observational study performed external validation [56]. The participants were patients with lung cancer who typically had relatively clear primary tumor boundaries. In this study, 3D U-Net models were employed to segment lungs, primary tumors and involved lymph nodes. The architecture and model hyperparameters were fine-tuned using nnU-Net, a deep-learning-based segmentation method that does not create a new network architecture, loss function or training scheme (hence its clever name: ‘no new net’), including preprocessing, network architecture, training and postprocessing for any new task [57, 58]. An expert radiation oncologist delineated the target to create discovery data, and validation was performed using external sources. Volumetric dice and surface dice were used for the assessment. Although the models demonstrated enhanced performance compared to the inter-observer benchmark and achieved results within the intra-observer benchmark during internal data validation (performed by the same expert), their performance did not surpass the benchmark when evaluated using external data (segmented by different experts). This outcome may indicate the presence of variations in segmentation styles and preferences among experts, as substantial variability exists in the manual delineation of tumors [20]. However, AI assistance leads to a 65% reduction in segmentation time (5.4 min) and a 32% reduction in inter-observer variability. Therefore, it may be very useful in helping residents create segmentations that are satisfactory to senior radiation oncologists at the facility where the residents work or at a satellite hospital where experts are not always present.
RADIOTHERAPY PLANNING
AI has long been widely used in RT planning. Second-generation AI, a system that responds to conditioned reflexes by teaching AI knowledge in the form of rules, called an expert system, has allowed the widespread use of IMRT. Treatment planning for IMRT involves inverse planning, in which dose distribution (or fluence map) optimization calculations are performed to determine the behavior of the multi-leaf collimator and to calculate the final dose distribution [59]. However, this optimization process requires repeated trial and error by the treatment planner and a treatment planning time of several hours. Another drawback is that the quality of treatment planning is influenced by the planner’s skill level [60]. Knowledge-based treatment planning, a machine learning model, has been implemented in commercial treatment planning systems (TPSs) since 2014 and is widely used today [61]. Knowledge-based treatment planning is a system that registers past treatment planning data with the TPS and creates a semiautomatic treatment plan. This technology has significantly reduced the planning time from days to hours. Then, DL emerged as a promising new approach to treatment planning. In the past 5 years, there have been 257 reports, including 26 reviews, on DL optimization. DL methods learn the contour and dose distribution inputs to the CT for treatment planning and automatically generate dose distributions by inputting new contours [62, 63]. DL methods have been reported to generate treatment plans comparable to knowledge-based treatment planning [64]. Therefore, DL has the potential to fully automate planning from segmentation to optimization in hours, minutes or even seconds.
Quality control and quality assurance
The next step in planning is the quality control (QC) and quality assurance (QA) of the treatment plan. Once IMRT treatment planning is complete, dose verification is required. Currently, this is performed by actual measurements using dosimeters, films or multidimensional detectors, which can take a few hours for staff [65]. Recently, research was conducted to predict the results of gamma analysis results for IMRT QA using machine learning of the gamma analysis results measured using a 2D detector [66]. This shortens the QA time, and the results suggest that QA using DL is a promising direction for clinical radiotherapy.
DELIVERING THE PLAN
Image-guided radiation therapy
AI has also been studied in the irradiation process in radiotherapy treatment rooms. Before irradiation with the treatment beam, the patient’s position is verified using an image-guided radiation therapy (IGRT) system. Cone-beam computed tomography (CBCT) images used for IGRT have a lower soft tissue contrast and a higher noise ratio than CT images, which affects the accuracy of image registration [67]. Recent studies have been conducted to improve CBCT image quality using U-Net and CycleGAN [68, 69]. Generating CT-like images from CBCT by learning the conversion between treatment-planning CT images and CBCT images has been proposed. In addition, dynamic tumor-tracking irradiation is sometimes used for moving tumors, such as lung or liver cancer. To date, irradiation has been performed while observing a gold marker placed near the target, or using a correlation model created from the movement of an external marker placed on the abdominal wall [70–73]. However, these methods have some limitations, such as being invasive and not always correlating the external marker with the tumor position [74]. Consequently, research is underway to achieve real-time image tracking using X-ray projection images without markers, and to improve the accuracy of tumor position prediction using AI to compensate for the time delay of the device [75, 76].
Adaptive radiotherapy
Finally, a technology called online adaptive radiotherapy, which completes image acquisition, treatment planning (modification) and irradiation of the treatment beam while the patient lies on the treatment bed, is currently in practical use. There are two major methods for implementing adaptive radiotherapy: using magnetic resonance imaging (MRI)-equipped equipment and using CT (or CBCT)-equipped equipment [77–79]. The former method was initially put into practical use through an MR-Linac, which can accurately obtain MRI images of tumors and soft tissues prior to treatment. However, CT value data is essential for dose calculation, and technology to generate virtual CT images from MRI scans using AI has been developed and implemented in clinical settings [80]. Although the conversion is limited to certain areas and the accuracy is not perfect, this technology has significant future potential. On the other hand, in the latter method, CBCT is used for adaptation; however, dose calculation cannot be performed directly on the CBCT images acquired on the treatment table either, because CBCT has a limited field of view, incorrect CT value, or increased amount of image artifact. At present, the system generates virtual CT image by deforming the pretreatment planning CT to the CBCT using mutual information, and AI support segmentation and adaptive planning workflow to shorten the time. Over the past 5 years, 257 papers have been published in this area, including 15 reviews, showing high expectations for the future development of this technology (Fig. 1).
Predicting prognosis with AI
Reports on radiomics and prognostication have increased considerably in recent years, and many reports have been published in the field of radiotherapy. Although numerous studies have addressed lung cancer and head and neck cancer, in comparison to overall survival (OS), rectal cancer has been the most widely reported [81–86]. This trend may be due to the National Comprehensive Cancer Network guidelines that recommend concurrent chemoradiotherapy followed by surgical resection for locally advanced rectal cancer after a systematic review by the Colorectal Cancer Collaborative Group revealed that preoperative radiotherapy reduces the risk of local recurrence and death from rectal cancer, especially in young, high-risk patients [87, 88]. Due to this approach, patients with locally advanced rectal cancer usually have pretreatment and posttreatment MRI scans and pathological results. The overall trend is to examine whether pathologic complete response at surgery can be predicted using pre- or post-CRT images [42, 89, 90].
Numerous studies have constructed prediction models using textural features within a retrospective single-institution analysis, utilizing T2W images or DWI/ADC map in MRI and employing a training and validation set. Although some highly promising outcomes have been reported, the regions of interest are typically contoured manually; the single-center nature with no external validation means that their versatility is limited, and it is not yet possible to advocate a standardized radiomics efficacy assessment with the data currently available [91–93]. However, manual segmentation can be replaced by auto-segmentation. A recent report by Li et al. constructed an automatic pipeline from tumor segmentation to outcome prediction using pretreatment MRI. U-Net with a codec structure was used for segmentation, and a three-layer CNN was used to build the prediction models and achieve a DSC segmentation accuracy of 0.79, complete clinical response (cCR) prediction accuracy of 0.789, specificity of 0.725 and sensitivity of 0.812 [94]. With the recent introduction of total neoadjuvant therapy for rectal cancer and the ability to watch and wait for surgery in cCR cases, prognostication in this area is expected to become even more important in the future. Conversely, although there are many reports on OS prediction models and a subset of these yield promising results, it is difficult to predict OS only from image data, as it is significantly affected by treatment methods and patient-specific factors. Therefore, big data processing, which includes data other than images, is necessary for prediction.
Future perspectives of AI in radiation oncology: What will AI bring and what is required?
The ability to significantly shorten the time from segmentation to planning using AI is a major advantage, but efforts to address inter-observer differences in tumor segmentation are still needed, which persists as a challenge at the current stage. The ability to use AI for OAR segmentation is a great advantage in daily practice; however, it can also play a major role in standardizing treatment when conducting large-scale clinical trials. In a study using data from the RTOG0617 trial, which aimed to assess the impact of radiation dose escalation on OS in patients with inoperable non-small cell lung cancer, Thor et al. compared cardiac segmentation in patients enrolled in a study with auto-segmentation using a DL algorithm and found that cardiac doses calculated by auto-segmentation tended to be higher and correlated more strongly with OS than those obtained in clinical trials [95]. Radiotherapy planning attempts to standardize the dose to the tumor while imposing dose constraints on the OAR, but differences in segmentation at the initial stage can affect the evaluation of the treatment. To conduct appropriate clinical trials, a considerable amount of time is spent centrally evaluating treatment plans. If auto-segmentation of OARs with dose constraints can be achieved, not only will data collection be simplified, but it may also allow for the correct evaluation of treatment efficacy. Finally, this approach could facilitate an evaluation of the true dose adequacy and provide a foundation for considering the appropriate prescribed dose corresponding to the heterogeneity within the tumor.
The use of MRI for treatment planning not only avoids unnecessary radiation exposure but also allows for precise contouring of the rectum and uterus, which are difficult to isolate with CT. However, it is necessary to convert MRI images to electron density because it is not possible to create a simple conversion table between electron density and signal intensity for MRI images. CT-MRI conversion using AI is being promoted to overcome this challenge [96, 97]. Currently, many challenges need to be overcome, such as the fact that bone density varies among individuals. However, as this, MRI to CT conversion research progresses, clearly delineating the boundaries of soft tissues like rectal and others from the subtle differences in density in CT may become possible, as in MRI. This may enable the same level of segmentation as with MRI, even in countries with limited medical resources that do not have MRI and must use only CT for treatment planning. The successful incorporation of AI into radiotherapy has the potential to standardize cancer treatment worldwide [98].
Another promising area is the development of large-scale language models (LLMs) and their applications in RT. Language understanding has been a central research topic in the field of AI for many years, with its history taking many forms, from early rule-based systems to modern highly sophisticated models. LLMs learn patterns from vast amounts of textual data to understand and generate natural languages. Significant progress has been made in the development of LLMs, primarily in the past few years. One example is the Generative Pretraining Transformer (GPT) series trained using OpenAI. Since the introduction of the first GPT, its successors have been rapidly scaled up, from GPT-3 to GPT-4 [99, 100]. This rapid progress has given the models highly sophisticated natural language understanding and generation capabilities, resulting in diverse applications such as question-and-answer systems, document creation, code generation and even the generation of poetry and creative writing.
In the medical field, the potential of LLMs has been widely recognized, and their range of applications has expanded. These applications include diagnostic support, medical document generation and organization, research support, drug selection, telemedicine support, image analysis support, medical education, preventive medicine, lifestyle improvement and clinical trial design and analysis [101, 102]. These applications are made possible by combining the ability to find patterns in large amounts of data with the ability of the LLM to generate natural language. However, challenges remain in the medical applications of LLMs, such as data privacy, model interpretability, risk of misdiagnosis and misinformation and consistency. Overcoming these ‘AI hallucinations’ and other challenges requires not only technological advances but also the establishment of appropriate regulations and guidelines.
In radiotherapy, further development of LLMs is expected to make a significant contribution to the area of prognosis prediction, where we must consider how to accumulate big data by integrating data other than images, such as concomitant medications and other patient background information. Furthermore, they may not only predict adverse events and effects but may also be able to conduct a preliminary consultation. LLMs have the potential to encompass the dissemination of information to patients, elucidation of terminology and addressing commonly posed patient inquiries [103–105].
CONCLUSION
Although there are still many issues to be addressed regarding the use of AI in RT, the introduction of AI in treatment is a step toward standardizing RT. Auto-OAR segmentation is nearly complete and DL has the potential to fully automate planning from segmentation to optimization within very short time. Adaptive radiotherapy is now available, and LLMs may guide patients with necessary information. Once AI can help with planning, delivery and data collection, radiation oncologists can devote more time to patient care. This will allow us to have more meaningful conversations with patients, which will lead to improved treatment outcomes.
CONFLICT OF INTEREST
The authors declare that they have no conflicts of interest.
FUNDING
This work did not receive any grant from funding agencies in the public, commercial or not-for-profit sectors.
PRESENTATION AT A CONFERENCE
None.
