PACS (Picture Archiving and Communication Systems) refers to the software and hardware required to scan, store, and view medical images through a digital network. This system of image archiving first appeared in radiology and eventually was adopted by other medical branches when the digital slide viewers and the scanning hardware evolved. These systems provide remote access to case studies and their associated images for medical applications, such as image viewers [15].

The need to store patient data along with the image, as well as remote access through a web protocol, and the necessity to share data with other institutions, led to the development of the DICOM standard that defined the image format, communication processes, query, and retrieval, for each of the medical branches [16].

Overview of Dicoogle application. The core of the application handles core DICOM services, while plugins extend the application with further functionalities. The new additions to Dicoogle are highlighted in blue

The DICOM format splits files into a header, where the file, image, and patient metadata are stored, and a body where the pixel data is kept [17]. This data is encoded using key-value pairs named tags. DICOM supplements extend the format, defining new tags to integrate more imaging modalities or implementing new functionalities. DICOM defines four data objects that form a hierarchy: patient, study, series, and instance or image. DICOM files always contain the pixel data of one single instance. Several instances can be linked together through the file metadata using patient study and series identifiers.

In the context of computer-assisted diagnosis, the DICOM standard has several supplements, fitted for multiple modalities like radiology or pathology. Supplement 23 [18] defines structured reports, which contain the diagnostic report of a study or image, including annotations and biomarkers. Supplement 172 [19] proposes parametric maps, which can be used to display heatmaps over the images, that can encode image classifications with a probability assigned to each image pixel. Supplement 111 [20] proposes image overlays, which are used to draw regions such as segmentations from analysis algorithms. These overlays can be exported to separate files, denominated DICOMSeg (DICOM Segmentation) files. Recently, the new DICOM supplement 222 for simple bulk annotations proposes a compressed format to communicate simple image annotations such as circles or rectangles. This is useful when the analysis outputs thousands of objects, such as can be the case when dealing with slide microscopy images [21].

The DICOM workgroup 23, artificial intelligence/application hosting [22], has been continuously working on the DICOM standard to normalize the integration of image analysis services. They have recently been refactoring the standard for structured reports to support the JSON format [23] (supplement 219), with the intent of improving the integration of this data in information systems. Supplement 251, which proposes integration of third-party applications using REST services for image analysis, is an ongoing work that has been targeting integration with other standards and hospital information systems. Currently, only the inference use case has been targeted, and it is not compatible with existing solutions that might not work with DICOM.

Development lifecycle of a machine learning model

As PACS became more prevalent in the healthcare industry, new usage paradigms emerged that forced the development of new architectures and functionalities to satisfy the new requirements. In this area, testing and integration of new features are very complex, and their development, deployment, and testing are not fast enough.

Dicoogle (Fig. 1),Footnote 1 an open-source DICOM-compliant PACS, addresses these needs through a plugin-based architecture, which allows easy development of new features or changes to its core functionality [24]. A Dicoogle plugin implements one of the available interfaces to control a specific aspect of the system. Indexing plugins, for example, can control how the files are to be indexed.

This architecture allows Dicoogle to bridge the gap between researchers and DICOM networks, allowing easy deployment and testing of new ideas. The development of new features is facilitated by the Software Development Kit (SDK) present in the framework, allowing another programmer to easily add new modules to Dicoogle without changing its core functionality. It also serves as a bridge between the PACS repository and third-party applications. Dicoogle is a research project of our group at Aveiro University. It has been successfully used in clinical environments for different medical specialties such as radiology and pathology. This work expands Dicoogle’s functionalities with a new interface and additions to the SDK.

Currently, there are several tools and frameworks to develop AI algorithms that vary in features and application domains. Tools such as Cellpose [25], ImageJ2 [26], and Cellprofiler [27] provide out-of-the-box tools and algorithms to perform image analysis and object identification. These tools have been integrated into applications such as QuPath [28]. Other tools, such as [29], the open-source biomedical image segmentation framework, go a step further, enabling researchers to use semi-automatic image segmentation pipelines and fully automatic segmentation through DL.

The next tier of tools encompasses the machine learning (ML) lifecycle, typically divided into three core phases (Fig. 2): data preparation, involving the collection, curation, and labeling of datasets; model development, where model architectures are designed, hyperparameters are tuned, and performance is evaluated; and model deployment, in which the trained model artifact is packaged and made available for inference in production environments.

There exist several technologies, such as Google Vertex AI (GVA), Amazon Sagemaker (AS), Azure Machine Learning (AZML), MONAI [30], and WebDL [31]. These tools focus on providing interfaces and technologies for users to build and deploy their algorithms, as opposed to providing a set of out-of-the-box tools. MONAI differs from the other platforms in that it was designed specifically for medical standards, integrating DICOM and HL7 directly into its interfaces to communicate with PACS and EHR (Electronic Health Records) systems. MONAI has been integrated into radiology [32] and pathology [28] tools.

The process of developing AI-based analysis tools starts with a data collection and preparation step. Items within a dataset can be assigned labels for classification tasks or annotated with regions of interest and used in object segmentation tasks. Several strategies exist to define datasets and labels, the most common one being manifest files, which link items on a dataset with a set of labels for training. An alternative approach relies on automatic labeling. Some systems, like GVA, AzureML, and WebDL, support automatic labelling of a dataset through its folder structure, where the name of each subfolder maps to a class. This strategy however is less flexible as it requires a very strict folder organization and is not practical to use in segmentation tasks.

MONAI improves this process by offering direct connection to PACS servers, allowing users to build datasets using image metadata (such as patient or study information) offering semi-automatic labeling systems, where users can correct the labelling using positive or negative clicks to guide the algorithms in an active learning loop.

Once the dataset is constructed and labeled, a classification or segmentation model, depending on the label type, can be developed. This is an iterative process, where the algorithm can be trained several times using different parameters or configurations. AutoML can automate much of this process, where only the training data needs to be supplied and the platform tries to find the best module architecture and hyperparameters automatically.

The last step when developing an AI application is to deploy it to production, so it can be used by third-party systems for inference. All systems behave similarly in this matter; any model with any output structure can be deployed to an endpoint and be used for inference. When requesting inferences from the models, they optionally can return explanation metrics, which may contain, for example, the most relevant features used in the prediction.

This makes these platforms very flexible, but without interoperability between different endpoints. Additionally, they do not provide interfaces to connect to medical systems such as EHR or PACS. MONAI addresses this issue with MONAI Deploy, a set of tools and interfaces to standardize model deployment and integrate them in medical production environments. Through an SDK, researchers can produce a MAP (MONAI App Package), which packages the trained model in a standardized format. This package can be deployed to a hosting server, connecting to the MONAI Deploy Workflow Manager.

Table 1 highlights the main features of the analyzed frameworks alongside the proposed framework. All frameworks, with varying degrees of complexity, implement tools to address the three main stages in the ML lifecycle. Cloud-based frameworks like GVA, AZML, AS, and WebDL provide infrastructure for model deployment, while for Dicoogle and MONAI, deployment is handled separately. On the other hand, Dicoogle and MONAI fit better in a medical image analysis context given their compatibility with protocols like DICOM and HL7, which allow them to be interoperable with existing information systems. Dicoogle stands out from MONAI in that it allows integration of existing third-party services.

This section presents how image analysis services are currently being integrated into imaging software. In [28], the authors present QuPath, an open-source digital pathology application. The viewer provides an SDK and plugin system that allows developers to extend the functionality of QuPath with new features. QuPath’s API is not directly compatible with DICOM formats, so reusing the algorithms in other applications will require adaptation efforts. Additionally, QuPath does not interface with PACS systems, meaning that using this system in production environments might not be feasible.

In [33], the authors describe Omero, a multidisciplinary microscopy data management tool. It integrates a scripting engine that allows users to write image-processing scripts and store the data generated by these scripts on the server so that they can be reused. This tool is designed specifically for microscopy data, and the authors do not mention support for other modalities. Additionally, it does not work with the DICOM standard, making it impractical to use with DICOM-based PACS.

In [34], Orthanc is presented as an open-source, easy-to-install DICOM PACS with a rich REST API for third-party control and integration. It includes a Lua scripting engine for creating processing pipelines upon new DICOM file arrival [35] and supports extensibility through plugins for its web interface and API. Unlike other solutions presented in this section, Orthanc does not address the pathological sector directly. Such shortcomings however can be overcome through the Lua scripting engine. The solutions explored thus far lack compatibility with other common hospital standards such as HL7, which might limit their applicability.

Sectra is a multidisciplinary PACS that supports both radiology and digital pathology workflows. It offers a marketplaceFootnote 2 for integrating AI solutions in these areas, where developers must apply to have their applications listed. Users can access these AI tools directly through the PACS, with all algorithms hosted on Sectra’s infrastructure. While this ensures solution quality, as the algorithms need to be certified, it limits flexibility for research use, where testing uncertified or experimental algorithms may be desired. Other commercial solutions, such as GE Healthcare Edison True PACS or IntelliSpace from Philips, among others, also provide cloud-based PACS with AI integrations.

In [36, 37], IHE (Integrating the Healthcare Enterprise) explores the use of AI in radiology and proposes a trial supplement for integrating AI services into healthcare networks. The supplement defines communication workflows between EHR and PACS systems for image analysis requests and provides detailed structures for image data exchange. Although it does not cover the entire ML lifecycle, it establishes a key foundation for interoperability between standards like HL7 and DICOM.

In [38], the authors present an API-based approach to integrate image analysis services within the open-source Dicoogle PACS framework. AI services were incorporated using Dicoogle web plugins that adhere to a specified web API protocol. Although this approach successfully enabled the integration of AI algorithms from various vendors, the proposed interface lacked type definitions, which did not enforce standardized input and output data types.

The works presented in this section integrate image analysis services through scripting engines, SDKs, APIs, or marketplaces. While these systems provide great extensibility and flexibility, the lack of a standard interface means these algorithms cannot be easily reused in different contexts. Without normalized interfaces, these services cannot be used across distinct viewer applications or imaging archives. Using as a base of work the developments from DICOM WG-23 and the platforms described in this section, this work proposes a prototype framework to integrate image analysis applications on a DICOM-compliant PACS relying on modern web-based technologies, to address the ongoing lack of standardization in medical image analysis.

Adopting a standard format to display analysis results makes these easily shareable with different systems and institutions. Existing DICOM datasets can be enhanced with this data by producing artifacts such as annotation or segmentation files that reference the original images, improving data quality for training the next generation of algorithms.

Overview of the proposed system architecture. Viewer applications connect to ML services through a single interface on the PACS archive