{"id":17391,"date":"2026-04-07T15:00:32","date_gmt":"2026-04-07T18:00:32","guid":{"rendered":"https:\/\/rtmedical.com.br\/voxtell-varian-eclipse-esapi-radiotherapy-research\/"},"modified":"2026-04-07T15:00:32","modified_gmt":"2026-04-07T18:00:32","slug":"voxtell-varian-eclipse-esapi-radiotherapy-research","status":"publish","type":"post","link":"https:\/\/rtmedical.com.br\/en\/voxtell-varian-eclipse-esapi-radiotherapy-research\/","title":{"rendered":"VoxTell and Varian Eclipse ESAPI: AI in Radiotherapy Research"},"content":{"rendered":"

VoxTell<\/strong> is a 3D vision-language model that segments structures in volumetric medical images from free-text prompts. Trained on over 62,000 CT, MRI, and PET volumes covering more than a thousand anatomical and pathological classes, the model represents a concrete advance in automatic segmentation. Gustavo Gomes Formento, a researcher at RT Medical Systems, developed two open-source integrations that connect VoxTell to interactive web interfaces and to the Varian Eclipse ESAPI<\/strong>, creating research prototypes that bring academic models closer to the real radiotherapy workflow.<\/p>\n

This article details the model architecture, both public integrations \u2014 web and ESAPI \u2014 and the DICOM coordinate conversion pipeline that makes it all possible. All content refers exclusively to research and technical evaluation tools, never to clinical software<\/strong>.<\/p>\n

What VoxTell Changes in 3D Segmentation<\/h2>\n
\"VoxTell
VoxTell architecture based on the project’s public materials.<\/figcaption><\/figure>\n

Conventional segmentation models operate with fixed labels. If the model was not trained for “posterior fossa tumor”, it simply does not segment it. VoxTell replaces this paradigm with free-text prompts: the operator types the desired structure \u2014 from “liver” to “left kidney with cortical cyst” \u2014 and the model generates the corresponding volumetric mask.<\/p>\n

The architecture combines a 3D image encoder with Qwen3-Embedding-4B<\/strong> as a frozen text encoder. A prompt decoder transforms text queries and latent image representations into multi-scale text features. The image decoder fuses visual and textual information at multiple resolutions using MaskFormer-style query-image fusion with deep supervision. The result: zero-shot segmentation with state-of-the-art performance on familiar structures and reasonable generalization to never-seen classes.<\/p>\n

The original paper (arXiv:2511.11450) documents training on 158 public datasets covering brain, head and neck, thorax, abdomen, pelvis, and musculoskeletal system \u2014 including vascular structures, organ sub-structures, and lesions. A foundation that reflects the migration of AI from isolated algorithms to workflow integration<\/a>.<\/p>\n

Web Interface: 3D Viewer, RTStruct, and Engineering for Limited GPU<\/h2>\n
\"VoxTell
VoxTell web interface with interactive viewer and text-based segmentation workflow.<\/figcaption><\/figure>\n

Developed by Gustavo Gomes Formento (RT Medical Systems), the voxtell-web-plugin<\/strong> is a FastAPI + React\/TypeScript application that puts the model behind an accessible interface. The operator uploads a volume (.nii, .nii.gz or DICOM), types a prompt such as “liver” or “prostate tumor”, and receives the 3D mask overlaid on the NiiVue viewer in real time.<\/p>\n

Low VRAM engineering is the practical differentiator. The Qwen3-Embedding-4B text encoder runs in float16, reducing memory usage from ~15 GB to ~7.5 GB. The memory allocator uses expandable_segments=True<\/code> to reduce fragmentation, and the sliding window operates with perform_everything_on_device=False<\/code> for partial CPU offload. As a result, 12 GB GPUs can already run inference \u2014 hardware found in research workstations, not just clusters.<\/p>\n

The viewer supports accumulation of multiple segmentations (liver + spleen + kidneys in the same session), manual drawing for refinement, and export in NIfTI and RTStruct<\/strong>. RTStruct export is particularly relevant: it produces a DICOM-RT file that can be imported into planning systems for comparative evaluation \u2014 always in a research context.<\/p>\n

Orientation note:<\/strong> images must be in RAS orientation for correct left\/right anatomical localization. Orientation mismatches produce mirrored or incorrect results. PyTorch 2.9.0 has an OOM bug in 3D convolutions; the recommended version is 2.8.0 or earlier.<\/p>\n

Varian Eclipse ESAPI: How the Integration Works<\/h2>\n
\"Conceptual
Public visual of promptable segmentation scenarios in the Eclipse context.<\/figcaption><\/figure>\n

Also created by Gustavo (RT Medical Systems), the VoxTell-ESAPI<\/strong> adds two components to the ecosystem: a Python\/FastAPI server that receives CT data via HTTP and runs inference on GPU, and a C# ESAPI plugin that extracts CT from Varian Eclipse, sends it to the server, and re-imports the resulting contours as RT structures.<\/p>\n

The complete workflow works as follows:<\/p>\n

    \n
  1. The operator opens a patient in Eclipse with CT and an existing structure set<\/li>\n
  2. The plugin creates a session on the server, sending volume geometry (origin, row\/column\/slice direction, spacing)<\/li>\n
  3. For each Z slice, voxels are extracted as ushort[xSize, ySize]<\/code>, converted to int32, serialized in little-endian, compressed with gzip, and encoded in base64 \u2014 reducing payload by ~4\u00d7<\/li>\n
  4. After all slices are sent, the server assembles the NIfTI volume with LPS\u2192RAS conversion<\/li>\n
  5. The operator types prompts (e.g., “liver, left kidney, spleen”) and submits<\/li>\n
  6. Inference runs asynchronously \u2014 Eclipse does not freeze<\/li>\n
  7. The server extracts 2D contours from the masks and returns coordinates in LPS (patient space)<\/li>\n
  8. The plugin imports via structure.AddContourOnImagePlane(contour_points_lps, z_index)<\/code><\/li>\n<\/ol>\n

    Existing structures are matched by name (exact, case-insensitive, or fuzzy). Structures not found are auto-created with DICOM type CONTROL. Names are sanitized to 16 characters (e.g., “left kidney” \u2192 “left_kidney”).<\/p>\n

    This plugin is intended exclusively for non-clinical environments: ECNC (External Calculation and Non-Clinical) and Varian TBOX (training box).<\/strong> It must never be run in a clinical environment.<\/p>\n

    DICOM Conversion Pipeline: LPS, RAS, and the Mathematics of Coordinates<\/h2>\n
    \"Workstation
    Photo: MART PRODUCTION \/ Pexels<\/figcaption><\/figure>\n

    The coordinate conversion between DICOM (LPS) and NIfTI (RAS) is the most critical technical point of the entire integration. An error at this stage produces mirrored volumes, anteroposterior inverted contours, or structures on the wrong side of the patient. The pipeline implements the transformation rigorously.<\/p>\n

    DICOM Geometry \u2192 LPS Affine<\/h3>\n

    Eclipse exposes the image geometry (origin, row direction, column direction, slice direction, spacing). The server builds the 4\u00d74 affine matrix that maps voxel indices to millimeter positions in the DICOM LPS (Left, Posterior, Superior) system:<\/p>\n

    $$x_{LPS} = A_{LPS} \\begin{bmatrix} i \\\\ j \\\\ k \\\\ 1 \\end{bmatrix}$$<\/p>\n

    Where the columns of $A_{LPS}$ are formed by:<\/p>\n