Circulating tumor cell detection in cancer patients using in-flow deep learning holography

Finding Cancer Clues in a Simple Blood Draw

Cancer often spreads long before a tumor shows up on a scan, by shedding stray cells into the bloodstream. These roaming “circulating tumor cells” are rare but powerful clues: they can signal cancer’s presence, track how it responds to treatment, and hint at what might happen next. This paper describes a new technology that can sift through billions of blood cells to find these few dangerous strays quickly, gently, and with high accuracy—raising the prospect of routine blood tests that track cancer in real time.

Why These Rare Cells Matter

Circulating tumor cells (CTCs) are cancer cells that have broken away from a primary or metastatic tumor and slipped into the bloodstream. Even in advanced disease, a milliliter of blood can contain billions of normal red and white blood cells but only a handful of CTCs—sometimes fewer than one. Yet their number and behavior are closely tied to how aggressive a tumor is, whether it resists therapy, and how long a patient is likely to live. Standard ways to find them often rely on surface markers such as EpCAM, proteins on the outside of many epithelial cancer cells. But as tumors evolve and change state, these markers can fade, meaning some of the most dangerous cells may escape detection.

Shortcomings of Current Tests

Many existing blood tests for CTCs try to catch cells by their molecular “nametags” or by physical traits like size and stiffness. Marker-based systems, such as those that magnetically pull out EpCAM-bearing cells, can miss tumor cells that have switched off these markers during a shape-shifting process linked to metastasis. Size- or density-based filters can also fail, because some tumor cells are similar in size and feel to normal white blood cells. Devices that combine several tricks—complex microfluidic chips, magnets, or acoustic forces—gain sensitivity but often at the cost of higher complexity, expense, and cell loss. As a result, many platforms struggle to be both simple enough for routine clinical use and sensitive enough to capture the full diversity of tumor cells.

A New Way to See Cells in Flow

The team introduces a compact system that chains together three key ideas: smart fluid handling, holographic imaging, and deep learning, with a helpful dose of fluorescence. First, whole blood is run through a “saw-tooth” microfluidic chip that strips out more than 99.999% of red blood cells and nearly all white blood cells, while keeping the vast majority of tumor cells. This turns a crowded sample into one where rare cells have a better chance to stand out. The enriched stream then flows through a simple straight channel under a pulsed violet laser. Instead of taking flat pictures, the system records holograms—interference patterns that encode both the brightness and the optical thickness of each cell, capturing subtle three‑dimensional structure in a single snapshot.

Teaching a Neural Network to Spot the Odd Ones Out

Because true CTCs are scarce, directly collecting enough real patient examples to train an algorithm is impractical. The researchers instead taught a deep learning model to recognize anything that does not look like ordinary blood. They fed it millions of holograms from healthy donors, showing the full variety of white and red blood cells and debris, and balanced them with images from many different cancer cell lines grown in the lab. A specialized neural network converts each hologram into a heatmap of likely tumor locations, treating cells as glowing key points instead of boxes. To scale up training without endless manual labeling, a helper model first marks all cells in pure cancer-cell images; the main model then uses those “pseudo-labels,” plus a loss function tuned to reflect how rare real CTCs are, to learn to be strict about false alarms while still catching unusual cells.

Putting the System to the Test

In controlled experiments where known numbers of prostate cancer cells were spiked into healthy blood, the platform recovered about 60% of these cells at a chosen operating point, while keeping false positives under one per milliliter—even at liberal detection thresholds. To further sharpen its decisions, the system also measures two fluorescent signals from antibodies against PSMA, a prostate‑specific marker, and EpCAM. During patient testing, a cell counted as a CTC had to both pass the holographic deep learning threshold and show a strong PSMA signal, combining label‑free shape information with targeted molecular confirmation. When the researchers applied this to 13 men with advanced prostate cancer and 8 healthy male donors, the cancer group showed much higher CTC counts (median 12.5 cells per mL) than the controls (median 1.5 cells per mL). Strikingly, only about a third of PSMA‑positive cells carried EpCAM, suggesting that tests relying solely on EpCAM would miss the majority of tumor cells in these patients.

What This Could Mean for Patients

By turning a standard blood draw into a rich, automated scan for rare cancer cells, this platform points toward more practical “liquid biopsies.” Its mix of label‑free holography, targeted fluorescence, and deep learning reaches a low false‑positive rate while capturing CTCs that traditional marker‑based systems overlook. Today, that means better separation of cancer patients from healthy individuals in a pilot prostate cancer study. Looking ahead, the approach could be adapted to many tumor types and fluids, and may eventually work without any fluorescent labels at all, relying purely on how cells look in holographic space. If fully proven in larger, diverse trials, such systems could allow doctors to track cancer’s rise and fall over time with a simple blood test, adjust treatments sooner, and recover living tumor cells for detailed molecular analysis—all without a scalpel.

Citation: Mallery, K., Bristow, N.R., Heller, N. et al. Circulating tumor cell detection in cancer patients using in-flow deep learning holography. npj Biosensing 3, 23 (2026). https://doi.org/10.1038/s44328-026-00084-z

Keywords: circulating tumor cells, liquid biopsy, digital holographic microscopy, deep learning diagnostics, prostate cancer