Medical Data Is Transforming Disease Research

A patient file, which is a collection of lab values, imaging results, and doctor notes gathered over years of routine care, sits in a database somewhere inside a hospital server in London, Chicago, or São Paulo. It is unremarkable on its own. Multiply that number by tens of millions of patients. Include genetic sequences. Add information from wearable devices that monitor blood oxygen levels, heart rate, and sleep patterns in real time. You receive more than just a stack of records. Researchers are just now learning how to use it, and the initial findings are truly unexpected.

Disease research is changing due to medical data. The pace varies greatly by nation, institution, and disease category, and the mechanisms are still being figured out. However, the path is obvious. Big data in healthcare is combining data from clinical records, electronic health records, molecular expression studies, diagnostic imaging, and IoT-connected devices in ways that produce insight at a scale no conventional clinical trial could match. Big data is characterized by its volume, velocity, variety, veracity, and variability. By 2025, it was projected that the global big data market for the healthcare sector would surpass $70 billion. According to a number of private analysis firms, the amount of research data produced daily is now on par with what was produced over a ten-year period.

The strongest argument for the practical implications of this change concerns illnesses that are virtually undetectable until they have advanced too far to be effectively treated. The most often mentioned example is pancreatic cancer. The reason it kills at such high rates is that symptoms don’t usually show up until the disease has progressed, and by then, the window for intervention is frequently closed. AI systems that analyze medical records and health histories on a large scale have started to spot possible early diagnostic signals, such as patterns in lab results, incidental imaging findings, and risk factors that, when combined across large populations, start to seem meaningful. The clinical validation process is still in its early stages. However, the enormous investment in this research is driven by the possibility that a disease with a five-year survival rate of about 11% could become detectable years earlier.

At a more established level, machine learning has already shown that it can diagnose cardiovascular diseases with accuracy that rivals and occasionally surpasses that of human specialists. One study that classified hyperlipidemia from clinical data using a complementary model based on support vector machines and neural networks achieved roughly 90% accuracy; if this figure were applied throughout a health system, it would significantly lower both misdiagnosis and the associated follow-up costs. Working specifically on the issue of medical data heterogeneity—the fact that patient records come in wildly different formats, from structured lab databases to plain text physician notes—the researchers at Lodz University of Technology discovered that natural language processing could help standardize and interpret these disparate inputs automatically, making them available for machine learning without the time-consuming manual transformation that currently causes bottlenecks.

The heterogeneity issue is worth considering because it is more indicative of the actual friction and less glamorous than AI’s ability to detect cancer. Hospital systems don’t communicate effectively with one another. It is frequently impossible to directly compare the electronic health records kept by different institutions without doing a lot of processing work. Free-form text is used to write patient notes. Different datasets have different units. Even when they describe the same thing, data gathered in Brazil and Germany appear different. These are structural obstacles that slow down all research applications built on top of the data, not small annoyances. Though it is gradual and largely undetectable to the general public, progress is being made.

Data is being produced by the scientific community at a rate that would have been nearly unthinkable to researchers in earlier generations. Omics technologies, such as transcriptomics, proteomics, metabolomics, and genomics, are producing molecular-level data about specific patients at low cost and high throughput. Ten years ago, whole-genome sequencing cost tens of thousands of dollars per patient. Today, it is getting close to a price point where it could be used in routine clinical practice for specific conditions. These massive data libraries are being organized and made available for research use by large-scale cloud platforms such as ELIXIR, the Global Alzheimer’s Association Interactive Network, and the National Cancer Institute’s Genomics Data Commons. Even though the precise clinical applications are still in the early stages of development, the infrastructure is being constructed piece by piece.

It’s difficult to ignore the tension that permeates all of this: between the institutions producing this data and the question of who ultimately controls and benefits from it, between the rapid pace of data accumulation and the slower pace of clinical validation, and between the promise of personalized medicine and the practical reality of data fragmentation. Health data platforms are being developed by the big tech companies, including Google, Microsoft, and Amazon. The computing infrastructure they provide is actually helpful, so that’s not shocking. However, the research community and regulators are still figuring out how to properly frame, let alone respond to, the questions raised by the concentration of health information in private commercial hands. The information is astounding. It’s still up for debate what will be built on top of it and for whom.