Formal thought disorder (FTD) is one of the central diagnostic features of psychosis and related psychiatric conditions. Clinically, FTD often presents as a combination of derailment - abrupt shifts in topic or loss of coherent structure - and perseveration, where an individual becomes stuck on specific ideas, phrases, or semantic clusters. These two forms of disorganization frequently co-occur, yet they operate in opposite directions from a linguistic perspective. As a result, their combined presence has historically been difficult to measure using standard natural language processing techniques.
The new study, published in Schizophrenia Bulletin, examines this paradox by using generative language simulations to create controlled examples of disorganized speech. The researchers, led by Robin Quillivic, Raymond J. Dolan, and Isaac Fradkin, set out to solve a core methodological challenge: derailment and perseveration push standard language metrics - particularly cosine distance measures - in opposite directions. While derailment increases semantic distance across sentences, perseveration decreases it by producing repeated and semantically similar content. When both occur together, the metrics can cancel each other out, masking clinically significant symptoms.
To address this problem, the team constructed a synthetic dataset using large language models. These models were programmed to simulate narrative responses while independently manipulating levels of derailment and perseveration. Derailment was induced by increasing the stochasticity of word selection, causing the text to wander unpredictably. Perseveration was introduced by inserting paraphrases or repeated concepts at controlled intervals during generation. This experimental control allowed the researchers to test how different NLP metrics respond to each type of linguistic disturbance.
Traditional cosine-distance metrics performed as expected for derailment, reliably detecting disorganization when semantic content shifted unpredictably. However, the same metrics failed to detect perseveration. Because cosine distance is designed to capture divergence rather than repetition, repeated themes or semantic loops produced artificially low scores, which could easily be misinterpreted as coherent speech. This finding highlights a longstanding limitation in clinical NLP approaches: the tendency to equate low semantic distance with organized thought, even when it reflects compulsive or pathological repetition.
To overcome this limitation, the authors developed new density-based language metrics. Instead of measuring the distance between successive sentences, these metrics assess the overall density and clustering of semantic content within a narrative. Using methods like dimensionality reduction and cluster-size analysis, density metrics are able to reveal when speech becomes overly constrained, repetitive, or locked into narrow regions of semantic space. Such patterns are hallmarks of perseveration, even when the text superficially appears coherent.
The results were clear. Density-based metrics outperformed cosine metrics in detecting perseveration in the synthetic datasets, accurately distinguishing it from derailment even when both were present in the same narrative. More importantly, these metrics also demonstrated superior performance when tested on real-world language data.
The researchers validated their approach in a transdiagnostic sample of 811 individuals drawn from the general population. Here, density metrics again proved more sensitive in capturing perseverative tendencies, particularly among participants with symptoms linked to repetitive or stuck thoughts. These included depressive symptoms, rumination, and difficulty initiating conversation - an aspect of negative formal thought disorder often overlooked by traditional models.
The ability to dissociate derailment and perseveration has meaningful implications for clinical assessment. Both features contribute differently to functional impairment and treatment response, yet clinicians currently rely on subjective interpretations or tools that may obscure the interplay between the two. By enabling quantitative differentiation, density metrics could strengthen diagnostic precision, illuminate cognitive mechanisms underlying FTD, and help identify specific symptom profiles in conditions such as schizophrenia, bipolar disorder, OCD, and severe depression.
The study also demonstrates the emerging role of theory-driven generative simulations in psychiatric research. Rather than depending solely on patient speech - where symptom severities, medication states, and contextual factors vary widely - researchers can now model specific cognitive disturbances with controlled parameters. This allows them to test analytic tools more rigorously and identify where conventional metrics fail to generalize across the spectrum of disordered thought.
Within this broader context, the findings highlight a deep structural insight: language reflects the underlying organization of cognition. When cognitive fields lose coherence, become fragmented, or collapse around repetitive loops, linguistic patterns shift accordingly. The new metrics capture these shifts directly, revealing a more refined picture of how thought becomes disordered.
Viewed through the lens of Seven Reflections' Dimensional Systems Architecture, the study illustrates how derailment and perseveration map onto distinct structural disturbances within cognitive fields. Derailment reflects a breakdown of field continuity - an inability to maintain stable pathways across conceptual space, leading to excessive dispersion and weakened coherence. Perseveration, by contrast, represents a collapse of the field into narrow semantic basins, where Cognitive Field Saturation (CFS) rises sharply due to repetitive activation that limits access to alternative trajectories. Density-based metrics detect this collapse by identifying overly dense, overly narrow semantic clusters. These two failure modes - excessive dispersion and excessive saturation - operate differently but can co-occur, producing the paradox long recognized in clinical practice. By separating these dynamics, the new analytic tools allow researchers to see the structural signatures of disordered cognition more clearly, revealing where the system loses flexibility, bandwidth, or integrative capacity.
Ultimately, this metric innovation advances the understanding of formal thought disorder beyond surface-level linguistic symptoms. It shifts attention toward the underlying architecture of cognition, where stability depends on the system's ability to maintain coherent trajectories while avoiding saturation-driven collapse. The authors note that further clinical validation is needed, but their approach provides a promising framework for more precise, mechanistic interpretations of disorganized thought across psychiatric conditions.
