Underneath the phrases of the Loss of life in Custody Reporting Act, the Justice Division is required to gather details about everybody who dies in prisons and jails throughout america. The intention behind this effort is that, via the aggregation of particulars about individuals who die within the custody of hundreds of legislation enforcement businesses, the ensuing dataset might inform future life-saving coverage adjustments.
Details about these deaths is usually not made public at a stage of granularity that may present perception into particular person instances. Nevertheless, as a result of what was possible a configuration error on a Division of Justice web site, we obtained unprecedented entry to the total, deanonymized dataset.
By way of an evaluation carried out earlier this 12 months, we have been in a position to present severe deficiencies throughout the dataset that basically name into query its capability to supply correct insights into traits in in-custody mortality.
This evaluation is meant to rectify a kind of issues in what, regardless of its myriad points, remains to be possible the one greatest supply for understanding in-custody loss of life in america.
Every loss of life listed within the dataset accommodates a “Method of Loss of life” discipline that shows a broad label of why somebody died — like “Pure Causes” or “Suicide” — chosen from a listing of eight pre-selected classes. Deaths additionally include a “Transient Circumstances” discipline, which accommodates a free-text description of the loss of life, starting from a single phrase to a number of paragraphs.
We found frequent mismatches between what was described in an entry’s “Transient Circumstances” and the “Method of Loss of life” that will logically comply with from that description. For instance, a number of “Transient Circumstances” describing capital punishment did not be marked as executions.
Our objective with this challenge was to make use of a suggestions loop of superior synthetic intelligence functions and handbook, human labeling to assign an correct “Method of Loss of life” to every loss of life within the dataset based mostly on what’s current within the “Transient Circumstances” discipline. The ensuing evaluation supplies each a clearer view of high-level traits of how individuals are dying in custody, in addition to an evaluation for a way incessantly the “Method of Loss of life” discipline within the dataset didn’t align with the opposite info offered.
What’s the information?
In November 2024, a web page on the web site of the Bureau of Justice Help, the workplace within the Justice Division managing the in-custody deaths information assortment, displayed a number of tables displaying high-level summarizations of the information — resembling counts of deaths by location kind and method.
Whereas the tables didn’t show this info on the particular person stage, it was attainable to click on via a sequence of menus to make the visualization instrument show the total, unredacted dataset. We downloaded that dataset on Nov. 20, 2024.
Justice Division officers didn’t reply once we requested if this publicity was intentional. Nevertheless, shortly after we downloaded the information, the web site was reconfigured to make subsequent downloads of the total dataset unattainable.
The info we downloaded contained details about 25,393 deaths that occurred in prisons, jails, neighborhood correction applications, and whereas legislation enforcement officers have been making arrests, stretching from Oct. 1, 2019 via Sept. 30, 2023.
An preliminary evaluation of the dataset that we revealed this 12 months confirmed a number of systemic issues.
Over 680 people who we all know died in legislation enforcement custody based mostly on info collected by advocacy teams or via media studies have been lacking. For comparability, we reviewed a listing of 1,847 identified deaths, largely targeted in Louisiana, Alabama and South Carolina.
When deaths have been listed, the descriptions have been typically insufficient. A random pattern of roughly 1,000 deaths discovered that, in over 75% of instances, the “Transient Circumstances” didn’t meet the Bureau of Justice Help’s personal commonplace for completeness.
For extra details about how The Marshall Challenge acquired the dataset and what it accommodates, please consult with our earlier submit on the information evaluation. The columns used for this evaluation have been “Method of Loss of life” and “Transient Circumstances.”
Evaluation
Choosing information
The dataset we downloaded from the Bureau of Justice Help web site included not solely individuals who died in prisons and jails, however individuals who died whereas being arrested by law enforcement officials or sheriff’s deputies, in addition to individuals who died whereas collaborating in neighborhood corrections applications, like in a midway home.
Since we needed our evaluation to focus solely on prisons and jails, we omitted the three,716 deaths that have been labeled as occurring throughout arrest, in neighborhood corrections, or the place the placement was unknown. That left us with 21,675 entries.
Broad categorization clustering
We began by clustering on the “Transient Circumstances” column, the place we grouped textual content strings by their similarity to 1 one other as a way to reveal normal traits within the dataset.
Embeddings flip clauses or sentences into significant vector representations in semantic area (i.e. two clauses with related meanings ought to exist in roughly the identical space of the vector area). Two “Transient Circumstances” with roughly related meanings, like “coronary heart assault” and “cardiac arrest,” must be shut in distance within the vector area. We used OpenAI’s “text-embedding-3-large” embedding mannequin to transform each entry within the “Transient Circumstances” column right into a vector with a size of three,072.
OpenAI’s mannequin doesn’t prepare on the information we entered into its system.
Subsequent, we clustered these vectors in multi-dimensional area to grasp the information’s form and roughly what number of clusters we must always create. For this process, we used a Uniform Manifold Approximation and Projection (UMAP) method to cut back the dimensionality of our area after which the HDBSCAN clustering algorithm to label every row within the dataset with an assigned cluster labeled by the algorithm. To guage the space between any two vectors, which is critical for clustering, we used the HDBSCAN’s cosine similarity metric.
We selected HDBSCAN over a Okay-means clustering algorithm, since we weren’t initially positive what number of clusters have been splendid for the dataset, which might be essential to outline when utilizing a k-means clustering algorithm. HDBSCAN is ready to decide by itself a perfect variety of clusters based mostly on the general form and distribution of the vectors throughout the dataset.
The algorithm recognized roughly 20 to 30 clusters, relying on the parameter picks for the minimal cluster measurement and the variety of neighbors anticipated. We then ran TF-IDF on all of the “Transient Circumstances” for every cluster as a way to label every cluster with a possible title. Some examples of subject labels have been “Cardiopulmonary Arrest / Cardiovascular Illness / Failure” and “Fentanyl Toxicity / Fentanyl Intoxication / Fentanyl.”
We manually reviewed the deaths assigned to every of the clusters, studying via “Transient Circumstances” to find out whether or not every loss of life belonged within the cluster by which HDBSCAN had assigned it.
As soon as we had manually gone via every cluster, we froze the entries in every cluster the human reviewer had determined belonged after which re-clustered the unfrozen entries as a way to obtain a greater understanding of the attainable titles for every cluster. We additionally tried to take away from the clustering cycles as many outcomes indicating unknown causes of loss of life or have been nonetheless pending post-mortem outcomes earlier than a reason for loss of life willpower might be made.
This step was made harder by the a number of misspellings of the phrase “unknown.”
Whereas every spherical helped our understanding of the clusters, we have been nonetheless left with a relatively broad class of “-1”, representing HDBSCAN’s reject pile of unclassified entries, which weren’t utilized in assigning cluster labels.
This course of resulted in a listing of named clusters into which we might then type the deaths within the dataset. We mapped every of the clusters we had created onto the unique “Method of Loss of life” classes outlined by the Justice Division.
Zeroshot Classification
Subsequent, we used the identical OpenAI embedding mannequin to do “zero-shot classification,” which compelled every of the “Transient Circumstances” into one of many clusters we outlined by calculating the cosine similarity between the vector of a given “Transient Circumstance” and the vector for every of the clusters and choosing the best rating.
This course of generated a spreadsheet containing every “Transient Circumstance,” its predicted cluster, the cosine similarity rating between the entry and its assigned cluster, its corrected method of loss of life, and some flags for human assessment.
We added a flag calling for a human to assessment it if:
-
The cosine similarity rating was low, lower than 0.3, indicating a weak match
-
The hole between the cluster with the best similarity rating and the cluster with the second-highest similarity rating was below 0.02, indicating that both of these two clusters might probably be a very good match for an outline of the temporary circumstance.
-
There weren’t overlapping phrases within the predicted label and the temporary circumstances.
A low similarity rating and a scarcity of overlapping phrases, for instance, was an indication to a human reviewer that the categorization is probably not correct.
Deaths within the dataset that have been flagged on this method acquired a human assessment for potential reclassification.
Entries with inadequate descriptions, like “Gunshot wound to the chest,” which might conceivably be positioned in a number of classes — like murder, use of drive by legislation enforcement or suicide — have been left to their unique ”Method of Loss of life” characterization.
Counting
As soon as we felt assured that each one entries have been categorized appropriately, we used the Python library pandas to group entries and depend the quantity in every cluster and replace the “Method of Loss of life” class as needed.
Limitations
There are a number of limitations to our evaluation that might potential have an effect on how precisely our outcomes replicate the scope of how individuals are dying in America’s prisons and jails:
-
Our prior reporting on the difficulty revealed that there are various deaths lacking from this dataset. We weren’t in a position to decide if these lacking deaths have been distributed randomly throughout the methods by which individuals died, as a result of we didn’t have a extra full dataset of in-custody deaths for comparability. If deaths of a sure kind have been systematically underreported, our outcomes might be skewed.
-
Our classifications are depending on the standard of the “Transient Circumstances” entries, which have wild variation within the stage of element current from one description to the following. As well as, some might be deliberately written to obscure particulars in regards to the loss of life that might be embarrassing or incriminating for the company holding the incarcerated particular person on the time of their loss of life.
-
Round 8,600 rows indicated the reason for loss of life was unknown as a result of a pending post-mortem or toxicology report and the knowledge was by no means subsequently up to date. If sure manners of deaths usually took longer than others to find out trigger, our outcomes might be undercounting these varieties of deaths.
-
California, which has the second-largest variety of incarcerated individuals of any state within the nation, doesn’t checklist a ”Transient Circumstances” for any deaths, as a result of state privateness guidelines. We categorized all of these deaths as being unknown of their “Method of Loss of life” as a result of it’s unattainable for us, or anybody utilizing the federal information, to evaluate the credibility of determinations based mostly on info submitted by the state.
Learn how to work with us
We have now determined to not publicly launch your complete checklist of names, as a result of privateness concerns for the households of incarcerated people. Nevertheless, if you’re a journalist or researcher considering reporting on, or researching, deaths in custody utilizing this dataset, please fill out this manner.
If you happen to’re considering studying extra about reporting on in-custody deaths, try our information for journalists, revealed as a part of The Marshall Challenge’s Examine This sequence.
Acknowledgements
Due to Jeff Kao at Bloomberg for advising on the technical points of the clustering evaluation.
