The Ontario Human Rights Commission recently released a report on racial profiling in the Toronto Police Services, and the results were quite damning. In the media there was an immediate and overwhelming consensus that the findings were beyond reproach and action must be undertaken immediately. Because of the nature of the topic no one seems willing to question the analysis. That doesn’t mean it is without flaws so I have gone through the effort of highlighting some of them below.
Before I get started I want to point out that I have no interest in disproving the findings of the report. I simply have years of experience scrutinizing statistical results and I recognize the importance of maintaining the integrity of any statistical findings. Perhaps even more-so when they are being used as a hammer. Statistics are a means to discovering truth and therefore they must be held to the highest standards.
On the Most Shocking Result
The first line in the report is especially eye-catching.
Between 2013 and 2017, a Black person in Toronto was nearly 20 times more likely than a White person to be involved in a fatal shooting by the Toronto Police Service (TPS).
To better understand this finding I’ve provided Table Seventeen from the report, the chart that it was based on. Look in the far right column on the table below labelled ‘SIU Investigation Rate’. If you divide 2.92 by 0.15 you get that distressing number that has been quoted in virtually every news story about the report.
One other very notable number in the chart is the low number of SIU investigations. Between 2013 and 2017 there was a total of 10 instances of civilian deaths caused by police shootings. It’s absolutely true that any number of civilian deaths is too many and incredibly tragic. But statistically, is this a strong enough finding to lead the report? Don’t get me wrong, the figures are true. This is essentially an attempted census of every SIU instance so the data represents exactly what happened. The question remains as to whether this particular finding meaningfully points to systemic racial profiling.
To elaborate further on this I performed two sensitivity analyses. First, I examined how much would I have to artificially increase the number of white people involved in these types of investigations in order to eliminate the difference between racial groups (the current value is ‘2’). The findings were somewhat startling. The number 2 would have to be increased to 38 to produce such a result. This does add a certain gravitas to the OHRC finding.
However when, under the same premise, I artificially adjusted the number of black people in this chart it raised further uncertainty. Even if I reduced this number to a single occurrence the analysis would still indicate that black people were nearly 3 times as likely to be involved in an altercation with police that led to a civilian death. This highlights the sensitivity problem when you have a low number of instances in your data.
Given the sensitivity of this particular analysis I don’t know how appropriate it was to lead the report with this result. If I was presenting a series of statistical analyses and had only 10 data points to back up a particular finding I would certainly frame it with a number of caveats, and I definitely wouldn’t lead with it.
A Big Assumption
An underlying assumption in this analysis is that the factors that are correlated to increased criminal activity are distributed evenly across every ethnic group residing in Toronto. In other words it assumes that, as a result of criminal activity, every ethnicity interacts with police at a ratio that is reflective of their population distribution. It is a very big assumption and doesn’t consider any of the following variables that may play a role.
- Is there a difference in the ratio of men to women as you examine each ethnic group?
- Is there a difference in socio-economic conditions?
- Do they live in disadvantaged communities?
- Are there differences in population density?
- Are there varying levels of transience?
If it was understood how each of these factors presented themselves within each ethnic group the TPS, and the OHRC, would have a much better idea of how much racial profiling is occurring. And they would also be better equipped to design an improvement plan that is catered to the most pressing need of each racial group.
Limitations of their Community Outreach
Probably the most convincing part of the report came out of discussions the OHRC had with 130 members of the black community in Toronto. They describe experiences where they are stopped without cause, undergo unjustified searches, and are arrested without merit. These certainly shed light on some disturbing practices that need to be addressed and rectified.
But is this pattern occurring across racial backgrounds or is it only happening to members of the black community? Maybe it seems obvious that these types of things happen disproportionately to the black community, but here was an opportunity to capture the data that proves it and it was neglected.
There could have been focus groups across all communities and ethnic backgrounds so they would have been equipped to make comparisons between them. Instead we are left with data from only one section of the population, as upsetting as it is. This could be a systemic issue with policing strategy, or it could be racial profiling. Again there was a missed opportunity to get the data that proves it.
Bias in the OHRC Leadership
We must hold institutions like the OHRC to a high standard so that we are informed correctly by the results of their work. There’s evidence that they were primarily focused on proving themselves right instead of delivering a statistically sound result. Here’s a quote from the OHRC chief commissioner Renu Mandhane at the launch of the inquiry on Nov 30, 2017:
We know, and the community knows, that racial profiling is real. This inquiry isn’t about establishing whether the community’s concerns are founded ….We know they are.
This goes against the mandate of the inquiry outlined on another page of the OHRC website which indicates that the question hadn’t yet been resolved:
The OHRC is inquiring into the practices and activities of the TPS between January 1, 2010 and June 30, 2017, to assess whether they are consistent with racial profiling and racial discrimination against the Black community
We have to be careful in taking these results at their face value when the study was conducted under leadership that believed they already knew what the results would be. When approaching statistics with a desired outcome firmly planted in your mind it is inevitable for that motivation to influence a large part of your work. The irony is quite thick that the OHRC claimed they knew the Toronto Police force was guilty of racial profiling before they even set out to find the proof.
As I stated at the beginning my objective here is not to dispute the existence of racial profiling. I am here first and foremost to defend the integrity of these types of exercises. As morally virtuous as the OHRC believes their objectives to be, that doesn’t exempt them from the appropriate scientific method. Because even if you are right, the damage done to the integrity of the pursuit of truth is too great a price to pay.