2008 At the beginning of the financial crisis, Cathy O’Neil was still working in the financial industry. She experienced firsthand how much people trust algorithm and how much damage it caused.
She jumped to the tech industry in frustration, but she discovered something similar: practitioners have the same blind belief in targeted advertising and mortgage-Backed Security risk assessment models. So she left.
"I don't think what we did at that time was trustworthy," she said.
A kind of "being accomplice" sentiment prompted her to write a book about the negative impact of big data on society.
The book was published in 2016, breaking the mainstream view that "algorithms are objective". She used examples after example to reveal how algorithms can aggravate social inequality.
Cathy said that before her book was published, “people didn’t really understand algorithms, they were not predictions, they were classification… It was not a mathematical problem, it was a political problem, a trust problem.”
Cathy tried to prove that many algorithms rely on historical data for training and recognition patterns, and also optimized for specific “concepts of success”: “People like you were successful in the past, so (it) tend to predict that you will succeed in the future.” Or “People like you used to fail, so you are likely to be a loser in the future.”
(Source: CHRISTOPHER CHURCHILL)
This seems to be a wise approach. But Cathy reveals how these algorithms collapse in significant, destructive ways.
For example, in prison systems, some algorithms are designed to predict the probability of a prisoner being arrested again, which can lead to unfair results, and it is usually targeted at people of color, poor people, "addicts", people living in bad communities, or those with psychological problems.
Cathy said: "For the prison system, we never defined what success is. We are just predicting and then continuing to focus on a group of people in the future because this is our past success experience.
This is very sad, and unfortunately, it also proves that we will transfer the responsibilities that should be borne by society to the victims."
Gradually, Cathy began to recognize another factor that exacerbates these inequality: shame.
"Someone has done something that he could choose not to do, and will we humiliate him because of it? For example, some people's genes determine that they are more likely to get fat, but every dietary management agency will use it to create shame.
Can you choose not to be addicted? This is much harder than you think. Have you had a chance to explain yourself? Many people are slight and cannot speak out, but we have been using this to humiliate them."
I recently had a conversation with Cathy about her new book "The Humiliation Machine: Who Makes a Profit in the New Era of Humiliation." This book delves into many ways in which shame is “weaponized” in American society and culture, and how we might fight back.
"MIT Technology Review": From algorithm to sense of shame, this is a big leap. How did you connect these two points?
Kathy O'Neal: I've studied the power behind the weaponization of algorithms. It is usually based on the idea that your knowledge is not enough to question the scientific mathematical formulas behind the algorithm. This is a humiliation in itself.
This is more obvious to me because I can still feel this way as a doctor of mathematics, which makes me confused.
The bad algorithm is a violation of trust, but it is also a humiliation: you are not even qualified to ask questions.
I interviewed a friend before. She was a principal and her teachers were evaluated by the New York City teacher value increase model.
I asked her if she could get the specific formula used to evaluate the teacher, but when she asked, she not only had to submit a lot of applications, but every time someone told her: "This is mathematics - you won't understand it."
"MIT Technology Review": In the book, you think that shame is a huge structural problem in society. Can you explain it?
Kathy O'Neal: Shame is a powerful mechanism that uses a systematic injustice to target specific goals. Some people might say, “It’s your fault (for the poor or addicted person)” or “This (the algorithm) is beyond your understanding”, giving people worthless labels are often enough to make people feel ashamed and shamed, and then stop asking questions.
As an example, I talked to a prisoner named Duane Townes, who was sentenced to jail for the second time. In prison, he was asked to work under an armed prison guard with a meager salary. If he had any complaints or had used the toilet for more than five minutes, the prison guard would inform his parole officer.
This is very embarrassing, he feels that he has not been treated like a human being. However, this is a specially designed project to train people who are serving their sentences to become "good workers".
This is equivalent to paralyzing their self-awareness and making them temporarily feel helpless and unable to defend their rights.
"MIT Technology Review": Has the COVID-19 pandemic exacerbated the problems you emphasized in your book?
Kathy O'Neal: The epidemic has brought more new norms and faster changes, including wearing masks, maintaining social distancing and getting vaccinated, so in this sense, it makes shame more common.
It is also obvious that small groups born in social media and political groups have completely different attitudes towards these changes and new norms, which has triggered a large-scale "war of shame" both online and offline.
Shame keeps those who have some differences away from each other. In other words, when the entire community lacks trust, shame can exacerbate the division: the more anger and insults the two sides express to each other, the further the distance between people.
2021, California became the first state in the United States to provide free lunch to all students, and this is not only helping vulnerable groups financially, but also helping them reduce their long-standing shame (coming from poverty).
"MIT Technology Review": How should we design a system to reduce shame? Is there a way for us to use our sense of shame to carry out social reform?
Kathy O'Neal: This is a good question! My advice is to qualify more people for benefits, or provide universal basic income for all and reduce student debt.
The United States has systematically humiliated the poor, and there is almost no unity among the poor.
This is almost entirely attributed to a series of social activities that make people feel ashamed. If we do not have a machine that can shape a “shaming” so successfully, the poor will proactively propose debt relief and universal basic income benefits.
"MIT Technology Review": The chapter on "Internet Shame" in the book discusses how companies such as Facebook, Google and other companies constantly optimize algorithms to trigger conflicts between people.How good is this for them? How can I offset it?
Kathy O'Neal: That's the result they want! If we don’t defend our sense of value from anger and then get likes and retweets through exaggerated, destructive humiliation, they won’t make a fortune.
I hope we start to see the manipulation of large tech companies as a "let us work for them for free" activity. We shouldn't do that. We should aim for higher goals, and that means targeting them.
On a personal level, if possible, we should not attack each other on social media, or even resist platforms that encourage such behavior.
At the system level, we must insist on auditing the platform's design and algorithms, and monitor toxic content.
This is not a simple suggestion. We know that Facebook tried to do this in 2018 and they found that it was feasible, but there was no profit , so they gave up.
"MIT Technology Review": You founded ORCA, an algorithm auditing company. What job does the company do?
Kathy O'Neal: At least in my company, algorithmic audit means asking a question: "Who is this algorithmic system ignoring?" The people who are ignored may be older applicants under the recruitment algorithm, or obese people in life insurance, or black people applying for student loans.
We must define the results we care about, the stakeholders who may be harmed, and what is fair. We also need to determine when the algorithm crosses the boundary threshold .
"MIT Technology Review": So, is there a "good" algorithm?
Kathy O'Neal: It depends on which field. In the recruitment field, I think it's OK, but if we don't define the outcome of the interest well, then stakeholders can be hurt.
The most important thing about it is (definition) the concept of fairness and the threshold. If none of this is done well, then we may end up with a meaningless and abused recruitment algorithm.
And in the judicial system, the confusion of criminal data is a too tricky issue, let alone unify everyone's opinions on the matter of "what is successful imprisonment".
Support: Ren
Original text:
1.https://www.technologyreview.com/2022/06/29/1053985/society-shame-book-review/
I interviewed a friend before. She was a principal and her teachers were evaluated by the New York City teacher value increase model.
I asked her if she could get the specific formula used to evaluate the teacher, but when she asked, she not only had to submit a lot of applications, but every time someone told her: "This is mathematics - you won't understand it."
"MIT Technology Review": In the book, you think that shame is a huge structural problem in society. Can you explain it?
Kathy O'Neal: Shame is a powerful mechanism that uses a systematic injustice to target specific goals. Some people might say, “It’s your fault (for the poor or addicted person)” or “This (the algorithm) is beyond your understanding”, giving people worthless labels are often enough to make people feel ashamed and shamed, and then stop asking questions.
As an example, I talked to a prisoner named Duane Townes, who was sentenced to jail for the second time. In prison, he was asked to work under an armed prison guard with a meager salary. If he had any complaints or had used the toilet for more than five minutes, the prison guard would inform his parole officer.
This is very embarrassing, he feels that he has not been treated like a human being. However, this is a specially designed project to train people who are serving their sentences to become "good workers".
This is equivalent to paralyzing their self-awareness and making them temporarily feel helpless and unable to defend their rights.
"MIT Technology Review": Has the COVID-19 pandemic exacerbated the problems you emphasized in your book?
Kathy O'Neal: The epidemic has brought more new norms and faster changes, including wearing masks, maintaining social distancing and getting vaccinated, so in this sense, it makes shame more common.
It is also obvious that small groups born in social media and political groups have completely different attitudes towards these changes and new norms, which has triggered a large-scale "war of shame" both online and offline.
Shame keeps those who have some differences away from each other. In other words, when the entire community lacks trust, shame can exacerbate the division: the more anger and insults the two sides express to each other, the further the distance between people.
2021, California became the first state in the United States to provide free lunch to all students, and this is not only helping vulnerable groups financially, but also helping them reduce their long-standing shame (coming from poverty).
"MIT Technology Review": How should we design a system to reduce shame? Is there a way for us to use our sense of shame to carry out social reform?
Kathy O'Neal: This is a good question! My advice is to qualify more people for benefits, or provide universal basic income for all and reduce student debt.
The United States has systematically humiliated the poor, and there is almost no unity among the poor.
This is almost entirely attributed to a series of social activities that make people feel ashamed. If we do not have a machine that can shape a “shaming” so successfully, the poor will proactively propose debt relief and universal basic income benefits.
"MIT Technology Review": The chapter on "Internet Shame" in the book discusses how companies such as Facebook, Google and other companies constantly optimize algorithms to trigger conflicts between people.How good is this for them? How can I offset it?
Kathy O'Neal: That's the result they want! If we don’t defend our sense of value from anger and then get likes and retweets through exaggerated, destructive humiliation, they won’t make a fortune.
I hope we start to see the manipulation of large tech companies as a "let us work for them for free" activity. We shouldn't do that. We should aim for higher goals, and that means targeting them.
On a personal level, if possible, we should not attack each other on social media, or even resist platforms that encourage such behavior.
At the system level, we must insist on auditing the platform's design and algorithms, and monitor toxic content.
This is not a simple suggestion. We know that Facebook tried to do this in 2018 and they found that it was feasible, but there was no profit , so they gave up.
"MIT Technology Review": You founded ORCA, an algorithm auditing company. What job does the company do?
Kathy O'Neal: At least in my company, algorithmic audit means asking a question: "Who is this algorithmic system ignoring?" The people who are ignored may be older applicants under the recruitment algorithm, or obese people in life insurance, or black people applying for student loans.
We must define the results we care about, the stakeholders who may be harmed, and what is fair. We also need to determine when the algorithm crosses the boundary threshold .
"MIT Technology Review": So, is there a "good" algorithm?
Kathy O'Neal: It depends on which field. In the recruitment field, I think it's OK, but if we don't define the outcome of the interest well, then stakeholders can be hurt.
The most important thing about it is (definition) the concept of fairness and the threshold. If none of this is done well, then we may end up with a meaningless and abused recruitment algorithm.
And in the judicial system, the confusion of criminal data is a too tricky issue, let alone unify everyone's opinions on the matter of "what is successful imprisonment".
Support: Ren
Original text:
1.https://www.technologyreview.com/2022/06/29/1053985/society-shame-book-review/
