2.3 Beneficial and Harmful Effects of Computing Innovations
Key Takeaways
- Most computing innovations are dual-use: the same capability that creates a benefit can also enable a harm.
- Analyze an innovation by asking who benefits, who could be harmed, and whether each effect is social, economic, or cultural, intended or unintended.
- Algorithmic bias can come from unrepresentative training data, historical bias in the data, proxy variables such as ZIP code, and feedback loops.
- The 2018 Gender Shades study found commercial gender classifiers erred about 0.8% of the time on lighter-skinned men and up to 34.7% on darker-skinned women.
- Common trade-offs pair convenience with privacy, security with civil liberties, and automation's productivity with job displacement.
What this competency asks
ETS asks you to understand beneficial and harmful effects of computing innovations and the trade-offs between them:
- Analyze computing innovations in terms of their social, economic, and cultural impacts, both beneficial and harmful.
- Identify trade-offs between beneficial and harmful effects.
A typical item describes a new technology and asks which uses are possible, which effect is harmful, or which trade-off is involved. One ETS sample question describes an application that combines a text file with recordings of a person's voice to produce realistic speech. It asks which uses are possible, and the correct answer is all three options: re-creating an unrecorded historical speech, fabricating slanderous audio, and fooling voice-activated security. The lesson is that one capability can produce both benefits and harms.
A framework for analyzing any innovation
| Question to ask | Why it matters |
|---|---|
| What does the innovation make easier, faster, or cheaper? | Identifies the intended benefit |
| Who are the stakeholders (users, non-users, workers, communities)? | Harms often fall on people who never chose the technology |
| What are the social effects (relationships, safety, civic life)? | Communication, misinformation, polarization, isolation |
| What are the economic effects (jobs, costs, markets)? | New industries and roles; displacement of others |
| What are the cultural effects (language, art, norms)? | Global sharing, preservation, homogenization |
| What unintended uses or consequences are possible? | Misuse, dual use, and side effects |
Worked examples
| Innovation | Beneficial effects | Harmful effects |
|---|---|---|
| Social media | Connects distant family and communities; organizing and activism | Misinformation spreads quickly; harassment; privacy loss; engagement-driven feeds can amplify outrage |
| GPS and location services | Navigation, emergency location, logistics efficiency | Tracking of people without consent; location data sold to third parties |
| E-commerce | Wider selection, lower prices, access for rural and homebound shoppers | Pressure on local stores; warehouse labor conditions; packaging waste |
| Voice and image generation (AI) | Accessibility tools, creative production, restoring lost audio | Deepfakes, fraud, impersonation that defeats voice authentication |
| Facial recognition | Finding missing persons; unlocking devices | Mass surveillance; higher error rates for some groups; wrongful identification |
| Automation and robotics | Fewer injuries in dangerous jobs; higher productivity | Displacement of routine jobs; need for retraining |
| Streaming and digital media | Global access to music, film, and educational content | Weaker income for some artists; dominance of a few platforms and languages |
Algorithmic bias: an important class of harm
Algorithmic bias is systematic unfairness in the output of an automated system, often against a demographic group. It usually arises without anyone intending it.
Where bias comes from
- Unrepresentative training data. If a group is underrepresented in the data, a model learns less about that group and makes more errors on it. The Gender Shades study (Buolamwini and Gebru, 2018) tested commercial gender classifiers. They erred about 0.8% of the time for lighter-skinned men but up to 34.7% for darker-skinned women, and the benchmark data sets they examined were overwhelmingly lighter-skinned.
- Historical bias. Data can accurately record a biased past. Reuters reported in 2018 that Amazon abandoned an experimental résumé-screening model. The model had learned from ten years of mostly male hiring data and penalized résumés containing the word "women's," as in "women's chess club captain."
- Proxy variables. Removing race or sex from the inputs does not guarantee fairness, because other features such as ZIP code, school attended, or shopping patterns can correlate with them. A widely used health-risk algorithm studied by Obermeyer and colleagues (Science, 2019) used past health-care spending as a proxy for health need. Because less money had historically been spent on Black patients with the same conditions, the model underestimated their need.
- Feedback loops. A predictive-policing tool trained on past arrests sends more officers to the same neighborhoods. More patrols produce more recorded arrests there, which the tool then reads as confirmation.
Consequences
ProPublica's 2016 analysis of the COMPAS recidivism tool found that Black defendants who did not reoffend were almost twice as likely as white defendants who did not reoffend to be labeled higher risk. The tool's makers disputed the analysis. The debate illustrates a deeper point: common mathematical definitions of fairness (equal error rates versus equally accurate risk scores, for example) generally cannot all be satisfied at once when groups have different base rates. Choosing among them is an ethical and policy decision, not only a technical one.
Mitigation strategies
- Collect representative data and measure accuracy separately for each group.
- Audit outcomes before and after deployment. For example, U.S. employment guidance uses a four-fifths (80%) rule as a rough flag: a selection rate for any group below 80% of the highest group's rate suggests adverse impact.
- Include diverse, interdisciplinary teams and affected communities in design.
- Prefer explainable models for high-stakes decisions, and give people a way to understand and appeal automated decisions.
- Keep a human in the loop for consequential decisions such as benefits, hiring, and criminal justice.
Recognizing trade-offs
A trade-off exists when gaining one benefit costs something else.
| Trade-off | One side | Other side |
|---|---|---|
| Convenience vs. privacy | Free, personalized apps and services | Behavioral tracking and data sale |
| Security vs. civil liberties | Camera networks and license-plate readers help solve crimes | Loss of anonymity; chilling effects on lawful assembly |
| Productivity vs. employment | Automation lowers costs and injuries | Routine jobs disappear faster than workers can retrain |
| Speed vs. due process | Automated eligibility decisions in seconds | Errors are hard to see and slow to appeal |
| Openness vs. safety | Open platforms encourage creativity and free expression | Harder to prevent harassment and misinformation |
When a question asks for the best statement about an innovation, prefer the answer that names both a benefit and a cost. Avoid answers that claim a technology is purely good or purely harmful.
A bank removes race and ethnicity from the inputs to a credit-approval model, yet an audit shows qualified minority applicants are still denied more often than similar applicants from other groups. What is the most likely explanation?
An app can clone a person's voice from a few minutes of recordings. Which statement best describes this innovation's effects?
A city decides where to send pothole-repair crews based only on reports submitted through a smartphone app. Crews end up concentrated in wealthier neighborhoods even though roads in lower-income areas are in worse condition. What caused this outcome?