Algorithms run everything from Uber to advertising. They’re used to sift through our CVs, check our credit and decide whether we get health insurance. But when they turn into “black boxes” that don’t offer up their secrets, we can’t hold them accountable.
Only now is this issue starting to be taken seriously. The London School of Economics recently introduced a masters degree in data and society, aiming to understand the human consequences of a data-driven society.
“Our own lives are open books,” said Frank Pasquale, law professor at the University of Maryland, at the launch of the new degree course. “While those firms who would scrutinise us are black boxes.”
Look at Uber, Pasquale suggested. When drivers fall below a 4.6 user rating, they could lose their job — but they’re given little information about how that’s calculated. There’s no recourse, the algorithm simply crunches the numbers. “The overall system isn’t rendered accountable, but the drivers are, in some ways, rendered hyper-accountable through constant surveillance,” he added.
Algorithms aren’t inherently evil: they’re tools used to simplify decisions, increase efficiency and offer convenience. But when they’re locked away we can’t understand how they work or even if they work at all. Consider data accuracy: if a data broker includes a false fact in a person’s profile, or its analysis makes an incorrect assumption, it could negatively affect someone’s life.
Pasquale pointed to assumptions made by social sciences researchers, including the belief that if you own a large vehicle but have no children, you’re more likely to be overweight. If a healthinsurer purchases that data package on you — likely paying pennies to get it — then good luck getting a fair deal on your insurance.
And if that data is incorrect (the wrong car model) or a false assumption (you inherited the car, or use it to haul sports equipment), there’s no way of correcting it. “One casual slur about you could end up in a random database without your knowledge,” he said. “Then, go on to populate hundreds of other digital dossiers or into a report on your health status, competence or criminal record. This new digital underground can ruin reputations.”
It needn’t be that way. “The data sharing infrastructure is built such that it can so easily propagate […] but they didn’t build into the architecture the ability to bring that [data] back if they need it.”
It’s defective by design, Pasquale argued, to build infrastructure that can slur people but not issue corrections. “Future reputation systems must enable the removal of stigma as fast as it spreads. This is not an insolvable problem.” The right to be forgotten, the EU’s controversial attempt to allow people to remove irrelevant personal information from search results, attempts to address part of this idea.
Algorithms can also be used to hide faults. A company accused of racist hiring policies may point to its CV sorting software, saying HR or executives can’t be at fault as it’s the machine that did the work.
But algorithms can be encoded with our own discrimination, particularly via “big data proxies”. Rather than filtering out people by skin colour or background, it removes CVs from certain neighbourhoods or with other indirect signifiers. “They can just encode the legal violation into code that’s very difficult for regulators to inspect – as we saw with Volkswagen,” Pasquale said.
RECIPE FOR DISASTER
The immediate response to a “black box” may well be transparency and consent: let’s crack algorithms open and approve each use of data. But companies will argue such technology is a trade secret.
“These types of information flows can be very complex,” Pasquale explained, showing a convoluted slide tracking data travelling from publishers to advertisers, via data brokers, marketing agencies, social media and so on. It’s impossible for individuals to keep track of all these “galaxies” of data. “If the regulatory model is built on notice and consent of where your data is going, given current practises of online sharing that makes things very, very difficult.”
Understanding an algorithm’s stated intent might be simple, but understanding how it’s expressed in code and what outcomes it causes will require being able to read and understand the language it’s written in. That’s tough for regulators, let alone the rest of us. “Code is the operationalisation of an algorithm,” said Evelyn Ruppert, sociology professor at Goldsmiths College. “It’s hard to even find where the algorithm is in a programme.”
According to Pasquale, such action lets us pry open the black box and get a better view. “The fact that we have such law creates certain leverage that creates opportunities for social scientists, journalists and researchers to understand what’s going on,” he said. “Oftentimes, it’s this very symbiotic relationship between lawyers and social scientists, between social scientists and journalists… to maintain a flow of information about things that could have been black boxes.”
Read more: http://www.wired.co.uk/news/archive/2016-01/29/make-algorithms-accountable