In the wake of the ongoing kerfuffle between the American Civil Liberties Union and Amazon, the tech giant has now responded more fully to accusations that its facial recognition system, Rekognition, wrongly identified 28 members of Congress as arrestees in an ACLU test last week.
Matt Wood, Amazon’s general manager for deep learning and artificial intelligence, wrote in a blog post last Friday that while the ACLU used the default setting of an 80-percent confidence level, the company recommends a far higher setting "for use cases where highly accurate face similarity matches are important."
"We continue to recommend that customers do not use less than 99 percent confidence levels for law enforcement matches, and then to only use the matches as one input across others that make sense for each agency," he wrote.
Wood went on to explain that Amazon tried to duplicate the ACLU’s test involving portraits of the 535 members of Congress. However, Amazon used a far larger dataset of faces for comparison: 850,000 over the ACLU’s 25,000 mugshots.
"When we set the confidence threshold at 99 percent (as we recommend in our documentation), our misidentification rate dropped to zero despite the fact that we are comparing against a larger corpus of faces (30x larger than the ACLU test)," he wrote.
"This illustrates how important it is for those using the technology for public safety issues to pick appropriate confidence levels, so they have few (if any) false positives."
FURTHER READING
The Amazon scientist said that Rekognition is meant to help law enforcement.
"While being concerned it’s applied correctly, we should not throw away the oven because the temperature could be set wrong and burn the pizza," Wood concluded. "It is a very reasonable idea, however, for the government to weigh in and specify what temperature (or confidence levels) it wants law enforcement agencies to meet to assist in their public safety work."
Amazon still has not said which law enforcement agencies are evaluating or using the Rekognition service, something that members of Congress have recently demanded.
Also last Friday, Sen. Ron Wyden (D-Ore.), Sen. Cory Booker (D-N.J.), and Sen. Ed Markey (D-Mass.) sent a letter to 39 federal law enforcement agencies asking about their use of the controversial technology.
For its part, Abdullah Hasan, an ACLU spokesman, said that the precise confidence level setting is essentially missing the forest for the trees, as the advocacy group outlined in its own July 27 blog post: "Because that argument attempts to sidestep the very real civil rights concerns raised by Amazon actively marketing Rekognition to law enforcement agencies," Hasan emailed Ars. "I will mention that per a blog post Amazon and Washington County wrote together instructing people on how to use Rekognition to find people in arrest photos, (just as the ACLU did in its test), they set a confidence threshold of 85 percent (not 95 percent and certainly not 99 percent)."
Copyright: https://arstechnica.com/tech-policy/2018/07/amazon-cops-should-set-confidence-level-on-facial-recognition-to-99/