WEBVTT

1
00:00:00.000 --> 00:00:34.210
 

2
00:00:34.230 --> 00:00:37.310
Vivek: Good morning, welcome to the Forty Eighth

3
00:00:37.330 --> 00:00:40.980
New York Area Science and Technology form.

4
00:00:41.000 --> 00:00:43.170
The webinar is hosted by the

5
00:00:43.190 --> 00:00:44.630
Department of Homeland Security's,

6
00:00:44.650 --> 00:00:46.460
National Urban Security Laboratory,

7
00:00:46.480 --> 00:00:47.459
pronounced new steal.

8
00:00:47.479 --> 00:00:50.810
My name is Vivek Agnish and I'm the test

9
00:00:50.830 --> 00:00:53.654
and evaluation division director for First

10
00:00:53.674 --> 00:00:55.510
Responder Technologies at NUSTL.

11
00:00:55.530 --> 00:00:56.947
Our NYAST forums

12
00:00:56.967 --> 00:00:59.821
Provides a platform for cross collaboration

13
00:00:59.841 --> 00:01:03.250
and information exchange between federal,

14
00:01:03.270 --> 00:01:05.334
state, and local government,

15
00:01:05.354 --> 00:01:06.380
emergency responders,

16
00:01:06.400 --> 00:01:07.842
academic institutions,

17
00:01:07.862 --> 00:01:10.770
and private sector groups.

18
00:01:10.790 --> 00:01:12.640
New steel started denious membership

19
00:01:12.660 --> 00:01:15.286
back in two thousand four and since

20
00:01:15.306 --> 00:01:17.422
then we've hosted dozens of forms

21
00:01:17.442 --> 00:01:19.857
addressing some of the most pressing

22
00:01:19.877 --> 00:01:21.940
topics in Homeland Security applications.

23
00:01:21.960 --> 00:01:24.628
Our topic today is the one that is

24
00:01:24.648 --> 00:01:26.254
constantly evolving the advantages

25
00:01:26.274 --> 00:01:28.480
of biometrics and face recognition.

26
00:01:28.500 --> 00:01:30.425
Technology is gaining more popularity

27
00:01:30.445 --> 00:01:33.100
day by day all around the world,

28
00:01:33.120 --> 00:01:35.410
and these systems are incredibly complex.

29
00:01:35.430 --> 00:01:37.780
Driving need for government testing

30
00:01:37.800 --> 00:01:40.150
of new scenarios and capabilities.

31
00:01:40.170 --> 00:01:42.350
Our speakers will examine the technical,

32
00:01:42.370 --> 00:01:43.962
operational and testing considerations

33
00:01:43.982 --> 00:01:46.440
and address some of the concerns

34
00:01:46.460 --> 00:01:47.860
that come with the wide

35
00:01:47.880 --> 00:01:50.040
spread, adoption of biometric,

36
00:01:50.060 --> 00:01:51.680
and face recognitions.

37
00:01:51.700 --> 00:01:54.040
Since the specific NYAST forum  is webinar only,

38
00:01:54.060 --> 00:01:56.032
we will interact through the chat

39
00:01:56.052 --> 00:01:58.790
box on the left side of your screen.

40
00:01:58.810 --> 00:02:01.170
Please start typing in the chat

41
00:02:01.190 --> 00:02:03.940
box as you think of any questions.

42
00:02:03.960 --> 00:02:06.348
My fellow MC Abby Hooper will

43
00:02:06.368 --> 00:02:08.653
call out questions to our speakers

44
00:02:08.673 --> 00:02:10.663
throughout their presentations if we

45
00:02:10.683 --> 00:02:13.602
have time at the end we will take

46
00:02:13.622 --> 00:02:19.640
additional questions from the chat pod.

47
00:02:19.660 --> 00:02:22.004
Our first speaker today.

48
00:02:22.024 --> 00:02:27.060
are  Arun Vemury and John Howard.

49
00:02:27.080 --> 00:02:29.909
Who are from our very own DHS

50
00:02:29.929 --> 00:02:32.150
science and Technology Directorate.

51
00:02:32.170 --> 00:02:35.190
Arun Vemury is DHS S&T's 

52
00:02:35.210 --> 00:02:37.149
Biometric and identity

53
00:02:37.169 --> 00:02:39.110
Technology center director.

54
00:02:39.130 --> 00:02:41.530
The center supports robust testing

55
00:02:41.550 --> 00:02:44.546
and evaluation at its Maryland test

56
00:02:44.566 --> 00:02:47.151
facility to inform applicants of

57
00:02:47.171 --> 00:02:49.312
biometric technology to specific

58
00:02:49.332 --> 00:02:51.770
operational use cases across DHS.

59
00:02:51.790 --> 00:02:54.540
And the Homeland Security community.

60
00:02:54.560 --> 00:02:55.992
As a center's director,

61
00:02:56.012 --> 00:02:58.235
Arun leads to developmental of

62
00:02:58.255 --> 00:03:00.560
Bleeding Edge and cost effective

63
00:03:00.580 --> 00:03:02.949
biometric identity and privacy enhancing

64
00:03:02.969 --> 00:03:04.860
technologies and capabilities.

65
00:03:04.880 --> 00:03:07.842
John Howard is a computer scientist

66
00:03:07.862 --> 00:03:09.900
specializing in biometrics research

67
00:03:09.920 --> 00:03:11.860
and is a fellow at the SMU AT

68
00:03:11.880 --> 00:03:14.240
and T Center for Virtualization.

69
00:03:14.260 --> 00:03:17.036
John has served as the principle

70
00:03:17.056 --> 00:03:18.978
investigation on numeral research

71
00:03:18.998 --> 00:03:21.062
and development efforts across

72
00:03:21.082 --> 00:03:23.150
the industry and government.

73
00:03:23.170 --> 00:03:25.580
His work is regularly cited by

74
00:03:25.600 --> 00:03:28.149
media outlets such as New York

75
00:03:28.169 --> 00:03:30.320
Times and The Washington Post.

76
00:03:30.340 --> 00:03:33.078
He is currently a principal scientist and

77
00:03:33.098 --> 00:03:36.230
the lead data scientists at DHS S&T's

78
00:03:36.250 --> 00:03:37.415
Maryland test facility.

79
00:03:37.435 --> 00:03:37.810
Together,

80
00:03:37.830 --> 00:03:40.239
our speakers will present a high

81
00:03:40.259 --> 00:03:42.479
level overview of face recognition

82
00:03:42.499 --> 00:03:43.830
and biometric components.

83
00:03:43.850 --> 00:03:45.950
Different types of testing

84
00:03:45.970 --> 00:03:47.540
and common misconceptions.

85
00:03:47.560 --> 00:03:48.236
Over to you,

86
00:03:48.256 --> 00:03:54.230
John.

87
00:03:54.250 --> 00:04:01.770
[silence]

88
00:04:01.790 --> 00:04:03.870
John Howard: OK, thank you for that introduction

89
00:04:03.890 --> 00:04:05.950
and thank you everyone for joining

90
00:04:05.970 --> 00:04:08.106
us today I'm John Howard and

91
00:04:08.126 --> 00:04:09.834
select mentioned computer scientists

92
00:04:09.854 --> 00:04:12.100
at the Maryland test facility.

93
00:04:12.120 --> 00:04:16.200
And today we're going to talk about.

94
00:04:16.220 --> 00:04:17.646
How we use faces to establish

95
00:04:17.666 --> 00:04:19.012
a person 's identity process

96
00:04:19.032 --> 00:04:20.780
generally known as face recognition?

97
00:04:20.800 --> 00:04:22.376
I'm happy if you want to

98
00:04:22.396 --> 00:04:27.420
go to the next slide.

99
00:04:27.440 --> 00:04:28.855
So face recognition systems have

100
00:04:28.875 --> 00:04:31.080
seen sort of an uptick in deployment,

101
00:04:31.100 --> 00:04:32.610
not just in EU S,

102
00:04:32.630 --> 00:04:34.130
But really around the world,

103
00:04:34.150 --> 00:04:35.960
and particularly in sort of law

104
00:04:35.980 --> 00:04:37.180
enforcement and travel context.

105
00:04:37.200 --> 00:04:38.690
They're all hosted reasons why

106
00:04:38.710 --> 00:04:39.898
face recognition systems seem

107
00:04:39.918 --> 00:04:41.450
to be gaining in popularity,

108
00:04:41.470 --> 00:04:44.195
but at least one sort of, in my opinion,

109
00:04:44.215 --> 00:04:46.330
is that humans do this as well.

110
00:04:46.350 --> 00:04:48.130
We look at other peoples faces

111
00:04:48.150 --> 00:04:49.990
and we know who they are.

112
00:04:50.010 --> 00:04:52.120
We do this, you know, her friends,

113
00:04:52.140 --> 00:04:53.622
foes, wives, children, parents, etc.

114
00:04:53.642 --> 00:04:55.659
So it's kind of natural for us

115
00:04:55.679 --> 00:04:57.760
to think that maybe a computer.

116
00:04:57.780 --> 00:04:59.390
Can do the same thing,

117
00:04:59.410 --> 00:05:01.588
but in order to get the computer

118
00:05:01.608 --> 00:05:03.300
to recognize who someone is,

119
00:05:03.320 --> 00:05:05.260
it's actually a fairly complicated process,

120
00:05:05.280 --> 00:05:07.339
and that's kind of what you see

121
00:05:07.359 --> 00:05:09.358
here on the first slide that

122
00:05:09.378 --> 00:05:11.130
we wanted to go through,

123
00:05:11.150 --> 00:05:13.320
it starts with an unknown person

124
00:05:13.340 --> 00:05:15.115
or someone whose identity you

125
00:05:15.135 --> 00:05:16.990
want to know here on the left,

126
00:05:17.010 --> 00:05:18.971
and the first thing we really need

127
00:05:18.991 --> 00:05:21.698
to do is get a digital picture of

128
00:05:21.718 --> 00:05:23.510
that person into the computer.

129
00:05:23.530 --> 00:05:26.030
At some level that's going to involve a

130
00:05:26.050 --> 00:05:28.490
camera. The camera produces face sample.

131
00:05:28.510 --> 00:05:30.690
Come.

132
00:05:30.710 --> 00:05:33.420
And feed that into sort of my area,

133
00:05:33.440 --> 00:05:35.120
which is a an algorithm.

134
00:05:35.140 --> 00:05:36.825
Algorithm creates what we call

135
00:05:36.845 --> 00:05:37.850
a biometric template.

136
00:05:37.870 --> 00:05:40.434
This is sort of a mathematical form of

137
00:05:40.454 --> 00:05:43.310
what a face looks like to a computer.

138
00:05:43.330 --> 00:05:45.010
Some sort of, you know,

139
00:05:45.030 --> 00:05:46.720
maybe this nice face sample,

140
00:05:46.740 --> 00:05:48.850
the information that's over here is

141
00:05:48.870 --> 00:05:50.810
something like distance between the eyes,

142
00:05:50.830 --> 00:05:52.170
or distance between particular

143
00:05:52.190 --> 00:05:53.880
person nose and their eyes.

144
00:05:53.900 --> 00:05:55.585
Once we have this template

145
00:05:55.605 --> 00:05:56.950
from an unknown person,

146
00:05:56.970 --> 00:05:59.470
we compare it to other templates

147
00:05:59.490 --> 00:06:01.930
in a database or gallery.

148
00:06:01.950 --> 00:06:04.560
And people in the gallery usually have

149
00:06:04.580 --> 00:06:06.330
other information associated with them,

150
00:06:06.350 --> 00:06:08.890
so something like a name and address

151
00:06:08.910 --> 00:06:11.167
a Social Security number and what

152
00:06:11.187 --> 00:06:13.369
the computer is suggesting when it

153
00:06:13.389 --> 00:06:15.566
maybe finds a match between this

154
00:06:15.586 --> 00:06:17.694
unknown template and one in the

155
00:06:17.714 --> 00:06:19.464
gallery is that that information

156
00:06:19.484 --> 00:06:21.699
in the gallery should also be

157
00:06:21.719 --> 00:06:23.580
associated with the unknown person.

158
00:06:23.600 --> 00:06:25.362
All of that ideally goes into

159
00:06:25.382 --> 00:06:27.415
some kind of report suggestion and

160
00:06:27.435 --> 00:06:29.010
then hopefully some information

161
00:06:29.030 --> 00:06:31.410
about how that face recognition

162
00:06:31.430 --> 00:06:33.396
program came to this conclusion in

163
00:06:33.416 --> 00:06:35.699
that gets picked up by yet another

164
00:06:35.719 --> 00:06:37.535
human who's sort decides what to

165
00:06:37.555 --> 00:06:39.500
do with that new information.

166
00:06:39.520 --> 00:06:41.708
So it's a high level overview of sort

167
00:06:41.728 --> 00:06:44.220
of how face recognition systems work.

168
00:06:44.240 --> 00:06:50.210
We can go to the next slide.

169
00:06:50.230 --> 00:06:53.800
[Silence]

170
00:06:53.820 --> 00:06:56.200
So hopefully you know that all made sense,

171
00:06:56.220 --> 00:06:58.090
and really the takeaway I kind of

172
00:06:58.110 --> 00:06:59.723
wanted people to come away with

173
00:06:59.743 --> 00:07:01.559
from that last slide is it based

174
00:07:01.579 --> 00:07:03.400
recognition is fairly complicated.

175
00:07:03.420 --> 00:07:05.104
It may seem simple because sort

176
00:07:05.124 --> 00:07:07.230
of human brains do this so often

177
00:07:07.250 --> 00:07:08.800
and so easily almost every day,

178
00:07:08.820 --> 00:07:09.583
probably every day.

179
00:07:09.603 --> 00:07:11.149
But to get a computer to

180
00:07:11.169 --> 00:07:12.700
do that it's complicated.

181
00:07:12.720 --> 00:07:13.956
And because it's complicated,

182
00:07:13.976 --> 00:07:16.600
things can go wrong and they can go wrong.

183
00:07:16.620 --> 00:07:18.100
Sort of at multiple different

184
00:07:18.120 --> 00:07:19.600
places in that whole process.

185
00:07:19.620 --> 00:07:22.000
The first place we see things go wrong,

186
00:07:22.020 --> 00:07:23.640
and I'm sure this is.

187
00:07:23.660 --> 00:07:25.872
Familiar to at least a few folks on this

188
00:07:25.892 --> 00:07:28.220
call is the cameras don't always get the

189
00:07:28.240 --> 00:07:30.216
nicest picture of people's faces, right?

190
00:07:30.236 --> 00:07:31.650
Maybe it's a CCTV camera.

191
00:07:31.670 --> 00:07:32.790
The angle is poor,

192
00:07:32.810 --> 00:07:34.220
or maybe it's at night.

193
00:07:34.240 --> 00:07:36.264
Got a little light, or maybe someone

194
00:07:36.284 --> 00:07:38.230
just wearing a hat or COVID mask.

195
00:07:38.250 --> 00:07:40.318
All of these things are going to cause

196
00:07:40.338 --> 00:07:42.230
problems for a face recognition system.

197
00:07:42.250 --> 00:07:44.120
The second spot that things can go

198
00:07:44.140 --> 00:07:46.230
wrong is really at the algorithm level.

199
00:07:46.250 --> 00:07:48.520
This is where you see most people think.

200
00:07:48.540 --> 00:07:50.362
The vast majority of problems lives

201
00:07:50.382 --> 00:07:51.950
where you've always hear a lot.

202
00:07:51.970 --> 00:07:54.410
You know it's a problem with the algorithms.

203
00:07:54.430 --> 00:07:55.679
From the algorithm,

204
00:07:55.699 --> 00:07:57.794
that's really just one piece

205
00:07:57.814 --> 00:07:59.760
that can have an issue.

206
00:07:59.780 --> 00:08:01.335
The problems you can encounter

207
00:08:01.355 --> 00:08:02.600
here are two fold.

208
00:08:02.620 --> 00:08:05.130
So say you have two pictures of me.

209
00:08:05.150 --> 00:08:07.258
One may be taken today and one

210
00:08:07.278 --> 00:08:09.240
from five or ten years ago.

211
00:08:09.260 --> 00:08:11.034
A computer might think that you

212
00:08:11.054 --> 00:08:13.030
know those don't look very similar.

213
00:08:13.050 --> 00:08:14.610
Maybe I've aged, maybe I've

214
00:08:14.630 --> 00:08:16.190
something how changed my appearance.

215
00:08:16.210 --> 00:08:19.470
Going to beard new haircut.

216
00:08:19.490 --> 00:08:20.965
Computer thinks that two pictures

217
00:08:20.985 --> 00:08:23.090
of the same person don't look alike.

218
00:08:23.110 --> 00:08:26.010
We call that a false non match.

219
00:08:26.030 --> 00:08:28.230
Which is this first error right here?

220
00:08:28.250 --> 00:08:28.800
And then,

221
00:08:28.820 --> 00:08:29.085
secondly,

222
00:08:29.105 --> 00:08:31.489
probably more of an issue from a law

223
00:08:31.509 --> 00:08:33.244
enforcement context is when computer

224
00:08:33.264 --> 00:08:35.363
thinks the two pictures of different

225
00:08:35.383 --> 00:08:37.010
people are actually the same.

226
00:08:37.030 --> 00:08:39.056
So maybe computer algorithm says that

227
00:08:39.076 --> 00:08:42.024
a picture of me and a picture of my

228
00:08:42.044 --> 00:08:43.830
brother look enough alike to be.

229
00:08:43.850 --> 00:08:47.180
You know, the computer thinks it's the same person.

230
00:08:47.200 --> 00:08:48.965
We call that a false match in

231
00:08:48.985 --> 00:08:50.598
determining sort of why and when

232
00:08:50.618 --> 00:08:51.140
that happens.

233
00:08:51.160 --> 00:08:53.228
It's something we're going to talk

234
00:08:53.248 --> 00:08:56.400
about a little bit later in the presentation.

235
00:08:56.420 --> 00:08:57.363
And then finally,

236
00:08:57.383 --> 00:08:59.289
people often forget that there's this

237
00:08:59.309 --> 00:09:01.147
third sort of piece of the puzzle

238
00:09:01.167 --> 00:09:03.140
that I went through in the last slide,

239
00:09:03.160 --> 00:09:05.106
the image gallery. So the databases, right?

240
00:09:05.126 --> 00:09:06.486
So face recognition systems aren't

241
00:09:06.506 --> 00:09:08.486
going to be able to match people

242
00:09:08.506 --> 00:09:09.890
that they've never seen before.

243
00:09:09.910 --> 00:09:10.416
For example,

244
00:09:10.436 --> 00:09:12.257
you could also have sort of data

245
00:09:12.277 --> 00:09:13.820
integrity problems in your database.

246
00:09:13.840 --> 00:09:15.510
Maybe it's a picture of me,

247
00:09:15.530 --> 00:09:17.710
but it's a sign that somebody

248
00:09:17.730 --> 00:09:19.900
else's is name and address.

249
00:09:19.920 --> 00:09:21.652
Or maybe you know this data

250
00:09:21.672 --> 00:09:22.820
has been corrupted somehow.

251
00:09:22.840 --> 00:09:25.756
All of those are going to cause issues

252
00:09:25.776 --> 00:09:28.710
with face recognition systems as well.

253
00:09:28.730 --> 00:09:34.190
Next slide.

254
00:09:34.210 --> 00:09:37.140
Uh, OK. So first slide sort of said you know,

255
00:09:37.160 --> 00:09:38.320
face recognition systems are

256
00:09:38.340 --> 00:09:39.500
complicated pieces of technology.

257
00:09:39.520 --> 00:09:40.960
Second slide says here's you

258
00:09:40.980 --> 00:09:42.772
know high level overview of think

259
00:09:42.792 --> 00:09:44.520
places where things can go wrong.

260
00:09:44.540 --> 00:09:46.290
This slide sort answers the question.

261
00:09:46.310 --> 00:09:48.060
How do we know all that?

262
00:09:48.080 --> 00:09:50.823
And the answer is we have lots of different

263
00:09:50.843 --> 00:09:53.253
kinds of tests we use to measure the

264
00:09:53.273 --> 00:09:55.710
performance of a face recognition system.

265
00:09:55.730 --> 00:09:57.890
The first is called a technology test.

266
00:09:57.910 --> 00:10:00.380
This is when you take just the algorithm.

267
00:10:00.400 --> 00:10:02.390
piece out of that larger workflow

268
00:10:02.410 --> 00:10:04.359
shown on slide three and using

269
00:10:04.379 --> 00:10:06.600
sort of a fixed set of face images.

270
00:10:06.620 --> 00:10:08.466
You answer those questions I showed

271
00:10:08.486 --> 00:10:10.330
in the last slide, right? How?

272
00:10:10.350 --> 00:10:12.500
What is the false non match rate?

273
00:10:12.520 --> 00:10:14.370
What is the false match rate?

274
00:10:14.390 --> 00:10:16.683
You can do this sort of again and again

275
00:10:16.703 --> 00:10:18.852
for different algorithms and you can

276
00:10:18.872 --> 00:10:21.355
maybe from a different vendor in answer

277
00:10:21.375 --> 00:10:23.406
the question who seems to have the

278
00:10:23.426 --> 00:10:25.960
best performance and you can also do this.

279
00:10:25.980 --> 00:10:28.468
Year after year and sort of

280
00:10:28.488 --> 00:10:30.140
track performance over time.

281
00:10:30.160 --> 00:10:33.070
There's a really nice characteristic.

282
00:10:33.090 --> 00:10:34.890
At NIST, National Institute of

283
00:10:34.910 --> 00:10:36.767
Standards and Technology is probably

284
00:10:36.787 --> 00:10:38.363
the sort of world authority on

285
00:10:38.383 --> 00:10:39.920
doing this kind of testing.

286
00:10:39.940 --> 00:10:41.870
When it comes to face recognition,

287
00:10:41.890 --> 00:10:43.500
I think that's what Patrick

288
00:10:43.520 --> 00:10:45.130
is going to talk about.

289
00:10:45.150 --> 00:10:45.786
Just shortly,

290
00:10:45.806 --> 00:10:48.082
the other kind of testing or another

291
00:10:48.102 --> 00:10:50.228
kind of testing is what we actually

292
00:10:50.248 --> 00:10:52.300
do at the Maryland test facility,

293
00:10:52.320 --> 00:10:54.260
and it's this scenario testing option.

294
00:10:54.280 --> 00:10:56.220
That's when we take full system.

295
00:10:56.240 --> 00:10:58.621
So not just the algorithm component that

296
00:10:58.641 --> 00:11:00.939
also camera in the database and we

297
00:11:00.959 --> 00:11:03.490
run them through a sort of real life.

298
00:11:03.510 --> 00:11:04.698
People against simulated workflow,

299
00:11:04.718 --> 00:11:07.227
so for us at the moment as facility

300
00:11:07.247 --> 00:11:08.867
usually focused on travel environment

301
00:11:08.887 --> 00:11:10.834
this could be we simulate something

302
00:11:10.854 --> 00:11:12.680
like an airport or land border.

303
00:11:12.700 --> 00:11:14.265
These are really nice because

304
00:11:14.285 --> 00:11:15.220
they multiple systems,

305
00:11:15.240 --> 00:11:17.596
so you sort of get a more close

306
00:11:17.616 --> 00:11:19.583
to real-world idea of what the

307
00:11:19.603 --> 00:11:21.240
error rates would look like,

308
00:11:21.260 --> 00:11:22.825
and they involve real people

309
00:11:22.845 --> 00:11:24.410
and sort of new data,

310
00:11:24.430 --> 00:11:26.642
so it's sort of new questions come up

311
00:11:26.662 --> 00:11:29.170
about how a face recognition system works.

312
00:11:29.190 --> 00:11:31.270
You have that opportunity to sort

313
00:11:31.290 --> 00:11:33.202
of refresh your data holdings

314
00:11:33.222 --> 00:11:35.070
and answer those questions.

315
00:11:35.090 --> 00:11:35.366
Finally,

316
00:11:35.386 --> 00:11:38.030
you know the last kind of testing down here,

317
00:11:38.050 --> 00:11:39.510
the bottom is operational testing.

318
00:11:39.530 --> 00:11:40.910
Both scenario and technology testing

319
00:11:40.930 --> 00:11:42.991
are usually done in labs or conducted

320
00:11:43.011 --> 00:11:44.250
by engineers and scientists.

321
00:11:44.270 --> 00:11:44.762
They're great,

322
00:11:44.782 --> 00:11:46.873
but at the end of the day there's

323
00:11:46.893 --> 00:11:48.538
not really a substitute for

324
00:11:48.558 --> 00:11:49.870
just answering the question.

325
00:11:49.890 --> 00:11:51.942
You know how well is this working

326
00:11:51.962 --> 00:11:52.830
at JFK airport,

327
00:11:52.850 --> 00:11:54.900
or how well is this working on?

328
00:11:54.920 --> 00:11:57.108
Sort of the same and table were crossing

329
00:11:57.128 --> 00:11:59.340
these kind of tests are really necessary.

330
00:11:59.360 --> 00:12:01.116
They can also be a challenge

331
00:12:01.136 --> 00:12:02.970
because data in the real world

332
00:12:02.990 --> 00:12:04.520
is much messier than in

333
00:12:04.540 --> 00:12:07.770
Our labs.

334
00:12:07.790 --> 00:12:11.920
Next slide.

335
00:12:11.940 --> 00:12:14.683
Uh, OK. So just a couple more slides here

336
00:12:14.703 --> 00:12:17.618
and then I think we'll open it up for

337
00:12:17.638 --> 00:12:19.970
questions and turn it over to Patrick,

338
00:12:19.990 --> 00:12:21.440
but you've probably seen information

339
00:12:21.460 --> 00:12:23.336
in the news or social media

340
00:12:23.356 --> 00:12:24.800
about face recognition system.

341
00:12:24.820 --> 00:12:26.410
It's becoming fairly popular topic,

342
00:12:26.430 --> 00:12:28.340
and some of it is accurate.

343
00:12:28.360 --> 00:12:29.950
Sort of well thought out,

344
00:12:29.970 --> 00:12:32.560
but there's also a lot of misconceptions out

345
00:12:32.580 --> 00:12:35.100
there and misconceptions that we sort of bring.

346
00:12:35.120 --> 00:12:38.000
Some of these up and sort of address them.

347
00:12:38.020 --> 00:12:40.359
The first mistake we see people make

348
00:12:40.379 --> 00:12:42.290
fairly frequently actually is too confused.

349
00:12:42.310 --> 00:12:44.943
face recognition with a different kind of

350
00:12:44.963 --> 00:12:47.470
computer program that we call base analysis,

351
00:12:47.490 --> 00:12:49.732
so face recognition is a computer

352
00:12:49.752 --> 00:12:52.009
program is really designed to answer

353
00:12:52.029 --> 00:12:54.130
one question and one question only,

354
00:12:54.150 --> 00:12:57.578
and that's who is this right now hasn't

355
00:12:57.598 --> 00:13:00.640
been answer like this is John Howard.

356
00:13:00.660 --> 00:13:02.400
There's other kinds of computer

357
00:13:02.420 --> 00:13:04.535
programs that do things like guess

358
00:13:04.555 --> 00:13:06.400
someone 's gender or their age,

359
00:13:06.420 --> 00:13:08.434
or if they're happy or not

360
00:13:08.454 --> 00:13:09.790
in that particular photo.

361
00:13:09.810 --> 00:13:12.261
All sorts of kind of characteristics that

362
00:13:12.281 --> 00:13:15.220
you think we can infer from a face image.

363
00:13:15.240 --> 00:13:17.932
That's a much harder task than the work

364
00:13:17.952 --> 00:13:20.640
through that I went through on slide three,

365
00:13:20.660 --> 00:13:22.674
which was just to figure out

366
00:13:22.694 --> 00:13:24.030
someone 's individual identity,

367
00:13:24.050 --> 00:13:26.798
but we often have people that point the

368
00:13:26.818 --> 00:13:29.485
studies of high error rates in sort of

369
00:13:29.505 --> 00:13:31.612
this age, gender emotion classification.

370
00:13:31.632 --> 00:13:33.388
And say, oh, there,

371
00:13:33.408 --> 00:13:36.540
it must exist in face recognition as well.

372
00:13:36.560 --> 00:13:38.100
That's not really the case.

373
00:13:38.120 --> 00:13:39.650
It's not really even close.

374
00:13:39.670 --> 00:13:41.205
Their rates on face recognition

375
00:13:41.225 --> 00:13:42.760
much lower than those tasks,

376
00:13:42.780 --> 00:13:44.315
but it's probably the most

377
00:13:44.335 --> 00:13:44.940
often misconception.

378
00:13:44.960 --> 00:13:47.700
We sort of hear about face

379
00:13:47.720 --> 00:13:49.540
recognition as a whole.

380
00:13:49.560 --> 00:13:50.083
Uh,

381
00:13:50.103 --> 00:13:54.560
next slide.

382
00:13:54.580 --> 00:13:56.160
Uh, OK. Misconception number two,

383
00:13:56.180 --> 00:13:58.078
we sometimes see out there in the

384
00:13:58.098 --> 00:13:59.980
wild is that face recognition?

385
00:14:00.000 --> 00:14:02.305
Is this untested technology and

386
00:14:02.325 --> 00:14:04.630
therefore we shouldn't use it?

387
00:14:04.650 --> 00:14:06.630
This one always kind of makes me scratch

388
00:14:06.650 --> 00:14:08.519
my head because NIST has been

389
00:14:08.539 --> 00:14:10.463
doing some form of face recognition

390
00:14:10.483 --> 00:14:12.170
testing since far as I can tell,

391
00:14:12.190 --> 00:14:13.682
1995 would be a

392
00:14:13.702 --> 00:14:15.092
little earlier and they've been

393
00:14:15.112 --> 00:14:16.520
constantly testing face recognition.

394
00:14:16.540 --> 00:14:18.194
Since 2006 DHS has

395
00:14:18.214 --> 00:14:19.900
been doing this since the early

396
00:14:19.920 --> 00:14:21.778
2010's, so that's two

397
00:14:21.798 --> 00:14:23.349
separate bodies of scientists that

398
00:14:23.369 --> 00:14:25.220
have really been looking at this

399
00:14:25.240 --> 00:14:28.630
for over a couple of decades now.

400
00:14:28.650 --> 00:14:29.910
As a class of computer program,

401
00:14:29.930 --> 00:14:32.488
face recognition might be one of the most

402
00:14:32.508 --> 00:14:34.600
sort of studied computer codes out there.

403
00:14:34.620 --> 00:14:36.165
Now that doesn't mean that

404
00:14:36.185 --> 00:14:37.420
every algorithm is tested,

405
00:14:37.440 --> 00:14:39.499
and it doesn't mean that you know

406
00:14:39.519 --> 00:14:41.500
even when we run these tests,

407
00:14:41.520 --> 00:14:43.390
we find out things we like.

408
00:14:43.410 --> 00:14:44.960
Some algorithms do perform poorly,

409
00:14:44.980 --> 00:14:47.390
but as a general statement face

410
00:14:47.410 --> 00:14:49.927
recognition has come through a pretty

411
00:14:49.947 --> 00:14:52.050
good amount of scientific scrutiny.

412
00:14:52.070 --> 00:14:57.920
Next slide.

413
00:14:57.940 --> 00:14:59.304
OK, last misconception here.

414
00:14:59.324 --> 00:15:01.380
So based recognition is a computer,

415
00:15:01.400 --> 00:15:02.415
computers are smart,

416
00:15:02.435 --> 00:15:04.490
should always listen to face recognition.

417
00:15:04.510 --> 00:15:06.668
Hopefully this doesn't come as a

418
00:15:06.688 --> 00:15:09.023
surprise to anyone who's sort of

419
00:15:09.043 --> 00:15:11.123
informing along face recognition can

420
00:15:11.143 --> 00:15:13.418
make mistakes because it was a nice

421
00:15:13.438 --> 00:15:15.161
work at DHS and NIST and others.

422
00:15:15.181 --> 00:15:16.907
We actually know a fair amount

423
00:15:16.927 --> 00:15:19.123
about how and why face recognition

424
00:15:19.143 --> 00:15:20.410
algorithms make mistakes.

425
00:15:20.430 --> 00:15:22.496
So things like pose, angle, lighting,

426
00:15:22.516 --> 00:15:24.980
blur and those images will cause face

427
00:15:25.000 --> 00:15:27.780
search to face images not to match right?

428
00:15:27.800 --> 00:15:29.768
Finish it and then other things like

429
00:15:29.788 --> 00:15:32.331
sort of look alikes or I mentioned

430
00:15:32.351 --> 00:15:33.567
brothers familiar relationships

431
00:15:33.587 --> 00:15:36.015
earlier may cause images of two

432
00:15:36.035 --> 00:15:37.940
different people to accidentally match.

433
00:15:37.960 --> 00:15:40.316
So For these reasons it's really

434
00:15:40.336 --> 00:15:43.030
important for that last step in the

435
00:15:43.050 --> 00:15:45.577
process that I showed on slide through

436
00:15:45.597 --> 00:15:48.177
the human adjudicator to be at the end

437
00:15:48.197 --> 00:15:50.605
of that workflow and this will review

438
00:15:50.625 --> 00:15:52.811
these results and use hopefully what

439
00:15:52.831 --> 00:15:55.010
we call sort of tangential information.

440
00:15:55.030 --> 00:15:57.950
Something that the face recognition.

441
00:15:57.970 --> 00:16:00.026
Algorithm didn't have available to it

442
00:16:00.046 --> 00:16:02.916
when that person is deciding what to do

443
00:16:02.936 --> 00:16:05.070
with the information from the computer.

444
00:16:05.090 --> 00:16:09.430
So that last step is really important.

445
00:16:09.450 --> 00:16:10.309
And with that,

446
00:16:10.329 --> 00:16:12.980
I think I'm almost right on time and I'll

447
00:16:13.000 --> 00:16:15.620
turn it back over to the deck to Abby to

448
00:16:15.640 --> 00:16:20.600
either do questions or to keep moving.

449
00:16:20.620 --> 00:16:22.670
Vivek: Alright, uh, thank you John.

450
00:16:22.690 --> 00:16:23.800
Appreciate it Abby,

451
00:16:23.820 --> 00:16:26.530
How much time do we have left

452
00:16:26.550 --> 00:16:28.803
for questions. Abby: We have about

453
00:16:28.823 --> 00:16:30.600
four minutes, maybe five.

454
00:16:30.620 --> 00:16:33.270
Everyone is encouraged to enter any

455
00:16:33.290 --> 00:16:35.920
questions for John into the chat pod.

456
00:16:35.940 --> 00:16:38.797
Bottom left of your screen and will

457
00:16:38.817 --> 00:16:41.710
call them out for John to answer.

458
00:16:41.730 --> 00:16:43.780
So please use the chat

459
00:16:43.800 --> 00:16:50.710
pod to ask any questions.

460
00:16:50.730 --> 00:16:52.402
And if we don't have any

461
00:16:52.422 --> 00:16:53.461
questions, that's OK too.

462
00:16:53.481 --> 00:16:54.746
There will be other opportunities

463
00:16:54.766 --> 00:16:55.800
throughout today's presentation.

464
00:16:55.820 --> 00:16:58.212
UM, and we hope to have more time at

465
00:16:58.232 --> 00:17:04.202
the end for additional Q and A as well.

466
00:17:04.222 --> 00:17:10.280


467
00:17:10.300 --> 00:17:12.390
And it looks like Joe Moda.

468
00:17:12.410 --> 00:17:14.226
He's with the Port Authority of

469
00:17:14.246 --> 00:17:16.204
New York and New Jersey asked

470
00:17:16.224 --> 00:17:18.268
if there are any standards or

471
00:17:18.288 --> 00:17:20.110
guidance on camera placement,

472
00:17:20.130 --> 00:17:21.860
and he's still typing John.

473
00:17:21.880 --> 00:17:24.860
But are there any standards or

474
00:17:24.880 --> 00:17:26.860
guidance on camera placement?

475
00:17:26.880 --> 00:17:31.200
Or angle to the person's face.

476
00:17:31.220 --> 00:17:33.528
John: So a lot of it's going to depend

477
00:17:33.548 --> 00:17:35.350
on algorithm that you're using,

478
00:17:35.370 --> 00:17:37.387
so it's hard to come up with

479
00:17:37.407 --> 00:17:38.860
sort of standardized guidance.

480
00:17:38.880 --> 00:17:41.037
The ones that I'm most familiar with

481
00:17:41.057 --> 00:17:43.360
or actually sort of use case specific

482
00:17:43.380 --> 00:17:45.730
there in their travel environment so IKAI

483
00:17:45.750 --> 00:17:47.828
has sort of standards about what

484
00:17:47.848 --> 00:17:49.702
photo should look like when they're

485
00:17:49.722 --> 00:17:51.300
in things like travel documents,

486
00:17:51.320 --> 00:17:52.890
passports, ID, things like that,

487
00:17:52.910 --> 00:17:55.123
but those are sort of a little

488
00:17:55.143 --> 00:17:56.080
bit more controlled.

489
00:17:56.100 --> 00:17:58.230
That sounds like this question is maybe

490
00:17:58.250 --> 00:18:00.400
more related to unconstrained like CCTD

491
00:18:00.420 --> 00:18:02.164
Type face capture and I'm not aware

492
00:18:02.184 --> 00:18:04.184
of any sort of standards around

493
00:18:04.204 --> 00:18:05.744
that they're developing some

494
00:18:05.764 --> 00:18:08.030
standards on how to do testing.

495
00:18:08.050 --> 00:18:09.070
Evaluation of surveillance

496
00:18:09.090 --> 00:18:09.770
type face applications.

497
00:18:09.790 --> 00:18:12.682
Maybe Patrick can speak a little bit

498
00:18:12.702 --> 00:18:16.690
more to that. In the next section.

499
00:18:16.710 --> 00:18:18.460
But these are just coming

500
00:18:18.480 --> 00:18:20.230
out sort of more recently.

501
00:18:20.250 --> 00:18:21.838
Arun you wanna, uh?

502
00:18:21.858 --> 00:18:23.848
You are also  fairly adept on

503
00:18:23.868 --> 00:18:28.490
the standard side of house.

504
00:18:28.510 --> 00:18:30.850
Arun: Oh yeah, thanks John.

505
00:18:30.870 --> 00:18:32.530
So yeah, I don't know about that.

506
00:18:32.550 --> 00:18:35.022
There are probably more like best practices

507
00:18:35.042 --> 00:18:36.920
rather than standards at this point,

508
00:18:36.940 --> 00:18:39.576
and I think the point with the question

509
00:18:39.596 --> 00:18:42.306
that I guess happy is asking is when

510
00:18:42.326 --> 00:18:45.030
you get too high and the angle you

511
00:18:45.050 --> 00:18:47.686
know seeing the faces is too off angle.

512
00:18:47.706 --> 00:18:48.998
Face recognition systems can

513
00:18:49.018 --> 00:18:50.730
struggle to recognize a person.

514
00:18:50.750 --> 00:18:51.744
So yeah, general.

515
00:18:51.764 --> 00:18:53.930
Generally speaking, you want to

516
00:18:53.950 --> 00:18:57.480
get it as frontal as possible.

517
00:18:57.500 --> 00:18:58.840
I can't find these.

518
00:18:58.860 --> 00:19:00.896
For some reason the number you know less

519
00:19:00.916 --> 00:19:02.915
than ten percent angle of inclination to

520
00:19:02.935 --> 00:19:05.100
the faces I was popping into my head,

521
00:19:05.120 --> 00:19:07.820
but I don't know if to be honest with you.

522
00:19:07.840 --> 00:19:09.780
I had to go back and figure out

523
00:19:09.800 --> 00:19:11.620
where that number is coming from.

524
00:19:11.640 --> 00:19:13.530
There's a good chance that you know.

525
00:19:13.550 --> 00:19:15.462
Again, there is are some standards

526
00:19:15.482 --> 00:19:17.561
that are developing right now on the

527
00:19:17.581 --> 00:19:19.073
use of face recognition in video

528
00:19:19.093 --> 00:19:20.990
and we can go check to see whether

529
00:19:21.010 --> 00:19:22.503
or not those documents site

530
00:19:22.523 --> 00:19:26.250
something like that.

531
00:19:26.270 --> 00:19:28.434
John: Yeah, but I would say the best sort

532
00:19:28.454 --> 00:19:31.085
of source for that is going to be

533
00:19:31.105 --> 00:19:32.508
your particular face recognition

534
00:19:32.528 --> 00:19:34.270
algorithm vendor right there.

535
00:19:34.290 --> 00:19:36.496
No, what they've built these face

536
00:19:36.516 --> 00:19:38.610
recognition systems to sort of handle

537
00:19:38.630 --> 00:19:40.610
in terms of access and stuff,

538
00:19:40.630 --> 00:19:42.950
and some of them are very good,

539
00:19:42.970 --> 00:19:44.670
actually driven by some activities

540
00:19:44.690 --> 00:19:47.001
over in IARPA at doing this sort

541
00:19:47.021 --> 00:19:48.905
of very off axis sort of profile

542
00:19:48.925 --> 00:19:54.760
to sideview type matching.

543
00:19:54.780 --> 00:19:58.770
Arun: Yeah, just to comment on what John just said.

544
00:19:58.790 --> 00:20:00.252
I think in general you know

545
00:20:00.272 --> 00:20:02.090
with a lot of the products.

546
00:20:02.110 --> 00:20:03.760
You guys are probably dealing with,

547
00:20:03.780 --> 00:20:05.700
your vendors will be your best source,

548
00:20:05.720 --> 00:20:07.362
not only because of the matching

549
00:20:07.382 --> 00:20:08.470
algorithms with the hardware,

550
00:20:08.490 --> 00:20:10.542
so they probably tested it and it done

551
00:20:10.562 --> 00:20:12.436
some level of integration work here before

552
00:20:12.456 --> 00:20:15.000
it's good to go in with a little bit

553
00:20:15.020 --> 00:20:16.780
of skepticism people usually you know,

554
00:20:16.800 --> 00:20:18.710
always say they have the best stuff.

555
00:20:18.730 --> 00:20:19.430
It's always true,

556
00:20:19.450 --> 00:20:21.160
but they'll at least be able to

557
00:20:21.180 --> 00:20:23.009
give you some things that

558
00:20:23.029 --> 00:20:24.250
hopefully they've tested before.

559
00:20:24.270 --> 00:20:26.194
And if they and one of the things

560
00:20:26.214 --> 00:20:27.694
you should definitely ask them

561
00:20:27.714 --> 00:20:29.610
is where have you deployed this.

562
00:20:29.630 --> 00:20:31.160
How did you deploy it?

563
00:20:31.180 --> 00:20:33.071
I think that'll help get you

564
00:20:33.091 --> 00:20:39.061
closer to the right direction.

565
00:20:39.081 --> 00:20:42.550


566
00:20:42.570 --> 00:20:48.540
Abby: Alright, Vivek back over to you.

567
00:20:48.560 --> 00:20:49.820


568
00:20:49.840 --> 00:20:53.320
Vivek: Thank you Abby. And thank you,

569
00:20:53.340 --> 00:20:56.260
John and Arun for the question.

570
00:20:56.280 --> 00:20:59.820
Thank you Mister Moda. Uh,

571
00:20:59.840 --> 00:21:03.060
so we still have a lot of ground to cover,

572
00:21:03.080 --> 00:21:05.447
so I'd like to introduce our next

573
00:21:05.467 --> 00:21:07.481
speaker from the National Institute

574
00:21:07.501 --> 00:21:10.300
of Standards and Technology or NIST.

575
00:21:10.320 --> 00:21:13.345
Patrick Grother he is a scientist

576
00:21:13.365 --> 00:21:16.204
at NIST responsible for biometric

577
00:21:16.224 --> 00:21:19.140
algorithm evaluation and biometric

578
00:21:19.160 --> 00:21:21.350
performance testing standardization.

579
00:21:21.370 --> 00:21:24.885
He leads face recognition vendor tests which

580
00:21:24.905 --> 00:21:27.665
constitute the world's largest independent

581
00:21:27.685 --> 00:21:31.230
public test of face recognition algorithms.

582
00:21:31.250 --> 00:21:33.422
This gives quantitative support

583
00:21:33.442 --> 00:21:35.620
to developers and users,

584
00:21:35.640 --> 00:21:37.820
and policymakers face with

585
00:21:37.840 --> 00:21:38.920
algorithm selections.

586
00:21:38.940 --> 00:21:40.016
Performance adequacy,

587
00:21:40.036 --> 00:21:41.660
assessment and Procurement

588
00:21:41.680 --> 00:21:43.310
specification. Patrick cultures

589
00:21:43.330 --> 00:21:45.740
NIST'S biannual international

590
00:21:45.760 --> 00:21:48.170
Face performance conference (IFPC)

591
00:21:48.190 --> 00:21:50.360
On measurement metrics and certification,

592
00:21:50.380 --> 00:21:52.540
and has since 2018.

593
00:21:52.560 --> 00:21:55.192
He had served as the

594
00:21:55.212 --> 00:21:57.780
chairman of the ISO IEC JTC.

595
00:21:57.800 --> 00:22:01.010
One subcommittee, thirty seven on biometrics.

596
00:22:01.030 --> 00:22:02.096
Without further ado,

597
00:22:02.116 --> 00:22:03.906
I'll let Patrick speaks to

598
00:22:03.926 --> 00:22:05.434
the performance and testing

599
00:22:05.454 --> 00:22:10.150
of face recognition system.

600
00:22:10.170 --> 00:22:14.678
Patrick: OK, uh, yeah. Thank you very much

601
00:22:14.698 --> 00:22:16.720
for that introduction and uh.

602
00:22:16.740 --> 00:22:19.240
Thank you for all the organizational work.

603
00:22:19.260 --> 00:22:20.320
Putting this together.

604
00:22:20.340 --> 00:22:22.840
Yeah, so my name is Patrick Grother.

605
00:22:22.860 --> 00:22:25.060
I'm a scientist at a government

606
00:22:25.080 --> 00:22:26.800
lab just outside Washington DC.

607
00:22:26.820 --> 00:22:30.070
It's part of the Department of Commerce.

608
00:22:30.090 --> 00:22:32.260
And we don't do regulation.

609
00:22:32.280 --> 00:22:34.372
We don't do policy.

610
00:22:34.392 --> 00:22:37.020
What we do with measurement,

611
00:22:37.040 --> 00:22:39.175
and we've been involved in

612
00:22:39.195 --> 00:22:41.860
biometrics going on sixty plus years.

613
00:22:41.880 --> 00:22:43.868
Initially fingerprints later things

614
00:22:43.888 --> 00:22:46.880
like DNA and face and voice.

615
00:22:46.900 --> 00:22:48.149
And iris recognition.

616
00:22:48.169 --> 00:22:50.270
The topic today of course,

617
00:22:50.290 --> 00:22:52.180
his face recognition.

618
00:22:52.200 --> 00:22:58.171
And at its heart, as John articulated.

619
00:22:58.191 --> 00:22:59.230


620
00:22:59.250 --> 00:23:00.673
Face recognition compares.

621
00:23:00.693 --> 00:23:03.550
Two faces are compared to face

622
00:23:03.570 --> 00:23:06.874
with a set of previous faces

623
00:23:06.894 --> 00:23:08.789
so that fundamental comparison

624
00:23:08.809 --> 00:23:10.810
is sort of embedded in in.

625
00:23:10.830 --> 00:23:13.477
In the goal of using face recognition

626
00:23:13.497 --> 00:23:16.380
so you can do that with a human.

627
00:23:16.400 --> 00:23:19.308
And you can do that with a face

628
00:23:19.328 --> 00:23:22.690
recognition algorithm, so you see.

629
00:23:22.710 --> 00:23:24.610
The two photos on the right

630
00:23:24.630 --> 00:23:26.470
there and the question becomes,

631
00:23:26.490 --> 00:23:29.990
is it the same person or not?

632
00:23:30.010 --> 00:23:31.856
And I'll give you the answer

633
00:23:31.876 --> 00:23:34.050
in a couple of slides time.

634
00:23:34.070 --> 00:23:36.880
But it's quite a challenging task generally,

635
00:23:36.900 --> 00:23:39.300
and particularly with faces unfamiliar faces,

636
00:23:39.320 --> 00:23:43.084
so people you don't know we're very good

637
00:23:43.104 --> 00:23:46.100
at recognizing people that we do know.

638
00:23:46.120 --> 00:23:48.980
But most operations aren't really like that,

639
00:23:49.000 --> 00:23:51.100
so.

640
00:23:51.120 --> 00:23:51.916
I'll go on.

641
00:23:51.936 --> 00:23:53.894
So this was a test that test

642
00:23:53.914 --> 00:23:55.899
of human capability that was

643
00:23:55.919 --> 00:23:57.982
administered a couple of years

644
00:23:58.002 --> 00:24:00.230
ago and the photos are available.

645
00:24:00.250 --> 00:24:03.580
And there's two paths and you can see

646
00:24:03.600 --> 00:24:06.780
quality is not what we would like.

647
00:24:06.800 --> 00:24:07.803
There's some blurred.

648
00:24:07.823 --> 00:24:09.170
There's some poor illumination,

649
00:24:09.190 --> 00:24:11.557
and you've got some sunlight down on

650
00:24:11.577 --> 00:24:14.280
the right, so the panel on the left.

651
00:24:14.300 --> 00:24:16.520
Is it the same person or not

652
00:24:16.540 --> 00:24:18.030
there on the right?

653
00:24:18.050 --> 00:24:20.420
Is it the same person or not?

654
00:24:20.440 --> 00:24:22.526
And nowadays we won the photos

655
00:24:22.546 --> 00:24:24.957
used in that evaluation through all

656
00:24:24.977 --> 00:24:26.870
our face recognition algorithms.

657
00:24:26.890 --> 00:24:27.728
And, uh,

658
00:24:27.748 --> 00:24:30.731
and most of them excel on that

659
00:24:30.751 --> 00:24:33.900
task better than typical humans.

660
00:24:33.920 --> 00:24:36.280
Some humans also excel.

661
00:24:36.300 --> 00:24:38.660
Human capability is buried.

662
00:24:38.680 --> 00:24:43.080
So I'll tell you the answers, here we go.

663
00:24:43.100 --> 00:24:46.326
And there are training classes available

664
00:24:46.346 --> 00:24:49.660
and aptitude varies naturally by individual.

665
00:24:49.680 --> 00:24:50.563
But that is,

666
00:24:50.583 --> 00:24:51.770
that is the answers,

667
00:24:51.790 --> 00:24:56.460
and it's not a trivial thing to do that.

668
00:24:56.480 --> 00:24:59.622
As a test designed by a

669
00:24:59.642 --> 00:25:01.210
university in Australia.

670
00:25:01.230 --> 00:25:02.186
I put the URL.

671
00:25:02.206 --> 00:25:03.406
These slides will be available

672
00:25:03.426 --> 00:25:04.470
to you afterwards,

673
00:25:04.490 --> 00:25:06.493
so you can actually try this at

674
00:25:06.513 --> 00:25:08.809
home and you can get a score and

675
00:25:08.829 --> 00:25:10.896
you can put that score in context

676
00:25:10.916 --> 00:25:12.820
of how well other people do.

677
00:25:12.840 --> 00:25:14.830
And there are some exceptional performers.

678
00:25:14.850 --> 00:25:15.787
in human comparison,

679
00:25:15.807 --> 00:25:17.701
but I'm not here to really

680
00:25:17.721 --> 00:25:19.190
to talk about humans.

681
00:25:19.210 --> 00:25:21.500
I'm here to talk about.

682
00:25:21.520 --> 00:25:25.270
Automated face recognition so.

683
00:25:25.290 --> 00:25:27.800
The answer I promised you earlier,

684
00:25:27.820 --> 00:25:29.488
same identity, different identity.

685
00:25:29.508 --> 00:25:31.180
They are actually sisters.

686
00:25:31.200 --> 00:25:34.220
My colleague and her sister.

687
00:25:34.240 --> 00:25:36.295
And you know that obviously

688
00:25:36.315 --> 00:25:37.130
genetically related,

689
00:25:37.150 --> 00:25:40.270
so they look like similar.

690
00:25:40.290 --> 00:25:42.979
And so you know the correct answer

691
00:25:42.999 --> 00:25:44.890
that is different identity.

692
00:25:44.910 --> 00:25:47.810
There's of course a whole set of technologies

693
00:25:47.830 --> 00:25:50.501
that use that sort of comparison

694
00:25:50.521 --> 00:25:52.870
function based on automated algorithms,

695
00:25:52.890 --> 00:25:55.852
and they used quite widely in

696
00:25:55.872 --> 00:25:57.840
airports and immigration settings.

697
00:25:57.860 --> 00:26:00.220
And you can see some photos on the left.

698
00:26:00.240 --> 00:26:01.804
There may be using a PIV

699
00:26:01.824 --> 00:26:02.860
card for access control,

700
00:26:02.880 --> 00:26:04.700
or using a passport for access control,

701
00:26:04.720 --> 00:26:08.020
or maybe even a driving license.

702
00:26:08.040 --> 00:26:10.590
Images of red from those credentials.

703
00:26:10.610 --> 00:26:13.188
They are templated as John described

704
00:26:13.208 --> 00:26:16.242
and the algorithm makes a measure of

705
00:26:16.262 --> 00:26:18.320
similarity between the two faces.

706
00:26:18.340 --> 00:26:20.272
If they're very similar.

707
00:26:20.292 --> 00:26:23.301
Then it's taken to be that

708
00:26:23.321 --> 00:26:26.378
they're the same person.

709
00:26:26.398 --> 00:26:29.490
And of course there can be some errors.

710
00:26:29.510 --> 00:26:32.010
The larger market segment is when we

711
00:26:32.030 --> 00:26:34.393
do what's called one too many instead

712
00:26:34.413 --> 00:26:36.745
of just one to one access control

713
00:26:36.765 --> 00:26:38.960
where you present a credential.

714
00:26:38.980 --> 00:26:41.560
You can instead just present your face.

715
00:26:41.580 --> 00:26:44.620
No credential, no passport, no phone.

716
00:26:44.640 --> 00:26:47.554
And you can pay a live face to a

717
00:26:47.574 --> 00:26:50.200
database of people that are expected.

718
00:26:50.220 --> 00:26:52.060
So this is a DHS.

719
00:26:52.080 --> 00:26:54.468
I think it's still a pilot for

720
00:26:54.488 --> 00:26:56.520
people departing the United States,

721
00:26:56.540 --> 00:26:59.130
so this these systems take a face.

722
00:26:59.150 --> 00:27:01.515
Compare it with everybody that's

723
00:27:01.535 --> 00:27:03.900
expected to board a plane.

724
00:27:03.920 --> 00:27:06.900
And they match if there's a match,

725
00:27:06.920 --> 00:27:09.462
then it's taken as a biometric

726
00:27:09.482 --> 00:27:10.750
evidence of departure.

727
00:27:10.770 --> 00:27:13.740
So a visa holder records their departure.

728
00:27:13.760 --> 00:27:16.659
It's also serving double duty as access

729
00:27:16.679 --> 00:27:20.430
control to the plane, typically so.

730
00:27:20.450 --> 00:27:22.270
Those systems are out there

731
00:27:22.290 --> 00:27:24.540
and that's called one to many,

732
00:27:24.560 --> 00:27:27.160
and that's quite a large market segment,

733
00:27:27.180 --> 00:27:30.260
so you are the sort of more

734
00:27:30.280 --> 00:27:31.980
focused topic today is.

735
00:27:32.000 --> 00:27:33.084
His maybe investigation

736
00:27:33.104 --> 00:27:33.820
criminal investigation.

737
00:27:33.840 --> 00:27:36.963
So this is a pending case coming

738
00:27:36.983 --> 00:27:39.910
out of Maryland a few years ago.

739
00:27:39.930 --> 00:27:42.990
A shooting at a newspaper office.

740
00:27:43.010 --> 00:27:44.110
Undocumented individual.

741
00:27:44.130 --> 00:27:45.790
They, the investigators,

742
00:27:45.810 --> 00:27:47.114
immediately took his photo,

743
00:27:47.134 --> 00:27:48.769
searched it against a Maryland

744
00:27:48.789 --> 00:27:50.010
driving license database,

745
00:27:50.030 --> 00:27:51.418
and retrieved essentially the

746
00:27:51.438 --> 00:27:53.880
biographic record so they got a name.

747
00:27:53.900 --> 00:27:54.930
They gotta dress.

748
00:27:54.950 --> 00:27:58.012
They got his age, that kind of thing,

749
00:27:58.032 --> 00:28:00.040
and so face recognition in that

750
00:28:00.060 --> 00:28:01.974
circumstance is, you know,

751
00:28:01.994 --> 00:28:03.740
very useful to an investigator,

752
00:28:03.760 --> 00:28:06.390
and it is extraordinarily accurate.

753
00:28:06.410 --> 00:28:08.560
Because you control the photos,

754
00:28:08.580 --> 00:28:11.578
so ID credentials and driving license

755
00:28:11.598 --> 00:28:13.663
databases are typically populated

756
00:28:13.683 --> 00:28:15.829
with frontal photos and you would

757
00:28:15.849 --> 00:28:18.181
arrange to take a frontal photo

758
00:28:18.201 --> 00:28:20.680
in an investigation such as this.

759
00:28:20.700 --> 00:28:22.845
Of course some investigations you

760
00:28:22.865 --> 00:28:25.010
don't really control the camera,

761
00:28:25.030 --> 00:28:27.545
you don't control the presentation

762
00:28:27.565 --> 00:28:30.030
of the subject. And then things.

763
00:28:30.050 --> 00:28:32.270
Arrows can start creeping in the calls.

764
00:28:32.290 --> 00:28:35.130
The image is not ideal anymore.

765
00:28:35.150 --> 00:28:37.660
So yeah, so that's one too many,

766
00:28:37.680 --> 00:28:40.200
and those databases can be quite large,

767
00:28:40.220 --> 00:28:42.552
so I just I'm not sure what

768
00:28:42.572 --> 00:28:44.540
the population of Maryland is,

769
00:28:44.560 --> 00:28:48.020
but you know, it's a few million and.

770
00:28:48.040 --> 00:28:50.462
A face recognition will succeed

771
00:28:50.482 --> 00:28:53.770
as long as the photo is quite good.

772
00:28:53.790 --> 00:28:56.200
Frontal view photo.

773
00:28:56.220 --> 00:28:58.510
So what's involved in doing this and

774
00:28:58.530 --> 00:29:00.870
and if John covered some of these

775
00:29:00.890 --> 00:29:03.256
things earlier I hope I reinforced

776
00:29:03.276 --> 00:29:05.850
that the points that he made so.

777
00:29:05.870 --> 00:29:08.060
We've got a subject in an environment.

778
00:29:08.080 --> 00:29:09.950
On the left there's a camera.

779
00:29:09.970 --> 00:29:12.165
Camera is very in their

780
00:29:12.185 --> 00:29:13.940
capability and then there.

781
00:29:13.960 --> 00:29:16.210
The speed at which they operate and

782
00:29:16.230 --> 00:29:19.140
and the illumination that they use.

783
00:29:19.160 --> 00:29:20.630
And and the quality of

784
00:29:20.650 --> 00:29:22.120
the image that comes out.

785
00:29:22.140 --> 00:29:24.766
And then that goes off to a

786
00:29:24.786 --> 00:29:25.900
face recognition algorithm.

787
00:29:25.920 --> 00:29:28.870
I've drawn it as a black box on this

788
00:29:28.890 --> 00:29:32.190
slide because most of them are proprietary.

789
00:29:32.210 --> 00:29:32.958
They are.

790
00:29:32.978 --> 00:29:35.262
They sort of embed high-end intellectual

791
00:29:35.282 --> 00:29:37.300
property that is not commoditized.

792
00:29:37.320 --> 00:29:39.156
Algorithms vary in their

793
00:29:39.176 --> 00:29:41.940
capability as I will show you.

794
00:29:41.960 --> 00:29:44.600
And they are protected IP.

795
00:29:44.620 --> 00:29:47.240
Uh, what would come out of that in

796
00:29:47.260 --> 00:29:49.357
an investigation would be a sort of

797
00:29:49.377 --> 00:29:51.270
a list of candidates are hypothesized

798
00:29:51.290 --> 00:29:53.990
candidates who might this person be?

799
00:29:54.010 --> 00:29:55.376
And then that would get

800
00:29:55.396 --> 00:29:57.160
passed off to a human reviewer.

801
00:29:57.180 --> 00:29:59.350
And as I mentioned earlier.

802
00:29:59.370 --> 00:30:03.440
Tests exist for that human capability.

803
00:30:03.460 --> 00:30:05.980
Tests obviously exist for the

804
00:30:06.000 --> 00:30:08.520
algorithm capability as I'll describe.

805
00:30:08.540 --> 00:30:10.320
And on the right side,

806
00:30:10.340 --> 00:30:13.260
you've got sort of two.

807
00:30:13.280 --> 00:30:15.600
Some possible errors that can happen.

808
00:30:15.620 --> 00:30:17.550
Usually it will be successful,

809
00:30:17.570 --> 00:30:17.901
but,

810
00:30:17.921 --> 00:30:20.007
but sometimes you can have a

811
00:30:20.027 --> 00:30:22.035
false negative where we've got

812
00:30:22.055 --> 00:30:23.400
a criminal investigation.

813
00:30:23.420 --> 00:30:26.560
The person is in the database.

814
00:30:26.580 --> 00:30:28.480
And yet we miss him because

815
00:30:28.500 --> 00:30:29.760
quality or because of,

816
00:30:29.780 --> 00:30:32.290
you know some other problem.

817
00:30:32.310 --> 00:30:35.220
And that would be a false negative.

818
00:30:35.240 --> 00:30:37.470
It's also possible that the person isn't.

819
00:30:37.490 --> 00:30:39.410
It is not in the database,

820
00:30:39.430 --> 00:30:41.984
and if the outcome of this was that we

821
00:30:42.004 --> 00:30:45.200
we said that the person was in the database,

822
00:30:45.220 --> 00:30:47.370
then that would be an improper association

823
00:30:47.390 --> 00:30:50.030
of the person with somebody in the database.

824
00:30:50.050 --> 00:30:51.318
That's a false positive.

825
00:30:51.338 --> 00:30:52.930
We want to avoid those,

826
00:30:52.950 --> 00:30:54.862
so we have benchmarks called the

827
00:30:54.882 --> 00:30:56.150
face recognition vendor test,

828
00:30:56.170 --> 00:30:59.240
which I'll give you information on next.

829
00:30:59.260 --> 00:31:01.500
And we've got a system,

830
00:31:01.520 --> 00:31:01.958
essentially,

831
00:31:01.978 --> 00:31:05.160
we've got a system that's built of

832
00:31:05.180 --> 00:31:07.540
automated FR Plus human review.

833
00:31:07.560 --> 00:31:09.892
And there's been a few cases

834
00:31:09.912 --> 00:31:12.568
that being quite a high profile

835
00:31:12.588 --> 00:31:15.130
cases documented in the press.

836
00:31:15.150 --> 00:31:17.020
Where by the algorithm finds

837
00:31:17.040 --> 00:31:19.504
some face and then the human

838
00:31:19.524 --> 00:31:21.580
reviewer mistakenly says yes.

839
00:31:21.600 --> 00:31:23.068
It's this person.

840
00:31:23.088 --> 00:31:25.550
For false positives can happen,

841
00:31:25.570 --> 00:31:29.386
and they tend to cause bad press

842
00:31:29.406 --> 00:31:31.030
and monetary compensation.

843
00:31:31.050 --> 00:31:32.294
Depends on the application.

844
00:31:32.314 --> 00:31:34.247
In an airport application is different

845
00:31:34.267 --> 00:31:35.880
than a bank robbery investigation,

846
00:31:35.900 --> 00:31:36.313
obviously.

847
00:31:36.333 --> 00:31:38.911
So we'll go on so face

848
00:31:38.931 --> 00:31:40.210
recognition vendor test.

849
00:31:40.230 --> 00:31:43.648
This is a set of benchmarks that I've been

850
00:31:43.668 --> 00:31:46.440
involved with now for twenty plus years.

851
00:31:46.460 --> 00:31:48.800
And they are large scale

852
00:31:48.820 --> 00:31:50.688
independent public assessments of

853
00:31:50.708 --> 00:31:52.970
face recognition algorithms,

854
00:31:52.990 --> 00:31:55.690
the core technology.

855
00:31:55.710 --> 00:31:57.419
And that logo on the right there

856
00:31:57.439 --> 00:31:59.221
is something that was designed by

857
00:31:59.241 --> 00:32:00.560
Department of Defense Counterdrug

858
00:32:00.580 --> 00:32:02.220
Agency in 1999.

859
00:32:02.240 --> 00:32:04.229
And you can see that mesh that

860
00:32:04.249 --> 00:32:05.790
sort of on the face,

861
00:32:05.810 --> 00:32:07.548
that that is not how algorithms

862
00:32:07.568 --> 00:32:09.060
work anymore at that time.

863
00:32:09.080 --> 00:32:11.172
It was a sort of a famous algorithm

864
00:32:11.192 --> 00:32:13.210
that was used for matching faces.

865
00:32:13.230 --> 00:32:15.919
You would put a mesh on two faces and

866
00:32:15.939 --> 00:32:18.430
see how different the meshes are.

867
00:32:18.450 --> 00:32:21.765
But nowadays algorithms work by

868
00:32:21.785 --> 00:32:24.433
neural networks consuming pixels

869
00:32:24.453 --> 00:32:27.800
and doing some quite opaque

870
00:32:27.820 --> 00:32:30.400
calculations on pixel values.

871
00:32:30.420 --> 00:32:31.786
Without doing something

872
00:32:31.806 --> 00:32:34.100
explicit like forming a mesh.

873
00:32:34.120 --> 00:32:37.022
And of course, capability is

874
00:32:37.042 --> 00:32:40.130
much better today than it was then.

875
00:32:40.150 --> 00:32:42.921
So the FRVT benchmark face

876
00:32:42.941 --> 00:32:44.960
recognition vendor tech vendor test.

877
00:32:44.980 --> 00:32:46.432
Currently consists of four

878
00:32:46.452 --> 00:32:47.170
different benchmarks,

879
00:32:47.190 --> 00:32:49.599
one on that one for one application

880
00:32:49.619 --> 00:32:51.230
for safe passport control,

881
00:32:51.250 --> 00:32:55.030
one too many for things like investigation.

882
00:32:55.050 --> 00:32:56.622
Our box number three.

883
00:32:56.642 --> 00:32:58.620
There is about morph detection.

884
00:32:58.640 --> 00:33:00.750
This is where computer graphics

885
00:33:00.770 --> 00:33:03.510
are used to combine two photos.

886
00:33:03.530 --> 00:33:06.947
Into one photo such that both two

887
00:33:06.967 --> 00:33:09.800
people can authenticate the same photo.

888
00:33:09.820 --> 00:33:11.335
That would be a security

889
00:33:11.355 --> 00:33:12.570
vulnerability in some systems,

890
00:33:12.590 --> 00:33:13.802
and there's an interest

891
00:33:13.822 --> 00:33:15.040
in detecting such things.

892
00:33:15.060 --> 00:33:17.190
Box number four is about quality assessment.

893
00:33:17.210 --> 00:33:19.675
Can you look at an image and

894
00:33:19.695 --> 00:33:22.050
say is it a good image?

895
00:33:22.070 --> 00:33:25.844
From that we produce a set of

896
00:33:25.864 --> 00:33:27.470
benchmark outputs products.

897
00:33:27.490 --> 00:33:30.700
These are reports that go on

898
00:33:30.720 --> 00:33:34.290
the website and also web

899
00:33:34.310 --> 00:33:36.930
pages with performance data.

900
00:33:36.950 --> 00:33:40.150
And I won't walk walking through all of them,

901
00:33:40.170 --> 00:33:41.940
but these are regularly updated.

902
00:33:41.960 --> 00:33:45.100
Their free to download no charge.

903
00:33:45.120 --> 00:33:47.155
And they sort of constitute

904
00:33:47.175 --> 00:33:49.210
the largest sort of public.

905
00:33:49.230 --> 00:33:51.018
Benchmarks of face recognition

906
00:33:51.038 --> 00:33:53.730
technology and they name vendor names.

907
00:33:53.750 --> 00:33:57.594
So if you're a developer there is

908
00:33:57.614 --> 00:34:00.510
some reputational risk to actually

909
00:34:00.530 --> 00:34:03.400
participating in such a test.

910
00:34:03.420 --> 00:34:06.522
And so we're going to list accuracy

911
00:34:06.542 --> 00:34:09.020
and performance of an algorithm.

912
00:34:09.040 --> 00:34:09.493
Uh,

913
00:34:09.513 --> 00:34:10.912
alongside other developers

914
00:34:10.932 --> 00:34:13.280
algorithms and that sort of.

915
00:34:13.300 --> 00:34:14.018
Uh.

916
00:34:14.038 --> 00:34:18.450
So the foster their competitive ecosystem.

917
00:34:18.470 --> 00:34:19.600
For technology developers

918
00:34:19.620 --> 00:34:21.910
to sort of play in,

919
00:34:21.930 --> 00:34:24.500
and we've looked now at upwards of

920
00:34:24.520 --> 00:34:26.072
eight hundred different algorithms

921
00:34:26.092 --> 00:34:28.022
from upwards of two hundred

922
00:34:28.042 --> 00:34:30.350
developers in the last four years.

923
00:34:30.370 --> 00:34:32.660
So it's quite a busy marketplace,

924
00:34:32.680 --> 00:34:34.964
and there's a very wide spectrum

925
00:34:34.984 --> 00:34:36.500
of performance across algorithms,

926
00:34:36.520 --> 00:34:39.570
which I'll show you in just a second.

927
00:34:39.590 --> 00:34:41.490
We'll do some more different

928
00:34:41.510 --> 00:34:43.562
benchmarks in the future, maybe,

929
00:34:43.582 --> 00:34:47.290
but how do we do those so?

930
00:34:47.310 --> 00:34:50.188
We have images we are privileged

931
00:34:50.208 --> 00:34:52.650
to be able to use.

932
00:34:52.670 --> 00:34:55.566
U S government imagery and so

933
00:34:55.586 --> 00:34:57.661
passports and visas, mugshots,

934
00:34:57.681 --> 00:35:00.820
DHS images of various kinds.

935
00:35:00.840 --> 00:35:01.646
Come and

936
00:35:01.666 --> 00:35:04.537
And they vary in their sort of

937
00:35:04.557 --> 00:35:06.788
appearance and in how constrained

938
00:35:06.808 --> 00:35:08.110
the capture is.

939
00:35:08.130 --> 00:35:10.392
So the top left is something

940
00:35:10.412 --> 00:35:12.100
like the gold standard,

941
00:35:12.120 --> 00:35:13.910
as appears on passports and

942
00:35:13.930 --> 00:35:15.783
visas you you probably applied

943
00:35:15.803 --> 00:35:17.727
for a passport before or driving

944
00:35:17.747 --> 00:35:19.893
license and had the image rejected

945
00:35:19.913 --> 00:35:21.900
because it wasn't good enough,

946
00:35:21.920 --> 00:35:24.486
so that's the sort of the

947
00:35:24.506 --> 00:35:26.210
standardized frontal view photo.

948
00:35:26.230 --> 00:35:27.618
Mugshots also are collected

949
00:35:27.638 --> 00:35:29.378
according to standards in state

950
00:35:29.398 --> 00:35:31.190
and local police departments.

951
00:35:31.210 --> 00:35:34.186
The FBI was sort of instrumental in

952
00:35:34.206 --> 00:35:36.360
propagating those standards back in

953
00:35:36.380 --> 00:35:38.810
about 1998, 99.

954
00:35:38.830 --> 00:35:41.675
In anticipation of an automated

955
00:35:41.695 --> 00:35:43.400
face recognition market,

956
00:35:43.420 --> 00:35:47.420
DHS is a energetic and enthusiastic user.

957
00:35:47.440 --> 00:35:48.664
Face recognition.

958
00:35:48.684 --> 00:35:51.780
But in different operational constraints.

959
00:35:51.800 --> 00:35:53.476
So as John described,

960
00:35:53.496 --> 00:35:55.596
those scenario tests that are

961
00:35:55.616 --> 00:35:58.339
run are looking at accuracy but

962
00:35:58.359 --> 00:36:00.661
also looking at usability like

963
00:36:00.681 --> 00:36:02.900
ability and speed of capture.

964
00:36:02.920 --> 00:36:06.419
Is it a pleasure to use a sort of

965
00:36:06.439 --> 00:36:10.190
an immigration kiosk for example?

966
00:36:10.210 --> 00:36:11.032
Uh, yeah,

967
00:36:11.052 --> 00:36:13.137
and there's a graphic right

968
00:36:13.157 --> 00:36:14.492
in the center

969
00:36:14.512 --> 00:36:16.516
of this slide. Wild images.

970
00:36:16.536 --> 00:36:19.547
That's a term referring to images that

971
00:36:19.567 --> 00:36:22.790
could appear on Flickr or in social media,

972
00:36:22.810 --> 00:36:25.730
or in photojournalism and sort of ironically,

973
00:36:25.750 --> 00:36:29.950
that graphic has been. Uh.

974
00:36:29.970 --> 00:36:31.291
Compressed somewhat by.

975
00:36:31.311 --> 00:36:33.530
By I think Adobe Software,

976
00:36:33.550 --> 00:36:34.646
which is not intended,

977
00:36:34.666 --> 00:36:36.375
but it sort of ironically represents

978
00:36:36.395 --> 00:36:37.940
a challenge for face recognition,

979
00:36:37.960 --> 00:36:39.068
so I'll move on.

980
00:36:39.088 --> 00:36:40.815
We've got millions of these images

981
00:36:40.835 --> 00:36:42.640
and we use them for benchmarking.

982
00:36:42.660 --> 00:36:44.110
What comes out of that?

983
00:36:44.130 --> 00:36:45.580
And you don't need to

984
00:36:45.600 --> 00:36:47.050
look at the numbers here,

985
00:36:47.070 --> 00:36:48.820
but this is essentially a leaderboard.

986
00:36:48.840 --> 00:36:49.844
It's a, it's a

987
00:36:49.864 --> 00:36:50.868
It's a performance table

988
00:36:50.888 --> 00:36:52.350
that we update regularly.

989
00:36:52.370 --> 00:36:54.331
The URL is at the bottom there

990
00:36:54.351 --> 00:36:55.839
and the numbers are giving

991
00:36:55.859 --> 00:36:57.640
you one to many error rates.

992
00:36:57.660 --> 00:36:59.789
So how often somebody is searched against

993
00:36:59.809 --> 00:37:02.150
the database and you don't find the answer?

994
00:37:02.170 --> 00:37:04.390
The correct answer and those numbers,

995
00:37:04.410 --> 00:37:04.764
although,

996
00:37:04.784 --> 00:37:07.010
but I'll walk you through it.

997
00:37:07.030 --> 00:37:09.940
So corny lists the algorithm and there's

998
00:37:09.960 --> 00:37:11.870
some famous algorithms in box B1

999
00:37:11.890 --> 00:37:14.870
and B2 from Idemia and NEC.

1000
00:37:14.890 --> 00:37:17.397
This sort of big industry players and

1001
00:37:17.417 --> 00:37:20.187
then across the top we apply those

1002
00:37:20.207 --> 00:37:22.232
that different images are different

1003
00:37:22.252 --> 00:37:24.960
kinds of images that I just showed you.

1004
00:37:24.980 --> 00:37:28.560
FBI custodian of mug shot.

1005
00:37:28.580 --> 00:37:29.925
FBI Border Patrol,

1006
00:37:29.945 --> 00:37:33.110
Border Patrol search images against the FBI.

1007
00:37:33.130 --> 00:37:37.270
So we sort of mimic that application.

1008
00:37:37.290 --> 00:37:39.244
We look at profile views and I'll

1009
00:37:39.264 --> 00:37:41.150
show you those in just a second.

1010
00:37:41.170 --> 00:37:42.870
Umm

1011
00:37:42.890 --> 00:37:43.544
Within DHS,

1012
00:37:43.564 --> 00:37:45.903
we look at CIS against KDP images

1013
00:37:45.923 --> 00:37:48.510
and then kiosks other border images.

1014
00:37:48.530 --> 00:37:51.214
And on the right so we look at

1015
00:37:51.234 --> 00:37:53.400
the aging problem, which I,

1016
00:37:53.420 --> 00:37:55.280
which I'll tell you about.

1017
00:37:55.300 --> 00:37:57.758
So you have sort of dig into

1018
00:37:57.778 --> 00:37:59.410
these numbers quite deeply.

1019
00:37:59.430 --> 00:38:01.290
But the algorithms are listed

1020
00:38:01.310 --> 00:38:02.800
on the left there,

1021
00:38:02.820 --> 00:38:05.310
and different algorithms perform differently.

1022
00:38:05.330 --> 00:38:07.248
And some are very good and

1023
00:38:07.268 --> 00:38:09.320
some are really not very good.

1024
00:38:09.340 --> 00:38:11.056
And and you can sort those columns

1025
00:38:11.076 --> 00:38:13.352
and sort of pick a database that's

1026
00:38:13.372 --> 00:38:14.840
appropriate to your application.

1027
00:38:14.860 --> 00:38:17.126
But the takeaway from this is

1028
00:38:17.146 --> 00:38:19.130
that the algorithm matters a lot.

1029
00:38:19.150 --> 00:38:21.400
As I said, it's not commoditize.

1030
00:38:21.420 --> 00:38:24.272
You can't just go and pick one off

1031
00:38:24.292 --> 00:38:26.710
the shelf like a Wi-Fi router.

1032
00:38:26.730 --> 00:38:27.602
Or yeah,

1033
00:38:27.622 --> 00:38:30.278
algorithm procurement is A is a

1034
00:38:30.298 --> 00:38:32.612
delicate topic and and you've got

1035
00:38:32.632 --> 00:38:35.550
to try and get the best algorithm.

1036
00:38:35.570 --> 00:38:37.660
The kinds of images matter too.

1037
00:38:37.680 --> 00:38:39.760
If we don't have good images,

1038
00:38:39.780 --> 00:38:43.260
we will sync recognition performance.

1039
00:38:43.280 --> 00:38:43.954
This is

1040
00:38:43.974 --> 00:38:46.383
A slide that I showed to show

1041
00:38:46.403 --> 00:38:48.750
that error rates of improved.

1042
00:38:48.770 --> 00:38:53.770
So it's a graph and it goes 2017 to 2021.

1043
00:38:53.790 --> 00:38:56.070
The the the numbers are error rates,

1044
00:38:56.090 --> 00:38:58.710
so we want the low error rates low,

1045
00:38:58.730 --> 00:39:00.999
false negative error rates and as you

1046
00:39:01.019 --> 00:39:03.990
can see for algorithms from this company.

1047
00:39:04.010 --> 00:39:05.097
This is typical.

1048
00:39:05.117 --> 00:39:07.680
The error rates have come down by,

1049
00:39:07.700 --> 00:39:09.820
you know, a factor of about thirty,

1050
00:39:09.840 --> 00:39:12.592
so thirty times fewer errors today than

1051
00:39:12.612 --> 00:39:15.130
than as recently as four years ago.

1052
00:39:15.150 --> 00:39:20.250
And so the consequence of that is.

1053
00:39:20.270 --> 00:39:21.985
That on identical database that

1054
00:39:22.005 --> 00:39:24.752
if if this database was a set of

1055
00:39:24.772 --> 00:39:26.370
sort of unsolved criminal cases,

1056
00:39:26.390 --> 00:39:29.436
say. Uh, that you might realize hits

1057
00:39:29.456 --> 00:39:32.743
in a database today with an algorithm

1058
00:39:32.763 --> 00:39:36.150
today that you didn't know you had

1059
00:39:36.170 --> 00:39:39.010
using an algorithm four years ago.

1060
00:39:39.030 --> 00:39:42.040
And the FBI is actually seen this as

1061
00:39:42.060 --> 00:39:43.877
it's done it's technology refresh.

1062
00:39:43.897 --> 00:39:46.031
So the takeaway from this slide

1063
00:39:46.051 --> 00:39:48.329
is that algorithms improve and

1064
00:39:48.349 --> 00:39:49.740
they continuously improve,

1065
00:39:49.760 --> 00:39:51.950
and this is evidence for it.

1066
00:39:51.970 --> 00:39:55.240
And it's it's industry wide essentially.

1067
00:39:55.260 --> 00:39:57.862
Uh, and so it becomes incumbent

1068
00:39:57.882 --> 00:39:59.610
to do tech refresh.

1069
00:39:59.630 --> 00:40:02.249
And to do that, whatever organization you

1070
00:40:02.269 --> 00:40:05.200
work for would need to have some kind of.

1071
00:40:05.220 --> 00:40:07.660
Uh, contractual procurement process

1072
00:40:07.680 --> 00:40:11.450
where you could upgrade and take

1073
00:40:11.470 --> 00:40:14.620
advantage of the latest technology.

1074
00:40:14.640 --> 00:40:17.070
I'll move on.

1075
00:40:17.090 --> 00:40:22.450
A question remains, what is the?

1076
00:40:22.470 --> 00:40:24.130
What is the remaining error?

1077
00:40:24.150 --> 00:40:26.790
If we've got such improvements, what remains?

1078
00:40:26.810 --> 00:40:30.858
And in this database, which is a mug

1079
00:40:30.878 --> 00:40:33.470
shots and border crossing images.

1080
00:40:33.490 --> 00:40:34.940
There's a number of problems,

1081
00:40:34.960 --> 00:40:36.700
but injury is one of them,

1082
00:40:36.720 --> 00:40:37.780
especially in mugshots.

1083
00:40:37.800 --> 00:40:40.671
People have got into a bar fight and

1084
00:40:40.691 --> 00:40:42.810
their face doesn't look as it did.

1085
00:40:42.830 --> 00:40:43.572
Maybe temporarily,

1086
00:40:43.592 --> 00:40:44.334
maybe not.

1087
00:40:44.354 --> 00:40:46.683
Also aging and in criminal settings

1088
00:40:46.703 --> 00:40:47.270
you can't.

1089
00:40:47.290 --> 00:40:49.060
You might not encounter somebody

1090
00:40:49.080 --> 00:40:51.370
for maybe a twenty year period,

1091
00:40:51.390 --> 00:40:54.376
so aging matters and I'll show

1092
00:40:54.396 --> 00:40:57.450
you that in just a second.

1093
00:40:57.470 --> 00:40:59.610
Low quality is also an issue.

1094
00:40:59.630 --> 00:41:01.662
You could always produce images that

1095
00:41:01.682 --> 00:41:04.062
are poor enough such that you will

1096
00:41:04.082 --> 00:41:06.006
get recognition errors and and there

1097
00:41:06.026 --> 00:41:08.610
are some poor quality mugshots remaining,

1098
00:41:08.630 --> 00:41:11.130
but a small percentage as you can

1099
00:41:11.150 --> 00:41:13.339
see .3% so will

1100
00:41:13.359 --> 00:41:15.810
go on and I'll talk about aging,

1101
00:41:15.830 --> 00:41:20.100
so these are photos of me going back

1102
00:41:20.120 --> 00:41:22.104
Twenty almost twenty years.

1103
00:41:22.124 --> 00:41:26.290
And you know, despite best efforts, you know.

1104
00:41:26.310 --> 00:41:28.666
You know trying to do some exercise and

1105
00:41:28.686 --> 00:41:30.814
stay out of the sun and moisturizing

1106
00:41:30.834 --> 00:41:32.740
and all that kind of stuff.

1107
00:41:32.760 --> 00:41:35.420
You know, facial appearance changes and

1108
00:41:35.440 --> 00:41:38.048
We're all familiar with with meeting

1109
00:41:38.068 --> 00:41:41.170
people you know along time ago and wow,

1110
00:41:41.190 --> 00:41:43.870
you know the appearance changed.

1111
00:41:43.890 --> 00:41:45.770
And and of course,

1112
00:41:45.790 --> 00:41:48.730
the that matters to a face

1113
00:41:48.750 --> 00:41:51.900
recognition algorithm because.

1114
00:41:51.920 --> 00:41:54.400
If it's comparing two recent photos,

1115
00:41:54.420 --> 00:41:56.480
then they are quite similar,

1116
00:41:56.500 --> 00:41:59.204
but over enough time span then

1117
00:41:59.224 --> 00:42:01.600
you've got less similarity and

1118
00:42:01.620 --> 00:42:04.060
face recognition always works by.

1119
00:42:04.080 --> 00:42:08.560
Wanting similar inputs so similar viewpoints,

1120
00:42:08.580 --> 00:42:10.120
similar resolutions similar

1121
00:42:10.140 --> 00:42:13.843
1ighting and any sort of deviation from

1122
00:42:13.863 --> 00:42:16.600
similar will upset face recognition,

1123
00:42:16.620 --> 00:42:20.050
so an aging is quite an

1124
00:42:20.070 --> 00:42:22.350
important component of that.

1125
00:42:22.370 --> 00:42:24.498
And and it's sort of irreversible

1126
00:42:24.518 --> 00:42:25.930
we you know we,

1127
00:42:25.950 --> 00:42:27.720
we don't really improve our

1128
00:42:27.740 --> 00:42:29.510
appearance easily and population wide.

1129
00:42:29.530 --> 00:42:30.942
It will eventually undermine

1130
00:42:30.962 --> 00:42:31.660
face recognition.

1131
00:42:31.680 --> 00:42:33.808
So that's what this one is

1132
00:42:33.828 --> 00:42:35.240
showing that these things,

1133
00:42:35.260 --> 00:42:38.460
if we run instead of just looking at me,

1134
00:42:38.480 --> 00:42:41.211
we look at a few hundred thousand

1135
00:42:41.231 --> 00:42:44.482
people and we track them over a period

1136
00:42:44.502 --> 00:42:47.750
of up to eighteen years on this chart.

1137
00:42:47.770 --> 00:42:50.920
Those blue violin sort of cases.

1138
00:42:50.940 --> 00:42:54.630
That is a similarity scores going down,

1139
00:42:54.650 --> 00:42:57.600
so people are becoming less

1140
00:42:57.620 --> 00:43:00.570
similar to their old photos.

1141
00:43:00.590 --> 00:43:02.414
And that eventually submerges

1142
00:43:02.434 --> 00:43:03.340
face recognition.

1143
00:43:03.360 --> 00:43:05.536
But below these threshold

1144
00:43:05.556 --> 00:43:08.830
values and error rates creep up.

1145
00:43:08.850 --> 00:43:11.000
You can see the Microsoft algorithm on

1146
00:43:11.020 --> 00:43:13.364
the right is actually better than the

1147
00:43:13.384 --> 00:43:15.780
algorithm from Entec lab on the left.

1148
00:43:15.800 --> 00:43:17.435
So reinforcing the point that

1149
00:43:17.455 --> 00:43:19.090
capability varies across the industry,

1150
00:43:19.110 --> 00:43:22.420
but aging is a big problem.

1151
00:43:22.440 --> 00:43:25.654
The takeaway here is that aging gives

1152
00:43:25.674 --> 00:43:27.910
reduced similarity and therefore

1153
00:43:27.930 --> 00:43:30.280
eventually recognition failure.

1154
00:43:30.300 --> 00:43:33.016
So if you can arrange to get regular

1155
00:43:33.036 --> 00:43:35.774
new photos as passport agencies, do.

1156
00:43:35.794 --> 00:43:38.558
And do so. But criminal justice

1157
00:43:38.578 --> 00:43:41.470
doesn't really have that opportunity.

1158
00:43:41.490 --> 00:43:46.480
And so, Yep, aging matters.

1159
00:43:46.500 --> 00:43:49.406
Now this is a trying to distinguish

1160
00:43:49.426 --> 00:43:52.250
now the effect of aging from age.

1161
00:43:52.270 --> 00:43:55.074
So if we put this this is this

1162
00:43:55.094 --> 00:43:58.561
is a little heat map and what

1163
00:43:58.581 --> 00:44:01.240
it's showing is error rates.

1164
00:44:01.260 --> 00:44:03.704
For putting somebody in a particular age

1165
00:44:03.724 --> 00:44:06.299
group in a database and then searching

1166
00:44:06.319 --> 00:44:08.957
it ten years later so the youngest

1167
00:44:08.977 --> 00:44:11.680
age group here is twelve to fifteen,

1168
00:44:11.700 --> 00:44:14.690
so young teenage female put into a

1169
00:44:14.710 --> 00:44:17.210
database and searched a decade later.

1170
00:44:17.230 --> 00:44:18.950
We could see an 8.6%

1171
00:44:18.970 --> 00:44:20.014
miss rate.

1172
00:44:20.034 --> 00:44:22.140
We wouldn't find them in the database.

1173
00:44:22.160 --> 00:44:23.840
That contrasts to somebody at

1174
00:44:23.860 --> 00:44:25.920
age thirty to forty five when

1175
00:44:25.940 --> 00:44:27.810
they were put in the database.

1176
00:44:27.830 --> 00:44:30.390
Uh, and the error rate there is

1177
00:44:30.410 --> 00:44:32.530
about 2.1%.

1178
00:44:32.550 --> 00:44:35.358
So a factor of four fewer errors

1179
00:44:35.378 --> 00:44:38.300
in adults than in young teenagers.

1180
00:44:38.320 --> 00:44:42.090
And you can do that for men as well,

1181
00:44:42.110 --> 00:44:45.600
and it typical across a lot

1182
00:44:45.620 --> 00:44:47.360
of face recognition.

1183
00:44:47.380 --> 00:44:49.490
That men are typically better recognized

1184
00:44:49.510 --> 00:44:51.880
than women only by a small amount,

1185
00:44:51.900 --> 00:44:53.970
but by an amount that is,

1186
00:44:53.990 --> 00:44:56.210
That is quite commonly observed

1187
00:44:56.230 --> 00:44:58.010
in in in tests.

1188
00:44:58.030 --> 00:45:01.181
And it turns out that maybe there's

1189
00:45:01.201 --> 00:45:03.617
a small difference between young

1190
00:45:03.637 --> 00:45:07.080
teenage guys than than than than women.

1191
00:45:07.100 --> 00:45:08.688
Different obviously aging is

1192
00:45:08.708 --> 00:45:10.698
happening quite quickly during the

1193
00:45:10.718 --> 00:45:12.489
teen years and facial appearance

1194
00:45:12.509 --> 00:45:14.480
changes and and so you know,

1195
00:45:14.500 --> 00:45:16.874
it's a takeaway here is that you

1196
00:45:16.894 --> 00:45:18.791
know the application of face

1197
00:45:18.811 --> 00:45:20.030
recognition to children,

1198
00:45:20.050 --> 00:45:21.510
including children younger than

1199
00:45:21.530 --> 00:45:24.475
a shown on this slide is a sort

1200
00:45:24.495 --> 00:45:26.160
of a deliberate policy decision

1201
00:45:26.180 --> 00:45:28.432
based on how well face recognition

1202
00:45:28.452 --> 00:45:30.020
is expected to work,

1203
00:45:30.040 --> 00:45:32.554
and this sort of data informs those

1204
00:45:32.574 --> 00:45:35.047
decisions and it informs our decisions

1205
00:45:35.067 --> 00:45:38.170
about how often you would issue passports.

1206
00:45:38.190 --> 00:45:41.562
Canada is uses passports on a five year

1207
00:45:41.582 --> 00:45:44.756
timeline for all individuals. U S.

1208
00:45:44.776 --> 00:45:48.110
Does passports on the ten year timeline.

1209
00:45:48.130 --> 00:45:50.980
So this is now showing that previous

1210
00:45:51.000 --> 00:45:54.599
effect that I just showed you age up

1211
00:45:54.619 --> 00:45:57.720
enrollment and also now timelapse up there.

1212
00:45:57.740 --> 00:46:00.261
From ten years up to fifteen years

1213
00:46:00.281 --> 00:46:03.119
and you can see that there's a

1214
00:46:03.139 --> 00:46:05.230
slow increase in error rates.

1215
00:46:05.250 --> 00:46:08.219
This graceful sort of aging quoted error

1216
00:46:08.239 --> 00:46:10.760
rates to increase as time goes by,

1217
00:46:10.780 --> 00:46:13.520
and I won't dwell on this anymore,

1218
00:46:13.540 --> 00:46:17.132
but aging is something to be avoided

1219
00:46:17.152 --> 00:46:18.170
if possible.

1220
00:46:18.190 --> 00:46:18.489
Uh,

1221
00:46:18.509 --> 00:46:21.439
this one is a figure and it's trying to

1222
00:46:21.459 --> 00:46:24.410
sort of contrast three different things.

1223
00:46:24.430 --> 00:46:26.595
So accurate algorithms at the

1224
00:46:26.615 --> 00:46:28.343
bottom inaccurate algorithms at

1225
00:46:28.363 --> 00:46:30.260
the top on the left side.

1226
00:46:30.280 --> 00:46:32.756
What happens when we have a

1227
00:46:32.776 --> 00:46:34.940
database size that increases up to,

1228
00:46:34.960 --> 00:46:36.410
say, twelve million?

1229
00:46:36.430 --> 00:46:37.880
And you know,

1230
00:46:37.900 --> 00:46:40.008
I can, I can tell you that the

1231
00:46:40.028 --> 00:46:41.620
State Department and DHS have much,

1232
00:46:41.640 --> 00:46:44.150
much larger databases than 12 million.

1233
00:46:44.170 --> 00:46:46.270
But as you put more and more people

1234
00:46:46.290 --> 00:46:49.150
in the database, accuracy degrades.

1235
00:46:49.170 --> 00:46:51.598
But I want to compare that with the effect

1236
00:46:51.618 --> 00:46:53.930
of aging that I just told you about,

1237
00:46:53.950 --> 00:46:56.460
and so the more spread out these dots are,

1238
00:46:56.480 --> 00:46:58.398
the higher the error rates after

1239
00:46:58.418 --> 00:47:00.887
on the left you have a big database

1240
00:47:00.907 --> 00:47:03.260
and on the right a lot of aging.

1241
00:47:03.280 --> 00:47:05.310
So you can see that aging,

1242
00:47:05.330 --> 00:47:07.360
at least with this mugshot data,

1243
00:47:07.380 --> 00:47:09.850
is a more influential variable.

1244
00:47:09.870 --> 00:47:14.650
That is population size in your database.

1245
00:47:14.670 --> 00:47:16.486
But keeping that sort of the

1246
00:47:16.506 --> 00:47:19.030
eye on on the perspective here,

1247
00:47:19.050 --> 00:47:21.220
the algorithm matters a lot more.

1248
00:47:21.240 --> 00:47:24.140
So if we look at those algorithms listed,

1249
00:47:24.160 --> 00:47:25.652
bottom of this chart,

1250
00:47:25.672 --> 00:47:27.542
you can barely perceive the

1251
00:47:27.562 --> 00:47:29.620
increase in error rates for aging.

1252
00:47:29.640 --> 00:47:31.384
All for population size.

1253
00:47:31.404 --> 00:47:33.150
And so you know,

1254
00:47:33.170 --> 00:47:36.102
accurate at the algorithm is the most sort

1255
00:47:36.122 --> 00:47:38.400
of influential variable on this chart.

1256
00:47:38.420 --> 00:47:40.650
Long term aging is more influential

1257
00:47:40.670 --> 00:47:44.820
and database side, so mileage varies.

1258
00:47:44.840 --> 00:47:48.270
I think we need to switch presentation now.

1259
00:47:48.290 --> 00:47:51.320
Abby, please all right and I'm gonna.

1260
00:47:51.340 --> 00:47:53.536
Abby: Yes, we're going to make a quick transition

1261
00:47:53.556 --> 00:47:55.940
over to a new document share and Patrick.

1262
00:47:55.960 --> 00:47:58.830
Feel free if you want to turn on your webcam.

1263
00:47:58.850 --> 00:48:00.280
If anyone has any questions,

1264
00:48:00.300 --> 00:48:02.300
please send them through the chat pod.

1265
00:48:02.320 --> 00:48:03.750
Hope my webcam is on,

1266
00:48:03.770 --> 00:48:09.740
it looks like it's on.

1267
00:48:09.760 --> 00:48:11.230


1268
00:48:11.250 --> 00:48:13.450
And we can't see you yet.

1269
00:48:13.470 --> 00:48:15.670
It should be the icon next

1270
00:48:15.690 --> 00:48:17.320
to your speaker, yeah?

1271
00:48:17.340 --> 00:48:18.952
20 minutes ago.

1272
00:48:18.972 --> 00:48:22.680
So yeah, start sharing.

1273
00:48:22.700 --> 00:48:23.780
OK, no, that's OK.

1274
00:48:23.800 --> 00:48:26.234
I know we got to see your face

1275
00:48:26.254 --> 00:48:28.410
on some of your slides already.

1276
00:48:28.430 --> 00:48:34.400
We thought we saw you momentarily Packer.

1277
00:48:34.420 --> 00:48:37.020


1278
00:48:37.040 --> 00:48:40.952
There we go. OK, so we have a image of me

1279
00:48:40.972 --> 00:48:43.817
and now we're going to talk about image

1280
00:48:43.837 --> 00:48:46.540
quality and we touched on this earlier.

1281
00:48:46.560 --> 00:48:49.290
I need to go back a couple of slides.

1282
00:48:49.310 --> 00:48:51.040
OK, so image quality.

1283
00:48:51.060 --> 00:48:53.240
Across the top here.

1284
00:48:53.260 --> 00:48:56.390
I've got a sort of a spectrum of images.

1285
00:48:56.410 --> 00:48:58.364
We start off with those passport

1286
00:48:58.384 --> 00:49:00.590
style and mug shot style photos,

1287
00:49:00.610 --> 00:49:02.854
but on the right side we end up

1288
00:49:02.874 --> 00:49:05.539
with a sort of a black image

1289
00:49:05.559 --> 00:49:07.590
where there's very little signal.

1290
00:49:07.610 --> 00:49:09.872
And as you can see as we

1291
00:49:09.892 --> 00:49:11.440
go across this slide,

1292
00:49:11.460 --> 00:49:13.190
different things are going wrong.

1293
00:49:13.210 --> 00:49:14.990
In illumination is bad,

1294
00:49:15.010 --> 00:49:17.690
the presentation view angle is bad.

1295
00:49:17.710 --> 00:49:20.610
Compression is bad, illumination is bad,

1296
00:49:20.630 --> 00:49:23.850
so that matters and

1297
00:49:23.870 --> 00:49:25.675
You know an influential variable

1298
00:49:25.695 --> 00:49:28.242
in in figuring out whether you can

1299
00:49:28.262 --> 00:49:29.940
do face recognition is whether

1300
00:49:29.960 --> 00:49:31.770
you have subject cooperation.

1301
00:49:31.790 --> 00:49:33.650
If we're doing video surveillance

1302
00:49:33.670 --> 00:49:34.780
in a casino,

1303
00:49:34.800 --> 00:49:36.772
we typically would not

1304
00:49:36.792 --> 00:49:38.270
have subject cooperation.

1305
00:49:38.290 --> 00:49:39.702
If we're doing application

1306
00:49:39.722 --> 00:49:40.780
for unemployment benefits,

1307
00:49:40.800 --> 00:49:43.237
we would have very good subject

1308
00:49:43.257 --> 00:49:45.076
cooperation and so that matters

1309
00:49:45.096 --> 00:49:47.224
because people will look at the

1310
00:49:47.244 --> 00:49:50.462
camera and they will make a sort of a

1311
00:49:50.482 --> 00:49:52.378
conformant presentations of the camera.

1312
00:49:52.398 --> 00:49:54.952
But in those applications where we

1313
00:49:54.972 --> 00:49:58.060
don't have cooperation or maybe even

1314
00:49:58.080 --> 00:50:00.790
active non cooperation, un cooperation.

1315
00:50:00.810 --> 00:50:02.560
Then you can

1316
00:50:02.580 --> 00:50:06.130
You can subvert the system

1317
00:50:06.150 --> 00:50:09.010
by having poor quality photos and you know,

1318
00:50:09.030 --> 00:50:10.810
it depends on photographer skill.

1319
00:50:10.830 --> 00:50:13.100
It depends on whether we're attempting

1320
00:50:13.120 --> 00:50:15.850
to conform to a sort of a

1321
00:50:15.870 --> 00:50:19.080
standard or best practice imaging design.

1322
00:50:19.100 --> 00:50:19.498
Uh,

1323
00:50:19.518 --> 00:50:21.170
and uh and what?

1324
00:50:21.190 --> 00:50:23.480
The sort of reference databases if

1325
00:50:23.500 --> 00:50:26.425
if we're trying to search Facebook or

1326
00:50:26.445 --> 00:50:29.530
Google photo indexing thing on your phone,

1327
00:50:29.550 --> 00:50:31.198
then essentially those photo

1328
00:50:31.218 --> 00:50:33.283
qualities are sort of unregulated

1329
00:50:33.303 --> 00:50:35.380
or not very well controlled,

1330
00:50:35.400 --> 00:50:38.092
and so image quality matters and

1331
00:50:38.112 --> 00:50:40.813
ultimately you can force the error

1332
00:50:40.833 --> 00:50:43.291
rates up to a hundred percent

1333
00:50:43.311 --> 00:50:46.100
failure rate by using images with

1334
00:50:46.120 --> 00:50:48.451
with essentially no signal.

1335
00:50:48.471 --> 00:50:51.540
We run a quite a challenging benchmark.

1336
00:50:51.560 --> 00:50:53.696
That image in the center there is

1337
00:50:53.716 --> 00:50:55.846
a profile view and they've been

1338
00:50:55.866 --> 00:50:57.771
collected in criminal justice for

1339
00:50:57.791 --> 00:50:59.000
a hundred years.

1340
00:50:59.020 --> 00:51:01.360
And and to be able to match a

1341
00:51:01.380 --> 00:51:03.190
profile view against the frontal

1342
00:51:03.210 --> 00:51:05.540
view taken on a different day

1343
00:51:05.560 --> 00:51:07.826
has been a sort of a the Holy Grail

1344
00:51:07.846 --> 00:51:10.500
of academic research now for a decade,

1345
00:51:10.520 --> 00:51:12.264
and it's quite a challenging problem

1346
00:51:12.284 --> 00:51:15.150
if you if you tried to recognize a friend,

1347
00:51:15.170 --> 00:51:17.320
sometimes from a a side on view,

1348
00:51:17.340 --> 00:51:18.960
it actually is quite difficult

1349
00:51:18.980 --> 00:51:21.216
and in your brain sort of pauses

1350
00:51:21.236 --> 00:51:22.590
in trying to do that.

1351
00:51:22.610 --> 00:51:23.786
The same with algorithms,

1352
00:51:23.806 --> 00:51:25.640
and it turns out that there's

1353
00:51:25.660 --> 00:51:27.404
only a handful of algorithms that

1354
00:51:27.424 --> 00:51:29.305
are capable of doing this today

1355
00:51:29.325 --> 00:51:30.650
most algorithms will fail.

1356
00:51:30.670 --> 00:51:32.510
They won't find the face features

1357
00:51:32.530 --> 00:51:33.750
that they're looking for,

1358
00:51:33.770 --> 00:51:35.720
and they will give you very.

1359
00:51:35.740 --> 00:51:37.072
Very high error rates.

1360
00:51:37.092 --> 00:51:39.966
So what you see on the right there

1361
00:51:39.986 --> 00:51:42.312
is a list of algorithms that

1362
00:51:42.332 --> 00:51:44.230
are capable of doing this.

1363
00:51:44.250 --> 00:51:46.834
The the error rates here today

1364
00:51:46.854 --> 00:51:48.657
on profile view matching

1365
00:51:48.677 --> 00:51:50.860
where banks where. Frontal frontal

1366
00:51:50.880 --> 00:51:53.230
search was in about 2010,

1367
00:51:53.250 --> 00:51:55.750
so we ran a benchmark for the FBI

1368
00:51:55.770 --> 00:51:58.722
back in 2010 that was used

1369
00:51:58.742 --> 00:52:01.067
towards their procurement of their

1370
00:52:01.087 --> 00:52:03.910
first face recognition search engine.

1371
00:52:03.930 --> 00:52:07.960
And in 2010 we recorded accuracy and

1372
00:52:07.980 --> 00:52:10.294
the accuracy then was mugshot

1373
00:52:10.314 --> 00:52:13.099
to mugshot searching and now we see

1374
00:52:13.119 --> 00:52:15.070
these profiles or mugshot searching.

1375
00:52:15.090 --> 00:52:17.770
Error rates are about the same but

1376
00:52:17.790 --> 00:52:21.035
it remains a very very tough task

1377
00:52:21.055 --> 00:52:23.500
for most developers and and it's

1378
00:52:23.520 --> 00:52:26.155
evidence that there has been a

1379
00:52:26.175 --> 00:52:28.760
lot of progress in face recognition.

1380
00:52:28.780 --> 00:52:31.595
And particularly being able to do

1381
00:52:31.615 --> 00:52:33.920
face recognition at angles other

1382
00:52:33.940 --> 00:52:35.320
than frontal view,

1383
00:52:35.340 --> 00:52:38.840
and that would be useful in things like,

1384
00:52:38.860 --> 00:52:41.920
you know, video games or video surveillance,

1385
00:52:41.940 --> 00:52:43.524
people walking into casinos,

1386
00:52:43.544 --> 00:52:46.006
all sorts of views could occur

1387
00:52:46.026 --> 00:52:47.640
in different operations,

1388
00:52:47.660 --> 00:52:50.898
and being able to tolerate various

1389
00:52:50.918 --> 00:52:54.050
departures from frontal is important.

1390
00:52:54.070 --> 00:52:57.330
I'll continue.

1391
00:52:57.350 --> 00:53:00.220
It's possible to buy software from

1392
00:53:00.240 --> 00:53:02.920
various developers that will give you

1393
00:53:02.940 --> 00:53:05.134
image quality measures and they might

1394
00:53:05.154 --> 00:53:07.050
identify particular defects in images.

1395
00:53:07.070 --> 00:53:10.050
I've listed a few of those at the

1396
00:53:10.070 --> 00:53:16.040
bottom here, and you can think of.

1397
00:53:16.060 --> 00:53:31.520
[audio goes out] [silence]

1398
00:53:31.540 --> 00:53:33.963
Right, you know correcting an imaging system

1399
00:53:33.983 --> 00:53:36.631
if the if there's inadequate illumination

1400
00:53:36.651 --> 00:53:39.733
or nonuniform illumination if it's over

1401
00:53:39.753 --> 00:53:41.854
exposing or underexposing particular

1402
00:53:41.874 --> 00:53:44.920
people if it's systematically miss focused,

1403
00:53:44.940 --> 00:53:48.610
or is not correctly focusing on a subject.

1404
00:53:48.630 --> 00:53:51.844
If it's very low resolution, that would

1405
00:53:51.864 --> 00:53:55.540
be a problem on the right side you.

1406
00:53:55.560 --> 00:53:57.280
You're also subject to

1407
00:53:57.300 --> 00:54:00.690
what a person does when they're in front

1408
00:54:00.710 --> 00:54:04.057
of the camera and they might not look at

1409
00:54:04.077 --> 00:54:07.010
the camera so they can roll their head.

1410
00:54:07.030 --> 00:54:08.780
They could pitch it down,

1411
00:54:08.800 --> 00:54:11.062
they could roll it sideways and and

1412
00:54:11.082 --> 00:54:12.670
those are influential variables.

1413
00:54:12.690 --> 00:54:14.524
They might pull a funny expression

1414
00:54:14.544 --> 00:54:16.254
that can decrease the similarity

1415
00:54:16.274 --> 00:54:17.980
with their reference photos.

1416
00:54:18.000 --> 00:54:19.810
Wearing sunglasses is a particular

1417
00:54:19.830 --> 00:54:22.360
kind of occlusion of the face that

1418
00:54:22.380 --> 00:54:24.000
will cause trouble as well.

1419
00:54:24.020 --> 00:54:26.240
Motion blur their capture quality

1420
00:54:26.260 --> 00:54:28.040
standards and imaging standards.

1421
00:54:28.060 --> 00:54:32.300
For scanning for photography that

1422
00:54:32.320 --> 00:54:35.710
are published and underdevelopment.

1423
00:54:35.730 --> 00:54:37.320
So

1424
00:54:37.340 --> 00:54:39.180
With the the onset of

1425
00:54:39.200 --> 00:54:41.040
the pandemic a year ago.

1426
00:54:41.060 --> 00:54:43.840
And I think

1427
00:54:43.860 --> 00:54:44.892
Yevgeny will talk about

1428
00:54:44.912 --> 00:54:46.210
this in the next segment,

1429
00:54:46.230 --> 00:54:50.229
but the issue came up of can you

1430
00:54:50.249 --> 00:54:51.674
do face recognition when people

1431
00:54:51.694 --> 00:54:53.430
are still wearing a face mask?

1432
00:54:53.450 --> 00:54:59.590
And so we set up a synthetic database and

1433
00:54:59.610 --> 00:55:01.726
We didn't have time or money to be

1434
00:55:01.746 --> 00:55:04.449
able to go and collect hundreds of

1435
00:55:04.469 --> 00:55:06.150
thousands of individuals quickly,

1436
00:55:06.170 --> 00:55:08.220
so we took a quick look.

1437
00:55:08.240 --> 00:55:11.398
A first order guess at how well

1438
00:55:11.418 --> 00:55:13.110
face recognition holds up.

1439
00:55:13.130 --> 00:55:14.923
When you cover the face and you

1440
00:55:14.943 --> 00:55:17.060
can see on the right side there.

1441
00:55:17.080 --> 00:55:20.620
Uh, various masks and and.

1442
00:55:20.640 --> 00:55:24.350
On the next slide, I will show you some more.

1443
00:55:24.370 --> 00:55:26.650
And so we covered the face digitally.

1444
00:55:26.670 --> 00:55:29.098
We we painted a sort of solid colored

1445
00:55:29.118 --> 00:55:31.920
masks on their face and ask the question,

1446
00:55:31.940 --> 00:55:33.770
what happens when you cover

1447
00:55:33.790 --> 00:55:35.695
70% or 50%

1448
00:55:35.715 --> 00:55:37.180
of the face with a mask,

1449
00:55:37.200 --> 00:55:38.750
and that would never really

1450
00:55:38.770 --> 00:55:40.800
going to be a good thing.

1451
00:55:40.820 --> 00:55:42.244
Face recognition industry had

1452
00:55:42.264 --> 00:55:44.865
grown up expecting sort of a mostly

1453
00:55:44.885 --> 00:55:46.790
unencumbered view of the face.

1454
00:55:46.810 --> 00:55:49.250
And when they don't get it,

1455
00:55:49.270 --> 00:55:51.300
some algorithms failed almost catastrophic.

1456
00:55:51.320 --> 00:55:52.713
Completely unusable.

1457
00:55:52.733 --> 00:55:56.730
A handful did turn out to be useful.

1458
00:55:56.750 --> 00:55:58.062
Pre pandemic algorithms.

1459
00:55:58.082 --> 00:55:58.950
Of course,

1460
00:55:58.970 --> 00:56:01.378
the vendor community jumped

1461
00:56:01.398 --> 00:56:03.200
forward and various.

1462
00:56:03.220 --> 00:56:03.617
UH,

1463
00:56:03.637 --> 00:56:06.119
members of the industry developed a

1464
00:56:06.139 --> 00:56:08.880
capability to just look at that region.

1465
00:56:08.900 --> 00:56:11.370
He showed it on the previous slide.

1466
00:56:11.390 --> 00:56:13.477
That region in the green box there

1467
00:56:13.497 --> 00:56:15.695
on the right side where there's

1468
00:56:15.715 --> 00:56:17.780
an information rich region there,

1469
00:56:17.800 --> 00:56:20.097
and you can if you can just

1470
00:56:20.117 --> 00:56:22.050
do recognition on that region.

1471
00:56:22.070 --> 00:56:25.630
How well can you achieve face recognition so?

1472
00:56:25.650 --> 00:56:26.165
Again,

1473
00:56:26.185 --> 00:56:28.840
we published reports pre pandemic

1474
00:56:28.860 --> 00:56:31.061
algorithms and ones developed

1475
00:56:31.081 --> 00:56:33.620
since the pandemic and we continue

1476
00:56:33.640 --> 00:56:39.619
to track accuracy there.

1477
00:56:39.639 --> 00:56:41.160


1478
00:56:41.180 --> 00:56:44.470
And it turns out that for the.

1479
00:56:44.490 --> 00:56:47.242
The leading developers of mask enabled

1480
00:56:47.262 --> 00:56:49.838
algorithms that accuracy is usable and

1481
00:56:49.858 --> 00:56:52.530
it it it has reset the industry back.

1482
00:56:52.550 --> 00:56:53.706
Maybe two, three,

1483
00:56:53.726 --> 00:56:55.666
four years depending on which

1484
00:56:55.686 --> 00:56:57.370
algorithm you're talking about.

1485
00:56:57.390 --> 00:56:59.780
But error rates have come down,

1486
00:56:59.800 --> 00:57:03.280
which that's what these graphs are showing.

1487
00:57:03.300 --> 00:57:07.235
and pushing, putting a mask on

1488
00:57:07.255 --> 00:57:08.890
the face increases error rates.

1489
00:57:08.910 --> 00:57:10.322
Some algorithms are usable,

1490
00:57:10.342 --> 00:57:13.316
but again you would have to be quite

1491
00:57:13.336 --> 00:57:15.270
careful in choosing an algorithm.

1492
00:57:15.290 --> 00:57:16.890
And maybe testing it to make

1493
00:57:16.910 --> 00:57:18.420
sure that it was effective

1494
00:57:18.440 --> 00:57:22.900
on people wearing face masks.

1495
00:57:22.920 --> 00:57:24.908
Twins switching gears

1496
00:57:24.928 --> 00:57:27.420
now is a separate topic.

1497
00:57:27.440 --> 00:57:30.450
You can see a pair of photos here,

1498
00:57:30.470 --> 00:57:33.144
collected by the University of Notre

1499
00:57:33.164 --> 00:57:36.296
Dame at a festival in Ohio that

1500
00:57:36.316 --> 00:57:38.410
happens every year called Twins

1501
00:57:38.430 --> 00:57:41.150
Day and so twins and triplets

1502
00:57:41.170 --> 00:57:43.334
come to this thing and they talk

1503
00:57:43.354 --> 00:57:45.700
about all sorts of issues relatively,

1504
00:57:45.720 --> 00:57:47.368
you know important to twins and

1505
00:57:47.388 --> 00:57:50.037
and one is well can they do face

1506
00:57:50.057 --> 00:57:51.529
recognition or iris recognition

1507
00:57:51.549 --> 00:57:53.050
or fingerprint recognition.

1508
00:57:53.070 --> 00:57:55.075
But face recognition turns out

1509
00:57:55.095 --> 00:57:57.944
to be a challenge for twins and

1510
00:57:57.964 --> 00:57:59.840
so why does this matter?

1511
00:57:59.860 --> 00:58:02.416
Twins are about 3% of all

1512
00:58:02.436 --> 00:58:04.900
babies born in the United States.

1513
00:58:04.920 --> 00:58:07.230
In 2015 were a twin

1514
00:58:07.250 --> 00:58:08.790
140,000

1515
00:58:08.810 --> 00:58:11.580
out of 4 million births.

1516
00:58:11.600 --> 00:58:13.412
And CDC maintains good

1517
00:58:13.432 --> 00:58:15.710
statistics on that every year,

1518
00:58:15.730 --> 00:58:18.000
and identical twins are less

1519
00:58:18.020 --> 00:58:19.840
common than fraternal twins.

1520
00:58:19.860 --> 00:58:22.516
The the the frequency of twins

1521
00:58:22.536 --> 00:58:25.350
has gone up over the years.

1522
00:58:25.370 --> 00:58:27.070
It varies considerably worldwide.

1523
00:58:27.090 --> 00:58:30.520
Very different in Japan than in West Africa,

1524
00:58:30.540 --> 00:58:32.070
for example, between pants up,

1525
00:58:32.090 --> 00:58:33.630
and they're becoming more common,

1526
00:58:33.650 --> 00:58:35.723
so it matters because if we would

1527
00:58:35.743 --> 00:58:37.980
have put a twin in a database,

1528
00:58:38.000 --> 00:58:40.300
just one of the twins.

1529
00:58:40.320 --> 00:58:42.040
The gallery size of one point,

1530
00:58:42.060 --> 00:58:43.380
6 million I think.

1531
00:58:43.400 --> 00:58:45.763
Mugshot it photos and we put one

1532
00:58:45.783 --> 00:58:47.870
of the twins in the database and

1533
00:58:47.890 --> 00:58:50.390
then we search with the other twin.

1534
00:58:50.410 --> 00:58:52.434
What comes back on the candidate list

1535
00:58:52.454 --> 00:58:55.148
at the bottom there is is from three

1536
00:58:55.168 --> 00:58:56.930
different algorithms that we evaluated.

1537
00:58:56.950 --> 00:58:59.220
That twin comes back at rank one,

1538
00:58:59.240 --> 00:59:01.116
so an investigator would have to

1539
00:59:01.136 --> 00:59:03.117
look at the list of candidates

1540
00:59:03.137 --> 00:59:04.780
and would have to say,

1541
00:59:04.800 --> 00:59:07.390
well is that the same person or not?

1542
00:59:07.410 --> 00:59:08.698
'cause they wouldn't know

1543
00:59:08.718 --> 00:59:10.010
it to twin automatically.

1544
00:59:10.030 --> 00:59:11.780
Many databases that are that exist

1545
00:59:11.800 --> 00:59:13.930
in the world don't actually record

1546
00:59:13.950 --> 00:59:15.470
Are you a twin?

1547
00:59:15.490 --> 00:59:18.262
So an investigator would have to be

1548
00:59:18.282 --> 00:59:20.668
alert so that possibility and will

1549
00:59:20.688 --> 00:59:23.512
then have to figure out is it a

1550
00:59:23.532 --> 00:59:26.128
twin and if it's not a twin is it.

1551
00:59:26.148 --> 00:59:28.287
Is it the same person who just

1552
00:59:28.307 --> 00:59:29.850
remarkably similar person?

1553
00:59:29.870 --> 00:59:34.670
Or is it actually you know the same person?

1554
00:59:34.690 --> 00:59:37.090
Really, that wouldn't be known going in.

1555
00:59:37.110 --> 00:59:39.500
That's the situation with an identical twin,

1556
00:59:39.520 --> 00:59:41.225
and the takeaway is identical

1557
00:59:41.245 --> 00:59:42.610
twins give false positives,

1558
00:59:42.630 --> 00:59:45.658
and that applies to almost all

1559
00:59:45.678 --> 00:59:48.040
algorithms that we've looked at.

1560
00:59:48.060 --> 00:59:49.340
With the fraternal twin,

1561
00:59:49.360 --> 00:59:51.290
some fraternal twins are different sex,

1562
00:59:51.310 --> 00:59:53.474
and that turns out not to be a

1563
00:59:53.494 --> 00:59:55.190
problem for face recognition.

1564
00:59:55.210 --> 00:59:57.790
But for two of the three algorithms here,

1565
00:59:57.810 --> 01:00:00.555
they both put the other twin here

1566
01:00:00.575 --> 01:00:01.740
at rank one.

1567
01:00:01.760 --> 01:00:03.588
So it might be easier for reviewer

1568
01:00:03.608 --> 01:00:05.910
to look at those two twins and and

1569
01:00:05.930 --> 01:00:08.390
establish that it's not the same person,

1570
01:00:08.410 --> 01:00:09.302
but maybe not.

1571
01:00:09.322 --> 01:00:09.606
Uhm,

1572
01:00:09.626 --> 01:00:11.503
you know twins might live on

1573
01:00:11.523 --> 01:00:13.519
different sides of the country and

1574
01:00:13.539 --> 01:00:15.941
there could be all sorts of reasons

1575
01:00:15.961 --> 01:00:17.950
why an investigator would be able

1576
01:00:17.970 --> 01:00:20.160
to exonerate that twin and not

1577
01:00:20.180 --> 01:00:22.710
You know, pursue the investigation further.

1578
01:00:22.730 --> 01:00:25.930
But it is a potential source of error,

1579
01:00:25.950 --> 01:00:28.340
so we go on to same sex

1580
01:00:28.360 --> 01:00:30.746
same sex fraternal twins give false

1581
01:00:30.766 --> 01:00:33.340
positives, and that is something to

1582
01:00:33.360 --> 01:00:36.590
be aware of given the twins are

1583
01:00:36.610 --> 01:00:38.062
fairly common and that

1584
01:00:38.082 --> 01:00:40.270
is the end of my segment.

1585
01:00:40.290 --> 01:00:42.480
Here are, yeah, we've run benchmarks.

1586
01:00:42.500 --> 01:00:44.315
The face recognition vendor test

1587
01:00:44.335 --> 01:00:47.166
I described to you, but also.

1588
01:00:47.186 --> 01:00:49.910
Video surveillance facing video

1589
01:00:49.930 --> 01:00:53.360
evaluation in 2015 or so.

1590
01:00:53.380 --> 01:00:56.490
And we continue to work on other challenges

1591
01:00:56.510 --> 01:00:59.740
in face recognition with with our partners,

1592
01:00:59.760 --> 01:01:02.602
which is part of the intelligence

1593
01:01:02.622 --> 01:01:04.591
community and we're always

1594
01:01:04.611 --> 01:01:07.130
looking to do new groundbreaking

1595
01:01:07.150 --> 01:01:09.680
is the government and industry.

1596
01:01:09.700 --> 01:01:11.340
[Abby] We have five minutes for questions,

1597
01:01:11.360 --> 01:01:13.251
so please use your chat pod and

1598
01:01:13.271 --> 01:01:19.240
I will call them out for you.

1599
01:01:19.260 --> 01:01:20.820


1600
01:01:20.840 --> 01:01:22.748
Thank you and again we will have a

1601
01:01:22.768 --> 01:01:25.059
little bit of time at the end to

1602
01:01:25.079 --> 01:01:26.240
address any unanswered questions.

1603
01:01:26.260 --> 01:01:28.442
Please feel free to send them

1604
01:01:28.462 --> 01:01:30.870
through the chat pod at any time.

1605
01:01:30.890 --> 01:01:36.860
[Echo]

1606
01:01:36.880 --> 01:01:42.070
[Silence]

1607
01:01:42.090 --> 01:01:44.261
All right, we have a question from

1608
01:01:44.281 --> 01:01:45.790
physicist Paul Goldhagen who asked,

1609
01:01:45.810 --> 01:01:47.650
have you looked at error rates

1610
01:01:47.670 --> 01:01:53.640
as a function of race and sex?

1611
01:01:53.660 --> 01:01:57.540
[Silence]

1612
01:01:57.560 --> 01:01:59.780
[Echo]

1613
01:01:59.800 --> 01:02:08.070
[silence]

1614
01:02:08.090 --> 01:02:10.280
Patrick: The answer to that one is is yes.

1615
01:02:10.300 --> 01:02:11.996
That question of course is in

1616
01:02:12.016 --> 01:02:13.566
in the news quite prominently

1617
01:02:13.586 --> 01:02:15.710
for the last couple of years.

1618
01:02:15.730 --> 01:02:19.720
Uh, I touched on error rates.

1619
01:02:19.740 --> 01:02:21.830
The dependence on sex.

1620
01:02:21.850 --> 01:02:24.481
Men slightly more easily

1621
01:02:24.501 --> 01:02:26.860
recognized typically than women.

1622
01:02:26.880 --> 01:02:29.280
Uh, we have looked at race.

1623
01:02:29.300 --> 01:02:31.710
We published a report, 2019.

1624
01:02:31.730 --> 01:02:34.860
We're updating that report now.

1625
01:02:34.880 --> 01:02:37.688
My colleagues also on this call at

1626
01:02:37.708 --> 01:02:40.916
MDTF have looked at that in in in

1627
01:02:40.936 --> 01:02:43.596
exquisite detail also. Uhm? And.

1628
01:02:43.616 --> 01:02:47.390
It that discussion of demographics,

1629
01:02:47.410 --> 01:02:50.550
so age, sex, race, potentially subject,

1630
01:02:50.570 --> 01:02:53.710
height and sort of other

1631
01:02:53.730 --> 01:02:56.770
demographic variables. Uhm?

1632
01:02:56.790 --> 01:02:59.700
You always want to sort of ask two questions.

1633
01:02:59.720 --> 01:03:02.232
What is the influence of those

1634
01:03:02.252 --> 01:03:04.460
things on false negative rates?

1635
01:03:04.480 --> 01:03:06.560
And separately, what is the

1636
01:03:06.580 --> 01:03:08.660
influence on false positive rates?

1637
01:03:08.680 --> 01:03:11.020
And so.

1638
01:03:11.040 --> 01:03:14.320
The importance of whatever results you have,

1639
01:03:14.340 --> 01:03:16.670
false negatives and false positives

1640
01:03:16.690 --> 01:03:18.653
is application dependent, so

1641
01:03:18.673 --> 01:03:23.157
In criminal justice it would be a different

1642
01:03:23.177 --> 01:03:25.940
consideration of demographic effects.

1643
01:03:25.960 --> 01:03:27.095
Than in, say,

1644
01:03:27.115 --> 01:03:29.405
access control into a building will

1645
01:03:29.425 --> 01:03:31.650
access control into a cell phone.

1646
01:03:31.670 --> 01:03:33.830
Uh, and so different priorities,

1647
01:03:33.850 --> 01:03:34.828
different importance,

1648
01:03:34.848 --> 01:03:36.824
different impacts of false

1649
01:03:36.844 --> 01:03:38.820
negatives versus false positives.

1650
01:03:38.840 --> 01:03:41.370
Uh, yeah, there's a long discussion

1651
01:03:41.390 --> 01:03:43.760
we could have there.

1652
01:03:43.780 --> 01:03:47.210
And in very short detail,

1653
01:03:47.230 --> 01:03:51.630
if you've got reasonably good images.

1654
01:03:51.650 --> 01:03:54.485
The fourth negative variations across

1655
01:03:54.505 --> 01:03:58.560
race and sex are really quite small.

1656
01:03:58.580 --> 01:04:00.795
The press doesn't really cover

1657
01:04:00.815 --> 01:04:04.684
it to that sort of degree of fine

1658
01:04:04.704 --> 01:04:06.770
grained sort of analysis.

1659
01:04:06.790 --> 01:04:07.211
Uh,

1660
01:04:07.231 --> 01:04:10.740
the false positive story is a bit different,

1661
01:04:10.760 --> 01:04:13.917
so even with very good images you can see

1662
01:04:13.937 --> 01:04:17.350
high error rates in particular demographics.

1663
01:04:17.370 --> 01:04:21.006
So I looked at an algorithm just yesterday

1664
01:04:21.026 --> 01:04:24.638
and it was giving quite high false

1665
01:04:24.658 --> 01:04:28.313
positive rates in East Asian faces.

1666
01:04:28.333 --> 01:04:31.537
and why that occurs is is something probably

1667
01:04:31.557 --> 01:04:35.340
to do with the way the algorithm was trained.

1668
01:04:35.360 --> 01:04:38.004
But we don't sort of do tests to explicitly

1669
01:04:38.024 --> 01:04:40.640
try and figure out why something occurs.

1670
01:04:40.660 --> 01:04:43.304
We just see that there are

1671
01:04:43.324 --> 01:04:45.080
these error rate differences.

1672
01:04:45.100 --> 01:04:47.817
I should point out if if John

1673
01:04:47.837 --> 01:04:50.445
and Yevgeny won't point out that

1674
01:04:50.465 --> 01:04:53.181
there is a standard now under

1675
01:04:53.201 --> 01:04:54.950
development for evaluation.

1676
01:04:54.970 --> 01:04:56.340
And quantification of alright.

1677
01:04:56.360 --> 01:04:58.430
Abby: We've got a couple more minutes,

1678
01:04:58.450 --> 01:04:59.741
Patrick: just face recognition,

1679
01:04:59.761 --> 01:05:02.370
but any other biometrics as well?

1680
01:05:02.390 --> 01:05:04.440
Yeah I'll, I'll be quiet right

1681
01:05:04.460 --> 01:05:06.970
down and it looks like Stephanie,

1682
01:05:06.990 --> 01:05:09.650
who is an electronics engineer at another

1683
01:05:09.670 --> 01:05:12.094
DHS operational component has

1684
01:05:12.114 --> 01:05:13.930
a question here.

1685
01:05:13.950 --> 01:05:14.960
Happy to scroll up.

1686
01:05:14.980 --> 01:05:18.910
[Echo]

1687
01:05:18.930 --> 01:05:22.358
Ask Debbie that may be used to manipulate

1688
01:05:22.378 --> 01:05:25.085
images like change lighting, sharpness,

1689
01:05:25.105 --> 01:05:30.870
aging, etc to improve accuracy.

1690
01:05:30.890 --> 01:05:44.700
[Echo]

1691
01:05:44.720 --> 01:05:48.080
Patrick: Yes,so most

1692
01:05:48.100 --> 01:05:51.350
Developers will be happy to sell a sort

1693
01:05:51.370 --> 01:05:55.430
of a GUI investigation or work station.

1694
01:05:55.450 --> 01:05:58.822
And that is software for what you would

1695
01:05:58.842 --> 01:06:01.308
call operator LED investigation or

1696
01:06:01.328 --> 01:06:04.410
operator LED use of face recognition.

1697
01:06:04.430 --> 01:06:06.825
And so you could prepare a photo

1698
01:06:06.845 --> 01:06:08.568
and those GUI applications

1699
01:06:08.588 --> 01:06:11.090
typically include some tools.

1700
01:06:11.110 --> 01:06:12.572
So obviously you might want to

1701
01:06:12.592 --> 01:06:14.280
contact the person of interest from,

1702
01:06:14.300 --> 01:06:17.300
say, a video feed or from a photo.

1703
01:06:17.320 --> 01:06:19.870
You might also do an in plane rotation,

1704
01:06:19.890 --> 01:06:23.020
just to sort of put the eyes horizontally.

1705
01:06:23.040 --> 01:06:27.030
It becomes quite controversial when.

1706
01:06:27.050 --> 01:06:29.454
A software tool would be used to manipulate

1707
01:06:29.474 --> 01:06:31.520
the images in more complicated ways.

1708
01:06:31.540 --> 01:06:34.175
So if if you were to change the shape

1709
01:06:34.195 --> 01:06:36.980
of the eyes with a stretch the face,

1710
01:06:37.000 --> 01:06:39.210
or to add a mustache.

1711
01:06:39.230 --> 01:06:40.925
That depending on the application

1712
01:06:40.945 --> 01:06:42.703
could be considered sort of

1713
01:06:42.723 --> 01:06:44.280
tampering with the evidence.

1714
01:06:44.300 --> 01:06:48.480
And that there have been some.

1715
01:06:48.500 --> 01:06:50.820
Standard work has been done

1716
01:06:50.840 --> 01:06:53.249
on regulating what kind of

1717
01:06:53.269 --> 01:06:56.370
transformations that can be applied.

1718
01:06:56.390 --> 01:07:01.070
And so you would look to the the fizz wig,

1719
01:07:01.090 --> 01:07:02.356
facial identification,

1720
01:07:02.376 --> 01:07:05.570
scientific working group under the

1721
01:07:05.590 --> 01:07:09.680
OSAC group to look at allowed changes.

1722
01:07:09.700 --> 01:07:11.500
Changing the lighting and

1723
01:07:11.520 --> 01:07:12.870
changing the sharpness.

1724
01:07:12.890 --> 01:07:14.478
Age, regression progression.

1725
01:07:14.498 --> 01:07:16.090
Would be OK.

1726
01:07:16.110 --> 01:07:18.463
Abby: We have one minute left so we're

1727
01:07:18.483 --> 01:07:20.973
going to take a quick question

1728
01:07:20.993 --> 01:07:22.870
from scientist Matt Minetti.

1729
01:07:22.890 --> 01:07:25.260
He asked for different image types.

1730
01:07:25.280 --> 01:07:27.260
Is there a different algorithm

1731
01:07:27.280 --> 01:07:28.460
that works best?

1732
01:07:28.480 --> 01:07:30.956
Like is that information available and

1733
01:07:30.976 --> 01:07:35.480
is that information available to users?

1734
01:07:35.500 --> 01:07:36.920
Anyway, he went on to say

1735
01:07:36.940 --> 01:07:38.530
use of the best algorithm.

1736
01:07:38.550 --> 01:07:56.370
Go ahead, Patrick.

1737
01:07:56.390 --> 01:07:58.160
Patrick: Yeah yeah thanks for that.

1738
01:07:58.180 --> 01:08:01.412
So we we try to capture a accuracy in our

1739
01:08:01.432 --> 01:08:04.510
benchmarks across a number of different.

1740
01:08:04.530 --> 01:08:05.926
image types,

1741
01:08:05.946 --> 01:08:08.290
so mugshots versus kiosk images.

1742
01:08:08.310 --> 01:08:11.410
It's sort of like images.

1743
01:08:11.430 --> 01:08:13.034
A border crossing images.

1744
01:08:13.054 --> 01:08:15.070
They're all a bit different.

1745
01:08:15.090 --> 01:08:17.429
We we fail at capturing all possible

1746
01:08:17.449 --> 01:08:20.461
image types so we don't have a current

1747
01:08:20.481 --> 01:08:23.010
benchmark on video surveillance for example.

1748
01:08:23.030 --> 01:08:25.420
And so to answer your question,

1749
01:08:25.440 --> 01:08:27.780
are different algorithms sort of

1750
01:08:27.800 --> 01:08:30.710
appropriate for different kinds of images?

1751
01:08:30.730 --> 01:08:34.006
A short answer there is yes you can

1752
01:08:34.026 --> 01:08:36.759
find algorithms that are better on

1753
01:08:36.779 --> 01:08:39.064
certain sort of performance factors

1754
01:08:39.084 --> 01:08:43.350
or image qualities, but generally.

1755
01:08:43.370 --> 01:08:45.786
because it would be sort of

1756
01:08:45.806 --> 01:08:47.430
operationally cumbersome to feel two,

1757
01:08:47.450 --> 01:08:48.790
three, four different algorithms.

1758
01:08:48.810 --> 01:08:50.912
Abby: OK, now I know we're out of

1759
01:08:50.932 --> 01:08:52.960
time for questions back over to

1760
01:08:52.980 --> 01:08:59.550
you Vivek.

1761
01:08:59.570 --> 01:09:03.620
Vivek: Thank you Patrick and Abby.

1762
01:09:03.640 --> 01:09:06.745
Alright, now I like to

1763
01:09:06.765 --> 01:09:09.250
introduce our final speakers.

1764
01:09:09.270 --> 01:09:11.850
Yevgeniy Sirotin and Laura Rabbitt.

1765
01:09:11.870 --> 01:09:14.682
Yevgeniy is a senior scientist

1766
01:09:14.702 --> 01:09:17.274
manager at the identity and

1767
01:09:17.294 --> 01:09:19.110
Data Sciences Laboratory,

1768
01:09:19.130 --> 01:09:21.182
which supports S&T's  biometric,

1769
01:09:21.202 --> 01:09:23.260
and identity technology center.

1770
01:09:23.280 --> 01:09:24.904
Maryland test facility.

1771
01:09:24.924 --> 01:09:28.060
Yevgeniy's work focuses on developing

1772
01:09:28.080 --> 01:09:30.540
and applying operationally relevant

1773
01:09:30.560 --> 01:09:33.640
metrics to measure technology performance.

1774
01:09:33.660 --> 01:09:36.760
This includes his research on biometric,

1775
01:09:36.780 --> 01:09:39.950
AI bias, and how to best

1776
01:09:39.970 --> 01:09:42.510
Introduce biometric AI into human

1777
01:09:42.530 --> 01:09:45.070
workflows or human algorithm team.

1778
01:09:45.090 --> 01:09:48.022
He's currently a Co editor of a new

1779
01:09:48.042 --> 01:09:50.151
standard for measuring demographic

1780
01:09:50.171 --> 01:09:52.770
effects in biometric systems.

1781
01:09:52.790 --> 01:09:55.490
Laura is the lead human factors team

1782
01:09:55.510 --> 01:09:57.810
Scientists at the Maryland test

1783
01:09:57.830 --> 01:09:59.680
facility her most recent.

1784
01:09:59.700 --> 01:10:01.656
Research publication focuses on

1785
01:10:01.676 --> 01:10:04.126
measuring human algorithm teams to

1786
01:10:04.146 --> 01:10:06.532
understand how people perceive and

1787
01:10:06.552 --> 01:10:08.842
incorporate information provided by an

1788
01:10:08.862 --> 01:10:11.100
algorithm or another external source.

1789
01:10:11.120 --> 01:10:13.290
By understanding how people perceive

1790
01:10:13.310 --> 01:10:15.552
information from an algorithm can

1791
01:10:15.572 --> 01:10:17.707
aid the design and implementation

1792
01:10:17.727 --> 01:10:19.000
of biometric systems.

1793
01:10:19.020 --> 01:10:21.200
To foster human algorithm synergy.

1794
01:10:21.220 --> 01:10:24.336
Yevgeniy and Laura I passed the

1795
01:10:24.356 --> 01:10:30.326
virtual mic back over to you.

1796
01:10:30.346 --> 01:10:33.260


1797
01:10:33.280 --> 01:10:34.340
Yevgeny: Thank you, Vivek.

1798
01:10:34.360 --> 01:10:36.500
I think I'm going to start

1799
01:10:36.520 --> 01:10:38.928
from here and thank you for

1800
01:10:38.948 --> 01:10:40.920
everybody for joining us today.

1801
01:10:40.940 --> 01:10:44.144
And I hope that this part of the

1802
01:10:44.164 --> 01:10:46.970
talk will also be interesting.

1803
01:10:46.990 --> 01:10:50.370
It's going to switch gears a little bit.

1804
01:10:50.390 --> 01:10:53.340
We've talked a lot about algorithm

1805
01:10:53.360 --> 01:10:55.738
testing that NIST performs and

1806
01:10:55.758 --> 01:10:58.069
what I'm going to tell you about

1807
01:10:58.089 --> 01:11:00.536
is a different type of biometric

1808
01:11:00.556 --> 01:11:03.545
testing that we do at the Maryland

1809
01:11:03.565 --> 01:11:06.950
test facility at DHS S and T lab.

1810
01:11:06.970 --> 01:11:09.911
And then Laura will speak to some of the

1811
01:11:09.931 --> 01:11:12.881
work that we've been doing specifically

1812
01:11:12.901 --> 01:11:15.450
focused on human algorithm teaming.

1813
01:11:15.470 --> 01:11:18.480
So I'll overview just general testing that

1814
01:11:18.500 --> 01:11:20.706
we've done in some of the conclusions

1815
01:11:20.726 --> 01:11:22.862
that we've we've made about face

1816
01:11:22.882 --> 01:11:24.380
recognition from those tests,

1817
01:11:24.400 --> 01:11:25.074
but first,

1818
01:11:25.094 --> 01:11:27.850
a little bit about the test facility itself.

1819
01:11:27.870 --> 01:11:32.200
It's kind of a unique capability within DHS.

1820
01:11:32.220 --> 01:11:34.040
DHS S&T created the Maryland

1821
01:11:34.060 --> 01:11:35.880
test facility in twenty fourteen,

1822
01:11:35.900 --> 01:11:37.580
equipping it with the key

1823
01:11:37.600 --> 01:11:38.940
technology required to support

1824
01:11:38.960 --> 01:11:40.660
large scale biometric testing?

1825
01:11:40.680 --> 01:11:42.740
So overall the facility is actually

1826
01:11:42.760 --> 01:11:44.557
20,000 square feet of

1827
01:11:44.577 --> 01:11:46.377
reconfigurable lab space and were

1828
01:11:46.397 --> 01:11:48.271
equipped with video recording and

1829
01:11:48.291 --> 01:11:49.791
various environmental sensors that

1830
01:11:49.811 --> 01:11:51.700
are needed to precisely record

1831
01:11:51.720 --> 01:11:53.540
ambient conditions during the tests,

1832
01:11:53.560 --> 01:11:55.829
and the facility was designed from the

1833
01:11:55.849 --> 01:11:58.330
ground up for human subjects testing.

1834
01:11:58.350 --> 01:11:59.858
There are dedicated private

1835
01:11:59.878 --> 01:12:01.386
briefing rooms for performing

1836
01:12:01.406 --> 01:12:02.980
informed consent and interviews.

1837
01:12:03.000 --> 01:12:03.922
And in fact,

1838
01:12:03.942 --> 01:12:06.201
all of the work that I'm going

1839
01:12:06.221 --> 01:12:08.966
to tell you about was done under

1840
01:12:08.986 --> 01:12:10.520
informed consent with IRB.

1841
01:12:10.540 --> 01:12:12.844
It's very different than the kind of

1842
01:12:12.864 --> 01:12:15.190
testing that we talked about before,

1843
01:12:15.210 --> 01:12:17.188
which you know which really use

1844
01:12:17.208 --> 01:12:18.977
sequestered large databases of images

1845
01:12:18.997 --> 01:12:20.570
gathered from various sources.

1846
01:12:20.590 --> 01:12:23.951
We are always gathering new images from

1847
01:12:23.971 --> 01:12:27.280
people that we invite into our lab.

1848
01:12:27.300 --> 01:12:29.302
And and that makes it different

1849
01:12:29.322 --> 01:12:30.650
because first of all,

1850
01:12:30.670 --> 01:12:32.552
we're a little bit limited with

1851
01:12:32.572 --> 01:12:34.952
the number of people we can have

1852
01:12:34.972 --> 01:12:36.380
participate during each test.

1853
01:12:36.400 --> 01:12:36.721
However,

1854
01:12:36.741 --> 01:12:39.108
what we do gain is the ability

1855
01:12:39.128 --> 01:12:41.422
to really test out new conditions

1856
01:12:41.442 --> 01:12:43.832
and new scenarios and gather new

1857
01:12:43.852 --> 01:12:46.270
images that aren't available today.

1858
01:12:46.290 --> 01:12:52.260
So next slide please.

1859
01:12:52.280 --> 01:12:54.850


1860
01:12:54.870 --> 01:12:57.000
OK, so since opening its

1861
01:12:57.020 --> 01:12:58.720
doors in 2014,

1862
01:12:58.740 --> 01:13:01.840
the Maryland test facility or MDTF for short

1863
01:13:01.860 --> 01:13:04.310
has performed numerous scenario tests.

1864
01:13:04.330 --> 01:13:06.614
We initially focused on working out

1865
01:13:06.634 --> 01:13:08.630
the biometric modalities and methods

1866
01:13:08.650 --> 01:13:10.730
that are suitable for supporting

1867
01:13:10.750 --> 01:13:12.470
biometric operations and high

1868
01:13:12.490 --> 01:13:14.630
throughput applications at airports.

1869
01:13:14.650 --> 01:13:17.640
As the use of biometrics in particular,

1870
01:13:17.660 --> 01:13:18.540
face recognition.

1871
01:13:18.560 --> 01:13:21.690
Has expanded so have the industry offerings.

1872
01:13:21.710 --> 01:13:24.045
However, we notice that industry

1873
01:13:24.065 --> 01:13:25.458
marketing materials often

1874
01:13:25.478 --> 01:13:27.660
present only those metrics that

1875
01:13:27.680 --> 01:13:29.650
make their systems look good,

1876
01:13:29.670 --> 01:13:32.302
which makes it very difficult to

1877
01:13:32.322 --> 01:13:34.070
compare offerings and procurement.

1878
01:13:34.090 --> 01:13:36.060
So DHS launched the biometric

1879
01:13:36.080 --> 01:13:38.602
technology rallies as a yearly survey

1880
01:13:38.622 --> 01:13:40.514
of commercial biometric technology

1881
01:13:40.534 --> 01:13:42.910
market along a common footing.

1882
01:13:42.930 --> 01:13:45.562
And so we've been running these

1883
01:13:45.582 --> 01:13:49.500
rallies since twenty eighteen.

1884
01:13:49.520 --> 01:13:51.672
And we've tested primarily technologies in

1885
01:13:51.692 --> 01:13:53.800
a high throughput unstaffed environment,

1886
01:13:53.820 --> 01:13:56.110
so think TSA, checkpoint or something

1887
01:13:56.130 --> 01:13:58.100
similar but without any staff.

1888
01:13:58.120 --> 01:14:00.828
A single rally event can assess up to

1889
01:14:00.848 --> 01:14:03.580
about a dozen different face, fingerprint,

1890
01:14:03.600 --> 01:14:05.530
iris, or multimodal acquisition systems.

1891
01:14:05.550 --> 01:14:08.267
And this is a key point hasn't

1892
01:14:08.287 --> 01:14:09.440
come up before.

1893
01:14:09.460 --> 01:14:11.790
But in John 's earlier remarks,

1894
01:14:11.810 --> 01:14:14.376
he talked about the components of

1895
01:14:14.396 --> 01:14:17.005
a biometric system and a biometric

1896
01:14:17.025 --> 01:14:19.490
algorithm is just part of that.

1897
01:14:19.510 --> 01:14:21.686
There's always going to be other

1898
01:14:21.706 --> 01:14:23.213
components involved in determining

1899
01:14:23.233 --> 01:14:24.910
the performance of your system.

1900
01:14:24.930 --> 01:14:27.070
An acquisition is a crucial one,

1901
01:14:27.090 --> 01:14:29.240
and I'll focus on that today.

1902
01:14:29.260 --> 01:14:31.275
So these biometric technologies from

1903
01:14:31.295 --> 01:14:33.369
various vendors are assessed along

1904
01:14:33.389 --> 01:14:35.361
predetermined metrics which we tell to

1905
01:14:35.381 --> 01:14:37.900
all vendors well in advance of the test.

1906
01:14:37.920 --> 01:14:39.710
These fall into three categories.

1907
01:14:39.730 --> 01:14:42.590
We typically say we will test for efficiency,

1908
01:14:42.610 --> 01:14:45.956
which is how long it takes to really use

1909
01:14:45.976 --> 01:14:49.820
the technology to get the desired task done.

1910
01:14:49.840 --> 01:14:50.180
Effectiveness,

1911
01:14:50.200 --> 01:14:51.620
which is you know,

1912
01:14:51.640 --> 01:14:53.420
how effective is the outcome.

1913
01:14:53.440 --> 01:14:55.580
You know how well it works.

1914
01:14:55.600 --> 01:14:57.740
Something like a performance percent correct?

1915
01:14:57.760 --> 01:14:58.876
Or something like that.

1916
01:14:58.896 --> 01:15:01.082
We of course you try to stick

1917
01:15:01.102 --> 01:15:03.267
with some standards based metrics

1918
01:15:03.287 --> 01:15:04.580
and finally satisfaction.

1919
01:15:04.600 --> 01:15:06.380
It relates to positive attitudes

1920
01:15:06.400 --> 01:15:08.180
towards the technology you know.

1921
01:15:08.200 --> 01:15:10.346
Do people enjoy their interaction with

1922
01:15:10.366 --> 01:15:12.500
the technology or other gaps there?

1923
01:15:12.520 --> 01:15:16.670
Do they perceive it As for example invasive?

1924
01:15:16.690 --> 01:15:17.014
Importantly,

1925
01:15:17.034 --> 01:15:19.422
the rally tests are able to answer

1926
01:15:19.442 --> 01:15:21.247
questions about how the technology

1927
01:15:21.267 --> 01:15:23.378
would perform, not in the abstract,

1928
01:15:23.398 --> 01:15:25.140
but in a specific scenario.

1929
01:15:25.160 --> 01:15:26.199
So for example,

1930
01:15:26.219 --> 01:15:27.260
as I mentioned,

1931
01:15:27.280 --> 01:15:29.534
the TSA checkpoint performance data from

1932
01:15:29.554 --> 01:15:31.454
the rallies and additional information

1933
01:15:31.474 --> 01:15:33.860
is available at our website which we

1934
01:15:33.880 --> 01:15:36.090
maintain for this purpose mdt.org.

1935
01:15:36.110 --> 01:15:38.337
It also has a lot of information

1936
01:15:38.357 --> 01:15:40.044
about our publications and scholarly

1937
01:15:40.064 --> 01:15:42.078
works that we've done and we're

1938
01:15:42.098 --> 01:15:44.362
currently in the process of selecting

1939
01:15:44.382 --> 01:15:45.900
technologies to participate in

1940
01:15:45.920 --> 01:15:47.850
the 2021 biometric.

1941
01:15:47.870 --> 01:15:48.630
Technology rally,

1942
01:15:48.650 --> 01:15:52.190
which will be held in September of this year.

1943
01:15:52.210 --> 01:15:53.698
These scenario testing activities

1944
01:15:53.718 --> 01:15:55.583
are necessary compliments to the

1945
01:15:55.603 --> 01:15:57.177
NIST evaluations that Patrick

1946
01:15:57.197 --> 01:15:59.499
talked about because they can assess

1947
01:15:59.519 --> 01:16:01.175
acquisition performance and matching

1948
01:16:01.195 --> 01:16:02.811
performance with biometric samples

1949
01:16:02.831 --> 01:16:05.186
acquired within a specific use case.

1950
01:16:05.206 --> 01:16:07.478
So there were questions in the

1951
01:16:07.498 --> 01:16:09.532
chat about will this algorithm

1952
01:16:09.552 --> 01:16:11.540
work well with my images,

1953
01:16:11.560 --> 01:16:14.700
and I think that that's a good answer.

1954
01:16:14.720 --> 01:16:17.155
That question could be gleaned

1955
01:16:17.175 --> 01:16:19.120
from scenario testing and.

1956
01:16:19.140 --> 01:16:20.835
Synergy between algorithm testing and

1957
01:16:20.855 --> 01:16:22.987
this sort of full system scenario

1958
01:16:23.007 --> 01:16:24.880
testing is an important synergy.

1959
01:16:24.900 --> 01:16:25.594
For instance,

1960
01:16:25.614 --> 01:16:27.379
Patrick mentioned the five assessment

1961
01:16:27.399 --> 01:16:29.179
and in that assessment actually

1962
01:16:29.199 --> 01:16:31.517
video data from and drive test was

1963
01:16:31.537 --> 01:16:33.806
used to assess the performance of

1964
01:16:33.826 --> 01:16:35.320
biometric algorithms on videos.

1965
01:16:35.340 --> 01:16:36.376
And you know,

1966
01:16:36.396 --> 01:16:38.136
before those videos didn't exist

1967
01:16:38.156 --> 01:16:40.350
and these were acquired as part

1968
01:16:40.370 --> 01:16:41.800
of these scenario tests,

1969
01:16:41.820 --> 01:16:43.906
so there's a lot of value

1970
01:16:43.926 --> 01:16:45.760
to gathering this new data,

1971
01:16:45.780 --> 01:16:47.610
and these videos simulated security

1972
01:16:47.630 --> 01:16:49.090
cameras and patches injured.

1973
01:16:49.110 --> 01:16:50.800
Loading bridges within a controlled

1974
01:16:50.820 --> 01:16:52.829
environment where we actually know the

1975
01:16:52.849 --> 01:16:54.960
identity of all the individuals in the video.

1976
01:16:54.980 --> 01:16:56.415
This is something that's very

1977
01:16:56.435 --> 01:16:57.923
difficult to do in operational

1978
01:16:57.943 --> 01:17:00.250
settings, so these videos from the

1979
01:17:00.270 --> 01:17:02.600
field much harder to work with.

1980
01:17:02.620 --> 01:17:04.712
So this testing is allowed us to make

1981
01:17:04.732 --> 01:17:06.280
some important observations regarding

1982
01:17:06.300 --> 01:17:08.750
operational challenges and biometric systems,

1983
01:17:08.770 --> 01:17:10.934
and these will include information about

1984
01:17:10.954 --> 01:17:12.740
some underappreciated sources of error,

1985
01:17:12.760 --> 01:17:15.270
including some that can lead to bias,

1986
01:17:15.290 --> 01:17:17.804
or to use a more specific term

1987
01:17:17.824 --> 01:17:18.530
differential performance.

1988
01:17:18.550 --> 01:17:22.970
So next slide, please.

1989
01:17:22.990 --> 01:17:24.775
So if you recalls Johns

1990
01:17:24.795 --> 01:17:25.500
excellent introduction,

1991
01:17:25.520 --> 01:17:27.510
he showed that there are many

1992
01:17:27.530 --> 01:17:29.699
components of a biometric system in

1993
01:17:29.719 --> 01:17:31.660
addition to the matching algorithm.

1994
01:17:31.680 --> 01:17:33.795
Testing the algorithm in isolation

1995
01:17:33.815 --> 01:17:35.503
isn't sufficient to understand

1996
01:17:35.523 --> 01:17:37.490
the operational performance.

1997
01:17:37.510 --> 01:17:39.608
And in fact, modern face recognition

1998
01:17:39.628 --> 01:17:40.667
algorithms make exceedingly

1999
01:17:40.687 --> 01:17:42.330
few errors on mugshot style,

2000
01:17:42.350 --> 01:17:45.450
face photos and as part of our rally tests,

2001
01:17:45.470 --> 01:17:48.058
we assess the ability of a biometric system

2002
01:17:48.078 --> 01:17:50.437
to collect an image which is suitable

2003
01:17:50.457 --> 01:17:52.710
for matching without a human operator.

2004
01:17:52.730 --> 01:17:55.206
That's what we've done in the past few

2005
01:17:55.226 --> 01:17:57.560
years that we've run this evaluation.

2006
01:17:57.580 --> 01:17:59.930
So think of a completely unstaffed

2007
01:17:59.950 --> 01:18:02.320
TSA checkpoint with gates that can

2008
01:18:02.340 --> 01:18:04.534
determine whether to allow you into

2009
01:18:04.554 --> 01:18:07.565
the airport based in part on your

2010
01:18:07.585 --> 01:18:08.900
biometrically verified identity.

2011
01:18:08.920 --> 01:18:11.284
And what you can see in the chart

2012
01:18:11.304 --> 01:18:13.592
to my right, what I'm the point?

2013
01:18:13.612 --> 01:18:15.740
I want to make with this slide

2014
01:18:15.760 --> 01:18:17.980
is that when we measure the total

2015
01:18:18.000 --> 01:18:20.420
error of a system in this scenario,

2016
01:18:20.440 --> 01:18:22.660
we find that the primary cause of error

2017
01:18:22.680 --> 01:18:25.540
is not in the biometric match per say,

2018
01:18:25.560 --> 01:18:27.908
but the primary cause of error is often

2019
01:18:27.928 --> 01:18:30.340
failure to collect an image for matching.

2020
01:18:30.360 --> 01:18:31.240
So that is,

2021
01:18:31.260 --> 01:18:34.500
you know you go up in front of the system,

2022
01:18:34.520 --> 01:18:35.968
you do your best.

2023
01:18:35.988 --> 01:18:38.170
You follow the instructions but somehow.

2024
01:18:38.190 --> 01:18:40.055
There's no image collected from

2025
01:18:40.075 --> 01:18:41.190
the system either.

2026
01:18:41.210 --> 01:18:44.118
It doesn't find your face or you move

2027
01:18:44.138 --> 01:18:46.840
through it too quickly and it can't.

2028
01:18:46.860 --> 01:18:49.048
It catches an image of a

2029
01:18:49.068 --> 01:18:50.990
chair or something like that,

2030
01:18:51.010 --> 01:18:53.528
so these failures to acquire can

2031
01:18:53.548 --> 01:18:55.607
greatly outnumber like from by

2032
01:18:55.627 --> 01:18:57.770
a factor of five to one overall.

2033
01:18:57.790 --> 01:19:00.220
All other sources of error in a

2034
01:19:00.240 --> 01:19:02.300
biometric system in this scenario,

2035
01:19:02.320 --> 01:19:03.530
and these errors,

2036
01:19:03.550 --> 01:19:05.170
Just an acquisition persacon frequently

2037
01:19:05.190 --> 01:19:06.792
hit double digit percentages

2038
01:19:06.812 --> 01:19:08.770
that you could see that here.

2039
01:19:08.790 --> 01:19:10.545
I'm plotting data over our

2040
01:19:10.565 --> 01:19:11.970
last few scenario tests.

2041
01:19:11.990 --> 01:19:13.071
These different rallies,

2042
01:19:13.091 --> 01:19:16.040
and in fact you could see for tests

2043
01:19:16.060 --> 01:19:18.710
to a full 12% of the error.

2044
01:19:18.730 --> 01:19:19.430
Or rather,

2045
01:19:19.450 --> 01:19:21.230
there was a 12%

2046
01:19:21.250 --> 01:19:23.400
failure to acquire rate on average

2047
01:19:23.420 --> 01:19:25.460
across the systems in that test.

2048
01:19:25.480 --> 01:19:27.265
So face recognition algorithms can't

2049
01:19:27.285 --> 01:19:30.070
recognize you if an image is not acquired.

2050
01:19:30.090 --> 01:19:32.499
This puts a low ceiling on performance

2051
01:19:32.519 --> 01:19:35.349
so that you're not going to get any

2052
01:19:35.369 --> 01:19:37.880
better no matter which algorithm you choose,

2053
01:19:37.900 --> 01:19:39.035
so we can.

2054
01:19:39.055 --> 01:19:40.190
Expect similar difficulties

2055
01:19:40.210 --> 01:19:41.350
in other environments.

2056
01:19:41.370 --> 01:19:43.564
For instance when trying to identify

2057
01:19:43.584 --> 01:19:45.922
people going into a subway station

2058
01:19:45.942 --> 01:19:48.370
or when trying to identify people

2059
01:19:48.390 --> 01:19:51.003
going through some checkpoint in a car

2060
01:19:51.023 --> 01:19:53.510
trying to image them through car windshields.

2061
01:19:53.530 --> 01:19:55.790
So when planning biometric system deployment,

2062
01:19:55.810 --> 01:19:57.848
this really kind of urges some

2063
01:19:57.868 --> 01:19:59.722
careful consideration to how the

2064
01:19:59.742 --> 01:20:01.490
images acquired image acquisition,

2065
01:20:01.510 --> 01:20:04.150
and we've done some testing on that.

2066
01:20:04.170 --> 01:20:09.260
So next slide, please.

2067
01:20:09.280 --> 01:20:11.738
So, but there are more subtle effects

2068
01:20:11.758 --> 01:20:13.998
on image acquisition on face recognition

2069
01:20:14.018 --> 01:20:16.266
performance and these come from the

2070
01:20:16.286 --> 01:20:18.850
choice of camera used to take the picture.

2071
01:20:18.870 --> 01:20:21.487
So what I'm showing along the bottom is a

2072
01:20:21.507 --> 01:20:23.859
multi panel picture where there are images

2073
01:20:23.879 --> 01:20:26.989
of the same woman taken across a variety

2074
01:20:27.009 --> 01:20:29.140
of commercial face acquisition cameras,

2075
01:20:29.160 --> 01:20:31.436
and these are all products that are

2076
01:20:31.456 --> 01:20:33.400
specifically designed for face recognition.

2077
01:20:33.420 --> 01:20:35.300
These are not just surveillance

2078
01:20:35.320 --> 01:20:37.587
style cameras and you could see

2079
01:20:37.607 --> 01:20:39.520
that the quality of each image.

2080
01:20:39.540 --> 01:20:41.210
Is really different right there?

2081
01:20:41.230 --> 01:20:42.900
They're different in color tone.

2082
01:20:42.920 --> 01:20:44.260
They're different in lightness,

2083
01:20:44.280 --> 01:20:47.630
and if you take a look at just one measure,

2084
01:20:47.650 --> 01:20:50.276
her skin tone in these fixtures pictures

2085
01:20:50.296 --> 01:20:52.823
those values alone can vary by a factor

2086
01:20:52.843 --> 01:20:55.070
of two or more across these images,

2087
01:20:55.090 --> 01:20:56.960
so the properties of the individual

2088
01:20:56.980 --> 01:20:58.644
in the photo depend critically

2089
01:20:58.664 --> 01:21:00.810
on the camera taking the picture,

2090
01:21:00.830 --> 01:21:02.634
and our testing shows that this

2091
01:21:02.654 --> 01:21:04.870
can matter a lot for algorithms.

2092
01:21:04.890 --> 01:21:07.885
So what I'm going to show what I'm showing

2093
01:21:07.905 --> 01:21:10.680
on the right now is the error rate.

2094
01:21:10.700 --> 01:21:12.605
For just one algorithm across a set

2095
01:21:12.625 --> 01:21:14.540
of ten different acquisition systems,

2096
01:21:14.560 --> 01:21:16.150
S1 through S10.

2097
01:21:16.170 --> 01:21:17.074
And for context,

2098
01:21:17.094 --> 01:21:18.614
this algorithm had to identify

2099
01:21:18.634 --> 01:21:20.340
the same set of individuals.

2100
01:21:20.360 --> 01:21:22.086
About 300 based on images

2101
01:21:22.106 --> 01:21:23.678
of those same individuals taken

2102
01:21:23.698 --> 01:21:25.170
on different acquisition systems.

2103
01:21:25.190 --> 01:21:27.318
This is what we typically do

2104
01:21:27.338 --> 01:21:29.450
during one of our rallies.

2105
01:21:29.470 --> 01:21:31.215
The algorithm was extremely good

2106
01:21:31.235 --> 01:21:32.980
on images from some systems.

2107
01:21:33.000 --> 01:21:34.388
For example system one.

2108
01:21:34.408 --> 01:21:35.800
It made no errors,

2109
01:21:35.820 --> 01:21:37.570
but failed frequently for others.

2110
01:21:37.590 --> 01:21:38.980
For example, system four,

2111
01:21:39.000 --> 01:21:40.750
where there were 6.5%

2112
01:21:40.770 --> 01:21:41.954
error rate.

2113
01:21:41.974 --> 01:21:44.980
This is what we call a briddle algorithm.

2114
01:21:45.000 --> 01:21:46.919
I'll note that this is likely a

2115
01:21:46.939 --> 01:21:49.246
best case scenario since all of the

2116
01:21:49.266 --> 01:21:50.686
acquisition systems were current

2117
01:21:50.706 --> 01:21:52.333
commercial face acquisition system

2118
01:21:52.353 --> 01:21:54.510
designed for this specific scenario.

2119
01:21:54.530 --> 01:21:56.255
And yet this algorithm really

2120
01:21:56.275 --> 01:21:58.062
showed a variation in performance

2121
01:21:58.082 --> 01:21:59.560
depending on exactly which.

2122
01:21:59.580 --> 01:22:02.040
Acquisition system is used.

2123
01:22:02.060 --> 01:22:03.730
So what's the takeaway here?

2124
01:22:03.750 --> 01:22:04.735
Face recognition algorithms

2125
01:22:04.755 --> 01:22:06.080
are accurate but achievable.

2126
01:22:06.100 --> 01:22:08.440
Accuracy depends on how the images acquired.

2127
01:22:08.460 --> 01:22:10.120
Just because an algorithm can

2128
01:22:10.140 --> 01:22:11.810
accurately match 99.9%

2129
01:22:11.830 --> 01:22:13.300
of mugshot images

2130
01:22:13.320 --> 01:22:15.575
does not mean that it won't make

2131
01:22:15.595 --> 01:22:17.380
significant errors were matching a

2132
01:22:17.400 --> 01:22:19.560
grainy image from a security camera.

2133
01:22:19.580 --> 01:22:21.590
I think Patrick made that point,

2134
01:22:21.610 --> 01:22:23.270
probably better than I did,

2135
01:22:23.290 --> 01:22:25.412
but to know the performance of the

2136
01:22:25.432 --> 01:22:27.650
algorithm with grainy security camera images.

2137
01:22:27.670 --> 01:22:29.740
These same images must be used

2138
01:22:29.760 --> 01:22:31.460
to perform the assessment and

2139
01:22:31.480 --> 01:22:34.710
And we do this on an ongoing basis with with.

2140
01:22:34.730 --> 01:22:37.640
In this use case, and in this scenario test.

2141
01:22:37.660 --> 01:22:38.001
However,

2142
01:22:38.021 --> 01:22:40.167
you really want to perhaps think

2143
01:22:40.187 --> 01:22:42.219
about designing a scenario test for

2144
01:22:42.239 --> 01:22:44.385
your own use case if you really

2145
01:22:44.405 --> 01:22:46.485
want to know how well your system

2146
01:22:46.505 --> 01:22:48.690
would work with the images you have.

2147
01:22:48.710 --> 01:22:54.100
So next slide please.

2148
01:22:54.120 --> 01:22:56.008
So of course technology and biology

2149
01:22:56.028 --> 01:22:58.340
alone are not the only determinants

2150
01:22:58.360 --> 01:23:00.340
of biometric system performance.

2151
01:23:00.360 --> 01:23:03.126
There's always the H factor and human

2152
01:23:03.146 --> 01:23:05.800
behavior plays a major role as well.

2153
01:23:05.820 --> 01:23:09.320
Human behavior will invariably cause.

2154
01:23:09.340 --> 01:23:11.364
For two as fat tails in the

2155
01:23:11.384 --> 01:23:13.440
performance of a biometric system,

2156
01:23:13.460 --> 01:23:15.490
I'll tell you what that means.

2157
01:23:15.510 --> 01:23:17.428
So the visualization to the right

2158
01:23:17.448 --> 01:23:18.778
shows transaction times measured

2159
01:23:18.798 --> 01:23:20.300
for various fingerprint system,

2160
01:23:20.320 --> 01:23:22.010
and these are different ones.

2161
01:23:22.030 --> 01:23:24.760
They either collect one finger or two finger,

2162
01:23:24.780 --> 01:23:27.160
or they collect a slap for slap,

2163
01:23:27.180 --> 01:23:29.218
or there was a non contact

2164
01:23:29.238 --> 01:23:30.590
system tested as well.

2165
01:23:30.610 --> 01:23:32.624
So you could see that most

2166
01:23:32.644 --> 01:23:34.360
transaction times are really fast,

2167
01:23:34.380 --> 01:23:37.443
so they're low too on the Y axis, right?

2168
01:23:37.463 --> 01:23:39.850
A lot of them sort of low,

2169
01:23:39.870 --> 01:23:42.640
and then there's an inflection point.

2170
01:23:42.660 --> 01:23:44.600
Come and and at that point these

2171
01:23:44.620 --> 01:23:45.790
transactions become progressively slower,

2172
01:23:45.810 --> 01:23:47.918
and these are the so called fat tails

2173
01:23:47.938 --> 01:23:50.020
that I'm referring to and what they

2174
01:23:50.040 --> 01:23:52.143
indicate to us usually is that there

2175
01:23:52.163 --> 01:23:54.370
are gaps in the design of the system.

2176
01:23:54.390 --> 01:23:55.225
For most people,

2177
01:23:55.245 --> 01:23:56.080
they're working well,

2178
01:23:56.100 --> 01:23:58.370
and if you look at the median performance,

2179
01:23:58.390 --> 01:24:01.230
you could be like proof the system can do it,

2180
01:24:01.250 --> 01:24:02.950
and under in under ten seconds.

2181
01:24:02.970 --> 01:24:04.950
But then you look at some of these

2182
01:24:04.970 --> 01:24:06.947
and you know some of these people

2183
01:24:06.967 --> 01:24:08.756
spent upwards of a few minutes

2184
01:24:08.776 --> 01:24:10.950
trying to get this system to work.

2185
01:24:10.970 --> 01:24:13.200
So what does that mean?

2186
01:24:13.220 --> 01:24:13.561
Well,

2187
01:24:13.581 --> 01:24:15.727
in fact there are multiple causes

2188
01:24:15.747 --> 01:24:17.950
of issues that cause fat tails,

2189
01:24:17.970 --> 01:24:19.406
but these include issues

2190
01:24:19.426 --> 01:24:20.500
with system usability,

2191
01:24:20.520 --> 01:24:22.325
including high system complexity or

2192
01:24:22.345 --> 01:24:24.150
low affordance and inadequate signage.

2193
01:24:24.170 --> 01:24:26.670
So what I'm trying to show that

2194
01:24:26.690 --> 01:24:28.900
in the photo on the right,

2195
01:24:28.920 --> 01:24:30.720
so on the bottom right

2196
01:24:30.740 --> 01:24:32.180
I'm showing an example.

2197
01:24:32.200 --> 01:24:34.686
It's a system that includes both the

2198
01:24:34.706 --> 01:24:36.634
passport scanner and a fingerprint

2199
01:24:36.654 --> 01:24:39.046
scanner all-in-one kiosk of very common

2200
01:24:39.066 --> 01:24:40.881
thing that you've definitely find

2201
01:24:40.901 --> 01:24:43.446
out in the field and in this photo.

2202
01:24:43.466 --> 01:24:46.290
This is part of one of our control test.

2203
01:24:46.310 --> 01:24:47.858
In this photo the person is being

2204
01:24:47.878 --> 01:24:49.286
asked to scan his fingerprints

2205
01:24:49.306 --> 01:24:50.690
on the fingerprint scanner,

2206
01:24:50.710 --> 01:24:52.700
but instead he puts his hand

2207
01:24:52.720 --> 01:24:54.040
into the passport scanner.

2208
01:24:54.060 --> 01:24:56.112
And this common error results from high

2209
01:24:56.132 --> 01:24:58.170
system complexity and issues with affordance.

2210
01:24:58.190 --> 01:25:00.042
It just feels equally plausible to

2211
01:25:00.062 --> 01:25:01.985
put your fingers and either device

2212
01:25:02.005 --> 01:25:03.900
so this person isn't an expert.

2213
01:25:03.920 --> 01:25:06.384
He doesn't know that the fingerprint

2214
01:25:06.404 --> 01:25:09.805
scanner is that blue thing on the right.

2215
01:25:09.825 --> 01:25:11.330
additional issues that can cause

2216
01:25:11.350 --> 01:25:13.100
these fat tails or clothing.

2217
01:25:13.120 --> 01:25:15.062
Wearing a hat that obscures the

2218
01:25:15.082 --> 01:25:16.370
face or face mask.

2219
01:25:16.390 --> 01:25:19.106
I'm going to come back to that a

2220
01:25:19.126 --> 01:25:22.097
little bit as well as self styling

2221
01:25:22.117 --> 01:25:24.409
like heavy makeup or alterations

2222
01:25:24.429 --> 01:25:26.080
in make up over time.

2223
01:25:26.100 --> 01:25:28.054
And So what this all kind of

2224
01:25:28.074 --> 01:25:29.870
means is that these operational

2225
01:25:29.890 --> 01:25:31.670
biometric system are susceptible

2226
01:25:31.690 --> 01:25:34.750
to the so called Black Swan events.

2227
01:25:34.770 --> 01:25:37.782
This is a term coined by the probability

2228
01:25:37.802 --> 01:25:40.018
scientist Nasim Taleb and what the

2229
01:25:40.038 --> 01:25:42.586
Black Swans were once thought not to

2230
01:25:42.606 --> 01:25:44.902
exist until to everyone 's surprise

2231
01:25:44.922 --> 01:25:46.842
they were observed in Australia.

2232
01:25:46.862 --> 01:25:50.046
So this is same thing with these biometric

2233
01:25:50.066 --> 01:25:52.685
errors if you only look at the

2234
01:25:52.705 --> 01:25:54.276
median transaction time here

2235
01:25:54.296 --> 01:25:56.200
you would think this system.

2236
01:25:56.220 --> 01:25:57.412
Perform really well,

2237
01:25:57.432 --> 01:26:00.240
but because we cannot predict human behavior,

2238
01:26:00.260 --> 01:26:02.484
it's important to expect these outliers

2239
01:26:02.504 --> 01:26:04.680
that could transform biometric performance,

2240
01:26:04.700 --> 01:26:06.700
and often for the worst.

2241
01:26:06.720 --> 01:26:09.199
So I think that it's important for

2242
01:26:09.219 --> 01:26:11.705
system designers and system users to

2243
01:26:11.725 --> 01:26:13.980
understand that about biometric systems,

2244
01:26:14.000 --> 01:26:17.164
and I'll show you an example of a

2245
01:26:17.184 --> 01:26:20.040
Black Swan event in a few slides.

2246
01:26:20.060 --> 01:26:25.090
Next slide, please.

2247
01:26:25.110 --> 01:26:26.430
So from the beginning,

2248
01:26:26.450 --> 01:26:28.105
we've been testing technologies with

2249
01:26:28.125 --> 01:26:29.810
people from the local population,

2250
01:26:29.830 --> 01:26:32.498
which is diverse in race, gender and age.

2251
01:26:32.518 --> 01:26:34.178
Much more diverse than a

2252
01:26:34.198 --> 01:26:35.870
typical college campus would be.

2253
01:26:35.890 --> 01:26:37.560
So ages eighteen to eighty.

2254
01:26:37.580 --> 01:26:39.920
People from all work walks of life,

2255
01:26:39.940 --> 01:26:41.840
and we wanted to make sure

2256
01:26:41.860 --> 01:26:43.179
that these technologies worked

2257
01:26:43.199 --> 01:26:44.640
well across that spectrum.

2258
01:26:44.660 --> 01:26:46.865
And sometimes we've observed that

2259
01:26:46.885 --> 01:26:49.536
biometric performance was not equal across

2260
01:26:49.556 --> 01:26:52.190
one or more of these demographic groups.

2261
01:26:52.210 --> 01:26:53.840
This is something, of course.

2262
01:26:53.860 --> 01:26:55.160
Patrick has observed as

2263
01:26:55.180 --> 01:26:56.480
well with algorithm testing,

2264
01:26:56.500 --> 01:26:58.280
but I'll talk about the scenario

2265
01:26:58.300 --> 01:27:00.110
testing of acquisition in particular,

2266
01:27:00.130 --> 01:27:02.750
so this is sometimes referred to as bias,

2267
01:27:02.770 --> 01:27:04.616
but when you tell an engineer

2268
01:27:04.636 --> 01:27:06.710
to fix bias in the system,

2269
01:27:06.730 --> 01:27:08.957
it can be hard to know what

2270
01:27:08.977 --> 01:27:10.340
specific action to take,

2271
01:27:10.360 --> 01:27:11.895
and it's important to understand

2272
01:27:11.915 --> 01:27:13.970
the root cause of the problem.

2273
01:27:13.990 --> 01:27:15.804
So what this slide tries to show

2274
01:27:15.824 --> 01:27:18.198
is a couple of examples of some

2275
01:27:18.218 --> 01:27:19.746
demographic differences in the

2276
01:27:19.766 --> 01:27:21.344
performance we observed during

2277
01:27:21.364 --> 01:27:23.260
testing and some root causes.

2278
01:27:23.280 --> 01:27:25.870
Which can be fixed by a better system design.

2279
01:27:25.890 --> 01:27:27.830
In one case we observed a system

2280
01:27:27.850 --> 01:27:29.322
that performed more poorly for

2281
01:27:29.342 --> 01:27:30.510
women relative to men,

2282
01:27:30.530 --> 01:27:32.250
and when looking at the photos

2283
01:27:32.270 --> 01:27:33.120
from the system,

2284
01:27:33.140 --> 01:27:34.570
we saw that sometimes clipped

2285
01:27:34.590 --> 01:27:35.440
images of faces.

2286
01:27:35.460 --> 01:27:39.820
So if you see the photos on the top right.

2287
01:27:39.840 --> 01:27:40.900
There they're clipped,

2288
01:27:40.920 --> 01:27:43.492
so the bottom of the face is

2289
01:27:43.512 --> 01:27:45.610
sort of off the off the frame.

2290
01:27:45.630 --> 01:27:47.230
And this was because men were

2291
01:27:47.250 --> 01:27:48.680
on average taller than women.

2292
01:27:48.700 --> 01:27:50.350
This affected women more than men,

2293
01:27:50.370 --> 01:27:52.030
but affected people of a certain

2294
01:27:52.050 --> 01:27:52.590
height equally.

2295
01:27:52.610 --> 01:27:53.156
So actually,

2296
01:27:53.176 --> 01:27:54.854
I'm showing you a picture of

2297
01:27:54.874 --> 01:27:56.210
a man and a woman.

2298
01:27:56.230 --> 01:27:57.718
Both are clipped because

2299
01:27:57.738 --> 01:27:59.230
there are similar height.

2300
01:27:59.250 --> 01:28:00.630
So in this case,

2301
01:28:00.650 --> 01:28:00.980
again,

2302
01:28:01.000 --> 01:28:03.430
it was the system biased against women,

2303
01:28:03.450 --> 01:28:03.761
maybe,

2304
01:28:03.781 --> 01:28:05.747
but reality is that the system

2305
01:28:05.767 --> 01:28:07.653
didn't work well for people that

2306
01:28:07.673 --> 01:28:09.693
were of a certain height and a

2307
01:28:09.713 --> 01:28:11.385
better camera viewport configuration

2308
01:28:11.405 --> 01:28:14.233
could have solved the issue here in.

2309
01:28:14.253 --> 01:28:15.142
In another example,

2310
01:28:15.162 --> 01:28:17.377
we saw that some systems had lower

2311
01:28:17.397 --> 01:28:19.152
performance for people that self

2312
01:28:19.172 --> 01:28:21.630
identified as black or African American.

2313
01:28:21.650 --> 01:28:24.157
So the two photos on the lower

2314
01:28:24.177 --> 01:28:26.530
right show photos of the same man,

2315
01:28:26.550 --> 01:28:29.650
one taken on a system that performed worse.

2316
01:28:29.670 --> 01:28:31.075
For black or African American

2317
01:28:31.095 --> 01:28:32.553
participants and one taken on

2318
01:28:32.573 --> 01:28:33.860
a better performing system,

2319
01:28:33.880 --> 01:28:35.792
so we found that system performance

2320
01:28:35.812 --> 01:28:37.672
was actually more related to the

2321
01:28:37.692 --> 01:28:39.286
skin tone of the individual rather

2322
01:28:39.306 --> 01:28:41.686
than race and could be related to the

2323
01:28:41.706 --> 01:28:43.202
properties of the acquisition system.

2324
01:28:43.222 --> 01:28:45.074
So in this case you know

2325
01:28:45.094 --> 01:28:46.625
reengineering the system with a

2326
01:28:46.645 --> 01:28:48.310
better camera could fix the issue.

2327
01:28:48.330 --> 01:28:50.228
You could see that on the worst

2328
01:28:50.248 --> 01:28:52.237
system you know the image is really

2329
01:28:52.257 --> 01:28:54.334
washed out and and you don't have

2330
01:28:54.354 --> 01:28:56.440
really good contrast across the face,

2331
01:28:56.460 --> 01:28:57.985
whereas on the better system

2332
01:28:58.005 --> 01:28:59.880
you could really see the face.

2333
01:28:59.900 --> 01:29:02.210
Very well, so this is not.

2334
01:29:02.230 --> 01:29:05.036
This is not to say that all cases of

2335
01:29:05.056 --> 01:29:06.814
demographic differentials and face

2336
01:29:06.834 --> 01:29:09.610
recognition are due to image acquisition.

2337
01:29:09.630 --> 01:29:11.178
There are certainly important

2338
01:29:11.198 --> 01:29:13.587
considerations that are intrinsic to the

2339
01:29:13.607 --> 01:29:14.715
way matching algorithms

2340
01:29:14.735 --> 01:29:16.220
are designed and built,

2341
01:29:16.240 --> 01:29:19.487
and I think Patrick has done in his tests

2342
01:29:19.507 --> 01:29:22.440
has looked at that in great detail.

2343
01:29:22.460 --> 01:29:24.255
However, acquisition system design is

2344
01:29:24.275 --> 01:29:26.129
another important factor needed to

2345
01:29:26.149 --> 01:29:28.219
ensure equal performance for different

2346
01:29:28.239 --> 01:29:30.050
demographic groups, as an example.

2347
01:29:30.070 --> 01:29:31.370
From a different domain,

2348
01:29:31.390 --> 01:29:33.512
consider what would happen if the quality

2349
01:29:33.532 --> 01:29:35.780
of an acquisition system in some locations.

2350
01:29:35.800 --> 01:29:37.640
Is different than in others.

2351
01:29:37.660 --> 01:29:39.470
That means that face recognition

2352
01:29:39.490 --> 01:29:42.102
performance for people that live near those

2353
01:29:42.122 --> 01:29:44.340
different locations may also be different.

2354
01:29:44.360 --> 01:29:45.965
So it's important to really

2355
01:29:45.985 --> 01:29:47.265
consider image acquisition when

2356
01:29:47.285 --> 01:29:48.522
discussing demographic differences

2357
01:29:48.542 --> 01:29:50.290
and face recognition performance.

2358
01:29:50.310 --> 01:29:56.190
So next slide please.

2359
01:29:56.210 --> 01:29:58.750
So now we're getting to that Black Swan.

2360
01:29:58.770 --> 01:30:00.350
I told you about earlier.

2361
01:30:00.370 --> 01:30:01.586
Sometimes human behavior changes

2362
01:30:01.606 --> 01:30:03.130
dramatically in a matter not

2363
01:30:03.150 --> 01:30:04.510
anticipated by biometric technology.

2364
01:30:04.530 --> 01:30:06.350
So the COVID-19national emergency

2365
01:30:06.370 --> 01:30:08.990
hit just as we were planning last years,

2366
01:30:09.010 --> 01:30:09.908
rally event.

2367
01:30:09.928 --> 01:30:11.438
Suddenly, people were wearing face

2368
01:30:11.458 --> 01:30:13.150
masks in all public settings,

2369
01:30:13.170 --> 01:30:15.022
but there were no performance benchmarks

2370
01:30:15.042 --> 01:30:16.971
on how well face recognition works

2371
01:30:16.991 --> 01:30:19.230
in the presence of real face masks.

2372
01:30:19.250 --> 01:30:21.470
So in the middle of the pandemic,

2373
01:30:21.490 --> 01:30:23.346
we safely ran a rally test focused

2374
01:30:23.366 --> 01:30:24.656
on measuring the performance

2375
01:30:24.676 --> 01:30:26.000
of commercial systems.

2376
01:30:26.020 --> 01:30:27.436
With and without masks.

2377
01:30:27.456 --> 01:30:30.325
So you could see the results in the

2378
01:30:30.345 --> 01:30:33.049
infographic on the right there's a lot of

2379
01:30:33.069 --> 01:30:36.090
detail on that graphic that I won't go over,

2380
01:30:36.110 --> 01:30:37.200
but without masks,

2381
01:30:37.220 --> 01:30:39.060
the median system identified 93%

2382
01:30:39.080 --> 01:30:40.960
of the 582

2383
01:30:40.980 --> 01:30:43.750
people that try to use it.

2384
01:30:43.770 --> 01:30:45.490
That's very good with masks.

2385
01:30:45.510 --> 01:30:45.837
However,

2386
01:30:45.857 --> 01:30:47.220
the median system identified

2387
01:30:47.240 --> 01:30:48.620
just 77%

2388
01:30:48.640 --> 01:30:50.748
and that's that's a drop in performance

2389
01:30:50.768 --> 01:30:52.904
that was caused by people wearing

2390
01:30:52.924 --> 01:30:54.880
their personal masks during collection.

2391
01:30:54.900 --> 01:30:56.352
And, importantly, some systems.

2392
01:30:56.372 --> 01:30:59.450
We're able to work fairly well in both cases,

2393
01:30:59.470 --> 01:31:00.554
even with masks.

2394
01:31:00.574 --> 01:31:02.760
The best face recognition system identified

2395
01:31:02.780 --> 01:31:04.620
96% of the people,

2396
01:31:04.640 --> 01:31:07.005
but obviously there was a

2397
01:31:07.025 --> 01:31:08.440
significant performance hit.

2398
01:31:08.460 --> 01:31:14.880
So I'll go to the next slide now.

2399
01:31:14.900 --> 01:31:16.530
So can these changes in performance

2400
01:31:16.550 --> 01:31:18.515
affect the way the system performs

2401
01:31:18.535 --> 01:31:20.180
for different demographic groups,

2402
01:31:20.200 --> 01:31:22.154
for example with respect to race

2403
01:31:22.174 --> 01:31:24.501
and looking at the data from the

2404
01:31:24.521 --> 01:31:26.463
last rally test, we found that

2405
01:31:26.483 --> 01:31:28.120
the answer is unfortunately yes.

2406
01:31:28.140 --> 01:31:30.432
The chart on the right shows that without

2407
01:31:30.452 --> 01:31:32.420
face masks there were many systems

2408
01:31:32.440 --> 01:31:34.510
that identified more than 95%

2409
01:31:34.530 --> 01:31:36.400
of each demographic group.

2410
01:31:36.420 --> 01:31:39.136
So each of the dots on the plot on

2411
01:31:39.156 --> 01:31:42.208
the plot to them to my right is

2412
01:31:42.228 --> 01:31:44.400
a single combination of a camera.

2413
01:31:44.420 --> 01:31:46.610
An algorithm and the higher the dot,

2414
01:31:46.630 --> 01:31:48.820
the better the performance of the system.

2415
01:31:48.840 --> 01:31:50.703
The color of the dot shows the

2416
01:31:50.723 --> 01:31:52.930
race of the people being matched.

2417
01:31:52.950 --> 01:31:54.820
Finally, the number above each violin

2418
01:31:54.840 --> 01:31:56.760
plot shows the number of systems

2419
01:31:56.780 --> 01:31:58.340
that perform better than 95%

2420
01:31:58.360 --> 01:32:00.200
for each race group,

2421
01:32:00.220 --> 01:32:01.905
the rally performance threshold that

2422
01:32:01.925 --> 01:32:04.620
was that was what the one will be set.

2423
01:32:04.640 --> 01:32:06.366
There were lots of systems above

2424
01:32:06.386 --> 01:32:07.955
this threshold for each race

2425
01:32:07.975 --> 01:32:09.050
category without masks.

2426
01:32:09.070 --> 01:32:09.366
However,

2427
01:32:09.386 --> 01:32:11.580
when people put on their personal mask,

2428
01:32:11.600 --> 01:32:12.525
the picture changed.

2429
01:32:12.545 --> 01:32:13.790
The performance decreased overall,

2430
01:32:13.810 --> 01:32:14.783
but performance decreased

2431
01:32:14.803 --> 01:32:15.780
more for individuals.

2432
01:32:15.800 --> 01:32:17.200
Self identifying as black

2433
01:32:17.220 --> 01:32:18.270
or African American.

2434
01:32:18.290 --> 01:32:20.274
While there were some systems above

2435
01:32:20.294 --> 01:32:22.320
threshold for other groups now no

2436
01:32:22.340 --> 01:32:23.940
system performed above threshold for

2437
01:32:23.960 --> 01:32:26.100
black or African American individuals.

2438
01:32:26.120 --> 01:32:27.880
So what's the takeaway from

2439
01:32:27.900 --> 01:32:29.310
these last two slides?

2440
01:32:29.330 --> 01:32:29.666
Well,

2441
01:32:29.686 --> 01:32:30.378
biometric performance

2442
01:32:30.398 --> 01:32:32.160
isn't going to be fixed.

2443
01:32:32.180 --> 01:32:35.360
It will vary as facts on the ground change,

2444
01:32:35.380 --> 01:32:37.130
and it's therefore really important

2445
01:32:37.150 --> 01:32:39.286
to test the technologies on an

2446
01:32:39.306 --> 01:32:40.891
ongoing basis to verify that

2447
01:32:40.911 --> 01:32:42.637
vendor claims are correct and

2448
01:32:42.657 --> 01:32:44.260
that performance hasn't changed.

2449
01:32:44.280 --> 01:32:47.030
And this includes testing the full system.

2450
01:32:47.050 --> 01:32:48.884
For equitability and I think there's

2451
01:32:48.904 --> 01:32:50.980
some of these ongoing tests you know.

2452
01:32:51.000 --> 01:32:52.666
Certainly the FRVT tests that Patrick

2453
01:32:52.686 --> 01:32:54.330
talked about are really important,

2454
01:32:54.350 --> 01:32:56.450
but I think I also you know,

2455
01:32:56.470 --> 01:32:57.975
scenario testing really is important

2456
01:32:57.995 --> 01:33:00.174
as well to be able to capture

2457
01:33:00.194 --> 01:33:02.004
some of these other contexts like

2458
01:33:02.024 --> 01:33:03.834
failure to acquire images and and

2459
01:33:03.854 --> 01:33:06.115
changes in the quality of the way

2460
01:33:06.135 --> 01:33:08.840
that the images are required.

2461
01:33:08.860 --> 01:33:10.364
And with that I think I'm

2462
01:33:10.384 --> 01:33:12.160
done with my part of the talk.

2463
01:33:12.180 --> 01:33:14.710
So Vivek, I'm not sure what you want to do.

2464
01:33:14.730 --> 01:33:16.915
If you want to go for questions

2465
01:33:16.935 --> 01:33:19.220
now or switch to Laura 's.

2466
01:33:19.240 --> 01:33:22.901
Vivek: I think we can have questions in the

2467
01:33:22.921 --> 01:33:26.560
chat pod as we're going through the.

2468
01:33:26.580 --> 01:33:29.400
Presentation, so if there are any questions,

2469
01:33:29.420 --> 01:33:31.430
will continue to answer them.

2470
01:33:31.450 --> 01:33:34.270
So let's move on to Laura's presentation.

2471
01:33:34.290 --> 01:33:35.746
Laura: Thank you so much.

2472
01:33:35.766 --> 01:33:38.601
So we've talked a lot about how

2473
01:33:38.621 --> 01:33:40.845
algorithms recognize faces like

2474
01:33:40.865 --> 01:33:43.089
face recognition is something

2475
01:33:43.109 --> 01:33:45.340
that humans do naturally.

2476
01:33:45.360 --> 01:33:47.915
Humans have dedicated hardware that

2477
01:33:47.935 --> 01:33:50.490
specifically perceives and recognizes faces.

2478
01:33:50.510 --> 01:33:52.275
This hardware is actually specialized

2479
01:33:52.295 --> 01:33:54.551
areas of the brain that are

2480
01:33:54.571 --> 01:33:56.700
specifically activated when doing faces.

2481
01:33:56.720 --> 01:34:00.530
Then when viewing objects or other items.

2482
01:34:00.550 --> 01:34:03.060
While humans can do face recognition easily,

2483
01:34:03.080 --> 01:34:04.872
we however do not have special

2484
01:34:04.892 --> 01:34:07.020
hardware for in our brain dedicated

2485
01:34:07.040 --> 01:34:09.220
to fingerprint or iris recognition.

2486
01:34:09.240 --> 01:34:11.170
You do not recognize your grandmother

2487
01:34:11.190 --> 01:34:12.552
by the arches

2488
01:34:12.572 --> 01:34:14.883
or the loops of her fingerprints when

2489
01:34:14.903 --> 01:34:17.624
she waves hello because humans can do

2490
01:34:17.644 --> 01:34:19.720
face recognition easily and naturally.

2491
01:34:19.740 --> 01:34:22.248
This is why we tend to use face

2492
01:34:22.268 --> 01:34:27.650
recognition to verify identity.

2493
01:34:27.670 --> 01:34:29.970
When we talk about human face recognition,

2494
01:34:29.990 --> 01:34:31.625
it's important to know there

2495
01:34:31.645 --> 01:34:32.950
are two different types.

2496
01:34:32.970 --> 01:34:35.260
The first one is familiar face recognition,

2497
01:34:35.280 --> 01:34:37.580
which is exactly what it sounds like.

2498
01:34:37.600 --> 01:34:39.210
This is where you recognize

2499
01:34:39.230 --> 01:34:41.220
faces of people that you know,

2500
01:34:41.240 --> 01:34:42.548
such as your family,

2501
01:34:42.568 --> 01:34:43.394
friends, celebrities,

2502
01:34:43.414 --> 01:34:45.450
and other notable figures.

2503
01:34:45.470 --> 01:34:47.130
The other type of face recognition

2504
01:34:47.150 --> 01:34:48.250
is unfamiliar face recognition,

2505
01:34:48.270 --> 01:34:50.810
and this is the faces of people you

2506
01:34:50.830 --> 01:34:53.670
do not know or have not met before.

2507
01:34:53.690 --> 01:34:55.815
Most real-world applications of face

2508
01:34:55.835 --> 01:34:57.960
recognition deal with unfamiliar faces.

2509
01:34:57.980 --> 01:35:00.802
So what do we know about unfamiliar

2510
01:35:00.822 --> 01:35:02.973
face recognition research conducted in

2511
01:35:02.993 --> 01:35:05.173
Australia examined how well passport

2512
01:35:05.193 --> 01:35:07.644
officers were at unfamiliar face

2513
01:35:07.664 --> 01:35:10.360
matching compared to university students.

2514
01:35:10.380 --> 01:35:12.580
The unfamiliar face matching task officers

2515
01:35:12.600 --> 01:35:14.790
and students completed was the GFMT

2516
01:35:14.810 --> 01:35:17.520
or the Glasgal feast matching tasks,

2517
01:35:17.540 --> 01:35:20.184
which is the gold standard for

2518
01:35:20.204 --> 01:35:21.960
measuring face matching ability.

2519
01:35:21.980 --> 01:35:23.885
Results of the Australian study

2520
01:35:23.905 --> 01:35:26.262
found that passport officers were no

2521
01:35:26.282 --> 01:35:28.217
better at unfamiliar face matching

2522
01:35:28.237 --> 01:35:30.243
then university students and that

2523
01:35:30.263 --> 01:35:32.253
employment duration or years of

2524
01:35:32.273 --> 01:35:33.870
experience had no relationship.

2525
01:35:33.890 --> 01:35:36.610
An unfamiliar face matching tasks.

2526
01:35:36.630 --> 01:35:38.976
So their results showed that there is

2527
01:35:38.996 --> 01:35:41.840
a wide range of performance when it

2528
01:35:41.860 --> 01:35:44.060
comes to unfamiliar face mathcers.

2529
01:35:44.080 --> 01:35:44.373
However,

2530
01:35:44.393 --> 01:35:46.251
there is a subgroup of people

2531
01:35:46.271 --> 01:35:48.231
known as super recognizers and

2532
01:35:48.251 --> 01:35:50.191
super recognizers are individuals

2533
01:35:50.211 --> 01:35:52.215
with exceptional face recognition

2534
01:35:52.235 --> 01:35:54.015
abilities and can remember faces

2535
01:35:54.035 --> 01:35:56.363
with little exposure and are very

2536
01:35:56.383 --> 01:35:59.080
good at unfamiliar face matching.

2537
01:35:59.100 --> 01:36:01.292
So when we think that maybe we

2538
01:36:01.312 --> 01:36:03.070
should employ super recognizers,

2539
01:36:03.090 --> 01:36:05.610
wear face matching tasks are very common.

2540
01:36:05.630 --> 01:36:05.986
However,

2541
01:36:06.006 --> 01:36:07.490
super recognizers aren't very

2542
01:36:07.510 --> 01:36:10.108
common and they only make up a

2543
01:36:10.128 --> 01:36:11.790
small percentage of the population.

2544
01:36:11.810 --> 01:36:13.986
So how can we overcome the

2545
01:36:14.006 --> 01:36:19.170
unfamiliar face matching skill gap?

2546
01:36:19.190 --> 01:36:21.200
While humans are prone to errors,

2547
01:36:21.220 --> 01:36:23.584
a lot of this talk has demonstrated how

2548
01:36:23.604 --> 01:36:25.930
accurate face recognition technology can be.

2549
01:36:25.950 --> 01:36:28.541
So we've now arrived at an interesting

2550
01:36:28.561 --> 01:36:30.130
intersection where human ability

2551
01:36:30.150 --> 01:36:32.210
and phased algorithms are combined.

2552
01:36:32.230 --> 01:36:34.145
But there are still unanswered

2553
01:36:34.165 --> 01:36:35.693
questions about how humans

2554
01:36:35.713 --> 01:36:37.740
and algorithms work together.

2555
01:36:37.760 --> 01:36:40.008
What sort of effects do we see with human

2556
01:36:40.028 --> 01:36:42.180
and algorithm decisions are combined?

2557
01:36:42.200 --> 01:36:43.728
How do we measure the performance

2558
01:36:43.748 --> 01:36:45.628
of a system that integrates both

2559
01:36:45.648 --> 01:36:50.150
humans and algorithms together?

2560
01:36:50.170 --> 01:36:51.758
Human algorithm decisions can

2561
01:36:51.778 --> 01:36:53.770
be combined in multiple ways.

2562
01:36:53.790 --> 01:36:55.780
To date, research has focused

2563
01:36:55.800 --> 01:36:57.390
on two different workflows.

2564
01:36:57.410 --> 01:37:00.200
The first workflow is a parallel process,

2565
01:37:00.220 --> 01:37:03.014
and this is where a human adjudicator

2566
01:37:03.034 --> 01:37:04.220
reviews face information,

2567
01:37:04.240 --> 01:37:06.225
while an algorithm also reviews

2568
01:37:06.245 --> 01:37:08.640
face information at the same time.

2569
01:37:08.660 --> 01:37:10.650
Once once each entity reaches

2570
01:37:10.670 --> 01:37:12.660
a decision about the face,

2571
01:37:12.680 --> 01:37:14.670
their decisions are fused together,

2572
01:37:14.690 --> 01:37:17.080
which creates a biometric match result.

2573
01:37:17.100 --> 01:37:18.744
Parallel processes are commonly

2574
01:37:18.764 --> 01:37:20.830
seen in long term investigations.

2575
01:37:20.850 --> 01:37:23.035
Forensic scenarios another type of

2576
01:37:23.055 --> 01:37:25.723
workflow is a serial process and

2577
01:37:25.743 --> 01:37:28.159
this is where an algorithm reviews

2578
01:37:28.179 --> 01:37:30.483
of face information and provides a

2579
01:37:30.503 --> 01:37:32.890
result to human and then the human

2580
01:37:32.910 --> 01:37:35.098
ways the algorithm decision along with

2581
01:37:35.118 --> 01:37:37.944
contacts in their own decision and who

2582
01:37:37.964 --> 01:37:40.130
ultimately decides the biometric result.

2583
01:37:40.150 --> 01:37:42.610
Serial processes are commonly seen

2584
01:37:42.630 --> 01:37:44.600
and travel security scenarios.

2585
01:37:44.620 --> 01:37:46.412
We conducted a study focusing on

2586
01:37:46.432 --> 01:37:47.917
the serial process which I'll

2587
01:37:47.937 --> 01:37:49.590
go over in the next two slides.

2588
01:37:49.610 --> 01:37:50.970
An important note before continuing.

2589
01:37:50.990 --> 01:37:53.322
These are not the only ways that humans

2590
01:37:53.342 --> 01:37:55.349
and algorithms can be combined is just

2591
01:37:55.369 --> 01:37:57.340
the way research has looked at so far.

2592
01:37:57.360 --> 01:37:59.840
If you have other ways which you work within,

2593
01:37:59.860 --> 01:38:01.412
algorithms would be interested in hearing

2594
01:38:01.432 --> 01:38:06.300
from you and about your processes.

2595
01:38:06.320 --> 01:38:08.440
So as we said in our study,

2596
01:38:08.460 --> 01:38:10.869
we used to serial process where an

2597
01:38:10.889 --> 01:38:12.548
algorithm provided decision and then

2598
01:38:12.568 --> 01:38:14.585
we asked volunteers what did they think

2599
01:38:14.605 --> 01:38:16.809
are these fees is the same person or

2600
01:38:16.829 --> 01:38:18.313
are these faces of different people?

2601
01:38:18.333 --> 01:38:20.426
Our goal was to determine if the algorithms

2602
01:38:20.446 --> 01:38:22.210
decision influenced volunteer responses.

2603
01:38:22.230 --> 01:38:24.793
So here on this slide there are two examples

2604
01:38:24.813 --> 01:38:27.410
of some of the face pairs and decisions.

2605
01:38:27.430 --> 01:38:28.502
Volunteers were shown,

2606
01:38:28.522 --> 01:38:30.686
some volunteers were shown the face

2607
01:38:30.706 --> 01:38:32.770
here on the left where the computer

2608
01:38:32.790 --> 01:38:35.210
said that face is worth the same people.

2609
01:38:35.230 --> 01:38:36.890
While other volunteers saw the face

2610
01:38:36.910 --> 01:38:39.080
pair on the right where the computer

2611
01:38:39.100 --> 01:38:40.770
said they were different people.

2612
01:38:40.790 --> 01:38:42.715
Does the decision presented with

2613
01:38:42.735 --> 01:38:45.188
either face pair make you question

2614
01:38:45.208 --> 01:38:46.550
your own decision?

2615
01:38:46.570 --> 01:38:48.270
And I'll give you the answer right now.

2616
01:38:48.290 --> 01:38:49.350
These are of different people,

2617
01:38:49.370 --> 01:38:50.640
so if you've got it right,

2618
01:38:50.660 --> 01:38:55.680
you can give yourself a pat on the back.

2619
01:38:55.700 --> 01:38:57.804
So the results of our what

2620
01:38:57.824 --> 01:39:00.070
were the results of our study.

2621
01:39:00.090 --> 01:39:02.092
We found that the computer decisions

2622
01:39:02.112 --> 01:39:03.896
did in fact alter volunteers

2623
01:39:03.916 --> 01:39:05.444
responses when the computer

2624
01:39:05.464 --> 01:39:07.760
said the faces were the same,

2625
01:39:07.780 --> 01:39:09.570
false positive rates were much

2626
01:39:09.590 --> 01:39:11.440
higher than when the computer

2627
01:39:11.460 --> 01:39:13.250
said the faces were different,

2628
01:39:13.270 --> 01:39:15.920
so these results demonstrate how the

2629
01:39:15.940 --> 01:39:18.203
algorithm decisions can cognitively bias

2630
01:39:18.223 --> 01:39:20.370
of person space similarity judgments.

2631
01:39:20.390 --> 01:39:22.386
We recently also did a follow-up

2632
01:39:22.406 --> 01:39:24.258
study where we included a face

2633
01:39:24.278 --> 01:39:26.190
mask on one of the images in the

2634
01:39:26.210 --> 01:39:28.134
face pair to emulate the conditions

2635
01:39:28.154 --> 01:39:31.250
that we see during the pandemic.

2636
01:39:31.270 --> 01:39:33.870
We found that face mask reduced information,

2637
01:39:33.890 --> 01:39:36.271
which then increases the influence of the

2638
01:39:36.291 --> 01:39:41.360
algorithm results on peoples judgments.

2639
01:39:41.380 --> 01:39:42.676
So given these results,

2640
01:39:42.696 --> 01:39:44.650
where should we go from here?

2641
01:39:44.670 --> 01:39:46.666
First, users and operators should be

2642
01:39:46.686 --> 01:39:48.391
trained on biometric match outcomes

2643
01:39:48.411 --> 01:39:50.666
and to and to critically review these

2644
01:39:50.686 --> 01:39:52.550
decisions when there are non matches.

2645
01:39:52.570 --> 01:39:54.440
This might be due to poor image

2646
01:39:54.460 --> 01:39:56.738
quality or it may actually be because

2647
01:39:56.758 --> 01:39:58.929
these are different people and when

2648
01:39:58.949 --> 01:40:01.372
there are matches there may be false

2649
01:40:01.392 --> 01:40:03.216
matches present which are extremely

2650
01:40:03.236 --> 01:40:05.206
difficult to spot false matches

2651
01:40:05.226 --> 01:40:07.462
made by algorithms tend to be of

2652
01:40:07.482 --> 01:40:09.330
people who are of the same race,

2653
01:40:09.350 --> 01:40:11.650
age and gender.

2654
01:40:11.670 --> 01:40:14.122
And users and face recognition technology

2655
01:40:14.142 --> 01:40:16.178
should supplement algorithm results by

2656
01:40:16.198 --> 01:40:17.920
using their investigative tool belt.

2657
01:40:17.940 --> 01:40:19.488
Consider other identifiable information

2658
01:40:19.508 --> 01:40:21.452
such as birthmarks, moles, tattoos,

2659
01:40:21.472 --> 01:40:23.020
and other physical features,

2660
01:40:23.040 --> 01:40:25.370
as well as confirming personal information.

2661
01:40:25.390 --> 01:40:27.182
Asking about information related

2662
01:40:27.202 --> 01:40:29.447
onto personal documents such as

2663
01:40:29.467 --> 01:40:31.500
places around the neighborhood.

2664
01:40:31.520 --> 01:40:32.319
And with that,

2665
01:40:32.339 --> 01:40:33.411
that concludes my section

2666
01:40:33.431 --> 01:40:34.230
of the presentation,

2667
01:40:34.250 --> 01:40:36.057
and I will turn it back over

2668
01:40:36.077 --> 01:40:42.047
to Vivek and Abby.

2669
01:40:42.067 --> 01:40:46.820


2670
01:40:46.840 --> 01:40:50.330
Vivek: Thank you Laura. Appreciate it.

2671
01:40:50.350 --> 01:40:52.308
Alright, appreciate it. So thank you to

2672
01:40:52.328 --> 01:40:54.580
you and all our speakers.

2673
01:40:54.600 --> 01:40:56.724
This was very informative.

2674
01:40:56.744 --> 01:40:59.269
Uh, we have a little bit time left

2675
01:40:59.289 --> 01:41:02.070
for Q and A with all these speakers.

2676
01:41:02.090 --> 01:41:04.290
I invite everyone to send questions

2677
01:41:04.310 --> 01:41:10.280
through the chat chat bot.

2678
01:41:10.300 --> 01:41:18.130


2679
01:41:18.150 --> 01:41:20.706
Abby: And to our NYAST members, if you can't

2680
01:41:20.726 --> 01:41:22.858
think of any questions now, that's OK.

2681
01:41:22.878 --> 01:41:25.070
We are going to send out the speaker

2682
01:41:25.090 --> 01:41:27.230
slides that were approved for release

2683
01:41:27.250 --> 01:41:29.080
to everyone in attendance today,

2684
01:41:29.100 --> 01:41:31.432
so you're welcome to reach out after to

2685
01:41:31.452 --> 01:41:37.422
ask any questions you might think of.

2686
01:41:37.442 --> 01:41:38.740


2687
01:41:38.760 --> 01:41:40.215
Vivek: While we're waiting on other

2688
01:41:40.235 --> 01:41:41.400
people to ask questions,

2689
01:41:41.420 --> 01:41:45.980
I actually had one pop up in my mind.

2690
01:41:46.000 --> 01:41:49.990
So my question is to the speakers.

2691
01:41:50.010 --> 01:41:53.200
In terms of the aging process,

2692
01:41:53.220 --> 01:41:57.365
we talked about how aging changes stuff

2693
01:41:57.385 --> 01:42:00.980
do algorithms actually take into account?

2694
01:42:01.000 --> 01:42:02.940
The time frame when the

2695
01:42:02.960 --> 01:42:04.510
image is recorded saying,

2696
01:42:04.530 --> 01:42:06.470
hey, it's five years older,

2697
01:42:06.490 --> 01:42:08.490
ten years old and

2698
01:42:08.510 --> 01:42:11.033
age those images when they're

2699
01:42:11.053 --> 01:42:17.023
trying to do the match.

2700
01:42:17.043 --> 01:42:19.140


2701
01:42:19.160 --> 01:42:22.547
Patrick: I, I don't think that's explicitly

2702
01:42:22.567 --> 01:42:25.960
an aspect that they algorithms are using,

2703
01:42:25.980 --> 01:42:28.950
so they they typically don't

2704
01:42:28.970 --> 01:42:30.750
trust biographic data.

2705
01:42:30.770 --> 01:42:32.870
So if you if you would say it's

2706
01:42:32.890 --> 01:42:35.134
a woman and you would say that

2707
01:42:35.154 --> 01:42:37.140
they are forty three years old.

2708
01:42:37.160 --> 01:42:39.870
That data is sort of historically unreliable,

2709
01:42:39.890 --> 01:42:42.132
and the industry hasn't really focused

2710
01:42:42.152 --> 01:42:44.940
on trying to employ metadata like that.

2711
01:42:44.960 --> 01:42:48.202
Not not in not in terms of

2712
01:42:48.222 --> 01:42:49.600
using the algorithm.

2713
01:42:49.620 --> 01:42:54.478
Uh, so so there have been attempts to do

2714
01:42:54.498 --> 01:42:59.010
explicit age regression and age progression.

2715
01:42:59.030 --> 01:43:01.704
Particularly in Child

2716
01:43:01.724 --> 01:43:03.500
Exploitation investigation.

2717
01:43:03.520 --> 01:43:06.152
Uhm, where you want to try

2718
01:43:06.172 --> 01:43:07.920
and help the algorithm.

2719
01:43:07.940 --> 01:43:10.806
by synthesizing the appearance

2720
01:43:10.826 --> 01:43:14.160
of somebody at a different time.

2721
01:43:14.180 --> 01:43:17.128
And I I don't know well how

2722
01:43:17.148 --> 01:43:19.056
those algorithms actually work

2723
01:43:19.076 --> 01:43:21.360
at that progression regression.

2724
01:43:21.380 --> 01:43:22.370
But yeah,

2725
01:43:22.390 --> 01:43:25.400
the the face recognition algorithms today

2726
01:43:25.420 --> 01:43:29.677
are more tolerant of changes in age.

2727
01:43:29.697 --> 01:43:30.873
age related appearance changes

2728
01:43:30.893 --> 01:43:36.863
than they used to be.

2729
01:43:36.883 --> 01:43:38.360


2730
01:43:38.380 --> 01:43:41.209
Abby: Alright, one question from Roy Diekman he

2731
01:43:41.229 --> 01:43:43.248
asked, are there enough distinguishing

2732
01:43:43.268 --> 01:43:45.690
characteristics that some tests have tried?

2733
01:43:45.710 --> 01:43:52.370
The side of the head for confirmation?

2734
01:43:52.390 --> 01:43:55.943
Patrick: So I had a slide on this the the

2735
01:43:55.963 --> 01:43:59.230
profile view matched a frontal view,

2736
01:43:59.250 --> 01:44:02.270
that's ninety degree head rotation.

2737
01:44:02.290 --> 01:44:04.680
And that's a very challenging task.

2738
01:44:04.700 --> 01:44:07.936
Uh, it's a lot of algorithm developers have

2739
01:44:07.956 --> 01:44:10.440
pursued what's called pose invariants.

2740
01:44:10.460 --> 01:44:12.655
They they want to be

2741
01:44:12.675 --> 01:44:14.430
able to recognize people,

2742
01:44:14.450 --> 01:44:17.811
sort of independent of the the orientation

2743
01:44:17.831 --> 01:44:21.420
of the head relative to the camera.

2744
01:44:21.440 --> 01:44:24.295
And some of you know

2745
01:44:24.315 --> 01:44:27.170
remarkable way down that road.

2746
01:44:27.190 --> 01:44:31.520
It remains a challenging problem.

2747
01:44:31.540 --> 01:44:35.576
Yeah, I don't think these algorithms

2748
01:44:35.596 --> 01:44:38.280
are explicitly discovering features.

2749
01:44:38.300 --> 01:44:40.835
Uh, you know, you know the chin

2750
01:44:40.855 --> 01:44:43.880
outline or or the details of the ear.

2751
01:44:43.900 --> 01:44:46.673
I think it is the

2752
01:44:46.693 --> 01:44:49.330
gains that have been made are.

2753
01:44:49.350 --> 01:44:52.235
Uh, from sort of general

2754
01:44:52.255 --> 01:44:58.225
pose invariants research.

2755
01:44:58.245 --> 01:45:04.690


2756
01:45:04.710 --> 01:45:06.676
Abby: OK, great will pause and see

2757
01:45:06.696 --> 01:45:12.666
if any more questions pop in.

2758
01:45:12.686 --> 01:45:25.410


2759
01:45:25.430 --> 01:45:27.230
OK, and speakers feel free if you

2760
01:45:27.250 --> 01:45:29.071
have anything else that you haven't

2761
01:45:29.091 --> 01:45:30.840
discussed, now is a good time.

2762
01:45:30.860 --> 01:45:32.850
We have a few extra minutes left,

2763
01:45:32.870 --> 01:45:35.130
or if not, we can certainly end early.

2764
01:45:35.150 --> 01:45:36.395
But again, please use the

2765
01:45:36.415 --> 01:45:42.385
chat pod to ask any questions.

2766
01:45:42.405 --> 01:45:46.930


2767
01:45:46.950 --> 01:45:49.800
Arun: Hi, this is Arun  I'm just responding back.

2768
01:45:49.820 --> 01:45:52.256
I think there was a question earlier about

2769
01:45:52.276 --> 01:45:54.270
pitch angles for CCTV camera systems.

2770
01:45:54.290 --> 01:45:56.460
My original answer was 10% or

2771
01:45:56.480 --> 01:45:58.871
ten degrees which I went back and

2772
01:45:58.891 --> 01:46:00.690
double checked it was for passport

2773
01:46:00.710 --> 01:46:02.880
photos plus or minus 5%.

2774
01:46:02.900 --> 01:46:04.790
However, I kind of

2775
01:46:04.810 --> 01:46:07.614
I checked back with with one of our

2776
01:46:07.634 --> 01:46:10.157
colleagues at the FBI and they actually

2777
01:46:10.177 --> 01:46:12.570
also came back with ten degrees.

2778
01:46:12.590 --> 01:46:14.980
If you go beyond ten degree pitch angle,

2779
01:46:15.000 --> 01:46:16.972
it turns out to be quite a challenge

2780
01:46:16.992 --> 01:46:18.590
for facial recognition systems.

2781
01:46:18.610 --> 01:46:20.697
I think Patrick and and we've seen

2782
01:46:20.717 --> 01:46:23.410
this in in some of our tests as well.

2783
01:46:23.430 --> 01:46:25.810
One thing you could think about too though,

2784
01:46:25.830 --> 01:46:27.917
is if you are employing a camera

2785
01:46:27.937 --> 01:46:28.820
system like that.

2786
01:46:28.840 --> 01:46:30.630
If you can deploy an attractor,

2787
01:46:30.650 --> 01:46:32.912
basically a sign or something like that

2788
01:46:32.932 --> 01:46:34.840
might drive persons I to the camera.

2789
01:46:34.860 --> 01:46:35.896
You might have more.

2790
01:46:35.916 --> 01:46:41.886
Might be more likely to be successful.

2791
01:46:41.906 --> 01:46:48.100


2792
01:46:48.120 --> 01:46:50.110
Vivek: Thank you for that Arun.

2793
01:46:50.130 --> 01:46:52.042
Any of these speakers have

2794
01:46:52.062 --> 01:46:53.699
any other antidotes they'd like

2795
01:46:53.719 --> 01:46:57.560
to share with the community?

2796
01:46:57.580 --> 01:47:03.550
How they could utilize their systems better?

2797
01:47:03.570 --> 01:47:24.920


2798
01:47:24.940 --> 01:47:27.740
Abby: It doesn't look like we have any

2799
01:47:27.760 --> 01:47:30.010
further questions coming through Vivek.

2800
01:47:30.030 --> 01:47:32.330
Vivek: Thank you Abby.

2801
01:47:32.350 --> 01:47:35.550
So this concludes today's presentation.

2802
01:47:35.570 --> 01:47:37.910
We thank you all for joining us and

2803
01:47:37.930 --> 01:47:40.070
please stay tuned for announcements,

2804
01:47:40.090 --> 01:47:42.160
announcements of our next NIST forum.

2805
01:47:42.180 --> 01:47:44.038
We look forward to sharing those

2806
01:47:44.058 --> 01:47:45.746
details with you and appreciate

2807
01:47:45.766 --> 01:47:48.080
everybody 's time joining us today.

2808
01:47:48.100 --> 01:48:05.794
Have a great day.