WEBVTT

00:00:00.000 --> 00:00:01.344
Good morning folks.

00:00:01.344 --> 00:00:03.584
My name is Arun Vemury.

00:00:03.590 --> 00:00:05.105
I'm with the DHS science

00:00:05.105 --> 00:00:06.014
and Technology Directorate.

00:00:06.020 --> 00:00:08.315
I'd like to thank you for joining us for

00:00:08.315 --> 00:00:10.208
today to learn more about the results

00:00:10.208 --> 00:00:12.332
of our 2021 Biometric technology rally.

00:00:12.332 --> 00:00:15.410
As many of you are likely aware of,

00:00:15.410 --> 00:00:18.090
the results are already or have been live

00:00:18.090 --> 00:00:20.520
on the mdtf.org website for some time,

00:00:20.520 --> 00:00:22.976
but today you'll have a chance to hear

00:00:22.976 --> 00:00:25.446
more from the test team who oversaw

00:00:25.446 --> 00:00:27.649
and managed the evaluations to help

00:00:27.649 --> 00:00:30.064
explain what the results mean and help

00:00:30.064 --> 00:00:32.532
answer any questions you may have.

00:00:32.532 --> 00:00:34.554
With that, let's go ahead to the next slide.

00:00:40.070 --> 00:00:41.838
So we'll give you a general overview of

00:00:41.838 --> 00:00:43.479
what we're going to talk about today.

00:00:43.480 --> 00:00:45.744
We'll provide a very brief intro to the

00:00:45.744 --> 00:00:47.810
biometric and Identity Technology Center.

00:00:47.810 --> 00:00:49.706
We'll talk about the technology rallies,

00:00:49.710 --> 00:00:51.834
the the current one for the one we just

00:00:51.834 --> 00:00:54.198
completed for 2021 and previous rallies

00:00:54.198 --> 00:00:57.252
and how that differs, how we what,

00:00:57.252 --> 00:00:59.870
what the timeline for the rally was,

00:00:59.870 --> 00:01:01.326
what the acquisition systems

00:01:01.326 --> 00:01:03.510
did as part of the test,

00:01:03.510 --> 00:01:06.108
how the matching systems, you know,

00:01:06.110 --> 00:01:08.273
work were used and how they were

00:01:08.273 --> 00:01:10.008
evaluated as part of the test.

00:01:10.010 --> 00:01:12.824
An explanation of the overall test approach

00:01:12.824 --> 00:01:14.966
and information about service providers

00:01:14.966 --> 00:01:17.564
or technology providers will then dig,

00:01:17.570 --> 00:01:18.875
dig deeper into the actual

00:01:18.875 --> 00:01:19.919
results of the test,

00:01:19.920 --> 00:01:21.558
which I think is probably the

00:01:21.558 --> 00:01:23.214
primary interest of you all who

00:01:23.214 --> 00:01:24.732
are joining us today talking about

00:01:24.732 --> 00:01:27.180
how we analyze the data and how we

00:01:27.180 --> 00:01:28.440
measured the performance measures

00:01:28.440 --> 00:01:31.016
we we performed and we'll close out

00:01:31.016 --> 00:01:33.353
with any questions or or information

00:01:33.353 --> 00:01:35.293
that you're looking for following

00:01:35.293 --> 00:01:37.708
up on the presentation today.

00:01:37.710 --> 00:01:38.628
Let's go to the next slide.

00:01:43.130 --> 00:01:45.200
So the biometric and identity Technology

00:01:45.200 --> 00:01:48.033
Center again is a group of subject

00:01:48.033 --> 00:01:50.041
matter expertise with subject experts

00:01:50.041 --> 00:01:52.663
within the Department of Homeland Security.

00:01:52.670 --> 00:01:55.113
Our role is within the science and

00:01:55.113 --> 00:01:57.270
Technology Directorate to provide objective,

00:01:57.270 --> 00:01:58.958
quantifiable information to help

00:01:58.958 --> 00:02:01.490
provide data to DHS components and

00:02:01.553 --> 00:02:03.817
stakeholders about the technologies,

00:02:03.820 --> 00:02:07.084
about how they work, where they don't work,

00:02:07.090 --> 00:02:09.142
how to manage our risks associated

00:02:09.142 --> 00:02:11.232
with these technologies, and also to

00:02:11.232 --> 00:02:13.356
work back with industry and academia.

00:02:13.360 --> 00:02:14.970
To identify issues with the

00:02:14.970 --> 00:02:17.030
technologies to make them work better,

00:02:17.030 --> 00:02:18.942
more effectively for potential

00:02:18.942 --> 00:02:20.854
use in the future.

00:02:20.860 --> 00:02:21.410
Next slide.

00:02:25.440 --> 00:02:28.128
So our biometric technology rallies are

00:02:28.128 --> 00:02:31.297
focused are basically a set of industry

00:02:31.297 --> 00:02:33.467
challenges where we are orienting

00:02:33.467 --> 00:02:36.397
industry to specific types of use cases.

00:02:36.400 --> 00:02:38.856
In this case, what we've done for the

00:02:38.856 --> 00:02:41.211
last several years have focused on

00:02:41.211 --> 00:02:43.301
the challenges associated with high

00:02:43.301 --> 00:02:45.140
throughput biometric recognition,

00:02:45.140 --> 00:02:47.114
something where we have a situation

00:02:47.114 --> 00:02:49.980
where we have a group of unknown people

00:02:49.980 --> 00:02:52.212
we need to process them one at a time,

00:02:52.220 --> 00:02:54.020
verify their identity within the

00:02:54.020 --> 00:02:55.820
matter of a few seconds.

00:02:55.820 --> 00:02:58.004
Because of the large volume of people who

00:02:58.004 --> 00:03:00.590
are going through various types of high

00:03:00.590 --> 00:03:02.918
throughput checkpoints and as an example,

00:03:02.918 --> 00:03:05.210
the high throughput checkpoints are actually

00:03:05.276 --> 00:03:07.738
abstracted out from notional DHS processes.

00:03:07.738 --> 00:03:09.046
So, for example,

00:03:09.050 --> 00:03:11.780
there are similarities to border crossings,

00:03:11.780 --> 00:03:13.523
similarities to aviation

00:03:13.523 --> 00:03:15.847
checkpoints to accessing critical

00:03:15.847 --> 00:03:18.010
infrastructure sectors and more.

00:03:18.010 --> 00:03:18.554
And ultimately,

00:03:18.554 --> 00:03:20.730
what the goal really is is to screen

00:03:20.790 --> 00:03:22.614
hundreds or maybe thousands of people

00:03:22.614 --> 00:03:24.618
within the matter of a few minutes,

00:03:24.620 --> 00:03:26.380
and we'll talk a little bit more about

00:03:26.380 --> 00:03:28.389
what the goals were or how we expressed

00:03:28.389 --> 00:03:30.264
those goals to industry so that they

00:03:30.264 --> 00:03:31.890
could come up with effective solutions.

00:03:31.890 --> 00:03:32.634
Here again,

00:03:32.634 --> 00:03:35.238
our goal is to help define

00:03:35.238 --> 00:03:37.469
these methods of measurement,

00:03:37.470 --> 00:03:39.456
provide quantifiable data to DHS components

00:03:39.456 --> 00:03:41.394
and stakeholders so that they can

00:03:41.394 --> 00:03:43.062
perform an apples to apples comparison

00:03:43.062 --> 00:03:44.780
about how technologies work or where

00:03:44.780 --> 00:03:46.744
the state of the art currently lies,

00:03:46.744 --> 00:03:48.508
not only in terms of matching.

00:03:48.510 --> 00:03:49.920
Systems provide biometric

00:03:49.920 --> 00:03:50.860
acquisition systems.

00:03:50.860 --> 00:03:51.793
In this case,

00:03:51.793 --> 00:03:53.970
space or iris cameras and also provide

00:03:54.036 --> 00:03:56.478
actionable and useful feedback to industry

00:03:56.478 --> 00:03:58.590
to make technologies work better.

00:03:58.590 --> 00:03:59.457
And with that,

00:03:59.457 --> 00:04:01.884
I will kick it over to Doctor Yevgeny

00:04:01.884 --> 00:04:04.187
Strong can go to the next slide.

00:04:08.980 --> 00:04:09.766
Good morning, everyone.

00:04:09.766 --> 00:04:11.076
Thank you for joining us.

00:04:11.080 --> 00:04:14.020
Thank you, Arun for the introduction.

00:04:14.020 --> 00:04:16.975
The 2022, the 2021 Biometric

00:04:16.975 --> 00:04:19.126
Technology Rally, the mark,

00:04:19.126 --> 00:04:22.024
the 4th Biometric Technology Rally and

00:04:22.024 --> 00:04:25.008
this year in 2022 we're running the

00:04:25.008 --> 00:04:27.660
5th Anniversary Rally and since 2018

00:04:27.735 --> 00:04:30.355
the rallies have demonstrated progress

00:04:30.355 --> 00:04:33.628
in the performance and maturity of

00:04:33.628 --> 00:04:36.498
biometric acquisition and matching systems.

00:04:36.500 --> 00:04:38.446
The results of these rallies have provided

00:04:38.446 --> 00:04:40.070
insights into how people interact.

00:04:40.070 --> 00:04:41.718
With biometric technologies and

00:04:41.718 --> 00:04:44.190
have led to improvements in system

00:04:44.256 --> 00:04:45.879
usability and performance.

00:04:45.880 --> 00:04:46.270
Importantly,

00:04:46.270 --> 00:04:49.000
the rallies today have focused on one

00:04:49.000 --> 00:04:51.827
specific use case and that's high throughput,

00:04:51.830 --> 00:04:53.674
unstaffed biometric acquisition and

00:04:53.674 --> 00:04:56.917
we continue to see new challenges to

00:04:56.917 --> 00:04:59.377
system performance within this use case,

00:04:59.380 --> 00:05:01.310
most recently with the introduction

00:05:01.310 --> 00:05:03.778
of face masks which make biometric

00:05:03.778 --> 00:05:05.634
acquisition more difficult even

00:05:05.634 --> 00:05:08.494
when people remove them during the

00:05:08.494 --> 00:05:09.440
biometric process.

00:05:09.440 --> 00:05:10.080
Next slide.

00:05:10.080 --> 00:05:10.400
Please.

00:05:12.530 --> 00:05:13.770
So in the following slides,

00:05:13.770 --> 00:05:15.685
I'll introduce the 2021 Biometric

00:05:15.685 --> 00:05:18.320
Technology rally and go over the major

00:05:18.320 --> 00:05:20.270
parts of the rally test process.

00:05:20.270 --> 00:05:23.960
Next slide please. As mentioned.

00:05:23.960 --> 00:05:27.072
Earlier, the 2021 rally was a test of

00:05:27.072 --> 00:05:29.689
biometric systems within a high throughput,

00:05:29.690 --> 00:05:30.917
unattended use case.

00:05:30.917 --> 00:05:32.962
This test focused on the

00:05:32.962 --> 00:05:34.470
face biometric modality,

00:05:34.470 --> 00:05:37.032
but it did accept applications from

00:05:37.032 --> 00:05:39.130
multimodal biometric systems as well.

00:05:39.130 --> 00:05:41.930
We designed this rally so that companies

00:05:41.930 --> 00:05:43.910
could demonstrate performance of their

00:05:43.910 --> 00:05:45.705
systems with respect to efficiency,

00:05:45.710 --> 00:05:47.794
effectiveness and user satisfaction

00:05:47.794 --> 00:05:51.740
and what we mean by efficiency is the

00:05:51.740 --> 00:05:53.930
ability of acquisition systems to work.

00:05:53.930 --> 00:05:55.950
Quickly and effectiveness with respect

00:05:55.950 --> 00:05:58.987
to the error rates that these systems

00:05:58.987 --> 00:06:01.362
experience and finally satisfaction is

00:06:01.362 --> 00:06:04.099
people's opinions of these technologies.

00:06:04.100 --> 00:06:05.255
Our test approach,

00:06:05.255 --> 00:06:06.795
which I'll describe later,

00:06:06.800 --> 00:06:09.880
allowed for assessment of interoperability

00:06:09.880 --> 00:06:12.960
between acquisition and matching systems.

00:06:12.960 --> 00:06:15.410
And as the second rally

00:06:15.410 --> 00:06:16.880
performed during COVID-19,

00:06:16.880 --> 00:06:19.505
we also wanted to capture the improvement

00:06:19.505 --> 00:06:21.433
in system performance for individuals

00:06:21.433 --> 00:06:24.023
wearing face masks compared to the 2020.

00:06:24.030 --> 00:06:25.500
rally,

00:06:25.500 --> 00:06:28.056
from the very beginning the rallies

00:06:28.056 --> 00:06:29.334
assessed biometric technology

00:06:29.334 --> 00:06:30.621
performance with demographically

00:06:30.621 --> 00:06:33.039
diverse groups of people to address

00:06:33.039 --> 00:06:34.811
questions raised about technology

00:06:34.811 --> 00:06:36.615
performance for specific groups.

00:06:36.620 --> 00:06:39.483
We designed the 2021 rally to allow

00:06:39.483 --> 00:06:42.068
reporting of metrics in a manner

00:06:42.068 --> 00:06:43.780
that's dis-aggregated by race,

00:06:43.780 --> 00:06:44.167
gender,

00:06:44.167 --> 00:06:47.263
and skin tone to measure how well systems

00:06:47.263 --> 00:06:49.619
perform for each group individually.

00:06:49.620 --> 00:06:50.292
Next slide,

00:06:50.292 --> 00:06:50.628
please.

00:06:52.880 --> 00:06:55.776
So in order to qualify for the rally,

00:06:55.780 --> 00:06:58.126
biometric systems had to meet a

00:06:58.126 --> 00:06:59.690
number of technical requirements

00:06:59.759 --> 00:07:01.835
which are presented on this slide.

00:07:01.840 --> 00:07:03.334
For acquisition systems,

00:07:03.334 --> 00:07:05.326
the following minimum requirements

00:07:05.326 --> 00:07:07.867
held each system had to operate

00:07:07.867 --> 00:07:10.016
in an unmanned mode, that is,

00:07:10.016 --> 00:07:12.106
without any operator or instructor

00:07:12.106 --> 00:07:14.228
to tell people what to do.

00:07:14.230 --> 00:07:15.394
Folks were instructed by

00:07:15.394 --> 00:07:17.140
the test team to come in,

00:07:17.140 --> 00:07:19.822
come and try to use each system according to

00:07:19.822 --> 00:07:22.037
whatever instructions the system provided.

00:07:22.040 --> 00:07:24.015
All the systems operated within

00:07:24.015 --> 00:07:26.993
a 6 by 8 footprint where you

00:07:26.993 --> 00:07:29.288
know the system provider could

00:07:29.288 --> 00:07:32.008
deploy the system as they saw fit.

00:07:32.010 --> 00:07:33.970
This all systems were required

00:07:33.970 --> 00:07:35.930
to collect face biometric imagery

00:07:35.930 --> 00:07:40.382
and to provide one biometric probe

00:07:40.382 --> 00:07:42.608
per test volunteer.

00:07:42.610 --> 00:07:44.914
The probes had to be submitted

00:07:44.914 --> 00:07:46.940
within a particular time limit.

00:07:46.940 --> 00:07:47.676
That is,

00:07:47.676 --> 00:07:49.516
during the interaction between the

00:07:49.516 --> 00:07:51.475
volunteer and the biometric system

00:07:51.475 --> 00:07:53.455
before they left the station.

00:07:53.460 --> 00:07:55.610
We also had some functionality

00:07:55.610 --> 00:07:57.760
requirements with respect to masks.

00:07:57.760 --> 00:08:00.616
These systems were required to acquire

00:08:00.616 --> 00:08:03.400
images from people wearing face masks,

00:08:03.400 --> 00:08:04.588
so they have to they have

00:08:04.588 --> 00:08:06.100
to be able to achieve that.

00:08:06.100 --> 00:08:09.320
And as an optional requirement,

00:08:09.320 --> 00:08:11.030
systems could provide images from

00:08:11.030 --> 00:08:12.398
iris or fingerprint modality.

00:08:12.400 --> 00:08:13.808
And as you'll see,

00:08:13.808 --> 00:08:15.920
we had one multimodal face Iris

00:08:15.999 --> 00:08:17.407
system in this test.

00:08:17.410 --> 00:08:18.770
The requirements for matching

00:08:18.770 --> 00:08:20.470
systems are on the right.

00:08:20.470 --> 00:08:22.195
These matching systems were provided

00:08:22.195 --> 00:08:24.650
as sort of black box algorithms.

00:08:24.650 --> 00:08:27.686
They were packaged inside a docker

00:08:27.686 --> 00:08:30.278
container which was deployed for

00:08:30.278 --> 00:08:32.688
evaluation on our local cluster.

00:08:32.690 --> 00:08:34.760
These systems have to be commercially

00:08:34.760 --> 00:08:35.105
available.

00:08:35.110 --> 00:08:35.370
Actually,

00:08:35.370 --> 00:08:37.450
that was the same as for acquisition systems,

00:08:37.450 --> 00:08:38.662
where we're encouraged.

00:08:38.662 --> 00:08:41.490
This is a test of commercial technology.

00:08:41.490 --> 00:08:43.250
There was some limitations

00:08:43.250 --> 00:08:45.890
regarding the size of those images,

00:08:45.890 --> 00:08:48.320
1.5 gigabytes, maximum in size.

00:08:48.320 --> 00:08:51.032
And they had to be performing

00:08:51.032 --> 00:08:52.840
the biometric match operations

00:08:52.918 --> 00:08:54.350
in a timely manner.

00:08:54.350 --> 00:08:54.954
Work critically,

00:08:54.954 --> 00:08:56.464
they have to work without

00:08:56.464 --> 00:08:57.870
access to external networks,

00:08:57.870 --> 00:09:01.335
so these were not we would not

00:09:01.335 --> 00:09:04.407
consider Web hosted API because images

00:09:04.407 --> 00:09:08.370
gathered at the MDTF stay at the MDTF.

00:09:08.370 --> 00:09:11.238
We had a specific swagger specification

00:09:11.240 --> 00:09:13.536
which we asked these systems to conform to,

00:09:13.540 --> 00:09:17.340
and that's hosted on our

00:09:17.340 --> 00:09:18.461
website github.mdtf.org.

00:09:18.461 --> 00:09:20.627
And of course the matching systems

00:09:20.627 --> 00:09:22.402
were also like the acquisition

00:09:22.402 --> 00:09:23.987
systems required to be able

00:09:23.987 --> 00:09:25.600
to perform all operations,

00:09:25.600 --> 00:09:27.628
including probe including template

00:09:27.628 --> 00:09:29.656
extraction and matching on

00:09:29.656 --> 00:09:31.998
probe images that are required

00:09:31.998 --> 00:09:34.093
with people wearing face masks.

00:09:34.100 --> 00:09:36.130
The matching was performed with

00:09:36.130 --> 00:09:38.160
face Mask matching was performed

00:09:38.160 --> 00:09:39.468
based on probe images.

00:09:39.468 --> 00:09:41.103
required with face masks relative

00:09:41.103 --> 00:09:43.038
to reference images which were

00:09:43.038 --> 00:09:44.590
acquired without face masks.

00:09:44.590 --> 00:09:45.698
So next slide please.

00:09:48.200 --> 00:09:49.785
So all rally participants were

00:09:49.785 --> 00:09:51.862
evaluated by a panel of experts

00:09:51.862 --> 00:09:53.598
from government and industry,

00:09:53.600 --> 00:09:55.532
and there are a number of government

00:09:55.532 --> 00:09:56.084
organizations represented.

00:09:56.090 --> 00:09:58.400
You could see that on this slide.

00:09:58.400 --> 00:10:01.334
In the end, 5 acquisition systems

00:10:01.334 --> 00:10:03.720
and 13 matching systems were

00:10:03.720 --> 00:10:05.620
selected to participate in total

00:10:05.620 --> 00:10:07.520
of the five acquisition systems,

00:10:07.520 --> 00:10:10.369
one was a multi-modal face Iris acquisition

00:10:10.369 --> 00:10:12.978
system and three of the matching

00:10:12.978 --> 00:10:15.173
systems were iris matching systems.

00:10:15.180 --> 00:10:17.368
The performance of acquisition.

00:10:17.368 --> 00:10:20.650
And matching for Iris was performed

00:10:20.738 --> 00:10:23.188
separately than that for face.

00:10:23.190 --> 00:10:25.010
We did not attempt to do any

00:10:25.010 --> 00:10:26.845
fusion or any joint multi-modal

00:10:26.845 --> 00:10:28.825
performance in this assessment,

00:10:28.830 --> 00:10:30.126
and I'll report on.

00:10:30.126 --> 00:10:31.746
We'll report on the results

00:10:31.746 --> 00:10:32.827
for Iris separately,

00:10:32.827 --> 00:10:35.886
each system was given a unique alias,

00:10:35.890 --> 00:10:38.669
which in this for this evaluation was

00:10:38.669 --> 00:10:41.549
inspired by US rivers and mountains.

00:10:41.550 --> 00:10:42.558
Next slide please.

00:10:45.120 --> 00:10:47.232
So the 2021 rally was announced

00:10:47.232 --> 00:10:49.542
in May of 2021 and the technology

00:10:49.542 --> 00:10:51.107
providers had two months to

00:10:51.107 --> 00:10:53.123
submit applications to participate

00:10:53.123 --> 00:10:54.770
following conditional acceptance,

00:10:54.770 --> 00:10:56.280
technology providers had two months

00:10:56.280 --> 00:10:58.270
to develop their systems for the test,

00:10:58.270 --> 00:11:00.162
which included integration with

00:11:00.162 --> 00:11:02.618
cloud hosted MDTF API's prior to

00:11:02.618 --> 00:11:04.880
arrival at the test facility and

00:11:04.955 --> 00:11:06.839
this significantly decreases the

00:11:06.839 --> 00:11:09.665
number of issues that we observe

00:11:09.670 --> 00:11:11.794
during the test process prior to

00:11:11.794 --> 00:11:13.210
testing all acquisition system

00:11:13.275 --> 00:11:15.187
providers installed their systems.

00:11:15.190 --> 00:11:18.345
ATF and participated in a

00:11:18.345 --> 00:11:20.869
VIP day with stakeholders.

00:11:20.870 --> 00:11:22.930
The acquisition system testing was

00:11:22.930 --> 00:11:24.578
then performed between September

00:11:24.578 --> 00:11:27.150
29th and October 15th of 2021.

00:11:27.150 --> 00:11:28.620
Next slide please.

00:11:30.710 --> 00:11:32.790
So this image helps visualize

00:11:32.790 --> 00:11:34.870
the overall rally test process,

00:11:34.870 --> 00:11:38.890
which starts with a box #1 on the very left,

00:11:38.890 --> 00:11:40.794
which is informed consent.

00:11:40.794 --> 00:11:43.860
In total, we had 601 diverse people

00:11:43.860 --> 00:11:46.498
from the local area briefed about

00:11:46.498 --> 00:11:49.174
the rally and who consented to

00:11:49.174 --> 00:11:51.470
participate in this evaluation.

00:11:51.470 --> 00:11:53.540
And we could go to box

00:11:53.540 --> 00:11:54.920
#2 for each participant.

00:11:54.920 --> 00:11:57.575
We acquired ground truth information

00:11:57.575 --> 00:11:59.699
regarding their self reported

00:11:59.699 --> 00:12:01.352
demographic characteristics including

00:12:01.352 --> 00:12:03.977
their gender and their race,

00:12:03.980 --> 00:12:05.600
as well as you know,

00:12:05.600 --> 00:12:08.738
ground truth measures of their phenotypes,

00:12:08.740 --> 00:12:09.950
which in this case included

00:12:09.950 --> 00:12:11.160
our assessment of skin tone.

00:12:11.160 --> 00:12:12.408
And I'll have a little bit

00:12:12.408 --> 00:12:14.111
more to say on that later.

00:12:14.111 --> 00:12:17.777
The reason why we included this

00:12:17.780 --> 00:12:19.748
phenotype in our assessment is that

00:12:19.748 --> 00:12:21.679
our research shows that skin tone.

00:12:21.680 --> 00:12:23.740
Is related to face recognition

00:12:23.740 --> 00:12:26.315
performance and skin tone values are

00:12:26.315 --> 00:12:28.540
measured using a specialized color

00:12:28.540 --> 00:12:30.650
colorimeters and these values are,

00:12:30.650 --> 00:12:31.550
you know,

00:12:31.550 --> 00:12:34.250
are uniform and and can translate

00:12:34.250 --> 00:12:36.325
across different definitions of

00:12:36.325 --> 00:12:39.065
demographic categories that may exist

00:12:39.065 --> 00:12:41.759
in different regions of the world.

00:12:41.760 --> 00:12:43.744
So on point #3,

00:12:43.744 --> 00:12:45.728
we performed acquisition first

00:12:45.728 --> 00:12:49.071
without any face masks and so you

00:12:49.071 --> 00:12:51.680
can see here what's represented in,

00:12:51.680 --> 00:12:52.084
in, in,

00:12:52.084 --> 00:12:53.700
in the third box is we had a

00:12:53.756 --> 00:12:55.440
number of acquisition systems,

00:12:55.440 --> 00:12:56.584
in this case five.

00:12:56.584 --> 00:12:58.718
Each volunteer used each one of the

00:12:58.718 --> 00:13:00.493
systems in a counterbalanced order

00:13:00.493 --> 00:13:02.213
and all the acquisition systems

00:13:02.213 --> 00:13:04.373
have to do is return to us a probe

00:13:04.380 --> 00:13:06.500
face image from each volunteer.

00:13:06.500 --> 00:13:08.260
First we did that without

00:13:08.260 --> 00:13:10.320
masks and then in Box #4,

00:13:10.320 --> 00:13:12.336
we asked volunteers to keep their masks.

00:13:12.340 --> 00:13:14.296
On while using the acquisition systems

00:13:14.296 --> 00:13:16.993
and then we tested the ability of these

00:13:16.993 --> 00:13:18.985
systems to perform the same function,

00:13:18.990 --> 00:13:21.966
now with masks for each individual.

00:13:21.970 --> 00:13:23.640
The results these these probe

00:13:23.640 --> 00:13:25.932
images were sent to the matching

00:13:25.932 --> 00:13:27.588
system evaluation portion,

00:13:27.590 --> 00:13:29.949
which is box #5 where the matching

00:13:29.949 --> 00:13:32.722
systems had to find the face in each

00:13:32.722 --> 00:13:35.201
photo and compare this face to known

00:13:35.201 --> 00:13:38.065
people to identify the person in the photo.

00:13:38.070 --> 00:13:39.534
In this case,

00:13:39.534 --> 00:13:42.950
we had a gallery of 500 individuals

00:13:42.950 --> 00:13:45.645
and a total of these ten face

00:13:45.645 --> 00:13:47.333
matching systems were evaluated

00:13:47.333 --> 00:13:49.920
this way and the results the

00:13:49.920 --> 00:13:51.600
identification results were then.

00:13:51.600 --> 00:13:54.505
Combined and the way we're going to

00:13:54.505 --> 00:13:56.163
report performance throughout this

00:13:56.163 --> 00:13:58.578
test box #6 is by combinations of

00:13:58.578 --> 00:14:00.419
acquisition and matching systems.

00:14:00.420 --> 00:14:02.256
So we had five camera systems,

00:14:02.260 --> 00:14:04.936
5 acquisition systems in the evaluation,

00:14:04.940 --> 00:14:07.000
10 matching algorithms and the

00:14:07.000 --> 00:14:09.721
evaluation that's a total of 50

00:14:09.721 --> 00:14:12.069
different possible system combinations.

00:14:12.070 --> 00:14:13.726
That that we're gonna take a look at.

00:14:13.730 --> 00:14:15.538
And the reason why we do it this

00:14:15.538 --> 00:14:17.463
way is to be able to understand

00:14:17.463 --> 00:14:19.478
the impact of camera and algorithm

00:14:19.478 --> 00:14:21.610
separately on biometric performance.

00:14:21.610 --> 00:14:23.830
And you'll see those results.

00:14:23.830 --> 00:14:24.978
So next slide please.

00:14:27.460 --> 00:14:29.574
So at this point I'm going to

00:14:29.574 --> 00:14:31.540
turn things over to John Howard,

00:14:31.540 --> 00:14:33.832
who's going to kick off the

00:14:33.832 --> 00:14:35.360
explaining the rally results.

00:14:38.970 --> 00:14:40.570
Great. Thanks you. Kenny.

00:14:40.570 --> 00:14:42.190
Yes. So now comes, you know,

00:14:42.190 --> 00:14:43.730
sort of the fun part.

00:14:43.730 --> 00:14:46.054
What did we learn about all these

00:14:46.054 --> 00:14:47.866
systems from executing that process

00:14:47.866 --> 00:14:49.724
that Annie just described first,

00:14:49.724 --> 00:14:52.202
just a couple of high levels of

00:14:52.202 --> 00:14:54.629
framing points to keep in mind as I

00:14:54.629 --> 00:14:56.450
sort of move through these results.

00:14:56.450 --> 00:14:58.610
Some of these evaluation metrics just

00:14:58.610 --> 00:15:01.258
apply to acquisition systems. So namely,

00:15:01.258 --> 00:15:04.018
you were talking efficiency and satisfaction.

00:15:04.018 --> 00:15:08.050
And we also only evaluated those metrics.

00:15:08.050 --> 00:15:10.042
Volunteers weren't wearing masks.

00:15:10.042 --> 00:15:12.105
You know, I recognize some of you

00:15:12.105 --> 00:15:13.749
might think it would be a very

00:15:13.749 --> 00:15:15.340
interesting data point to compare, like,

00:15:15.340 --> 00:15:16.640
satisfaction across some people were

00:15:16.640 --> 00:15:19.410
wearing masks and they weren't, but.

00:15:19.410 --> 00:15:21.674
Sort of bound the analysis in some way,

00:15:21.680 --> 00:15:23.710
and that was one choice we sort

00:15:23.710 --> 00:15:25.599
of made was not to report.

00:15:25.600 --> 00:15:28.000
Speed and satisfaction with

00:15:28.000 --> 00:15:29.800
and without masks.

00:15:29.800 --> 00:15:30.930
Effectiveness, on the other hand,

00:15:30.930 --> 00:15:32.316
is not just an acquisition metric.

00:15:32.320 --> 00:15:34.210
This is something we're trying to highlight.

00:15:34.210 --> 00:15:35.474
Last couple of years,

00:15:35.474 --> 00:15:37.054
effectiveness is so dependent on

00:15:37.054 --> 00:15:38.909
a combination of an acquisition

00:15:38.909 --> 00:15:40.759
system and a matching system.

00:15:40.760 --> 00:15:43.088
And so the unique combination of those things

00:15:43.088 --> 00:15:45.319
will have unique effectiveness measures.

00:15:45.320 --> 00:15:46.850
And then also for effectiveness,

00:15:46.850 --> 00:15:48.668
since it was a focus of this year's rally,

00:15:48.670 --> 00:15:51.008
we did consider and volunteers were wearing

00:15:51.008 --> 00:15:53.338
masks and they weren't wearing masks.

00:15:53.340 --> 00:15:55.104
And then we've also continued to

00:15:55.104 --> 00:15:56.280
report effectiveness out sort

00:15:56.329 --> 00:15:57.799
of in these two different ways.

00:15:57.800 --> 00:16:00.836
The first people call operationally focused.

00:16:00.840 --> 00:16:03.156
This includes all sources of air

00:16:03.156 --> 00:16:05.728
that can sort of happen with the

00:16:05.728 --> 00:16:06.898
acquisition or the matching system.

00:16:06.900 --> 00:16:08.760
So that's maybe the acquisition system

00:16:08.760 --> 00:16:10.988
than sending an image with the matching.

00:16:10.990 --> 00:16:13.180
Process it or maybe did processing

00:16:13.180 --> 00:16:15.635
it didn't match back to that

00:16:15.635 --> 00:16:17.027
subjects historical images.

00:16:17.030 --> 00:16:18.926
We think that operational focus within

00:16:18.926 --> 00:16:21.560
its name is a little bit more relevant

00:16:21.560 --> 00:16:25.810
to what improved like HSBC and airport.

00:16:25.810 --> 00:16:28.978
But then we also report on matching focus

00:16:28.978 --> 00:16:30.820
effectiveness which were discounts.

00:16:30.820 --> 00:16:33.439
Yeah, there's nobody.

00:16:33.440 --> 00:16:35.155
An acquisition system to send

00:16:35.155 --> 00:16:36.360
them the matching systems can't

00:16:36.360 --> 00:16:37.435
really affect that at all,

00:16:37.440 --> 00:16:39.636
and so that gives us a better picture of

00:16:39.636 --> 00:16:41.997
how matching systems complete each other.

00:16:42.000 --> 00:16:43.089
And then lastly,

00:16:43.089 --> 00:16:45.267
how we report on demographic differentials?

00:16:45.270 --> 00:16:45.982
This is.

00:16:45.982 --> 00:16:47.406
Currently a hot topic.

00:16:47.410 --> 00:16:49.480
I think in the biometric world

00:16:49.480 --> 00:16:51.498
and the different groups taking

00:16:51.498 --> 00:16:53.622
different approaches and it sort of

00:16:53.622 --> 00:16:55.110
answers the question of how should

00:16:55.165 --> 00:16:56.936
we take the performance of Group A.

00:16:56.940 --> 00:16:58.340
And the performance of Group B and

00:16:58.340 --> 00:17:00.172
say you know, are they the same?

00:17:00.172 --> 00:17:01.096
Are they different?

00:17:01.100 --> 00:17:02.960
We did this in a way that we think makes

00:17:03.013 --> 00:17:05.020
the most sense in the context of the rally,

00:17:05.020 --> 00:17:07.258
which is to compare those two

00:17:07.258 --> 00:17:09.561
performance values to the goal and

00:17:09.561 --> 00:17:12.144
threshold values that we've had now for.

00:17:12.150 --> 00:17:16.148
Years I've got a little bit more on this

00:17:16.148 --> 00:17:18.267
towards the end about why we did this way,

00:17:18.270 --> 00:17:19.460
you know, some good properties,

00:17:19.460 --> 00:17:21.112
that's kind of conceptually

00:17:21.112 --> 00:17:21.938
and mathematically,

00:17:21.940 --> 00:17:24.796
but also some of the approaches

00:17:24.796 --> 00:17:25.748
of considered.

00:17:25.750 --> 00:17:25.966
OK,

00:17:25.966 --> 00:17:26.398
Next up.

00:17:28.880 --> 00:17:29.932
This is like 14.

00:17:29.932 --> 00:17:30.984
Moving the actual results.

00:17:30.990 --> 00:17:32.770
The first efficiency or speed.

00:17:32.770 --> 00:17:34.666
This is the average transaction time.

00:17:34.670 --> 00:17:37.130
Again per acquisition system.

00:17:37.130 --> 00:17:40.126
Higher on the bottom on the

00:17:40.126 --> 00:17:42.026
right here was worse. Faster.

00:17:42.026 --> 00:17:44.954
The goal for this year is 4 seconds.

00:17:44.960 --> 00:17:47.368
You can see that with the solid

00:17:47.368 --> 00:17:49.260
red line towards the bottom.

00:17:49.260 --> 00:17:50.548
And then you can see that three

00:17:50.548 --> 00:17:51.700
out of the five acquisition

00:17:51.700 --> 00:17:53.080
systems and that that goal,

00:17:53.080 --> 00:17:56.668
the fastest being acquisition system longer.

00:17:56.670 --> 00:18:01.374
******* just under 3 seconds per volunteer.

00:18:01.380 --> 00:18:03.066
And then the system granted and.

00:18:05.420 --> 00:18:07.400
We're sorry, system parading system away.

00:18:07.400 --> 00:18:10.870
Also make that 4 second goal and then

00:18:10.870 --> 00:18:12.830
we also have a threshold value 8 seconds

00:18:12.880 --> 00:18:14.680
which is some grand and that kind of

00:18:14.680 --> 00:18:17.095
see that with the solid and halfway

00:18:17.095 --> 00:18:19.110
filled in circles at the bottom.

00:18:21.200 --> 00:18:21.630
Next slide.

00:18:24.990 --> 00:18:25.814
Satisfaction for.

00:18:25.814 --> 00:18:27.874
Looked a little bit higher

00:18:27.874 --> 00:18:29.900
on this slide is better.

00:18:29.900 --> 00:18:30.916
So we want to be towards the top.

00:18:30.920 --> 00:18:32.560
You want to have lots of people that

00:18:32.560 --> 00:18:33.708
are satisfied with your system.

00:18:33.710 --> 00:18:35.740
This is measured by the percentage of

00:18:35.740 --> 00:18:38.194
people that pick either happy or the

00:18:38.194 --> 00:18:41.110
very happy option after me after using.

00:18:41.110 --> 00:18:42.545
The acquisition system again here

00:18:42.545 --> 00:18:44.440
you can see really good results.

00:18:44.440 --> 00:18:47.250
Almost all the systems fortified

00:18:47.250 --> 00:18:50.108
met the goal, which is 95%.

00:18:50.108 --> 00:18:52.736
That's solid bar at the top.

00:18:52.740 --> 00:18:55.116
That was the large long grain of them,

00:18:55.120 --> 00:18:56.924
48 at the highest,

00:18:56.924 --> 00:19:00.689
being granite at 97.8% of all.

00:19:00.690 --> 00:19:03.335
They're happier. They're very happy

00:19:03.335 --> 00:19:05.980
after interacting with the system.

00:19:05.980 --> 00:19:06.520
Next slide.

00:19:11.780 --> 00:19:13.720
Oops, yeah, there we go.

00:19:13.720 --> 00:19:14.780
So moving into effectiveness,

00:19:14.780 --> 00:19:16.370
this is where we're going to

00:19:16.417 --> 00:19:17.880
start to see a few more numbers.

00:19:17.880 --> 00:19:19.314
And that's because of what I

00:19:19.314 --> 00:19:20.560
mentioned a few slides back.

00:19:20.560 --> 00:19:22.440
We're not taking into account.

00:19:22.440 --> 00:19:23.529
Combination of matching

00:19:23.529 --> 00:19:24.618
and acquisition system.

00:19:24.620 --> 00:19:27.500
We also have the mask and no mask

00:19:27.500 --> 00:19:30.123
convention and we also have the

00:19:30.123 --> 00:19:32.865
operational versus matching system focus so.

00:19:32.870 --> 00:19:34.389
I'm going to go through these one

00:19:34.389 --> 00:19:36.150
at a time to explain the chart.

00:19:36.150 --> 00:19:38.376
I'll mention that all of these,

00:19:38.380 --> 00:19:39.280
including the last two,

00:19:39.280 --> 00:19:40.850
are at the bucket over right now.

00:19:40.850 --> 00:19:42.474
So if you don't quite get every

00:19:42.474 --> 00:19:43.830
number and get every trend,

00:19:43.830 --> 00:19:45.540
don't worry, you can go.

00:19:45.540 --> 00:19:47.682
Back to the site and sort of

00:19:47.682 --> 00:19:49.338
peruse these at your convenience.

00:19:49.338 --> 00:19:52.818
So how to read this is going down the road.

00:19:52.820 --> 00:19:55.424
You'll find the 10 face matching

00:19:55.424 --> 00:19:57.891
systems all aliases just getting

00:19:57.891 --> 00:19:59.920
mentioned going across the columns,

00:19:59.920 --> 00:20:01.645
you see the five acquisition

00:20:01.645 --> 00:20:03.266
systems and then the cell you

00:20:03.266 --> 00:20:04.940
have the value for the metric.

00:20:04.940 --> 00:20:08.279
And on this chart that metric is

00:20:08.279 --> 00:20:10.208
operationally focused trade notification

00:20:10.208 --> 00:20:13.512
rate with no masks as you see

00:20:13.512 --> 00:20:16.477
sort of on the left hand side there.

00:20:16.480 --> 00:20:18.937
And so that operational folks needs including

00:20:18.937 --> 00:20:21.129
things like that to submit an image.

00:20:21.130 --> 00:20:23.097
Then we added a little color coding

00:20:23.097 --> 00:20:25.374
to make this easier on the eyes and

00:20:25.374 --> 00:20:27.584
we added a little marker which is

00:20:27.584 --> 00:20:30.503
a diamond to sort of highlight the

00:20:30.503 --> 00:20:32.970
highest metric in each category.

00:20:32.970 --> 00:20:37.228
For this you can see the highest was 99.7%,

00:20:37.228 --> 00:20:37.866
opportunity,

00:20:37.866 --> 00:20:39.142
focus, trade,

00:20:39.142 --> 00:20:41.623
unification rate that was achieved

00:20:41.623 --> 00:20:43.227
by two matching systems,

00:20:43.230 --> 00:20:44.313
sun and paint,

00:20:44.313 --> 00:20:46.479
both with the images that were

00:20:46.479 --> 00:20:48.008
required of acquisition system

00:20:48.008 --> 00:20:50.192
for our stuff in the paper.

00:20:53.970 --> 00:20:54.600
Next slide.

00:20:59.860 --> 00:21:03.136
OK, so same general layout here.

00:21:03.140 --> 00:21:05.558
We're just changing the metric so.

00:21:05.560 --> 00:21:06.550
We're going to stay without

00:21:06.550 --> 00:21:07.969
masks as it says on the left,

00:21:07.970 --> 00:21:09.839
but now we're essentially what we're doing

00:21:09.839 --> 00:21:11.631
is ignoring an acquisition assistance,

00:21:11.631 --> 00:21:13.716
failure to submit an image.

00:21:13.720 --> 00:21:14.650
So the first thing I know,

00:21:14.650 --> 00:21:17.306
this is a lot more green and several

00:21:17.306 --> 00:21:19.936
more diamond symbols for this metric,

00:21:19.936 --> 00:21:22.428
the highest tier is actually 100%

00:21:22.428 --> 00:21:25.030
and that was achieved by a couple

00:21:25.030 --> 00:21:26.430
different system combinations systems.

00:21:26.430 --> 00:21:28.510
So that with three acquisition

00:21:28.510 --> 00:21:30.590
systems system painted with two

00:21:30.658 --> 00:21:32.940
and system design did it with one.

00:21:32.940 --> 00:21:35.310
So that means those matches.

00:21:35.310 --> 00:21:38.016
System we're able to correctly identify

00:21:38.020 --> 00:21:41.500
100% of the actual images that came from.

00:21:41.500 --> 00:21:43.380
Are those particular acquisition systems

00:21:43.380 --> 00:21:44.310
So we thought that was.

00:21:47.030 --> 00:21:47.690
Next slide.

00:21:55.550 --> 00:21:58.406
OK, so now I'm putting masks on.

00:21:58.410 --> 00:22:00.480
You'll probably need to notice

00:22:00.480 --> 00:22:02.550
there's no green on this chart.

00:22:02.550 --> 00:22:04.966
I'll also point out that this chart goes

00:22:04.966 --> 00:22:07.097
back again to including acquisition error.

00:22:07.097 --> 00:22:09.791
So the presence of masks made

00:22:09.791 --> 00:22:12.240
this unattended high throughput.

00:22:12.240 --> 00:22:14.450
Scenario more difficult both on

00:22:14.450 --> 00:22:17.131
the actors and matching side to

00:22:17.131 --> 00:22:19.513
the point that no rally system

00:22:19.513 --> 00:22:21.450
combination met that objective

00:22:21.450 --> 00:22:23.932
threshold objective we had set at 95%.

00:22:23.932 --> 00:22:26.060
So our highest performer.

00:22:26.060 --> 00:22:26.772
with mask with presence,

00:22:26.772 --> 00:22:28.329
I think it's right in the middle there.

00:22:28.330 --> 00:22:30.450
It's system Mazon along with

00:22:30.450 --> 00:22:32.570
acquisition system wrong and he

00:22:32.648 --> 00:22:35.256
was able to identify just 93%.

00:22:35.256 --> 00:22:37.526
Of the volunteers that transition,

00:22:37.530 --> 00:22:40.678
that system with masks.

00:22:43.870 --> 00:22:44.500
The next slide.

00:22:52.760 --> 00:22:55.100
OK. And then finally with masks,

00:22:55.100 --> 00:22:58.736
but now ignoring the failures of the

00:22:58.736 --> 00:23:00.500
acquisition system to send an image,

00:23:00.500 --> 00:23:02.020
we get the following,

00:23:02.020 --> 00:23:03.920
so more green than before,

00:23:03.920 --> 00:23:05.816
but certainly less than on that chart 2.

00:23:05.820 --> 00:23:07.648
And there's also no.

00:23:07.648 --> 00:23:09.476
Multiple instances of that

00:23:09.476 --> 00:23:10.878
system combination reaching

00:23:10.878 --> 00:23:12.824
that 100% performance mark,

00:23:12.824 --> 00:23:16.166
our best was assistant paint assistant

00:23:16.166 --> 00:23:18.774
period 99.6% of subjects have

00:23:18.774 --> 00:23:21.329
submitted images with masks identified.

00:23:21.330 --> 00:23:24.350
So still pretty good.

00:23:24.350 --> 00:23:25.946
But not as good as those 100%

00:23:25.950 --> 00:23:27.324
marks that were sort of across

00:23:27.324 --> 00:23:28.380
the Board on Chart 2.

00:23:31.170 --> 00:23:34.880
OK, you can go to next slide.

00:23:34.880 --> 00:23:36.984
So those are sort of the wrong results.

00:23:36.990 --> 00:23:38.313
As I mentioned, those are on the

00:23:38.313 --> 00:23:39.580
website the next couple of slides,

00:23:39.580 --> 00:23:41.015
including some of the things that are

00:23:41.015 --> 00:23:42.718
present sort of move off those raw results.

00:23:42.720 --> 00:23:44.435
And so try to provide some insights,

00:23:44.440 --> 00:23:45.910
some of which may have been obvious

00:23:45.910 --> 00:23:47.792
if you were paying attention as I ran

00:23:47.792 --> 00:23:49.360
through those last couple of charts,

00:23:49.360 --> 00:23:50.697
these actually aren't up on the website,

00:23:50.700 --> 00:23:52.510
but we will put the slides up.

00:23:52.510 --> 00:23:53.710
A little bit later this week,

00:23:53.710 --> 00:23:56.438
so you can go back and reference them.

00:23:56.440 --> 00:23:59.770
So number one, what this is showing is that.

00:23:59.770 --> 00:24:01.798
We think Image acquisition is really

00:24:01.798 --> 00:24:04.106
driving the error rates and the biometric

00:24:04.106 --> 00:24:06.230
systems as we test them today, so.

00:24:06.230 --> 00:24:08.790
How we read this chart on the right

00:24:08.790 --> 00:24:11.719
is that across the bottom X axis is

00:24:11.719 --> 00:24:13.700
sort of the percent of errors that

00:24:13.770 --> 00:24:16.086
were acquisition at those to submit.

00:24:16.090 --> 00:24:18.211
Across the Y axis you have the

00:24:18.211 --> 00:24:20.528
percent of the areas that were

00:24:20.528 --> 00:24:22.308
matching theirs subsequently failed

00:24:22.308 --> 00:24:24.448
to extract one match sample,

00:24:24.448 --> 00:24:27.234
and so if you are below this

00:24:27.234 --> 00:24:29.249
diagonal see in the middle,

00:24:29.250 --> 00:24:31.406
it means you had that system combination

00:24:31.406 --> 00:24:33.480
which is a dot on this chart.

00:24:33.480 --> 00:24:35.770
Had a harder time acquiring

00:24:35.770 --> 00:24:38.060
images into the matching images.

00:24:38.060 --> 00:24:39.930
And you'll see without masks

00:24:39.930 --> 00:24:42.410
that most almost all of the dots

00:24:42.410 --> 00:24:44.140
should have below that diagonal,

00:24:44.140 --> 00:24:45.676
and as soon as you put masks on,

00:24:45.680 --> 00:24:47.036
it's almost universally true.

00:24:47.036 --> 00:24:49.663
There's a couple outliers up at the top

00:24:49.663 --> 00:24:51.730
for a matching system didn't appear to

00:24:51.730 --> 00:24:54.040
work very well this year with masks,

00:24:54.040 --> 00:24:56.259
but every other system that to reasonably

00:24:56.259 --> 00:24:58.440
work with masks is to load that dock.

00:24:58.440 --> 00:25:01.152
And so we think that this is evidence

00:25:01.152 --> 00:25:03.700
that the best way to reduce error

00:25:03.700 --> 00:25:05.820
rates and scenario test is going to

00:25:05.820 --> 00:25:07.624
attend the higher throughput cases.

00:25:07.624 --> 00:25:10.516
It's for technology providers that sort

00:25:10.516 --> 00:25:13.960
of focus on getting high quality image

00:25:13.960 --> 00:25:16.032
error rates that are much higher than

00:25:16.032 --> 00:25:18.173
the ability of these matching systems

00:25:18.173 --> 00:25:20.171
actually match the images, right.

00:25:20.171 --> 00:25:22.176
These matching system error rates

00:25:22.176 --> 00:25:23.379
are sort of.

00:25:23.380 --> 00:25:24.223
One in thousand,

00:25:24.223 --> 00:25:26.190
One in a million kind of grades was

00:25:26.262 --> 00:25:28.118
the acquisition system error.

00:25:28.120 --> 00:25:30.148
It's going to be.

00:25:30.150 --> 00:25:32.210
3 4 5 percent.

00:25:32.210 --> 00:25:33.194
So that's kind of a recommendation

00:25:33.194 --> 00:25:34.060
we had in the past,

00:25:34.060 --> 00:25:35.362
but I think affairs repeating that

00:25:35.362 --> 00:25:37.210
we sort of see that year after year.

00:25:37.210 --> 00:25:39.046
We think there's still an opportunity

00:25:39.046 --> 00:25:40.270
there for technology providers.

00:25:42.550 --> 00:25:43.160
Next slide.

00:25:46.730 --> 00:25:49.286
This one also probably was obvious,

00:25:49.290 --> 00:25:50.529
but we wanted to sort of put

00:25:50.529 --> 00:25:52.032
it all in one spot and people

00:25:52.032 --> 00:25:53.406
have the slide is showing us.

00:25:53.410 --> 00:25:56.320
It's mass cause issues for face

00:25:56.320 --> 00:26:00.050
recognition masks of those reduced tier,

00:26:00.050 --> 00:26:02.330
so effectiveness measure.

00:26:02.330 --> 00:26:04.610
Including acquisition rates,

00:26:04.610 --> 00:26:07.050
that's the annual AUC there.

00:26:07.050 --> 00:26:09.626
Every dot on this chart is a system

00:26:09.626 --> 00:26:11.520
combination with its tier metric laid

00:26:11.520 --> 00:26:14.034
out and then the numbers at the top

00:26:14.034 --> 00:26:16.372
are the ones that crossed the threshold

00:26:16.372 --> 00:26:19.284
values of 95% and see that without

00:26:19.284 --> 00:26:21.390
masks 26 system combinations did.

00:26:21.390 --> 00:26:23.326
But as soon as we put masks on,

00:26:23.330 --> 00:26:24.898
none of them were able to and then

00:26:24.898 --> 00:26:26.875
the red bar is just sort of

00:26:26.875 --> 00:26:28.127
that median performance number,

00:26:28.130 --> 00:26:31.310
which is kinda delineated the left,

00:26:31.310 --> 00:26:32.890
you can see that.

00:26:32.890 --> 00:26:35.636
Mass caused about a 10% reduction

00:26:35.636 --> 00:26:38.430
in during median tier for us,

00:26:38.430 --> 00:26:41.250
about a 5% reduction in media matching tier,

00:26:41.250 --> 00:26:43.690
which is a canopy.

00:26:43.690 --> 00:26:45.167
Less of an impact on matching 2.

00:26:45.170 --> 00:26:46.830
That's still an impact.

00:26:46.830 --> 00:26:49.050
So we think that's evidence that

00:26:49.050 --> 00:26:50.170
there's some opportunities to

00:26:50.170 --> 00:26:51.905
do stuff on the acquisition side

00:26:51.905 --> 00:26:53.325
and on the matching side,

00:26:53.330 --> 00:26:55.445
sort of a note of optimism in all of

00:26:55.445 --> 00:26:57.516
this though is that when we went back

00:26:57.516 --> 00:26:59.614
and sort of pulled the median tier

00:26:59.614 --> 00:27:02.483
from the 2020 route and that was about 77%,

00:27:02.483 --> 00:27:02.916
so.

00:27:02.916 --> 00:27:04.648
Even though we're still,

00:27:04.650 --> 00:27:05.586
you know, not perfect,

00:27:05.586 --> 00:27:07.283
we're not seeing this with 100% numbers.

00:27:07.283 --> 00:27:09.041
We see without masks about 10

00:27:09.041 --> 00:27:10.165
percentage points better than

00:27:10.165 --> 00:27:11.544
we were just see where it go.

00:27:11.550 --> 00:27:13.044
So this opportunities

00:27:13.044 --> 00:27:15.036
continue to serve improve.

00:27:15.040 --> 00:27:16.979
Your systems and then again if it

00:27:16.979 --> 00:27:18.708
wasn't obvious from that last chart,

00:27:18.710 --> 00:27:21.028
this should also help make it sort

00:27:21.028 --> 00:27:23.114
of reinforce this point is that most

00:27:23.114 --> 00:27:25.152
errors with face masks were due to

00:27:25.152 --> 00:27:26.870
issues getting that in the drudge.

00:27:26.870 --> 00:27:28.358
They're not getting it at all.

00:27:28.360 --> 00:27:30.250
They're getting a poor quality image.

00:27:32.370 --> 00:27:33.936
I think we can go to the next item.

00:27:33.940 --> 00:27:34.685
I think I'm turning it

00:27:34.685 --> 00:27:35.430
back over to you again.

00:27:35.430 --> 00:27:37.158
It's going to talk about demographics.

00:27:41.680 --> 00:27:42.526
Thanks, John.

00:27:42.526 --> 00:27:45.064
So John's provided us an overview

00:27:45.064 --> 00:27:48.066
of the overall results for the full

00:27:48.066 --> 00:27:52.098
group of 601 diverse volunteers.

00:27:52.100 --> 00:27:54.550
I will now go over the disaggregated

00:27:54.550 --> 00:27:56.358
performance results focusing on our

00:27:56.358 --> 00:27:57.758
measure of system effectiveness,

00:27:57.760 --> 00:28:00.511
which is going to be tier we

00:28:00.511 --> 00:28:02.166
disaggregated tier performance for

00:28:02.166 --> 00:28:04.166
each of eight demographic groups

00:28:04.170 --> 00:28:06.530
based on self reported race.

00:28:06.530 --> 00:28:09.029
We had three groups based on race,

00:28:09.030 --> 00:28:11.260
those volunteers that self reported

00:28:11.260 --> 00:28:14.289
as Asian as black or as white.

00:28:14.290 --> 00:28:16.270
We had two gender groups.

00:28:16.270 --> 00:28:18.874
Those that self reported as male or

00:28:18.874 --> 00:28:22.428
female and we also performed the segregation.

00:28:22.430 --> 00:28:25.020
Based on our measure of skin tone,

00:28:25.020 --> 00:28:28.751
where we separated participants into

00:28:28.751 --> 00:28:31.950
3 tertiles and I'll explain more on

00:28:32.030 --> 00:28:34.934
that how we did that later into lighter

00:28:34.934 --> 00:28:37.480
medium and darker skin tone groups.

00:28:37.480 --> 00:28:40.378
Next slide please.

00:28:40.380 --> 00:28:42.816
So a little bit about skin tone.

00:28:42.820 --> 00:28:44.872
We assess skin tone based on

00:28:44.872 --> 00:28:46.240
direct measurement using a

00:28:46.310 --> 00:28:48.179
calibrated colorimeter device.

00:28:48.180 --> 00:28:50.094
Our research has indicated that skin

00:28:50.094 --> 00:28:52.348
tone is an important correlate of

00:28:52.348 --> 00:28:54.180
face recognition system performance,

00:28:54.180 --> 00:28:56.400
but cannot be reliably estimated

00:28:56.400 --> 00:28:58.176
from biometric images themselves.

00:28:58.180 --> 00:29:01.438
So we perform an independent measurement

00:29:01.440 --> 00:29:03.936
under controlled conditions for the rally,

00:29:03.940 --> 00:29:07.126
we measured the average skin tone

00:29:07.126 --> 00:29:09.916
taken from 2 temples from each.

00:29:09.916 --> 00:29:10.582
Volunteer on.

00:29:10.582 --> 00:29:12.998
You can see the schematic on the

00:29:12.998 --> 00:29:15.110
right which shows where we took

00:29:15.110 --> 00:29:17.134
the measurement just above the face

00:29:17.134 --> 00:29:19.241
mask and the device that we used.

00:29:19.250 --> 00:29:21.575
The rally scenario tests measured

00:29:21.575 --> 00:29:23.900
skin tone using this instrument

00:29:23.976 --> 00:29:26.406
and identified people with darker,

00:29:26.410 --> 00:29:28.566
medium and light skin tone and the

00:29:28.566 --> 00:29:31.252
chart on the right bottom helps you

00:29:31.252 --> 00:29:33.507
visualize that this chart shows

00:29:33.507 --> 00:29:36.784
the CIELAB color values for human

00:29:36.784 --> 00:29:38.869
skin measured within this test.

00:29:38.870 --> 00:29:40.280
Each dot on this chart.

00:29:40.280 --> 00:29:41.720
Represents a single individual and

00:29:41.720 --> 00:29:43.925
the hue of the dot represents the

00:29:43.925 --> 00:29:46.043
actual color reading from the device,

00:29:46.050 --> 00:29:48.360
and you could see that these

00:29:48.360 --> 00:29:50.705
readings form a Crescent within the

00:29:50.705 --> 00:29:52.883
LB plane in this color space.

00:29:52.890 --> 00:29:55.753
This the L part of this color

00:29:55.753 --> 00:29:56.980
space actually corresponds

00:29:57.059 --> 00:29:58.999
to the perceptual likeness,

00:29:59.000 --> 00:30:01.790
so this L star value is what we used

00:30:01.790 --> 00:30:04.766
to split into tertiles and the first

00:30:04.766 --> 00:30:07.640
tertile was the those on sort of

00:30:07.640 --> 00:30:10.146
the bottom of this spectrum with a

00:30:10.150 --> 00:30:11.844
darker skin than we split it off.

00:30:11.850 --> 00:30:12.978
To the middle tertile,

00:30:12.978 --> 00:30:15.049
which was going to be medium skin

00:30:15.049 --> 00:30:16.975
and the lighter tertile was folks

00:30:16.975 --> 00:30:18.906
that were separated out into those

00:30:18.906 --> 00:30:19.818
with lighter skin.

00:30:19.820 --> 00:30:22.004
And that's how we did that split.

00:30:22.010 --> 00:30:23.108
Next slide please.

00:30:26.980 --> 00:30:30.160
So this slide shows.

00:30:30.160 --> 00:30:33.988
A visualization of results based that's

00:30:33.988 --> 00:30:38.570
similar to what John presented earlier.

00:30:38.570 --> 00:30:41.842
So you can see that there are matching

00:30:41.842 --> 00:30:45.455
systems on the along the Y axis and

00:30:45.455 --> 00:30:47.929
acquisition systems along the X axis,

00:30:47.930 --> 00:30:49.820
and what this chart shows is a

00:30:49.820 --> 00:30:51.890
summary of the performance overall.

00:30:51.890 --> 00:30:53.760
And as I mentioned previously,

00:30:53.760 --> 00:30:55.490
we had eight different demographic

00:30:55.490 --> 00:30:57.909
groups that we had in the rally,

00:30:57.910 --> 00:31:00.475
and this visualization shows in

00:31:00.475 --> 00:31:03.040
green when a system combination

00:31:03.125 --> 00:31:05.569
met the performance benchmark.

00:31:05.570 --> 00:31:08.480
The goal for each demographic group.

00:31:08.480 --> 00:31:11.336
The shading in yellow is when the system

00:31:11.336 --> 00:31:14.220
met system combination met the threshold.

00:31:14.220 --> 00:31:16.710
The rally threshold 95% threshold

00:31:16.710 --> 00:31:19.200
benchmark for tier for each

00:31:19.200 --> 00:31:21.295
demographic group and those that

00:31:21.295 --> 00:31:24.338
are not shaded green or yellow means

00:31:24.338 --> 00:31:27.376
that that system failed to meet the

00:31:27.376 --> 00:31:29.216
threshold rally threshold performance

00:31:29.216 --> 00:31:32.310
for at least one of the groups.

00:31:32.310 --> 00:31:34.934
And what you could see in panel A

00:31:34.934 --> 00:31:37.580
is performance with respect to true

00:31:37.580 --> 00:31:39.975
identification rate inclusive of both

00:31:39.975 --> 00:31:41.528
acquisition and matching errors.

00:31:41.528 --> 00:31:44.321
So all errors within the system and

00:31:44.321 --> 00:31:47.029
the chart on the right shows matching

00:31:47.029 --> 00:31:49.482
tier which is basically all errors,

00:31:49.482 --> 00:31:51.702
but discounting errors where the

00:31:51.702 --> 00:31:53.722
acquisition system simply did not

00:31:53.722 --> 00:31:55.507
deliver an image for assessment.

00:31:55.510 --> 00:31:59.335
And what you could see on the left side

00:31:59.335 --> 00:32:02.658
here is that for a good number of.

00:32:02.660 --> 00:32:04.700
Uh systems system combinations.

00:32:04.700 --> 00:32:08.232
These were able to meet the 95%

00:32:08.232 --> 00:32:10.592
tier threshold for all demographic

00:32:10.592 --> 00:32:12.480
groups in the assessment,

00:32:12.480 --> 00:32:15.196
and you could also see the relative

00:32:15.196 --> 00:32:17.369
importance of acquisition and matching by

00:32:17.369 --> 00:32:19.940
looking at sort of the shape that this,

00:32:19.940 --> 00:32:23.625
this block of successful systems

00:32:23.625 --> 00:32:27.720
meets the shape that it makes where

00:32:27.720 --> 00:32:30.990
you could see that for most.

00:32:30.990 --> 00:32:33.580
Matching systems with acquisition systems,

00:32:33.580 --> 00:32:35.967
herard and long were in fact able

00:32:35.967 --> 00:32:38.589
to meet this 95% benchmark for

00:32:38.589 --> 00:32:40.188
all demographic groups.

00:32:40.190 --> 00:32:43.270
So all but system sun.

00:32:43.270 --> 00:32:45.898
And and So what we see is it looks

00:32:45.898 --> 00:32:48.492
like it's relatively more important

00:32:48.492 --> 00:32:51.277
which acquisition system is employed

00:32:51.280 --> 00:32:53.326
rather than which matching system with

00:32:53.326 --> 00:32:55.432
respect to meeting meeting these raw

00:32:55.432 --> 00:32:57.137
performance benchmarks and some of

00:32:57.137 --> 00:32:59.217
the information that I'll show you next,

00:32:59.220 --> 00:33:01.130
we'll delve into more details

00:33:01.130 --> 00:33:03.040
on the matching tier side.

00:33:03.040 --> 00:33:06.334
If we discount the errors and in just

00:33:06.334 --> 00:33:08.459
acquisition of the biometric images,

00:33:08.460 --> 00:33:11.008
you can see that both acquisition and

00:33:11.008 --> 00:33:13.460
matching is important for meeting the goal.

00:33:13.460 --> 00:33:15.245
For all groups and you could see

00:33:15.245 --> 00:33:16.790
that because the Green Zone forms

00:33:16.790 --> 00:33:18.825
a sort of a triangle shape on this

00:33:18.825 --> 00:33:20.939
chart you can see that some systems

00:33:20.939 --> 00:33:22.038
were particularly effective.

00:33:22.038 --> 00:33:24.966
Some matching systems in particular systems,

00:33:24.970 --> 00:33:26.888
paint and salt were able to meet

00:33:26.888 --> 00:33:29.322
the goal for all groups so long as

00:33:29.322 --> 00:33:31.269
images were acquired and they were

00:33:31.269 --> 00:33:33.572
able to do that with all acquisition

00:33:33.572 --> 00:33:36.020
systems with safe system to cocoa and

00:33:36.020 --> 00:33:38.823
and so that that sort of highlights

00:33:38.823 --> 00:33:40.362
that the importance of that.

00:33:40.362 --> 00:33:40.630
However,

00:33:40.630 --> 00:33:42.170
there are some acquisition

00:33:42.170 --> 00:33:43.710
systems that also worked.

00:33:43.710 --> 00:33:44.354
Very well,

00:33:44.354 --> 00:33:46.286
across a variety of matching systems

00:33:46.286 --> 00:33:48.094
on this front and that system

00:33:48.094 --> 00:33:50.160
or a one that acquired an image,

00:33:50.160 --> 00:33:52.939
that image worked with five of the,

00:33:52.940 --> 00:33:56.664
you know half of the matching systems.

00:33:56.670 --> 00:33:57.846
Let's go to the next slide.

00:33:57.850 --> 00:33:59.098
We'll, we'll break things

00:33:59.098 --> 00:34:00.970
down into a bit more detail.

00:34:00.970 --> 00:34:03.364
So this slide shows a slightly different

00:34:03.364 --> 00:34:04.890
visualizations of the results,

00:34:04.890 --> 00:34:07.548
but focuses on gender, in particular.

00:34:07.550 --> 00:34:09.526
In these violin plots that I'm showing you,

00:34:09.530 --> 00:34:11.660
each dot represents the performance

00:34:11.660 --> 00:34:13.790
of a single system combination.

00:34:13.790 --> 00:34:15.704
The numbers above each plot show

00:34:15.704 --> 00:34:17.526
the number of system combinations

00:34:17.526 --> 00:34:20.245
meeting the 95% rally threshold.

00:34:20.245 --> 00:34:23.170
So for tier without masks,

00:34:23.170 --> 00:34:25.235
that's going to be panel A more

00:34:25.235 --> 00:34:26.720
system combinations met the rally.

00:34:26.720 --> 00:34:28.708
Your thresholds for male,

00:34:28.708 --> 00:34:30.696
then for female volunteers.

00:34:30.700 --> 00:34:32.420
There was a trend toward

00:34:32.420 --> 00:34:33.796
better tier for males,

00:34:33.800 --> 00:34:35.752
which was also observed

00:34:35.752 --> 00:34:38.192
with masks in panel B,

00:34:38.200 --> 00:34:39.964
but no system at the tier

00:34:39.964 --> 00:34:40.846
threshold with masks.

00:34:40.850 --> 00:34:43.714
So you could see that the violins are

00:34:43.714 --> 00:34:46.190
squarely below that Gray shaded area and

00:34:46.190 --> 00:34:49.619
and you have zeros above each violin.

00:34:49.619 --> 00:34:52.072
For matching without for

00:34:52.072 --> 00:34:53.560
matching tier without masks,

00:34:53.560 --> 00:34:55.816
that's going to be panel C

00:34:55.820 --> 00:34:58.200
performance was overall very good.

00:34:58.200 --> 00:35:00.702
Most system combinations met the matching

00:35:00.702 --> 00:35:03.598
tier threshold for both males and females.

00:35:03.600 --> 00:35:05.320
There was hardly any difference.

00:35:05.320 --> 00:35:08.008
45 systems met it for females

00:35:08.008 --> 00:35:09.800
and 47 for males.

00:35:09.800 --> 00:35:14.294
Matching tier remained very high with masks,

00:35:14.300 --> 00:35:17.100
unlike a tier and a number of systems

00:35:17.100 --> 00:35:19.966
were able to meet the rally threshold.

00:35:19.970 --> 00:35:20.306
Panel.

00:35:20.306 --> 00:35:22.658
So that's panel D and again we

00:35:22.658 --> 00:35:24.284
saw similar performance with

00:35:24.284 --> 00:35:26.489
masks for males and females.

00:35:26.490 --> 00:35:29.502
It's 29 systems met the rally

00:35:29.502 --> 00:35:32.768
threshold for females and 28 for males.

00:35:32.770 --> 00:35:34.210
So what these results suggest is

00:35:34.210 --> 00:35:35.554
that most of the differentials

00:35:35.554 --> 00:35:37.528
based on gender in this scenario

00:35:37.528 --> 00:35:39.226
were associated with errors in

00:35:39.226 --> 00:35:40.922
image acquisition not matching.

00:35:40.922 --> 00:35:44.018
So you know this difference in

00:35:44.018 --> 00:35:46.949
tier without masks 20 versus 35

00:35:46.949 --> 00:35:49.983
systems goes away when you look at

00:35:49.983 --> 00:35:53.010
it without with with matching tier.

00:35:53.010 --> 00:35:54.318
Next slide please.

00:35:58.970 --> 00:36:00.914
So this slide uses the same

00:36:00.914 --> 00:36:02.210
visualization approach as before,

00:36:02.210 --> 00:36:04.085
but the disaggregation is now

00:36:04.085 --> 00:36:05.960
by self reported race group.

00:36:05.960 --> 00:36:07.785
Again, we had volunteers that

00:36:07.785 --> 00:36:10.450
self report as Asian as black or

00:36:10.450 --> 00:36:12.370
African American and as white.

00:36:12.370 --> 00:36:14.468
So for tier without masks, again panel,

00:36:14.468 --> 00:36:16.472
A more system combinations met the

00:36:16.472 --> 00:36:18.629
rally tier threshold for volunteers that

00:36:18.629 --> 00:36:21.186
self reported as white and fewer for

00:36:21.186 --> 00:36:23.310
volunteers that self reported as black,

00:36:23.310 --> 00:36:26.410
only 18 versus a 33.

00:36:26.410 --> 00:36:28.839
The trend toward lower tier for volunteers.

00:36:28.840 --> 00:36:30.994
Self reporting as Black was also

00:36:30.994 --> 00:36:33.019
observed with masks in panel B.

00:36:33.020 --> 00:36:33.924
Interestingly, however,

00:36:33.924 --> 00:36:35.732
the performance volunteer self

00:36:35.732 --> 00:36:37.970
reporting as Asian remained high

00:36:37.970 --> 00:36:39.810
even in this challenging condition.

00:36:39.810 --> 00:36:43.254
For matching tier without masks panel

00:36:43.254 --> 00:36:45.650
C performance was overall very good

00:36:45.650 --> 00:36:47.872
and most system combinations met the

00:36:47.872 --> 00:36:49.906
matching tier threshold for all groups.

00:36:49.910 --> 00:36:50.318
Again,

00:36:50.318 --> 00:36:52.358
matching tier remained high with

00:36:52.358 --> 00:36:55.371
masks and panel D with hardly any

00:36:55.371 --> 00:36:57.551
decline for volunteer self reporting

00:36:57.551 --> 00:36:59.922
as Asian and these results suggest

00:36:59.922 --> 00:37:01.787
that without masks acquisition was

00:37:01.787 --> 00:37:03.978
the main contributor to demographic

00:37:03.978 --> 00:37:05.730
differentials based on race,

00:37:05.730 --> 00:37:07.800
whereas with masks both acquisition

00:37:07.800 --> 00:37:10.318
and matching affected the number of

00:37:10.318 --> 00:37:13.530
systems meeting. Rally benchmark.

00:37:13.530 --> 00:37:14.478
Next slide please.

00:37:16.990 --> 00:37:19.540
So this slide again repurposes

00:37:19.540 --> 00:37:21.070
the same visualization,

00:37:21.070 --> 00:37:22.950
but now we're performing disaggregation

00:37:22.950 --> 00:37:25.250
based on skin tone tertiles.

00:37:25.250 --> 00:37:28.578
And I remember that T1 corresponds to the

00:37:28.578 --> 00:37:31.307
tertile with the darkest skin tone T2

00:37:31.307 --> 00:37:34.406
for medium and then T3 for the lightest

00:37:34.406 --> 00:37:36.198
skin tone for tier without masks,

00:37:36.198 --> 00:37:37.680
which I'm showing you in panel,

00:37:37.680 --> 00:37:40.592
a more system combinations met the rally

00:37:40.592 --> 00:37:43.748
tier threshold for volunteers self reporting.

00:37:43.750 --> 00:37:47.218
Sorry, volunteers with lighter skin T3.

00:37:47.220 --> 00:37:50.628
That's 36 systems and fewer for

00:37:50.628 --> 00:37:54.527
volunteers with darker skin T one that's

00:37:54.527 --> 00:37:57.587
18 system this difference between 18

00:37:57.587 --> 00:38:00.374
and 36 was the largest difference in

00:38:00.374 --> 00:38:02.340
performance observed in the rally,

00:38:02.340 --> 00:38:05.133
and the trend toward lower tier for

00:38:05.133 --> 00:38:07.434
volunteers with darker skin was also

00:38:07.434 --> 00:38:09.498
observed with masks in panel B,

00:38:09.500 --> 00:38:11.804
performance volunteers with medium

00:38:11.804 --> 00:38:12.956
skin tones,

00:38:12.960 --> 00:38:15.039
many of whom self identified as Asian,

00:38:15.040 --> 00:38:18.106
remained high even with face masks.

00:38:18.110 --> 00:38:20.696
For matching tier without masks panel

00:38:20.696 --> 00:38:22.935
C performance was overall again very

00:38:22.935 --> 00:38:25.106
good and most system combinations met

00:38:25.106 --> 00:38:27.290
the matching tier threshold for all groups.

00:38:27.290 --> 00:38:29.086
Again matching tier remained

00:38:29.086 --> 00:38:31.780
high with masks for many system

00:38:31.859 --> 00:38:34.329
combinations and panel D however,

00:38:34.330 --> 00:38:36.886
fewer system combinations met the matching

00:38:36.886 --> 00:38:39.267
tier threshold for volunteers with darker

00:38:39.267 --> 00:38:41.430
skin than for the other two groups,

00:38:41.430 --> 00:38:43.698
so suggesting that there's still in effect

00:38:43.698 --> 00:38:45.943
there and these results overall again

00:38:45.943 --> 00:38:47.968
show that without masks acquisition.

00:38:47.970 --> 00:38:49.878
Was the main contributor to demographic

00:38:49.878 --> 00:38:51.440
differentials based on skin tone.

00:38:51.440 --> 00:38:51.842
However,

00:38:51.842 --> 00:38:52.646
with masks,

00:38:52.646 --> 00:38:54.254
both acquisition and matching

00:38:54.254 --> 00:38:56.255
affected the number of systems

00:38:56.255 --> 00:38:58.090
meeting the Rally benchmark for

00:38:58.090 --> 00:38:59.890
people with different skin tones.

00:38:59.890 --> 00:39:00.925
We're currently investigating

00:39:00.925 --> 00:39:02.305
the complex results observed

00:39:02.305 --> 00:39:04.038
in the presence of face masks,

00:39:04.040 --> 00:39:07.148
so it's an interesting effect.

00:39:07.150 --> 00:39:09.799
Next slide please.

00:39:09.800 --> 00:39:11.984
So these last few slides that I showed

00:39:11.984 --> 00:39:13.838
were pretty information dense here.

00:39:13.840 --> 00:39:16.094
I'm going to summarize the key takeaways

00:39:16.094 --> 00:39:17.490
about the demographic differentials

00:39:17.490 --> 00:39:19.758
and face recognition that we observe.

00:39:19.760 --> 00:39:20.408
So first,

00:39:20.408 --> 00:39:22.028
some system combinations were able

00:39:22.028 --> 00:39:24.540
to meet the rally to your threshold

00:39:24.540 --> 00:39:26.016
for all demographic groups.

00:39:26.020 --> 00:39:27.810
This threshold was chosen before

00:39:27.810 --> 00:39:30.016
we started the rally execution to

00:39:30.016 --> 00:39:31.448
represent the performance level

00:39:31.448 --> 00:39:33.580
deemed suitable for this use case,

00:39:33.580 --> 00:39:36.244
namely the high throughput

00:39:36.244 --> 00:39:38.001
unattended system performance,

00:39:38.001 --> 00:39:40.948
a biometric system that meets this threshold.

00:39:40.950 --> 00:39:42.786
All groups would therefore still be

00:39:42.786 --> 00:39:44.529
deemed suitable for the use case.

00:39:44.530 --> 00:39:45.894
According to this benchmark,

00:39:45.894 --> 00:39:47.940
even if the user population is

00:39:48.002 --> 00:39:49.466
homogeneous composed solely of

00:39:49.466 --> 00:39:51.296
members of that one group,

00:39:51.300 --> 00:39:54.324
and in this test 18 of the 50,

00:39:54.330 --> 00:39:56.832
about a third of the acquisition

00:39:56.832 --> 00:39:58.500
matching system combination met

00:39:58.566 --> 00:40:00.129
this stringent criteria.

00:40:00.130 --> 00:40:02.105
So choosing a system combination

00:40:02.105 --> 00:40:04.080
without testing may not yield

00:40:04.146 --> 00:40:06.246
equitable results for all groups.

00:40:06.250 --> 00:40:08.506
If you're just picking out of a hat.

00:40:08.510 --> 00:40:12.129
2nd despite high performance by some systems,

00:40:12.130 --> 00:40:14.320
we observed some demographic differentials

00:40:14.320 --> 00:40:16.510
in median system performance for

00:40:16.574 --> 00:40:18.779
different groups that did align

00:40:18.779 --> 00:40:20.543
with pre-existing fairness concerns.

00:40:20.550 --> 00:40:22.920
We saw that median system performance

00:40:22.920 --> 00:40:24.963
without masks was lower for

00:40:24.963 --> 00:40:26.908
female than for male volunteers.

00:40:26.910 --> 00:40:28.620
Performance was also lower for

00:40:28.620 --> 00:40:29.988
volunteers that self identified

00:40:29.988 --> 00:40:31.884
as black or African American and

00:40:31.884 --> 00:40:33.742
differed most based on skin tone

00:40:33.742 --> 00:40:35.267
with lower performance for people

00:40:35.267 --> 00:40:36.487
with darker skin as

00:40:36.490 --> 00:40:37.630
compared to lighter skin.

00:40:37.630 --> 00:40:39.055
It's important to note the

00:40:39.055 --> 00:40:40.279
scale of these observed.

00:40:40.280 --> 00:40:42.264
Differences without masks differences

00:40:42.264 --> 00:40:45.240
in tier across groups were generally

00:40:45.313 --> 00:40:49.210
below 6% and below 1% for matching tier.

00:40:49.210 --> 00:40:51.592
Again highlighting the importance of image

00:40:51.592 --> 00:40:54.010
acquisition and shaping these differentials.

00:40:54.010 --> 00:40:56.500
The differences grew in the presence

00:40:56.500 --> 00:40:59.087
of masks now falling at 10% or

00:40:59.087 --> 00:41:01.172
below for differences in tier

00:41:01.172 --> 00:41:03.705
and below 4% in matching tier.

00:41:03.705 --> 00:41:06.255
So this complex interaction shows that

00:41:06.255 --> 00:41:08.832
equitability of a biometric system may

00:41:08.832 --> 00:41:10.922
change when operational conditions change.

00:41:10.930 --> 00:41:13.330
Obviating the need for operational

00:41:13.330 --> 00:41:14.770
monitoring and testing.

00:41:14.770 --> 00:41:15.658
Next slide please.

00:41:18.120 --> 00:41:20.143
So so far I've been discussing the

00:41:20.143 --> 00:41:22.140
performance of face recognition systems.

00:41:22.140 --> 00:41:24.485
However, the 2021 rally included

00:41:24.485 --> 00:41:27.332
one multimodal face Iris system and

00:41:27.332 --> 00:41:29.846
three Iris matching systems as well.

00:41:29.850 --> 00:41:32.139
And this slide is intended to summarize

00:41:32.139 --> 00:41:33.960
the observation for these systems,

00:41:33.960 --> 00:41:36.222
so I'll make a few points. First,

00:41:36.222 --> 00:41:40.296
no system combination with Iris met the

00:41:40.296 --> 00:41:43.152
rally threshold for any demographic group.

00:41:43.152 --> 00:41:44.421
Performance was generally

00:41:44.421 --> 00:41:46.220
fairly low compared to face,

00:41:46.220 --> 00:41:47.772
unlike for face recognition.

00:41:47.772 --> 00:41:50.272
However, masks did not appear to reduce

00:41:50.272 --> 00:41:51.680
median iris recognition performance.

00:41:51.680 --> 00:41:53.510
You can compare without masks,

00:41:53.510 --> 00:41:56.996
the median tier was 74.4 with masks.

00:41:57.000 --> 00:41:59.440
The median tier was 77.3,

00:41:59.440 --> 00:42:01.738
so very close, and again similar.

00:42:01.740 --> 00:42:05.340
Similarly, close results from matching tier.

00:42:05.340 --> 00:42:07.195
We did observe some demographic

00:42:07.195 --> 00:42:08.679
differentials for iris systems.

00:42:08.680 --> 00:42:10.810
However, these were distinct from

00:42:10.810 --> 00:42:12.940
those observed for face recognition.

00:42:12.940 --> 00:42:15.130
Without masks, performance was lower

00:42:15.130 --> 00:42:18.150
for male relative to female volunteers.

00:42:18.150 --> 00:42:19.774
So the opposite result.

00:42:19.774 --> 00:42:22.210
And it was lower for volunteers

00:42:22.285 --> 00:42:24.500
that self identified as Asian.

00:42:24.500 --> 00:42:25.106
In general,

00:42:25.106 --> 00:42:27.227
the size of the differentials was comparable

00:42:27.227 --> 00:42:29.377
to those observed in face recognition.

00:42:29.380 --> 00:42:29.994
However,

00:42:29.994 --> 00:42:33.678
we saw nearly a larger differentials

00:42:33.680 --> 00:42:36.578
min matching tier for IRIS as

00:42:36.578 --> 00:42:39.140
compared to face without masks.

00:42:39.140 --> 00:42:39.968
And at this point,

00:42:39.968 --> 00:42:41.841
I'll hand it off to John Howard to

00:42:41.841 --> 00:42:43.286
discuss our overall conclusions and

00:42:43.286 --> 00:42:45.064
our efforts on standardizing testing

00:42:45.064 --> 00:42:46.780
and measurement of demographic

00:42:46.780 --> 00:42:48.496
differentials for biometric systems.

00:42:51.970 --> 00:42:52.738
Thank you, Yergeniy.

00:42:52.738 --> 00:42:54.530
So we've kind of thrown a lot

00:42:54.585 --> 00:42:56.110
of information to you today.

00:42:56.110 --> 00:42:57.335
So we wanted to just leave you

00:42:57.335 --> 00:42:58.890
with a few high level conclusions.

00:42:58.890 --> 00:43:02.328
You can sort of take back.

00:43:02.330 --> 00:43:03.860
You just have to walk away with a few points,

00:43:03.860 --> 00:43:06.020
so the first is.

00:43:06.020 --> 00:43:07.740
When the main, passionate,

00:43:07.740 --> 00:43:09.380
sometimes people to deride

00:43:09.380 --> 00:43:11.720
biometric system to slow or clunky,

00:43:11.720 --> 00:43:13.778
and no one likes to use them,

00:43:13.780 --> 00:43:14.974
the actually fairly good data on

00:43:14.974 --> 00:43:16.458
this sort of the year after year,

00:43:16.460 --> 00:43:18.531
that's kind of indicates the opposite, right.

00:43:18.531 --> 00:43:20.699
And most of these systems are very fast.

00:43:20.700 --> 00:43:21.795
And the satisfaction

00:43:21.795 --> 00:43:23.255
rates reasoning very high.

00:43:23.260 --> 00:43:25.006
So you don't seem to mind

00:43:25.006 --> 00:43:28.259
interacting with them in this model?

00:43:28.260 --> 00:43:31.242
2nd. At this point,

00:43:31.242 --> 00:43:32.612
you know the biggest area

00:43:32.612 --> 00:43:33.809
of improvement that we see.

00:43:33.810 --> 00:43:37.548
It's possible right now in these systems.

00:43:37.550 --> 00:43:39.212
Is to make improvements in the

00:43:39.212 --> 00:43:40.895
acquisition process and I know this

00:43:40.895 --> 00:43:42.729
sort of goes against the current trends

00:43:42.729 --> 00:43:44.368
and computer science or technology

00:43:44.368 --> 00:43:46.023
companies in general doesn't want

00:43:46.023 --> 00:43:49.190
to focus on our more complicated

00:43:49.190 --> 00:43:51.460
CNN's with larger training sets.

00:43:51.460 --> 00:43:53.692
But where the performance is matching

00:43:53.692 --> 00:43:55.918
systems currently is the gains to be

00:43:55.918 --> 00:43:57.580
had there still especially for the

00:43:57.636 --> 00:43:59.624
best ones are really on the margins.

00:43:59.630 --> 00:44:02.668
We talking about small improvements point 1%,

00:44:02.670 --> 00:44:05.330
point 1%.

00:44:05.330 --> 00:44:06.495
But acquisition error rates going

00:44:06.495 --> 00:44:08.313
to have a supportive and I would be

00:44:08.313 --> 00:44:09.720
higher than that right up percentage or

00:44:09.767 --> 00:44:11.377
maybe as high as five percentage points.

00:44:11.380 --> 00:44:14.515
So getting better qualities of

00:44:14.515 --> 00:44:17.023
optics and feedback mechanisms.

00:44:17.030 --> 00:44:19.676
The acquisitions systems we think could

00:44:19.676 --> 00:44:22.630
really help through the overall process.

00:44:22.630 --> 00:44:24.255
3rd you know encouragingly mass

00:44:24.255 --> 00:44:26.142
performance got better in 2021 as

00:44:26.142 --> 00:44:28.470
compared to 2022 and not by sort of

00:44:28.547 --> 00:44:30.680
the trivial marginal percentage

00:44:30.680 --> 00:44:33.520
improvement of about 10%.

00:44:33.520 --> 00:44:35.368
We think this is a sign that they're

00:44:35.368 --> 00:44:36.865
still out there and acquisition

00:44:36.865 --> 00:44:38.500
improvements if could be made,

00:44:38.500 --> 00:44:38.948
you know,

00:44:38.948 --> 00:44:40.740
right now to focus on the mask scenario.

00:44:40.740 --> 00:44:42.198
But this may be sort of a hair burner.

00:44:44.220 --> 00:44:46.104
Other long standing problems

00:44:46.104 --> 00:44:47.988
with face recognition said

00:44:47.988 --> 00:44:50.200
mainly here basic collusions.

00:44:50.200 --> 00:44:51.436
Can you just made this point,

00:44:51.440 --> 00:44:53.381
but without masks?

00:44:53.381 --> 00:44:56.616
Sort of differences in performance?

00:44:56.620 --> 00:44:58.495
Median system performance across the

00:44:58.495 --> 00:45:00.600
different groups for less than 6%,

00:45:00.600 --> 00:45:02.658
less than 6% of the nation,

00:45:02.660 --> 00:45:05.054
as were largely driven by failures

00:45:05.054 --> 00:45:07.220
to submit images for certain

00:45:07.220 --> 00:45:09.700
demographic groups not driven by.

00:45:09.700 --> 00:45:13.270
There's not certainly graphic groups.

00:45:13.270 --> 00:45:15.818
And then without masks and the best

00:45:15.818 --> 00:45:18.114
systems can become pretty well across

00:45:18.114 --> 00:45:20.706
all those groups that have been

00:45:20.706 --> 00:45:23.886
maintained overall tier of 98.4.

00:45:23.886 --> 00:45:28.020
For each group tested male, female.

00:45:28.020 --> 00:45:30.800
Skin profiles as well.

00:45:30.800 --> 00:45:32.930
And iris?

00:45:32.930 --> 00:45:35.462
Right behind some of the face

00:45:35.462 --> 00:45:36.306
recognition performance.

00:45:36.310 --> 00:45:39.068
But we didn't have that many Iris

00:45:39.068 --> 00:45:40.761
acquisition systems or matching

00:45:40.761 --> 00:45:43.359
systems related to face this year.

00:45:43.360 --> 00:45:45.523
That's also just sort of the function

00:45:45.523 --> 00:45:48.288
of the high throughput scenario as well.

00:45:48.290 --> 00:45:48.970
Next slide.

00:45:54.550 --> 00:45:58.267
Yeah. So this is a fun new addition for

00:45:58.267 --> 00:46:00.583
this year who was interactive results

00:46:00.583 --> 00:46:03.062
look right down through the website that

00:46:03.062 --> 00:46:05.589
you're seeing is sort of a scrolling.

00:46:05.590 --> 00:46:07.024
Series of how you can manipulate

00:46:07.024 --> 00:46:08.828
and play with this data on your own.

00:46:08.830 --> 00:46:10.786
Some of these slides which screenshots

00:46:10.786 --> 00:46:12.590
haven't put in there as well.

00:46:12.590 --> 00:46:14.655
Would you could do things like see

00:46:14.655 --> 00:46:16.192
overall trade interpretation results.

00:46:16.192 --> 00:46:18.676
For example you can see them

00:46:18.676 --> 00:46:20.809
just for males just for females.

00:46:20.810 --> 00:46:22.987
Just for the race categories and toggle,

00:46:22.990 --> 00:46:24.735
it just didn't seem better

00:46:24.735 --> 00:46:26.950
compared with any of that masks.

00:46:26.950 --> 00:46:28.750
It's a lot of numbers,

00:46:28.750 --> 00:46:30.667
but we thought this was a good way for

00:46:30.667 --> 00:46:32.466
people to sort of choose their own

00:46:32.466 --> 00:46:34.053
adventure through them and find the

00:46:34.053 --> 00:46:35.859
data points that were important to them.

00:46:35.860 --> 00:46:38.196
In addition to viewing the data this way.

00:46:38.200 --> 00:46:39.180
You can see the video,

00:46:39.180 --> 00:46:42.400
there's also a downloadable PDF report with

00:46:42.400 --> 00:46:45.817
one giant table that has all the numbers.

00:46:45.820 --> 00:46:47.124
I will warn you,

00:46:47.124 --> 00:46:49.052
it's over 1600 total numbers and

00:46:49.052 --> 00:46:50.960
so the tables 24 pages long,

00:46:50.960 --> 00:46:53.252
but there's really some sort of

00:46:53.252 --> 00:46:55.451
analysis or piece of information

00:46:55.451 --> 00:46:57.660
that is important to your group.

00:46:57.660 --> 00:46:59.562
I'd encourage you have tried to

00:46:59.562 --> 00:47:01.656
make as much data available for

00:47:01.656 --> 00:47:03.912
the possible so that people can

00:47:03.912 --> 00:47:05.640
answer their own questions.

00:47:05.640 --> 00:47:06.100
Except.

00:47:12.280 --> 00:47:13.402
Oops. OK, perfect.

00:47:13.402 --> 00:47:16.594
And then so finally we wanted to suggest

00:47:16.594 --> 00:47:19.099
this kind of demographic testing.

00:47:19.100 --> 00:47:21.025
So the past couple of years we've

00:47:21.025 --> 00:47:22.847
been involved in trying to distill

00:47:22.847 --> 00:47:25.038
lessons learned from doing this kind of

00:47:25.101 --> 00:47:26.776
testing and to international standard

00:47:26.776 --> 00:47:29.525
so that others can sort of execute and

00:47:29.525 --> 00:47:32.106
learn from our lessons learned that

00:47:32.106 --> 00:47:35.260
standard is ISO/IEC 19795-10.

00:47:35.260 --> 00:47:37.790
It's currently in working drafts

00:47:37.790 --> 00:47:39.810
through the title of the standard.

00:47:39.810 --> 00:47:41.122
John, biometric performance testing

00:47:41.122 --> 00:47:42.762
across in the graphic groups.

00:47:42.770 --> 00:47:44.594
It deals with things about how

00:47:44.594 --> 00:47:46.270
do you find infographic groups,

00:47:46.270 --> 00:47:48.153
including some of the things that we

00:47:48.153 --> 00:47:50.088
need just talked about with skin time.

00:47:50.090 --> 00:47:52.386
What do you need to consider when planning

00:47:52.386 --> 00:47:54.508
an assessment of demographic differentials?

00:47:54.510 --> 00:47:57.146
And then once you've executed a study

00:47:57.146 --> 00:47:59.442
like the one we just reported on,

00:47:59.450 --> 00:48:00.866
how do you calculate error rates?

00:48:00.870 --> 00:48:03.012
And so we report those back and I think

00:48:03.012 --> 00:48:05.734
to sort of highlight some of the topics

00:48:05.734 --> 00:48:09.368
we discuss in that group and example here.

00:48:09.370 --> 00:48:10.586
What we're showing is.

00:48:10.586 --> 00:48:12.410
At the top of the table.

00:48:12.410 --> 00:48:15.812
The median tier for males and rather

00:48:15.812 --> 00:48:18.358
than 97% in the meadian tier for females

00:48:18.358 --> 00:48:20.280
for another 4% and so the next question

00:48:20.280 --> 00:48:21.979
is sort of ask yourself is how do you,

00:48:21.980 --> 00:48:23.065
what do you do with those numbers?

00:48:23.070 --> 00:48:25.270
How do you report them out different things

00:48:25.270 --> 00:48:27.160
that are proposed by different groups?

00:48:27.160 --> 00:48:29.533
One option is obviously just subtract them

00:48:29.533 --> 00:48:32.109
and say that's a difference of about 3%.

00:48:32.110 --> 00:48:35.158
Another option is the ratio difference.

00:48:35.160 --> 00:48:36.840
6% errors over 3% errors.

00:48:36.840 --> 00:48:38.490
That's about two times difference

00:48:38.490 --> 00:48:40.840
in the amount of errors experienced

00:48:40.840 --> 00:48:43.654
from that she knows which is males.

00:48:43.660 --> 00:48:45.412
Those are good options and certainly

00:48:45.412 --> 00:48:46.580
things to be considered,

00:48:46.580 --> 00:48:48.750
but I think you lose something there.

00:48:48.750 --> 00:48:51.238
That's the magnitude so.

00:48:51.240 --> 00:48:53.472
6% error rate over 3% error rate.

00:48:53.472 --> 00:48:55.036
The difference of 2X.

00:48:55.040 --> 00:48:56.140
That's the same thing.

00:48:56.140 --> 00:48:58.390
Come to the same conclusion with that 10%

00:48:58.390 --> 00:49:01.090
error rate or 20% error rate over 10% rate.

00:49:01.090 --> 00:49:03.889
So you sort of lose where you are relatively.

00:49:03.890 --> 00:49:05.300
And so for that reason,

00:49:05.300 --> 00:49:06.818
and also just for historical reasons,

00:49:06.820 --> 00:49:08.092
just sort of how we reported

00:49:08.092 --> 00:49:09.210
value results in the past,

00:49:09.210 --> 00:49:12.010
we chose this bottom option this year,

00:49:12.010 --> 00:49:14.190
which is we just reported.

00:49:14.190 --> 00:49:16.710
Error rates for success rates of

00:49:16.710 --> 00:49:19.245
different groups relative to our rally

00:49:19.245 --> 00:49:22.175
goals and thresholds and 95% and 99%.

00:49:22.175 --> 00:49:23.540
Seeing this scenario,

00:49:23.540 --> 00:49:25.948
you would have found a difference right

00:49:25.950 --> 00:49:28.472
tier from males is leading that 95%

00:49:28.472 --> 00:49:30.446
threshold while the tier for females is

00:49:30.446 --> 00:49:34.020
not short of arrived at the conclusion there.

00:49:34.020 --> 00:49:36.820
And a lot of other demographic groups,

00:49:36.820 --> 00:49:37.960
that wasn't the case.

00:49:37.960 --> 00:49:40.909
We think this is sort of easy to understand.

00:49:40.910 --> 00:49:43.277
Some of the other ones are using as well.

00:49:43.280 --> 00:49:45.280
It doesn't do as good a job capturing

00:49:45.280 --> 00:49:46.720
the magnitude of the difference,

00:49:46.720 --> 00:49:48.778
especially if you have league thresholds.

00:49:48.780 --> 00:49:51.110
If you're threshold success rate,

00:49:51.110 --> 00:49:53.091
you're looking for something like 80 would

00:49:53.091 --> 00:49:55.284
have one group of 81 and one group of 99.

00:49:55.290 --> 00:49:56.890
And you might not capture

00:49:56.890 --> 00:49:59.730
that approaching it this way.

00:49:59.730 --> 00:50:02.106
But this is how we sort of reported out

00:50:02.110 --> 00:50:04.350
in the rally and open to feedback on

00:50:04.350 --> 00:50:06.373
this and these kind of topics about

00:50:06.373 --> 00:50:09.204
how you do some of these things I just

00:50:09.204 --> 00:50:11.370
discussed are of interest to your group.

00:50:11.370 --> 00:50:14.340
Please consider contributing to the

00:50:14.340 --> 00:50:16.319
standard development in your eye.

00:50:18.390 --> 00:50:20.679
And let you know to get involved.

00:50:20.680 --> 00:50:22.204
We're certainly looking to have an

00:50:22.204 --> 00:50:23.679
inclusive group of people with lots

00:50:23.679 --> 00:50:25.128
of ideas and thoughts on this one.

00:50:27.230 --> 00:50:28.178
Alright, next slide.

00:50:31.160 --> 00:50:32.135
With that, I'm turning it

00:50:32.135 --> 00:50:33.110
back over to Arun.

00:50:37.770 --> 00:50:40.103
Alright. Well, thank you so much, John.

00:50:40.103 --> 00:50:42.047
I appreciate that and thanks you Yegeney .

00:50:42.050 --> 00:50:43.954
Uh, you guys did a great job presenting

00:50:43.954 --> 00:50:45.773
some of the results that we had

00:50:45.773 --> 00:50:47.604
from the rally result rally in 2021.

00:50:47.604 --> 00:50:50.676
Umm, so a couple of updates here about

00:50:50.676 --> 00:50:53.842
the the next biometric rally we just

00:50:53.842 --> 00:50:56.759
announced this a couple of weeks ago.

00:50:56.760 --> 00:50:59.874
We're still focusing in on the

00:50:59.874 --> 00:51:02.380
unattended high throughput use case

00:51:02.380 --> 00:51:05.160
for the acquisition systems.

00:51:05.160 --> 00:51:06.875
And the thing that we're going to

00:51:06.875 --> 00:51:08.553
be making a little bit different

00:51:08.553 --> 00:51:10.611
here is actually what we tried to

00:51:10.674 --> 00:51:13.740
do back in I guess 2020 pre COVID

00:51:13.740 --> 00:51:16.260
which was moved to small,

00:51:16.260 --> 00:51:18.528
you know group processing being able to.

00:51:18.530 --> 00:51:20.935
Process groups together without having

00:51:20.935 --> 00:51:23.724
to artificially force groups to kind

00:51:23.724 --> 00:51:26.228
of go through the checkpoint 1 by 1.

00:51:26.230 --> 00:51:28.960
So it still has this high

00:51:28.960 --> 00:51:30.259
throughput aspect to it,

00:51:30.259 --> 00:51:32.380
but the intention is to be able

00:51:32.448 --> 00:51:34.668
to process people as they travel.

00:51:34.670 --> 00:51:36.590
With friends, with colleagues,

00:51:36.590 --> 00:51:37.550
with families,

00:51:37.550 --> 00:51:39.350
so that you don't need to have children,

00:51:39.350 --> 00:51:40.352
go through separately.

00:51:40.352 --> 00:51:41.688
Adults go through separately.

00:51:41.690 --> 00:51:44.378
It all works together.

00:51:44.380 --> 00:51:46.836
The focus also will be on only the

00:51:46.836 --> 00:51:48.957
face biometric modality at this point,

00:51:48.960 --> 00:51:51.970
given the reduction or the the limited

00:51:51.970 --> 00:51:54.539
performance we've seen with Iris so far.

00:51:54.540 --> 00:51:55.975
I think what we're open to changing

00:51:55.975 --> 00:51:57.498
that back up again in the future,

00:51:57.500 --> 00:51:59.572
but right now it looks like the the

00:51:59.572 --> 00:52:01.307
focus really should be on on trying

00:52:01.307 --> 00:52:03.579
to make sure that we can do this

00:52:03.579 --> 00:52:05.163
evaluation with face at a minimum.

00:52:05.170 --> 00:52:05.471
Umm.

00:52:05.471 --> 00:52:07.277
I think the important things we

00:52:07.277 --> 00:52:09.603
want to talk about here are we'll

00:52:09.603 --> 00:52:11.631
talk about a few different things.

00:52:11.640 --> 00:52:14.400
We want to focus in on.

00:52:14.400 --> 00:52:15.920
Umm, you know again efficiency,

00:52:15.920 --> 00:52:17.972
effectiveness and user satisfaction

00:52:17.972 --> 00:52:20.537
with this biometric acquisition systems,

00:52:20.540 --> 00:52:23.033
you know being able to look at how well

00:52:23.033 --> 00:52:25.040
images acquired with an acquisition

00:52:25.040 --> 00:52:27.536
system will work with different matches.

00:52:27.540 --> 00:52:29.244
This happens when you might have

00:52:29.244 --> 00:52:31.137
a large enterprise or a large

00:52:31.137 --> 00:52:32.877
organization that has different matchers

00:52:32.877 --> 00:52:35.265
or maybe different matchers that are

00:52:35.265 --> 00:52:36.555
configured slightly differently.

00:52:36.560 --> 00:52:38.560
We want to make sure that the matches

00:52:38.560 --> 00:52:40.888
can take in imagery from a lot of

00:52:40.888 --> 00:52:42.560
different types of collection devices.

00:52:42.560 --> 00:52:44.541
Typically a lot of cameras that get

00:52:44.541 --> 00:52:46.917
deployed will be different from one another.

00:52:46.920 --> 00:52:48.534
One organization might buy some or

00:52:48.534 --> 00:52:50.169
even one location might have a few,

00:52:50.170 --> 00:52:51.958
or they might have a different

00:52:51.958 --> 00:52:53.620
number of other collection devices.

00:52:53.620 --> 00:52:55.545
We want to make sure we understand

00:52:55.545 --> 00:52:57.936
how well the matching system and the

00:52:57.936 --> 00:52:59.811
matching process works regardless of

00:52:59.811 --> 00:53:02.452
the cameras that might be submitting images,

00:53:02.452 --> 00:53:05.368
we plan to continue to disaggregate

00:53:05.368 --> 00:53:06.880
performance based off of.

00:53:06.880 --> 00:53:08.173
Characteristics of demographic

00:53:08.173 --> 00:53:11.819
characteristics like race, gender, skin tone.

00:53:11.820 --> 00:53:14.430
And we're also going to include

00:53:14.430 --> 00:53:16.940
these concepts of privacy measures,

00:53:16.940 --> 00:53:19.000
and this is kind of the idea of when we,

00:53:19.000 --> 00:53:21.428
when there are, you know, again large,

00:53:21.428 --> 00:53:23.600
high throughput scenarios,

00:53:23.600 --> 00:53:25.798
there are often large groups of people

00:53:25.798 --> 00:53:28.382
and not all of those people may have

00:53:28.382 --> 00:53:31.219
opted into the face photo collection process.

00:53:31.220 --> 00:53:33.404
So you might have people in the background,

00:53:33.410 --> 00:53:35.425
you may have people going

00:53:35.425 --> 00:53:36.634
through parallel lanes.

00:53:36.640 --> 00:53:38.146
So the intention here is also

00:53:38.146 --> 00:53:40.088
to not only make sure that we

00:53:40.088 --> 00:53:41.553
can evaluate how well systems.

00:53:41.560 --> 00:53:43.294
Or acquiring photos of people who

00:53:43.294 --> 00:53:45.348
are within the field of view or

00:53:45.348 --> 00:53:46.743
within the intended capture volume

00:53:46.743 --> 00:53:48.908
of the biometric acquisition system,

00:53:48.910 --> 00:53:50.954
but also make sure that they're not

00:53:50.954 --> 00:53:52.827
collecting photos of people who have

00:53:52.827 --> 00:53:54.392
not explicitly opted in

00:53:54.392 --> 00:53:56.352
and might be walking parallel or

00:53:56.352 --> 00:53:57.947
close to the acquisition system,

00:53:57.950 --> 00:54:00.351
but not explicitly in the capture volume

00:54:00.351 --> 00:54:02.589
of that specific acquisition system.

00:54:02.590 --> 00:54:03.722
So in this case,

00:54:03.722 --> 00:54:06.350
opt in is kind of assessed based off of,

00:54:06.350 --> 00:54:08.630
you know, your physical location and

00:54:08.630 --> 00:54:10.640
proximity relative to the camera.

00:54:10.640 --> 00:54:11.176
You know,

00:54:11.176 --> 00:54:13.320
there would be a specific type of captures.

00:54:13.320 --> 00:54:14.845
If you're interested in learning

00:54:14.845 --> 00:54:17.040
more about the rally for 2022,

00:54:17.040 --> 00:54:20.008
please go to mdtf.org/rally 2020.

00:54:20.008 --> 00:54:23.361
You can also e-mail us at people

00:54:23.361 --> 00:54:25.396
screening at hq.dhs.gov we have,

00:54:25.396 --> 00:54:27.732
we have, we've done our announcements

00:54:27.732 --> 00:54:29.184
webinar at this point,

00:54:29.190 --> 00:54:31.750
but we will be.

00:54:31.750 --> 00:54:34.366
Having our technical webinar in a few weeks,

00:54:34.370 --> 00:54:36.302
and if you're watching this on

00:54:36.302 --> 00:54:38.710
a recorded on a recorded video,

00:54:38.710 --> 00:54:40.312
chances are we've already had that

00:54:40.312 --> 00:54:41.733
webinar so I would definitely

00:54:41.733 --> 00:54:43.805
recommend that you go to the website

00:54:46.470 --> 00:54:47.622
mdtf.org/rally 2022 to learn

00:54:47.622 --> 00:54:49.732
more about the rally and see if

00:54:49.732 --> 00:54:51.676
it might be appropriate for your

00:54:51.676 --> 00:54:52.648
organization to participate.

00:54:52.650 --> 00:54:54.400
For our stakeholders who are

00:54:54.400 --> 00:54:55.450
not technology developers,

00:54:55.450 --> 00:54:57.142
please let us know if you're

00:54:57.142 --> 00:54:57.988
interested in participating.

00:54:57.990 --> 00:54:59.904
We are intending to do VIP

00:54:59.904 --> 00:55:01.700
day where we can bring.

00:55:01.700 --> 00:55:03.720
Groups of VIPs through see

00:55:03.720 --> 00:55:05.336
the systems and operations,

00:55:05.340 --> 00:55:06.866
see how they work and get the

00:55:06.866 --> 00:55:08.550
chance to meet with some of the

00:55:08.550 --> 00:55:09.765
vendors who are making these

00:55:09.765 --> 00:55:12.021
new and and cutting edge kind of

00:55:12.021 --> 00:55:12.999
technologies and capabilities.