Good morning folks.
My name is Arun Vemury.
I'm with the DHS science
and Technology Directorate.
I'd like to thank you for joining us for
today to learn more about the results
of our 2021 Biometric technology rally.
As many of you are likely aware of,
the results are already or have been live
on the mdtf.org website for some time,
but today you'll have a chance to hear
more from the test team who oversaw
and managed the evaluations to help
explain what the results mean and help
answer any questions you may have.
With that, let's go ahead to the next slide.
So we'll give you a general overview of
what we're going to talk about today.
We'll provide a very brief intro to the
biometric and Identity Technology Center.
We'll talk about the technology rallies,
the the current one for the one we just
completed for 2021 and previous rallies
and how that differs, how we what,
what the timeline for the rally was,
what the acquisition systems
did as part of the test,
how the matching systems, you know,
work were used and how they were
evaluated as part of the test.
An explanation of the overall test approach
and information about service providers
or technology providers will then dig,
dig deeper into the actual
results of the test,
which I think is probably the
primary interest of you all who
are joining us today talking about
how we analyze the data and how we
measured the performance measures
we we performed and we'll close out
with any questions or or information
that you're looking for following
up on the presentation today.
Let's go to the next slide.
So the biometric and identity Technology
Center again is a group of subject
matter expertise with subject experts
within the Department of Homeland Security.
Our role is within the science and
Technology Directorate to provide objective,
quantifiable information to help
provide data to DHS components and
stakeholders about the technologies,
about how they work, where they don't work,
how to manage our risks associated
with these technologies, and also to
work back with industry and academia.
To identify issues with the
technologies to make them work better,
more effectively for potential
use in the future.
Next slide.
So our biometric technology rallies are
focused are basically a set of industry
challenges where we are orienting
industry to specific types of use cases.
In this case, what we've done for the
last several years have focused on
the challenges associated with high
throughput biometric recognition,
something where we have a situation
where we have a group of unknown people
we need to process them one at a time,
verify their identity within the
matter of a few seconds.
Because of the large volume of people who
are going through various types of high
throughput checkpoints and as an example,
the high throughput checkpoints are actually
abstracted out from notional DHS processes.
So, for example,
there are similarities to border crossings,
similarities to aviation
checkpoints to accessing critical
infrastructure sectors and more.
And ultimately,
what the goal really is is to screen
hundreds or maybe thousands of people
within the matter of a few minutes,
and we'll talk a little bit more about
what the goals were or how we expressed
those goals to industry so that they
could come up with effective solutions.
Here again,
our goal is to help define
these methods of measurement,
provide quantifiable data to DHS components
and stakeholders so that they can
perform an apples to apples comparison
about how technologies work or where
the state of the art currently lies,
not only in terms of matching.
Systems provide biometric
acquisition systems.
In this case,
space or iris cameras and also provide
actionable and useful feedback to industry
to make technologies work better.
And with that,
I will kick it over to Doctor Yevgeny
Strong can go to the next slide.
Good morning, everyone.
Thank you for joining us.
Thank you, Arun for the introduction.
The 2022, the 2021 Biometric
Technology Rally, the mark,
the 4th Biometric Technology Rally and
this year in 2022 we're running the
5th Anniversary Rally and since 2018
the rallies have demonstrated progress
in the performance and maturity of
biometric acquisition and matching systems.
The results of these rallies have provided
insights into how people interact.
With biometric technologies and
have led to improvements in system
usability and performance.
Importantly,
the rallies today have focused on one
specific use case and that's high throughput,
unstaffed biometric acquisition and
we continue to see new challenges to
system performance within this use case,
most recently with the introduction
of face masks which make biometric
acquisition more difficult even
when people remove them during the
biometric process.
Next slide.
Please.
So in the following slides,
I'll introduce the 2021 Biometric
Technology rally and go over the major
parts of the rally test process.
Next slide please. As mentioned.
Earlier, the 2021 rally was a test of
biometric systems within a high throughput,
unattended use case.
This test focused on the
face biometric modality,
but it did accept applications from
multimodal biometric systems as well.
We designed this rally so that companies
could demonstrate performance of their
systems with respect to efficiency,
effectiveness and user satisfaction
and what we mean by efficiency is the
ability of acquisition systems to work.
Quickly and effectiveness with respect
to the error rates that these systems
experience and finally satisfaction is
people's opinions of these technologies.
Our test approach,
which I'll describe later,
allowed for assessment of interoperability
between acquisition and matching systems.
And as the second rally
performed during COVID-19,
we also wanted to capture the improvement
in system performance for individuals
wearing face masks compared to the 2020.
rally,
from the very beginning the rallies
assessed biometric technology
performance with demographically
diverse groups of people to address
questions raised about technology
performance for specific groups.
We designed the 2021 rally to allow
reporting of metrics in a manner
that's dis-aggregated by race,
gender,
and skin tone to measure how well systems
perform for each group individually.
Next slide,
please.
So in order to qualify for the rally,
biometric systems had to meet a
number of technical requirements
which are presented on this slide.
For acquisition systems,
the following minimum requirements
held each system had to operate
in an unmanned mode, that is,
without any operator or instructor
to tell people what to do.
Folks were instructed by
the test team to come in,
come and try to use each system according to
whatever instructions the system provided.
All the systems operated within
a 6 by 8 footprint where you
know the system provider could
deploy the system as they saw fit.
This all systems were required
to collect face biometric imagery
and to provide one biometric probe
per test volunteer.
The probes had to be submitted
within a particular time limit.
That is,
during the interaction between the
volunteer and the biometric system
before they left the station.
We also had some functionality
requirements with respect to masks.
These systems were required to acquire
images from people wearing face masks,
so they have to they have
to be able to achieve that.
And as an optional requirement,
systems could provide images from
iris or fingerprint modality.
And as you'll see,
we had one multimodal face Iris
system in this test.
The requirements for matching
systems are on the right.
These matching systems were provided
as sort of black box algorithms.
They were packaged inside a docker
container which was deployed for
evaluation on our local cluster.
These systems have to be commercially
available.
Actually,
that was the same as for acquisition systems,
where we're encouraged.
This is a test of commercial technology.
There was some limitations
regarding the size of those images,
1.5 gigabytes, maximum in size.
And they had to be performing
the biometric match operations
in a timely manner.
Work critically,
they have to work without
access to external networks,
so these were not we would not
consider Web hosted API because images
gathered at the MDTF stay at the MDTF.
We had a specific swagger specification
which we asked these systems to conform to,
and that's hosted on our
website github.mdtf.org.
And of course the matching systems
were also like the acquisition
systems required to be able
to perform all operations,
including probe including template
extraction and matching on
probe images that are required
with people wearing face masks.
The matching was performed with
face Mask matching was performed
based on probe images.
required with face masks relative
to reference images which were
acquired without face masks.
So next slide please.
So all rally participants were
evaluated by a panel of experts
from government and industry,
and there are a number of government
organizations represented.
You could see that on this slide.
In the end, 5 acquisition systems
and 13 matching systems were
selected to participate in total
of the five acquisition systems,
one was a multi-modal face Iris acquisition
system and three of the matching
systems were iris matching systems.
The performance of acquisition.
And matching for Iris was performed
separately than that for face.
We did not attempt to do any
fusion or any joint multi-modal
performance in this assessment,
and I'll report on.
We'll report on the results
for Iris separately,
each system was given a unique alias,
which in this for this evaluation was
inspired by US rivers and mountains.
Next slide please.
So the 2021 rally was announced
in May of 2021 and the technology
providers had two months to
submit applications to participate
following conditional acceptance,
technology providers had two months
to develop their systems for the test,
which included integration with
cloud hosted MDTF API's prior to
arrival at the test facility and
this significantly decreases the
number of issues that we observe
during the test process prior to
testing all acquisition system
providers installed their systems.
ATF and participated in a
VIP day with stakeholders.
The acquisition system testing was
then performed between September
29th and October 15th of 2021.
Next slide please.
So this image helps visualize
the overall rally test process,
which starts with a box #1 on the very left,
which is informed consent.
In total, we had 601 diverse people
from the local area briefed about
the rally and who consented to
participate in this evaluation.
And we could go to box
#2 for each participant.
We acquired ground truth information
regarding their self reported
demographic characteristics including
their gender and their race,
as well as you know,
ground truth measures of their phenotypes,
which in this case included
our assessment of skin tone.
And I'll have a little bit
more to say on that later.
The reason why we included this
phenotype in our assessment is that
our research shows that skin tone.
Is related to face recognition
performance and skin tone values are
measured using a specialized color
colorimeters and these values are,
you know,
are uniform and and can translate
across different definitions of
demographic categories that may exist
in different regions of the world.
So on point #3,
we performed acquisition first
without any face masks and so you
can see here what's represented in,
in, in,
in the third box is we had a
number of acquisition systems,
in this case five.
Each volunteer used each one of the
systems in a counterbalanced order
and all the acquisition systems
have to do is return to us a probe
face image from each volunteer.
First we did that without
masks and then in Box #4,
we asked volunteers to keep their masks.
On while using the acquisition systems
and then we tested the ability of these
systems to perform the same function,
now with masks for each individual.
The results these these probe
images were sent to the matching
system evaluation portion,
which is box #5 where the matching
systems had to find the face in each
photo and compare this face to known
people to identify the person in the photo.
In this case,
we had a gallery of 500 individuals
and a total of these ten face
matching systems were evaluated
this way and the results the
identification results were then.
Combined and the way we're going to
report performance throughout this
test box #6 is by combinations of
acquisition and matching systems.
So we had five camera systems,
5 acquisition systems in the evaluation,
10 matching algorithms and the
evaluation that's a total of 50
different possible system combinations.
That that we're gonna take a look at.
And the reason why we do it this
way is to be able to understand
the impact of camera and algorithm
separately on biometric performance.
And you'll see those results.
So next slide please.
So at this point I'm going to
turn things over to John Howard,
who's going to kick off the
explaining the rally results.
Great. Thanks you. Kenny.
Yes. So now comes, you know,
sort of the fun part.
What did we learn about all these
systems from executing that process
that Annie just described first,
just a couple of high levels of
framing points to keep in mind as I
sort of move through these results.
Some of these evaluation metrics just
apply to acquisition systems. So namely,
you were talking efficiency and satisfaction.
And we also only evaluated those metrics.
Volunteers weren't wearing masks.
You know, I recognize some of you
might think it would be a very
interesting data point to compare, like,
satisfaction across some people were
wearing masks and they weren't, but.
Sort of bound the analysis in some way,
and that was one choice we sort
of made was not to report.
Speed and satisfaction with
and without masks.
Effectiveness, on the other hand,
is not just an acquisition metric.
This is something we're trying to highlight.
Last couple of years,
effectiveness is so dependent on
a combination of an acquisition
system and a matching system.
And so the unique combination of those things
will have unique effectiveness measures.
And then also for effectiveness,
since it was a focus of this year's rally,
we did consider and volunteers were wearing
masks and they weren't wearing masks.
And then we've also continued to
report effectiveness out sort
of in these two different ways.
The first people call operationally focused.
This includes all sources of air
that can sort of happen with the
acquisition or the matching system.
So that's maybe the acquisition system
than sending an image with the matching.
Process it or maybe did processing
it didn't match back to that
subjects historical images.
We think that operational focus within
its name is a little bit more relevant
to what improved like HSBC and airport.
But then we also report on matching focus
effectiveness which were discounts.
Yeah, there's nobody.
An acquisition system to send
them the matching systems can't
really affect that at all,
and so that gives us a better picture of
how matching systems complete each other.
And then lastly,
how we report on demographic differentials?
This is.
Currently a hot topic.
I think in the biometric world
and the different groups taking
different approaches and it sort of
answers the question of how should
we take the performance of Group A.
And the performance of Group B and
say you know, are they the same?
Are they different?
We did this in a way that we think makes
the most sense in the context of the rally,
which is to compare those two
performance values to the goal and
threshold values that we've had now for.
Years I've got a little bit more on this
towards the end about why we did this way,
you know, some good properties,
that's kind of conceptually
and mathematically,
but also some of the approaches
of considered.
OK,
Next up.
This is like 14.
Moving the actual results.
The first efficiency or speed.
This is the average transaction time.
Again per acquisition system.
Higher on the bottom on the
right here was worse. Faster.
The goal for this year is 4 seconds.
You can see that with the solid
red line towards the bottom.
And then you can see that three
out of the five acquisition
systems and that that goal,
the fastest being acquisition system longer.
******* just under 3 seconds per volunteer.
And then the system granted and.
We're sorry, system parading system away.
Also make that 4 second goal and then
we also have a threshold value 8 seconds
which is some grand and that kind of
see that with the solid and halfway
filled in circles at the bottom.
Next slide.
Satisfaction for.
Looked a little bit higher
on this slide is better.
So we want to be towards the top.
You want to have lots of people that
are satisfied with your system.
This is measured by the percentage of
people that pick either happy or the
very happy option after me after using.
The acquisition system again here
you can see really good results.
Almost all the systems fortified
met the goal, which is 95%.
That's solid bar at the top.
That was the large long grain of them,
48 at the highest,
being granite at 97.8% of all.
They're happier. They're very happy
after interacting with the system.
Next slide.
Oops, yeah, there we go.
So moving into effectiveness,
this is where we're going to
start to see a few more numbers.
And that's because of what I
mentioned a few slides back.
We're not taking into account.
Combination of matching
and acquisition system.
We also have the mask and no mask
convention and we also have the
operational versus matching system focus so.
I'm going to go through these one
at a time to explain the chart.
I'll mention that all of these,
including the last two,
are at the bucket over right now.
So if you don't quite get every
number and get every trend,
don't worry, you can go.
Back to the site and sort of
peruse these at your convenience.
So how to read this is going down the road.
You'll find the 10 face matching
systems all aliases just getting
mentioned going across the columns,
you see the five acquisition
systems and then the cell you
have the value for the metric.
And on this chart that metric is
operationally focused trade notification
rate with no masks as you see
sort of on the left hand side there.
And so that operational folks needs including
things like that to submit an image.
Then we added a little color coding
to make this easier on the eyes and
we added a little marker which is
a diamond to sort of highlight the
highest metric in each category.
For this you can see the highest was 99.7%,
opportunity,
focus, trade,
unification rate that was achieved
by two matching systems,
sun and paint,
both with the images that were
required of acquisition system
for our stuff in the paper.
Next slide.
OK, so same general layout here.
We're just changing the metric so.
We're going to stay without
masks as it says on the left,
but now we're essentially what we're doing
is ignoring an acquisition assistance,
failure to submit an image.
So the first thing I know,
this is a lot more green and several
more diamond symbols for this metric,
the highest tier is actually 100%
and that was achieved by a couple
different system combinations systems.
So that with three acquisition
systems system painted with two
and system design did it with one.
So that means those matches.
System we're able to correctly identify
100% of the actual images that came from.
Are those particular acquisition systems
So we thought that was.
Next slide.
OK, so now I'm putting masks on.
You'll probably need to notice
there's no green on this chart.
I'll also point out that this chart goes
back again to including acquisition error.
So the presence of masks made
this unattended high throughput.
Scenario more difficult both on
the actors and matching side to
the point that no rally system
combination met that objective
threshold objective we had set at 95%.
So our highest performer.
with mask with presence,
I think it's right in the middle there.
It's system Mazon along with
acquisition system wrong and he
was able to identify just 93%.
Of the volunteers that transition,
that system with masks.
The next slide.
OK. And then finally with masks,
but now ignoring the failures of the
acquisition system to send an image,
we get the following,
so more green than before,
but certainly less than on that chart 2.
And there's also no.
Multiple instances of that
system combination reaching
that 100% performance mark,
our best was assistant paint assistant
period 99.6% of subjects have
submitted images with masks identified.
So still pretty good.
But not as good as those 100%
marks that were sort of across
the Board on Chart 2.
OK, you can go to next slide.
So those are sort of the wrong results.
As I mentioned, those are on the
website the next couple of slides,
including some of the things that are
present sort of move off those raw results.
And so try to provide some insights,
some of which may have been obvious
if you were paying attention as I ran
through those last couple of charts,
these actually aren't up on the website,
but we will put the slides up.
A little bit later this week,
so you can go back and reference them.
So number one, what this is showing is that.
We think Image acquisition is really
driving the error rates and the biometric
systems as we test them today, so.
How we read this chart on the right
is that across the bottom X axis is
sort of the percent of errors that
were acquisition at those to submit.
Across the Y axis you have the
percent of the areas that were
matching theirs subsequently failed
to extract one match sample,
and so if you are below this
diagonal see in the middle,
it means you had that system combination
which is a dot on this chart.
Had a harder time acquiring
images into the matching images.
And you'll see without masks
that most almost all of the dots
should have below that diagonal,
and as soon as you put masks on,
it's almost universally true.
There's a couple outliers up at the top
for a matching system didn't appear to
work very well this year with masks,
but every other system that to reasonably
work with masks is to load that dock.
And so we think that this is evidence
that the best way to reduce error
rates and scenario test is going to
attend the higher throughput cases.
It's for technology providers that sort
of focus on getting high quality image
error rates that are much higher than
the ability of these matching systems
actually match the images, right.
These matching system error rates
are sort of.
One in thousand,
One in a million kind of grades was
the acquisition system error.
It's going to be.
3 4 5 percent.
So that's kind of a recommendation
we had in the past,
but I think affairs repeating that
we sort of see that year after year.
We think there's still an opportunity
there for technology providers.
Next slide.
This one also probably was obvious,
but we wanted to sort of put
it all in one spot and people
have the slide is showing us.
It's mass cause issues for face
recognition masks of those reduced tier,
so effectiveness measure.
Including acquisition rates,
that's the annual AUC there.
Every dot on this chart is a system
combination with its tier metric laid
out and then the numbers at the top
are the ones that crossed the threshold
values of 95% and see that without
masks 26 system combinations did.
But as soon as we put masks on,
none of them were able to and then
the red bar is just sort of
that median performance number,
which is kinda delineated the left,
you can see that.
Mass caused about a 10% reduction
in during median tier for us,
about a 5% reduction in media matching tier,
which is a canopy.
Less of an impact on matching 2.
That's still an impact.
So we think that's evidence that
there's some opportunities to
do stuff on the acquisition side
and on the matching side,
sort of a note of optimism in all of
this though is that when we went back
and sort of pulled the median tier
from the 2020 route and that was about 77%,
so.
Even though we're still,
you know, not perfect,
we're not seeing this with 100% numbers.
We see without masks about 10
percentage points better than
we were just see where it go.
So this opportunities
continue to serve improve.
Your systems and then again if it
wasn't obvious from that last chart,
this should also help make it sort
of reinforce this point is that most
errors with face masks were due to
issues getting that in the drudge.
They're not getting it at all.
They're getting a poor quality image.
I think we can go to the next item.
I think I'm turning it
back over to you again.
It's going to talk about demographics.
Thanks, John.
So John's provided us an overview
of the overall results for the full
group of 601 diverse volunteers.
I will now go over the disaggregated
performance results focusing on our
measure of system effectiveness,
which is going to be tier we
disaggregated tier performance for
each of eight demographic groups
based on self reported race.
We had three groups based on race,
those volunteers that self reported
as Asian as black or as white.
We had two gender groups.
Those that self reported as male or
female and we also performed the segregation.
Based on our measure of skin tone,
where we separated participants into
3 tertiles and I'll explain more on
that how we did that later into lighter
medium and darker skin tone groups.
Next slide please.
So a little bit about skin tone.
We assess skin tone based on
direct measurement using a
calibrated colorimeter device.
Our research has indicated that skin
tone is an important correlate of
face recognition system performance,
but cannot be reliably estimated
from biometric images themselves.
So we perform an independent measurement
under controlled conditions for the rally,
we measured the average skin tone
taken from 2 temples from each.
Volunteer on.
You can see the schematic on the
right which shows where we took
the measurement just above the face
mask and the device that we used.
The rally scenario tests measured
skin tone using this instrument
and identified people with darker,
medium and light skin tone and the
chart on the right bottom helps you
visualize that this chart shows
the CIELAB color values for human
skin measured within this test.
Each dot on this chart.
Represents a single individual and
the hue of the dot represents the
actual color reading from the device,
and you could see that these
readings form a Crescent within the
LB plane in this color space.
This the L part of this color
space actually corresponds
to the perceptual likeness,
so this L star value is what we used
to split into tertiles and the first
tertile was the those on sort of
the bottom of this spectrum with a
darker skin than we split it off.
To the middle tertile,
which was going to be medium skin
and the lighter tertile was folks
that were separated out into those
with lighter skin.
And that's how we did that split.
Next slide please.
So this slide shows.
A visualization of results based that's
similar to what John presented earlier.
So you can see that there are matching
systems on the along the Y axis and
acquisition systems along the X axis,
and what this chart shows is a
summary of the performance overall.
And as I mentioned previously,
we had eight different demographic
groups that we had in the rally,
and this visualization shows in
green when a system combination
met the performance benchmark.
The goal for each demographic group.
The shading in yellow is when the system
met system combination met the threshold.
The rally threshold 95% threshold
benchmark for tier for each
demographic group and those that
are not shaded green or yellow means
that that system failed to meet the
threshold rally threshold performance
for at least one of the groups.
And what you could see in panel A
is performance with respect to true
identification rate inclusive of both
acquisition and matching errors.
So all errors within the system and
the chart on the right shows matching
tier which is basically all errors,
but discounting errors where the
acquisition system simply did not
deliver an image for assessment.
And what you could see on the left side
here is that for a good number of.
Uh systems system combinations.
These were able to meet the 95%
tier threshold for all demographic
groups in the assessment,
and you could also see the relative
importance of acquisition and matching by
looking at sort of the shape that this,
this block of successful systems
meets the shape that it makes where
you could see that for most.
Matching systems with acquisition systems,
herard and long were in fact able
to meet this 95% benchmark for
all demographic groups.
So all but system sun.
And and So what we see is it looks
like it's relatively more important
which acquisition system is employed
rather than which matching system with
respect to meeting meeting these raw
performance benchmarks and some of
the information that I'll show you next,
we'll delve into more details
on the matching tier side.
If we discount the errors and in just
acquisition of the biometric images,
you can see that both acquisition and
matching is important for meeting the goal.
For all groups and you could see
that because the Green Zone forms
a sort of a triangle shape on this
chart you can see that some systems
were particularly effective.
Some matching systems in particular systems,
paint and salt were able to meet
the goal for all groups so long as
images were acquired and they were
able to do that with all acquisition
systems with safe system to cocoa and
and so that that sort of highlights
that the importance of that.
However,
there are some acquisition
systems that also worked.
Very well,
across a variety of matching systems
on this front and that system
or a one that acquired an image,
that image worked with five of the,
you know half of the matching systems.
Let's go to the next slide.
We'll, we'll break things
down into a bit more detail.
So this slide shows a slightly different
visualizations of the results,
but focuses on gender, in particular.
In these violin plots that I'm showing you,
each dot represents the performance
of a single system combination.
The numbers above each plot show
the number of system combinations
meeting the 95% rally threshold.
So for tier without masks,
that's going to be panel A more
system combinations met the rally.
Your thresholds for male,
then for female volunteers.
There was a trend toward
better tier for males,
which was also observed
with masks in panel B,
but no system at the tier
threshold with masks.
So you could see that the violins are
squarely below that Gray shaded area and
and you have zeros above each violin.
For matching without for
matching tier without masks,
that's going to be panel C
performance was overall very good.
Most system combinations met the matching
tier threshold for both males and females.
There was hardly any difference.
45 systems met it for females
and 47 for males.
Matching tier remained very high with masks,
unlike a tier and a number of systems
were able to meet the rally threshold.
Panel.
So that's panel D and again we
saw similar performance with
masks for males and females.
It's 29 systems met the rally
threshold for females and 28 for males.
So what these results suggest is
that most of the differentials
based on gender in this scenario
were associated with errors in
image acquisition not matching.
So you know this difference in
tier without masks 20 versus 35
systems goes away when you look at
it without with with matching tier.
Next slide please.
So this slide uses the same
visualization approach as before,
but the disaggregation is now
by self reported race group.
Again, we had volunteers that
self report as Asian as black or
African American and as white.
So for tier without masks, again panel,
A more system combinations met the
rally tier threshold for volunteers that
self reported as white and fewer for
volunteers that self reported as black,
only 18 versus a 33.
The trend toward lower tier for volunteers.
Self reporting as Black was also
observed with masks in panel B.
Interestingly, however,
the performance volunteer self
reporting as Asian remained high
even in this challenging condition.
For matching tier without masks panel
C performance was overall very good
and most system combinations met the
matching tier threshold for all groups.
Again,
matching tier remained high with
masks and panel D with hardly any
decline for volunteer self reporting
as Asian and these results suggest
that without masks acquisition was
the main contributor to demographic
differentials based on race,
whereas with masks both acquisition
and matching affected the number of
systems meeting. Rally benchmark.
Next slide please.
So this slide again repurposes
the same visualization,
but now we're performing disaggregation
based on skin tone tertiles.
And I remember that T1 corresponds to the
tertile with the darkest skin tone T2
for medium and then T3 for the lightest
skin tone for tier without masks,
which I'm showing you in panel,
a more system combinations met the rally
tier threshold for volunteers self reporting.
Sorry, volunteers with lighter skin T3.
That's 36 systems and fewer for
volunteers with darker skin T one that's
18 system this difference between 18
and 36 was the largest difference in
performance observed in the rally,
and the trend toward lower tier for
volunteers with darker skin was also
observed with masks in panel B,
performance volunteers with medium
skin tones,
many of whom self identified as Asian,
remained high even with face masks.
For matching tier without masks panel
C performance was overall again very
good and most system combinations met
the matching tier threshold for all groups.
Again matching tier remained
high with masks for many system
combinations and panel D however,
fewer system combinations met the matching
tier threshold for volunteers with darker
skin than for the other two groups,
so suggesting that there's still in effect
there and these results overall again
show that without masks acquisition.
Was the main contributor to demographic
differentials based on skin tone.
However,
with masks,
both acquisition and matching
affected the number of systems
meeting the Rally benchmark for
people with different skin tones.
We're currently investigating
the complex results observed
in the presence of face masks,
so it's an interesting effect.
Next slide please.
So these last few slides that I showed
were pretty information dense here.
I'm going to summarize the key takeaways
about the demographic differentials
and face recognition that we observe.
So first,
some system combinations were able
to meet the rally to your threshold
for all demographic groups.
This threshold was chosen before
we started the rally execution to
represent the performance level
deemed suitable for this use case,
namely the high throughput
unattended system performance,
a biometric system that meets this threshold.
All groups would therefore still be
deemed suitable for the use case.
According to this benchmark,
even if the user population is
homogeneous composed solely of
members of that one group,
and in this test 18 of the 50,
about a third of the acquisition
matching system combination met
this stringent criteria.
So choosing a system combination
without testing may not yield
equitable results for all groups.
If you're just picking out of a hat.
2nd despite high performance by some systems,
we observed some demographic differentials
in median system performance for
different groups that did align
with pre-existing fairness concerns.
We saw that median system performance
without masks was lower for
female than for male volunteers.
Performance was also lower for
volunteers that self identified
as black or African American and
differed most based on skin tone
with lower performance for people
with darker skin as
compared to lighter skin.
It's important to note the
scale of these observed.
Differences without masks differences
in tier across groups were generally
below 6% and below 1% for matching tier.
Again highlighting the importance of image
acquisition and shaping these differentials.
The differences grew in the presence
of masks now falling at 10% or
below for differences in tier
and below 4% in matching tier.
So this complex interaction shows that
equitability of a biometric system may
change when operational conditions change.
Obviating the need for operational
monitoring and testing.
Next slide please.
So so far I've been discussing the
performance of face recognition systems.
However, the 2021 rally included
one multimodal face Iris system and
three Iris matching systems as well.
And this slide is intended to summarize
the observation for these systems,
so I'll make a few points. First,
no system combination with Iris met the
rally threshold for any demographic group.
Performance was generally
fairly low compared to face,
unlike for face recognition.
However, masks did not appear to reduce
median iris recognition performance.
You can compare without masks,
the median tier was 74.4 with masks.
The median tier was 77.3,
so very close, and again similar.
Similarly, close results from matching tier.
We did observe some demographic
differentials for iris systems.
However, these were distinct from
those observed for face recognition.
Without masks, performance was lower
for male relative to female volunteers.
So the opposite result.
And it was lower for volunteers
that self identified as Asian.
In general,
the size of the differentials was comparable
to those observed in face recognition.
However,
we saw nearly a larger differentials
min matching tier for IRIS as
compared to face without masks.
And at this point,
I'll hand it off to John Howard to
discuss our overall conclusions and
our efforts on standardizing testing
and measurement of demographic
differentials for biometric systems.
Thank you, Yergeniy.
So we've kind of thrown a lot
of information to you today.
So we wanted to just leave you
with a few high level conclusions.
You can sort of take back.
You just have to walk away with a few points,
so the first is.
When the main, passionate,
sometimes people to deride
biometric system to slow or clunky,
and no one likes to use them,
the actually fairly good data on
this sort of the year after year,
that's kind of indicates the opposite, right.
And most of these systems are very fast.
And the satisfaction
rates reasoning very high.
So you don't seem to mind
interacting with them in this model?
2nd. At this point,
you know the biggest area
of improvement that we see.
It's possible right now in these systems.
Is to make improvements in the
acquisition process and I know this
sort of goes against the current trends
and computer science or technology
companies in general doesn't want
to focus on our more complicated
CNN's with larger training sets.
But where the performance is matching
systems currently is the gains to be
had there still especially for the
best ones are really on the margins.
We talking about small improvements point 1%,
point 1%.
But acquisition error rates going
to have a supportive and I would be
higher than that right up percentage or
maybe as high as five percentage points.
So getting better qualities of
optics and feedback mechanisms.
The acquisitions systems we think could
really help through the overall process.
3rd you know encouragingly mass
performance got better in 2021 as
compared to 2022 and not by sort of
the trivial marginal percentage
improvement of about 10%.
We think this is a sign that they're
still out there and acquisition
improvements if could be made,
you know,
right now to focus on the mask scenario.
But this may be sort of a hair burner.
Other long standing problems
with face recognition said
mainly here basic collusions.
Can you just made this point,
but without masks?
Sort of differences in performance?
Median system performance across the
different groups for less than 6%,
less than 6% of the nation,
as were largely driven by failures
to submit images for certain
demographic groups not driven by.
There's not certainly graphic groups.
And then without masks and the best
systems can become pretty well across
all those groups that have been
maintained overall tier of 98.4.
For each group tested male, female.
Skin profiles as well.
And iris?
Right behind some of the face
recognition performance.
But we didn't have that many Iris
acquisition systems or matching
systems related to face this year.
That's also just sort of the function
of the high throughput scenario as well.
Next slide.
Yeah. So this is a fun new addition for
this year who was interactive results
look right down through the website that
you're seeing is sort of a scrolling.
Series of how you can manipulate
and play with this data on your own.
Some of these slides which screenshots
haven't put in there as well.
Would you could do things like see
overall trade interpretation results.
For example you can see them
just for males just for females.
Just for the race categories and toggle,
it just didn't seem better
compared with any of that masks.
It's a lot of numbers,
but we thought this was a good way for
people to sort of choose their own
adventure through them and find the
data points that were important to them.
In addition to viewing the data this way.
You can see the video,
there's also a downloadable PDF report with
one giant table that has all the numbers.
I will warn you,
it's over 1600 total numbers and
so the tables 24 pages long,
but there's really some sort of
analysis or piece of information
that is important to your group.
I'd encourage you have tried to
make as much data available for
the possible so that people can
answer their own questions.
Except.
Oops. OK, perfect.
And then so finally we wanted to suggest
this kind of demographic testing.
So the past couple of years we've
been involved in trying to distill
lessons learned from doing this kind of
testing and to international standard
so that others can sort of execute and
learn from our lessons learned that
standard is ISO/IEC 19795-10.
It's currently in working drafts
through the title of the standard.
John, biometric performance testing
across in the graphic groups.
It deals with things about how
do you find infographic groups,
including some of the things that we
need just talked about with skin time.
What do you need to consider when planning
an assessment of demographic differentials?
And then once you've executed a study
like the one we just reported on,
how do you calculate error rates?
And so we report those back and I think
to sort of highlight some of the topics
we discuss in that group and example here.
What we're showing is.
At the top of the table.
The median tier for males and rather
than 97% in the meadian tier for females
for another 4% and so the next question
is sort of ask yourself is how do you,
what do you do with those numbers?
How do you report them out different things
that are proposed by different groups?
One option is obviously just subtract them
and say that's a difference of about 3%.
Another option is the ratio difference.
6% errors over 3% errors.
That's about two times difference
in the amount of errors experienced
from that she knows which is males.
Those are good options and certainly
things to be considered,
but I think you lose something there.
That's the magnitude so.
6% error rate over 3% error rate.
The difference of 2X.
That's the same thing.
Come to the same conclusion with that 10%
error rate or 20% error rate over 10% rate.
So you sort of lose where you are relatively.
And so for that reason,
and also just for historical reasons,
just sort of how we reported
value results in the past,
we chose this bottom option this year,
which is we just reported.
Error rates for success rates of
different groups relative to our rally
goals and thresholds and 95% and 99%.
Seeing this scenario,
you would have found a difference right
tier from males is leading that 95%
threshold while the tier for females is
not short of arrived at the conclusion there.
And a lot of other demographic groups,
that wasn't the case.
We think this is sort of easy to understand.
Some of the other ones are using as well.
It doesn't do as good a job capturing
the magnitude of the difference,
especially if you have league thresholds.
If you're threshold success rate,
you're looking for something like 80 would
have one group of 81 and one group of 99.
And you might not capture
that approaching it this way.
But this is how we sort of reported out
in the rally and open to feedback on
this and these kind of topics about
how you do some of these things I just
discussed are of interest to your group.
Please consider contributing to the
standard development in your eye.
And let you know to get involved.
We're certainly looking to have an
inclusive group of people with lots
of ideas and thoughts on this one.
Alright, next slide.
With that, I'm turning it
back over to Arun.
Alright. Well, thank you so much, John.
I appreciate that and thanks you Yegeney .
Uh, you guys did a great job presenting
some of the results that we had
from the rally result rally in 2021.
Umm, so a couple of updates here about
the the next biometric rally we just
announced this a couple of weeks ago.
We're still focusing in on the
unattended high throughput use case
for the acquisition systems.
And the thing that we're going to
be making a little bit different
here is actually what we tried to
do back in I guess 2020 pre COVID
which was moved to small,
you know group processing being able to.
Process groups together without having
to artificially force groups to kind
of go through the checkpoint 1 by 1.
So it still has this high
throughput aspect to it,
but the intention is to be able
to process people as they travel.
With friends, with colleagues,
with families,
so that you don't need to have children,
go through separately.
Adults go through separately.
It all works together.
The focus also will be on only the
face biometric modality at this point,
given the reduction or the the limited
performance we've seen with Iris so far.
I think what we're open to changing
that back up again in the future,
but right now it looks like the the
focus really should be on on trying
to make sure that we can do this
evaluation with face at a minimum.
Umm.
I think the important things we
want to talk about here are we'll
talk about a few different things.
We want to focus in on.
Umm, you know again efficiency,
effectiveness and user satisfaction
with this biometric acquisition systems,
you know being able to look at how well
images acquired with an acquisition
system will work with different matches.
This happens when you might have
a large enterprise or a large
organization that has different matchers
or maybe different matchers that are
configured slightly differently.
We want to make sure that the matches
can take in imagery from a lot of
different types of collection devices.
Typically a lot of cameras that get
deployed will be different from one another.
One organization might buy some or
even one location might have a few,
or they might have a different
number of other collection devices.
We want to make sure we understand
how well the matching system and the
matching process works regardless of
the cameras that might be submitting images,
we plan to continue to disaggregate
performance based off of.
Characteristics of demographic
characteristics like race, gender, skin tone.
And we're also going to include
these concepts of privacy measures,
and this is kind of the idea of when we,
when there are, you know, again large,
high throughput scenarios,
there are often large groups of people
and not all of those people may have
opted into the face photo collection process.
So you might have people in the background,
you may have people going
through parallel lanes.
So the intention here is also
to not only make sure that we
can evaluate how well systems.
Or acquiring photos of people who
are within the field of view or
within the intended capture volume
of the biometric acquisition system,
but also make sure that they're not
collecting photos of people who have
not explicitly opted in
and might be walking parallel or
close to the acquisition system,
but not explicitly in the capture volume
of that specific acquisition system.
So in this case,
opt in is kind of assessed based off of,
you know, your physical location and
proximity relative to the camera.
You know,
there would be a specific type of captures.
If you're interested in learning
more about the rally for 2022,
please go to mdtf.org/rally 2020.
You can also e-mail us at people
screening at hq.dhs.gov we have,
we have, we've done our announcements
webinar at this point,
but we will be.
Having our technical webinar in a few weeks,
and if you're watching this on
a recorded on a recorded video,
chances are we've already had that
webinar so I would definitely
recommend that you go to the website
mdtf.org/rally 2022 to learn
more about the rally and see if
it might be appropriate for your
organization to participate.
For our stakeholders who are
not technology developers,
please let us know if you're
interested in participating.
We are intending to do VIP
day where we can bring.
Groups of VIPs through see
the systems and operations,
see how they work and get the
chance to meet with some of the
vendors who are making these
new and and cutting edge kind of
technologies and capabilities.