Biometrics Evaluation Centre (BEZ)
BEZ - Experiments and Analyses
Introduction of long-term test series
Since May 2022, the BEZ has been conducting regular long-term test series to examine biometric systems. Three times a week, both complete systems and individual components, such as sensors, are tested under realistic conditions. The objective is to evaluate the performance and usability of these systems and thereby identify opportunities for optimisation
The tests are conducted with registered participants. The current pool is composed primarily of university students and staff. However, in pursuit of greater diversity and more robust results, additional participants are continuously recruited. Further information on the procedure and registration can be found under Long-term test series.
Core research areas
In addition to investigating performance and usability, the BEZ is also engaged in other areas of research outside of the long-term test series:
- Resistance of biometric systems (presentation attack detection, PAD)
- Comparison of algorithms (commercial vs. open source)
- Influence of environmental factors and recording angle
- Comparison of real vs. synthetic data (face, finger)
Current experiments, e.g. on the spectrometer, with the LokiMk2 sensor or the time-of-flight camera, are making an important contribution to the further development of PAD methods. They form the basis for further investigations into PAD, for example with masks, prints or digital deepfakes.
Active experiments as part of long-term evaluations
- K13 multi-camera system: The K13 system simultaneously captures high-resolution facial images from 13 different angles. This standardised environment enables precise analyses of the influence of viewing direction, lighting and facial expressions on recognition performance.
- Finger scanner: Various types of finger scanners (optical/TFT, optical/FTIR, OCT/3D) capture high-quality fingerprints – both flat and rolled.
- Spectrometer: The spectrometer measures the reflectance spectrum of the skin in the visible and near-infrared range. This allows real skin and artificial materials such as masks to be reliably distinguished – regardless of skin type. The information obtained serves as a reference for the development of biometric PAD systems.
- Near-infrared point sensor (‘LokiMk2’): The LokiMk2 is a highly specialised sensor used to detect presentation attacks. It uses near-infrared light to detect skin reflection patterns, reliably distinguishing between real skin and fakes such as masks or prints.
- Remote photoplethysmography (rPPG) with time-of-flight camera (ToF): In this experiment, a ToF camera is combined with remote photoplethysmography (rPPG) to measure the pulse contactlessly via minimal colour or depth changes in the face. The aim is to identify reliable signs of vitality in order to strengthen defences against deepfakes and presentation attacks.
- Automated border control gates (ABC gates): The ABC gates simulate automated border control systems such as those used at airports. Here, the reliability of facial verification under realistic conditions is tested – e.g. in changing lighting conditions, with head coverings or depending on the person's condition on the day.
Short-term experiments
In addition to long-term studies, the BEZ also conducts separate short-term experiments in which special biometric systems are tested under controlled conditions. Such experiments can also be carried out on request as part of collaborations with industry partners or authorities.
- 3D finger: Contactless fingerprint capture using optical coherence tomography (OCT). This opto-electronic process provides a 3D scan of the fingertip, which, in addition to the external fingerprint, also reveals an internal imprint at the boundary between the dermis and epidermis as well as sweat glands, thus offering significantly increased protection against forgery.
- Back of the hand: In this special study, the BEZ supported the Federal Criminal Police Office in investigating the suitability of the back of the hand as a biometric modality. The aim was to build up a high-quality ground truth database of around 200 hands. To this end, hands were photographed in different gestures from different angles and in different lighting conditions using a special recording stand at the BEZ.
Age and gender distribution
Long-term analyses of face recognition
The long-term analysis examined how stable the recognition performance of facial recognition algorithms remains over a period of approximately 2.5 years. The aim was to record changes in the face (e.g. due to ageing, hairstyle, beard growth) and to evaluate their influence on the biometric comparison score .
For the analysis, frontal images of the participants (camera 5 at the K13 system) were compared with the original reference image. Both commercial off-the-shelf (COTS) and open-source algorithms were used.
Delta score over time
This graph shows the change in the delta score relative to the first image for a COTS algorithm (y-axis: delta score, x-axis: days since original reference image). The delta score describes the difference between the comparison score of an image and the original reference image of the same person. Each point represents an image comparison of the same person. The red trend line shows a slight decline in recognition performance over time. This decline was observed across all algorithms. While the magnitude of this decline varied considerably across algorithms, its overall impact is negligible, thereby empirically supporting the 10-year validity period of identity documents.
Score distribution per algorithm and year
The box plots illustrate the distribution of comparison scores for all five algorithms examined over three years. It is clear that commercial algorithms achieve higher scores, while open-source methods show greater performance losses over time. However, a direct comparison of the detection accuracy of the algorithms is only possible to a limited extent, as the threshold value for each algorithm can be set manually.
Angle analyses
The K13 multi-camera system was used to investigate how the angle at which an image is captured affects the recognition accuracy of facial recognition algorithms. To do this, a frontal image of a person (camera 5) is compared with images of the same person taken from horizontal (x-axis) and vertical (y-axis) angles. This allows the isolated influence of the respective angle on the comparison results to be analysed.
The following two figures illustrate the results of one of the algorithms examined.
Influence of the horizontal viewing angle on the biometric score (x-axis)
The figure shows the distribution of biometric scores for K13 images taken from four different horizontal angles compared to the front-facing photo. The scores decrease slightly as the x-angle increases.
Influence of the vertical viewing angle on the biometric score (y-axis)
The figure shows the distribution of biometric scores for K13 images taken from four different vertical angles compared to the front-facing photo. As the y-angle increases, the comparative values decrease significantly and in some cases fall below the threshold value T = 0.755.
Kurzzeitexperimente
Neben den langfristig angelegten Studien führt das BEZ auch gezielte Kurzzeitexperimente durch, bei denen spezielle biometrische Systeme unter kontrollierten Bedingungen getestet werden. Solche Experimente können auf Anfrage auch im Rahmen von Kooperationen mit Industriepartnern oder Behörden durchgeführt werden.
- 3D-Finger: Kontaktlose Erfassung von Fingerabdrücken mittels optischer Kohärenztomographie (OCT). Dieses opto-elektronische Verfahren liefert einen 3D-Scan der Fingerkuppe, der neben dem äußeren Fingerabdruck auch einen inneren Abdruck an der Grenze zwischen Dermis und Epidermis sowie Schweißdrüsen sichtbar macht und so eine deutlich erhöhte Fälschungssicherheit bietet.
Handrücken: In dieser Sonderuntersuchung unterstützte das BEZ das Bundeskriminalamt dabei, die Eignung des Handrückens als biometrische Modalität zu untersuchen. Ziel war der Aufbau einer hochwertigen Ground-Truth-Datenbank von etwa 200 Händen. Hierzu wurden mithilfe eines speziellen Aufnahmestandes im BEZ Hände in unterschiedlichen Gesten aus verschiedenen Winkeln und in verschiedenen Lichtverhältnissen aufgenommen.
Weitere Forschungsgebiete
Neben der Untersuchung der Performanz und der Usability widmet sich das BEZ außerhalb der Langzeittestreihen weiteren Forschungsgebieten:
- Überwindungssicherheit biometrischer Systeme (Presentation Attack Detection, PAD)
- Vergleich von Algorithmen (kommerziell vs. Open Source)
- Einfluss von Umgebungsfaktoren und Aufnahmewinkel
- Vergleich realer vs. synthetischer Daten (Gesicht, Finger)
Die Experimente am Spektrometer, mit dem LokiMk2-Sensor und der Time-of-Flight-Kamera leisten einen wichtigen Beitrag zur Weiterentwicklung von PAD-Verfahren. Sie bilden die Grundlage für weiterführende Untersuchungen zur Erkennung von Täuschungsversuchen, etwa durch Masken, Drucke oder digitale Deepfakes.