Joint learning of assignment and representation for biometric group membership

Joint learning of assignment and representation for biometric group membership


This paper proposes a framework for group membership protocols preventing the curious but honest server from reconstructing the enrolled biometric signatures and inferring the identity of querying clients. This framework learns the embedding parameters, group representations and assignments simultaneously. Experiments show the trade-off between security/privacy and verification/identification performances.


noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt \name Marzieh Gheisari, Teddy Furon, Laurent Amsaleg, 1 \addressUniv Rennes, Inria, CNRS, IRISA, France
{ marzieh.gheisari-khorasgani, teddy.furon},


Group Representation, Verification, Identification, Security, Data Privacy.

1 Introduction

Group membership verification is a procedure checking whether an item or an individual is a member of a group. If membership is positively established, then an access to some resources (buildings, wifi, payment, conveyor units, …) is granted; otherwise the access is refused. Being granted with this shared privileged access requires that the members of the group could be distinguished from non-members, but it does not require to distinguish members from one another. Indeed, privacy concerns suggest that the verification should be carried-out anonymously.

This paper studies group verification and also group identification. In this later setup, there are several groups of members and one needs to identify in which group the user is belonging to. This paper focuses on privacy preserving group identification procedure where group identity of a member is found without disclosing the identity of that individual.

In computer vision, it is very common to aggregate signals into one representation  [16, 7, 11], but they do not consider security or privacy. For instance, in [6], Iscen et al. use the group testing paradigm to pack a random set of image signatures into a unique high-dimensional vector where the similarities between the original non-aggregated signatures and a query signature is preserved through the aggregation.

Recently  [2, 3, 4] proposed a framework based on aggregation and embedding of several biometric signatures into a unique vector representing the members of a group. It has been demonstrated that this allows a good assessment of the membership property at test time provided that the groups are small. It has also been shown that this provides privacy and security. Privacy is enforced because it is impossible to infer from the aggregated feature which original signature matches the one used to probe the system. Security is preserved since nothing meaningful leaks from embedded data [12, 13].

This paper revisits the core mechanism proposed by [3]. That work, however, is deterministic in the sense that it learns group representations based on predefined groups. This paper shows that learning jointly the group representations and group assignments results in better performance without damaging the security. This adresses scenarios where the number of members is too big. Their signatures can not be packed into one unique group representation with a technique like [3]. Therefore, members are automatically assigned to different groups. A light cryptographic protocol is deployed to secure their privacy during group verification.

2 Group membership

2.1 Notations

The embedding, the assignment, and the group representations are learned jointly at enrolment, and given to a server. Biometric signatures are modelled as vectors in . is the matrix of the signatures to be enrolled into groups. The group representations are stored column wise in matrix . The group representations are quantized and sparse i.e., with and , .

At query time, the user computes a sparse representation of his biometric signature . For that purpose, function maps a vector to a sequence of discrete symbols. We use the sparsifying transform coding [12, 13]: . After projecting on the column vectors of , the output alphabet is imposed by the ternarization function : The components having the lowest amplitude are set to 0. The remaining ones are quantized to +1 or -1 according to their sign.

2.2 Formulation of the optimization problem

Our group membership protocol aims at jointly learning the partition, the embedding and the group representations. The key is to introduce the auxiliary data , …, the hash codes of enrolled signatures and the group indicator matrix ( if is assigned to -th group). Then, the optimization problem is composed of a cost for embedding and a cost for partitioning :


The embedding cost is the loss for quantizing signatures:


The assignment aims at grouping together signatures sharing similar hash codes: the overall dissimilarity between members and their group representation is minimized while the separation between two groups is maximized. Inspired by Linear Discriminant Analysis, we consider variance to measure dissimilarity. The within group scatter matrix and the between group scatter matrix are defined as

where . The cost for partitioning is for some , in .

In the end, the objective function is formulated as:


The constraint on ensures that each signature belongs to exactly one group.

2.3 Suboptimal solution

The solution of (3) is found by iterating the following steps:

-Step. We fix , , and update by solving:


This problem is a least square Procruste problem with orthogonality constraint. By setting [14] shows that , where contains the eigenvectors corresponding to the () largest eigenvalues of and contains the eigenvectors of .

E-Step. Given , and , (3) amounts to:


We first find the solution relaxing the constraints and then apply ternarization function to obtain sparse codes:


(R,Y)-Step. When fixing and , the assignment and group representations are found by minimizing:


As is fixed, is irrelevant to , thus minimizing (7) is equivalent to:


Relaxing the ternarization constraint, (8) is solved by a k-means clustering algorithm, i.e.iteratively:

  • Update assignments: Each item is assigned to its nearest group representative.

  • Update centroids: -th centroid is the mean of all in group .

Then the group representation is found by applying ternarization function on -th centroid.

Figure 1: Performances comparison for varying group size . at for group verification.

3 Experiments

This section presents the datasets used in our experiments and investigates the performance of the proposed method for two application scenarios. We compare our scheme with EoA-SP, AoE-SP [2] and EoA-ML, AoE-ML  [3]. For the baselines individuals of each dataset are enrolled into random groups but for our scheme the algorithm learns how to partition the enrolled templates.

3.1 Datasets

Face Datasets

Face descriptors are obtained from a pre-trained network based on VGG-Face architecture [10] followed by PCA and then -normalization with .

LFW [5]. These are pictures of celebrities in all sort of viewpoint and under an uncontrolled environment. We use pre-aligned LFW images. The enrollment set consists of individuals with at least two images in the LFW database. One random template of each individual is enrolled in the system, playing the role of . Some other individuals were randomly picked in the database to play the role of impostors.

CFP [15]. These are frontal and profile views of celebrities taken in an uncontrolled environnement. We use frontal images to be enrolled in the system. The impostor set is a random selection of other individuals.

IRIS Datasets

Iris images are prepossessed by the following steps: iris localization, iris normalization and image enhancement. Then the feature vectors are extracted by Gabor filters.

CASIA-IrisV1 [8]. The database includes 756 iris images from 108 eyes of Chinese persons. The images stored in the database were captured within a highly constrained capturing environment. 3 images were collected in a first session and 4 images in a second session. The database is created by randomly sampling individuals to be enrolled, and impostors.

MMU2 [17]. This dataset contains images corresponding to people with different age and nationality from Asia, Middle East, Africa and Europe. Each of them contributes to 5 iris images for each eye. We exclude 5 left eye iris images due to cataract disease.

3.2 Group Verification

A user claims she/he belongs to group . This claim is true under hypothesis and false under hypothesis (i.e. the user is an impostor). Her/his signature is embedded into , and is sent to the system, which compares to the group representation . The system accepts () or rejects () the claim. This is a two hypothesis test with two probabilities of errors: is the false positive rate and is the false negative rate. The figure of merit is when .

Fig. 1 compares the performance of our scheme with baselines for group membership verification. Totally our scheme gives a better verification performance especially on CASIA. Since our method tries to simultaneously learn group representations and assignment, it aggregates similar embedded vectors and this looses less information.

Note that, although LFW and CFP are difficult datasets due to the ”in the wild” variations, the group membership verification task is handled well even for large group sizes. This is not the case for iris datasets. As mentioned before, we make use of VGG-Face for face datasets while for iris, traditional feature extraction algorithms are used. So, the big difference in overall analysis shows how the feature space affect the performance of group membership tasks.

3.3 Group Identification

The scenario is an open set identification where the querying user is either enrolled or an impostor. The system proceeds in two steps. First, it decides whether or not this user is enrolled. This is verification as above, except that the group is unknow: The system computes , , and accepts if the minimum of these distances is below a given threshold . The figure of merit is when .

When , the system proceeds to the second step. The estimated group is given by . The figure of merit for this second step is or the Detection and Identification Rate .

Fig. 2 shows that our scheme brings improvement compared to the baselines and the improvement is also better as the size of groups increases.

The impact of the group size on DIR is illustrated in Fig. 3. Obviously, packing more signatures into one group representation is detrimental. It gets worse when the queries are not well correlated with the enrolled signature.

Figure 2: Performances comparison for varying group size on group identification for CFP(left) and LFW(right). at for the first step of group identification (solid) and for the second step of group identification (dashed).
Figure 3: The Detection and Identification Rate () vs.  for group identification on CASIA-IRISV1.

3.4 Security and Privacy Analysis

A curious server can only reconstruct a single vector from the group representation , and this vector serves as an estimation of any signature in the group. We measure the security by the mean square error over the dataset:


For the of privacy of query template, a curious server can reconstruct the query template from its embedding:


These reconstructions are possible only if matrix is known. This is not the case in practice, so we give here an extra advantage to the curious server. Figure 4 compares security with AoE-ML [3] where the assignment was imposed randomly, i.e. not learned. Different levels of sparsity are tested. The reconstruction error of queries are close in either case, yet learning the assignment improves verification performance. Reconstructing enrolled signatures is more difficult due to the aggregation. However, learning the assignment by similarity correspondence in the embedded domain decreases the security slightly while improving the performance a lot.

Figure 4: Investigation of trade-off between security and performance for varying sparsity level on CFP (with ) and CASIA-IrisV1 (with ).

4 Security Protocols

This section gives an example of a cryptographic protocol exploiting the group representations. The experimental section showed that grouping secures the enrolled signatures, but ternarization alone provides less protection to the query. Therefore, this protocol strengthens the protection of the querying user. For security reason, the server only manipulates query and the distances in the encrypted domain. For privacy reason, the server only learns that the query is close enough to one group representation, but it cannot tell which group exactly. We assume honest but curious user and server.

This protocol also justifies choices of our scheme: Queries and group representations are heavily quantized onto a small alphabet . They are long vectors but sparse: only components will be processed in the encrypted domain. Moreover, we have . These facts ease the use of partial homomorphic encryptions with limited module, whence a low complexity and expansion factor. The group representations remain in the clear on the server side, and we do not need fully homomorphic encryption.

The user generates a pair of secret and public keys for an additive homomorphic cryptosystem (say [9]), and sends the query encrypted component-wise. The server computes its correlation with group representation :


The server also generates a key pair for a multiplicative homomorphic cryptosystem (say [1]), and sends the user . The user randomly permutes the order of these quantities and masks them by multiplying them by . This yields another semantically secure version of the ciphertexts thanks to the multiplicative homomorphy of . The server decrypts , but the permutation prevents connecting back to the group index . Again thanks to homomorphy, the server computes where are random signed integers. The user decrypts and sends to the server. The user cannot guess the distances thanks to the masking , not even the sign of . The server can do this (since it knows ) and thus learns whether there is one group where is negative.

5 Conclusion

We proposed a method for group membership verification and identification jointly learning group representations and assignment. The idea is to minimize the overall distance between group members while maximizing the separation between groups in the embedded domain. Yet, the method still has some rigidness: the prototyping of the embedding (the sparse ternary quantization), considering mean as group centroids, and assigning a signature to only one group.


  1. thanks: Research supported by the ERA-Net project ID_IoT 20CH21_167534.


  1. T. Elgamal (1985-07) A public key cryptosystem and a signature scheme based on discrete logarithms. IEEE Transactions on Information Theory 31 (4), pp. 469–472. External Links: Document Cited by: §4.
  2. M. Gheisari, T. Furon, L. Amsaleg, B. Razeghi and S. Voloshynovskiy (2019) Aggregation and embedding for group membership verification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Cited by: §1, §3.
  3. M. Gheisari, T. Furon and L. Amsaleg (2019-06) Privacy preserving group membership verification and identification. In Proceedings of the The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Cited by: §1, §1, §3.4, §3.
  4. M. Gheisari, T. Furon and L. Amsaleg (2019) Group membership verification with privacy: sparse or dense?. In Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS), Cited by: §1.
  5. G. B. Huang, M. Mattar, T. Berg and E. Learned-Miller (2008) Labeled faces in the wild: a database forstudying face recognition in unconstrained environments. In Workshop on faces in’Real-Life’Images: detection, alignment, and recognition, Cited by: §3.1.1.
  6. A. Iscen, T. Furon, V. Gripon, M. Rabbat and H. Jégou (2017) Memory vectors for similarity search in high-dimensional spaces. IEEE Transactions on Big Data. External Links: Document, Link Cited by: §1.
  7. H. Jégou, F. Perronnin, M. Douze, J. Sánchez, P. Pérez and C. Schmid (2012) Aggregating local image descriptors into compact codes. IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (9), pp. 1704–1716. External Links: Document, Link Cited by: §1.
  8. C. A. of Sciences’ Institute of Automation Casia-irisv1 iris image database [online]. Available: Cited by: §3.1.2.
  9. P. Paillier (1999) Public-key cryptosystems based on composite degree residuosity classes. In Advances in Cryptology — EUROCRYPT ’99, J. Stern (Ed.), Berlin, Heidelberg, pp. 223–238. External Links: ISBN 978-3-540-48910-8 Cited by: §4.
  10. O. M. Parkhi, A. Vedaldi and A. Zisserman (2015) Deep face recognition.. In Proceedings of the British Machine Vision Conference, Cited by: §3.1.1.
  11. F. Perronnin and C. Dance (2007) Fisher kernels on visual vocabularies for image categorization. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, External Links: Document, ISSN 1063-6919 Cited by: §1.
  12. B. Razeghi, S. Voloshynovskiy, D. Kostadinov and O. Taran (2017) Privacy preserving identification using sparse approximation with ambiguization. In Proceedings of the IEEE International Workshop on Information Forensics and Security, Cited by: §1, §2.1.
  13. B. Razeghi and S. Voloshynovskiy (2018) Privacy-preserving outsourced media search using secure sparse ternary codes. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Cited by: §1, §2.1.
  14. P. H. Schönemann (1966) A generalized solution of the orthogonal procrustes problem. Psychometrika 31 (1), pp. 1–10. External Links: Document, ISSN 1860-0980, Link Cited by: §2.3.
  15. S. Sengupta, J. Chen, C. Castillo, V. M. Patel, R. Chellappa and D. W. Jacobs (2016) Frontal to profile face verification in the wild. In Proceeding of the IEEE Winter Conference on Applications of Computer Vision, Cited by: §3.1.1.
  16. J. Sivic and A. Zisserman (2003) Video google: a text retrieval approach to object matching in videos. In Proceedings of the IEEE International Conference on Computer Vision, External Links: Document Cited by: §1.
  17. T. M. University MMU2 iris image database [online]. Available: ccteo/. Cited by: §3.1.2.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description