Analysis of planar ornament patterns via motif asymmetry assumption and local connections
Abstract
Planar ornaments, a.k.a. wallpapers, are regular repetitive patterns which exhibit translational symmetry in two independent directions. There are exactly distinct planar symmetry groups. We present a fully automatic method for complete analysis of planar ornaments in of these groups, specifically, the groups called and . Given the image of an ornament fragment, we present a method to simultaneously classify the input into one of the groups and extract the so called fundamental domain (FD), the minimum region that is sufficient to reconstruct the entire ornament. A nice feature of our method is that even when the given ornament image is a small portion such that it does not contain multiple translational units, the symmetry group as well as the fundamental domain can still be defined. This is because, in contrast to common approach, we do not attempt to first identify a global translational repetition lattice. Though the presented constructions work for quite a wide range of ornament patterns, a key assumption we make is that the perceivable motifs (shapes that repeat) alone do not provide clues for the underlying symmetries of the ornament. In this sense, our main target is the planar arrangements of asymmetric interlocking shapes, as in the symmetry art of Escher.
keywords:
ornaments, wallpaper groups, mosaics, regular patterns, Escher style planar patterns1 Introduction
Planar ornaments, a.k.a. wallpapers, are repetitive patterns which exhibit translational symmetry in two independent directions. They form a tiling of the plane. They are created by repeating base unit in a predictable manner, using four primitive planar geometric operations: translation, rotation, reflection and glide reflection (Fig. 1). Using combinations of these primitive operations applied on a base unit, different patterns can be generated. An interesting fact is that the four primitive operations can be combined in exactly seventeen different ways to tile a plane, forming the so called Polya ().
We present an illustrative example in Fig. 2. Firstly, observe that the pattern Fig. 2 (a) can be generated by replicating the equilateral triangular fragment depicted in Fig. 2(b) by rotations in a systematic manner. This equilateral triangle fragment is the smallest fragment of the pattern that is sufficient to construct the entire pattern in Fig. 2(a) using four primitive operations. As such, it is referred as the fundamental domain (FD). Secondly, by rotating the FD twice around its top corner and then rotating around the middle point of its base yields the rhombus depicted in Fig. 2(c). This is the smallest translational unit; the pattern in (a) can be generated by simply translating it along two independent directions; hence, it is referred as the unit cell (UC). The abstracted unit cell of the tile, the relation between the fundamental domain and the unit cell along with the symmetries, is shown in Fig. 2 (d). The blue hexagons on the four corners of the rhombus indicate sixfold rotation centers, and the red triangles and the pink diamonds located on the centers and the side midpoints of the two triangles forming the rhombus respectively indicate threefold and twofold rotation centers. In the final illustration (Fig. 2(e)) a sample cell is shown as superimposed on the original ornament image.
Through a common naming provided by Crystallographic notation, the presented example happens to belong to a group called as , indicating six fold rotations. In the Crystallographic notation, the remaining groups are named as , , , , , , , , , , , , , , and . In Fig. 3, we depict the cell structures of each of the symmetry groups that we are interested in.
In the group name, each character position defines a group property: The first position is either the letter which stands for or the letter which stands for . The primitive cell is a unit cell with the centers of highest order of rotation at the vertices. The centered cell is encountered only in two cases ( and symmetry groups), and is chosen so that the reflection axis is normal to one or both sides of the cell. The digit that follows the letters or indicates the highest order of rotation, whereas the letter characters and respectively stand for mirror and glide reflections. When there are two positions containing either of and , it is understood that the first reflection is normal to axis and the second is at an angle . The digit denoting the highest order of rotation in a symmetry group can take only values 1, 2, 3, 4, and 6. This restriction is introduced by the , which states that the patterns repeating in two dimension can only exhibit , , , and rotations.
(a)  (b)  (c) 
In the present paper, given an image fragment from an ornament belonging to either of the mentioned groups, we present a robust method to extract the fundamental domain along with the underlying symmetry operations. That is, we provide a complete analysis of an ornamental pattern in groups based on an image. An important feature of our computational scheme is even when the given input image is only a small portion that a full unit cell does not fit to image (e.g., Fig. 4(a)), the symmetry group as well as the fundamental domain can still be defined. Of course, once the symmetry group and fundamental domain are defined, it becomes trivial to deduce the translational unit cell. We remark that most of the existing methods rely on first discovering the underlying lattice via a translational repetition structure. This requires global calculations, e.g., autocorrelations. In contrast, we do not search for a translational repetition or lattice. We directly look for local connections among motifs (protiles) from which we deduce symmetry clues that are later integrated via a decision tree.
Using a decision tree is indeed the classical method for grouping tiles into symmetry groups based on individual clues Doris (); the classical decision tree is depicted in Fig. 5; observe that the check for mirror reflections dominate the yes/no question set. The rational behind the given question sequence could be that it is easy for humans to spot mirror reflection and rotational symmetries while much harder to spot glides. While this is true, the robustness in the computational case might not be the same.
We propose an alternative decision tree, of which details are given in § 3.
Initially, we accumulate indirect clues to mirror reflection as opposed to searching for mirror reflection axis. We postpone mirror reflection control till the last stage, and then at the last stage use mirror reflection check about the predicted axes only to eliminate possible false alarms resulting from indirect clues.
That is, we use mirror reflection check only to eliminate false alarms, not to catch missed ones.
This means that we are willing to sacrifice mirror reflections in order not to wrongly assume that the tile has a mirror reflection.
Our rational is as follows: If we only miss the mirror reflection, say classify a tile as , we still get the correct unit cell and a redundant (twice the size) fundamental domain. Hence, the pattern can be correctly generated. If, however, we falsely classify a tile as , then the extracted fundamental domain is not sufficient, as we falsely assume the existence of a mirror reflection.
A consequence of not searching for mirror reflection is that we can only recognize groups that contain sufficient symmetries other than mirror. This leaves out of the groups, and gives us the groups listed above.
For the groups, our goal is to obtain the fundamental domain robustly up to a mirror reflection.
In accumulating symmetry clues, we only resort to local connections among motifs (protiles). Many of the classical ornaments, such as those found in Islamic art, are constructed from symmetric protiles such as stars that provide a clue to the symmetry group of the tile. In some works, the symmetries of the protiles themselves are used as a clue to the symmetry. However, we believe that inferring symmetries of the motifs are not robust due to possible noise in motif extraction. Moreover, the ornament artist may be using nearly symmetric motifs that do not reflect the symmetries of the ornament itself (several such examples exist in Escher’s art). Hence, we neither attempt to recover individual motifs correctly nor check their symmetries. Indeed, we even expect the patterns we analyze contain sufficient number of asymmetric motifs or at least the most significant symmetries are not completely swallowed by motifs having the same symmetry.
Motif asymmetry assumption is not as restrictive as it may strike at first.
Consider two valid examples shown in Fig. 6. The first one is of group and contains exactly three all symmetric protiles – brown, beige and gray– respectively having sixfold, threefold and twofold rotational symmetries. That is, the ornament does not contain any asymmetric motif and all three types of rotation centers coincide with the motif centers. Nevertheless, the local relations among the twofold rotational symmetric gray motifs reveal both the sixfold and threefold rotational centers, providing sufficient clues for . The second ornament, at a first glance, might give the impression that it contains only a single form that has all the symmetries of the ornament itself, i.e., threefold rotational symmetry and mirror reflection. Nevertheless, due to texture, there are also other detectable motifs in the form of circles or circle fragments, of which local relations provide clues for various threefold rotation centers.
In general, our constructions can not handle ornaments of single motif such that the motif contains all the symmetries of the symmetry group; for example, a checkerboard pattern or a uniform pack of triangles.
We remark, however, that such ornaments are typically the most obvious ones to recognize. Furthermore, even for those class of ornaments, it is possible to identify the translation grid though full analysis revealing FD can not be performed.
2 Related Work
Ornament patterns have always been a source of curiosity and interest, not only in arts and crafts but also other fields including mathematics, computation, cultural studies etc. Early researchers mostly examined ornaments in cultural contexts, e.g., WashburnBook (), with a goal of revealing social structures and their interaction via dominant symmetries used in the ornament designs of individual cultures or geographical regions. In mathematics, the ornament patterns are studied in terms of the groups formed by the symmetry operations. Few examples include Grunbaum86 (); Senechal88 (); Coxeter86 (). The Dutch artist, Escher took a particular interest in patterns formed by repeating asymmetric shapes and discovered a local structure leading to the same wallpaper patterns; his work on symmetry is examined in DorisBook (). Regular repetitive patterns such as Wallpaper and Frieze groups are even utilized in quite practical problems; for example, to analyze human gait Liu_Gait () or to achieve automatic fabric defect detection in 2D patterned textures Ngan2008 (); Asha2012 (); Ngan2010 ().
In the general pool of works in computational symmetry, the main focus has been finding symmetry axes in single objects.
Since a single object can exhibit only mirror reflections and rotational symmetries, the efforts are heavily focused on reflections and rotations Sun97 (); Sun99 (); Keller (); Loy (); Prasad2004 (); Prasad2005 (); Lee2008 (); Lee2010 () or finding local symmetries a.k.a shape skeletons. To our knowledge, Liu2010_2 (); Lee2012 () are the only works that address finding a glide reflection axis in an image, though the goal is to study one dimensional arrangements of symmetry, e.g., leaves. The works targeting shape symmetry, whether directly from an image or from a segmented region, fall out of our focus. Our focus is on the symmetries of planar patterns formed by regular repetition of shapes via four primitive geometric transformations.
In analyzing planar periodic patterns, translational symmetry, the most primitive repetition operation, is encountered in several works on recurring structure discovery LiuJ2013 (); Doubek2009 (); Doubek2010 (); Torii2013 (); Gao2009 (). The general flow of such works is to detect visual words and cluster them based on their appearance and spatial layout. Among these works, Doubek2009 (); Doubek2010 (); Torii2013 () further perform image retrieval based on the discovered recurring structures. In Doubek2009 (); Doubek2010 () instead of directly using the recurring structures for image matching, the authors first detect a translational repetition lattice of an image. There can be multiple lattices for an image. Thus, given a query image with various detected lattices, they search a database for images with equivalent lattices. In each search, the matching score between two lattices is a product of two measurements: the similarity of the grayscale mean of the representative unit cell and the similarity of the color histograms. In Park2008 (); Park2009 () detection of deformed lattice in a given pattern is proposed. They first propose a seed lattice from detected interest points. Using those interest points a commonly occurring lattice vectors are extracted. Subsequently, the seed lattice is refined, and grown outward until it covers whole pattern.
In Han2014 (), for translational symmetry, model based lattice estimation is performed where the model comparison for hypotheses generated via peaks of the autocorrelation is implemented using approximate marginal likelihood.
Some works move further beyond computing a translational lattice to address classifying repeated patterns according to the 17 plane symmetry groups (wallpapers) Liu98 (); Liu2000_1 (); Liu2000_5 (); Liu2004_pami (); Liu2001 (). As a first step, lattice detection is performed. After the lattice detection step, sequence of yes/no questions are answered, until the final symmetry group is determined. Since the possible lattice types that can be associated with are restricted by the symmetries the ornament pattern exhibits, detecting the lattice types reduces the number of symmetry groups. Though the lattice detection is commonly performed using peak heights in autocorrelation Liu98 (), in Liu2000_1 () an alternative peak detection algorithm based on so called regions of dominance is used to detect patterns translational lattice. The region of dominance is defined as the largest circle centered on the candidate peak such that no higher peaks are contained in the circle. The authors argue that the region of dominance is more important than the height of the peak. A Hough transform is used to detect two shortest translation vectors that best explains the majority of the point data. In order to test whether a pattern has certain symmetry, the conjectured symmetry is applied to entire pattern and then the similarity between the original image and the transformed one is computed. The representative motif is chosen to be the most symmetrical figure.
Recently, Albert2015 (), combined lattice extraction and point symmetry groups of individual motifs to analyse Islamic patterns in mosaics. This method specifically targets Islamic ornaments in which motifs such as nstars typically provide clues to the underlying plane symmetry group. As such, it is not readily applicable if motifs can not be robustly extracted or motifs do not reflect the symmetries. In, Nasri2015 (), rotation groups are detected to analyze Islamic rosette patterns.
It is also possible to perform a continuous characterization of the ornament by comparing ornament images. This is for example encountered in AgustiMelchor2013 (), where ornament images are classified according to a symmetry feature vector calculated based on a prior lattice extraction and yes/no questions; for lattice detection, they used method in AgustiMelchor2008 (). In Adanova2016 (), ornament images are directly compared in a transformed domain after applying a global transformation.
Note that among the works addressing planar patterns, there are also several interesting works on pattern synthesis, including how to generate an ornament in a certain symmetry group, how to use a given motif to tile the plane in a certain style, or how to map a given wallpaper pattern to a curved surface Kaplan (); vonGagern2009 ().
3 The Method
Our symmetry detection system has three modules: image processing module, local connectivity analysis, and final symmetry detection. Each of them are separately explained below.
3.1 Image Processing
The input to the image processing module is an arbitrary ornament which may be a noisy scanned image or screen shot of a part of an ornament drawn using a computer tool. That is, input ornament images are acquired in arbitrary imaging conditions. The processing proceeds in three stages: gamma correction, initial clustering, and refinement. At the end of the refinement step, binary images for each ornament are obtained. These binary images will be called masks for that ornament image.
The number of masks is a result of adaptive clustering. Quite often, the number of masks coincides with the number of colors in that ornament image. This, however, is not always so because the image processing module is not given the number of colors.
Gamma Correction. The first step of the image processing module is gammacorrection Dijk2006 () to brighten the black lines and shadows. It is performed on the component of color space using the following formula:
Resulting image depends on the parameter. If one chooses then the lightness of an image is higher than in original image and darker colors have more contrast. When no effect on original image is observed, while makes colors darker than in original image.
Initial Clustering. The next step is an iterative application of a clustering algorithm till the number of initial clusters drop below a predefined value . Here, we assume that the number of distinct colors is less than ; hence, if the number of colors is more, then some color groups are merged to form bigger motif groups. For clustering, we use fast and robust mean shift algorithm Fukunaga1975 (). It produces clusters based on a given feature space. In our case, the features are channels of the gamma corrected image. The appealing feature of this algorithm for us is that it does not require the number of clusters to be specificied. (This is unlike kmeans). However, it requires a bandwidth parameter which indirectly influence the number of detected clusters. We automate the clustering process by iteratively using mean shift clustering increasing the bandwidth at each iteration as follows.
At the initial step, the bandwidth parameter for mean shift algorithm is set to and the number of clusters is observed. Then at each iteration, the bandwidth is increased by . For all our images and . The iterations are stopped whenever the number of clusters drop below . The resulting clusters are used to define binary images. The resulting bandwidth is taken as the image dependent bandwidth estimate .
Refinement. The computation of initial clusters as outlined above is performed in the color space. Hence, spatial proximity of the pixels are not taken into account. In the next stage, a sequential combination of median filtering in the pixel space and mean shift clustering in the color space are applied iteratively for the fixed bandwidth . The median filtering is realized as follows: If a pixel of class is surrounded by a pixel of class , it is assigned to class , and channels of that pixel to the cluster center of the cluster . This sequential application of median filtering followed by mean shift with fixed bandwith is performed only few times. For all of our images, five iterations seemed sufficient. More iterations may cause the components of different colors to join.
Final clusters in the pixel space may have small holes. These holes may result either from an insufficient application of the iterative and sequential filtering step outlined above or simply from a small feature such as an eye of a bird. To remove holes, all background (foreground) connected components with radius smaller than a given threshold are converted to foreground (background) pixels. Performing elimination based on component radius rather than component area is more reliable, because it might be the case that all (or some) of the components join giving large areas causing the necessary but separate components to be eliminated.
A sample result of the image processing module is depicted in Fig.7.
3.2 Local connectivity analysis I: from masks to connections
The process of local connectivity analysis starts with consistent keypoint detection on connected components. An obvious means is to detect centroid of each foreground component. However, this will be very sensitive to the output of the image processing module. Separate repeating motifs may touch one another, for example. Furthermore, this joining of the motifs may be inconsistent throughout the ornament plane. What is important for the purpose of further analysis is that the detected keypoints are consistent throughout the pattern. It is, however, not critical whether they really coincide with the true motif centers. Hence, we call these keypoints as nodes.
Towards robustly locating nodes, a continuous labelling on each binary image, namely mask, is performed, yielding a continuous image , whose values are in . The labelling stage is detailed at the end of this subsection as a separate paragraph. The nodes are calculated as the centroids of the positive valued connected components of the image. Some sample label images are depicted in Fig.8. A node is merely a robustly computed keypoint.
Once the nodes are detected, a graph called connectivity graph is constructed via an iterative extraction of local node relations. In the first iteration, the minimal pairwise distance is found, and then all connections with similar distances are extracted (using a fixed tolerance ). In the next iteration, excluding the extracted connections, next minimal distance is computed to extract new connections. After iterations, connections of various sizes are obtained. Note that the connections with large distances are not much meaningful; at best they provide redundant information. Hence, it is better to keep small. Neither the choice of the parameter nor the tolerance is critical, because we later reclassify the connections using the mean shift algorithm.
Given a set of connections stored at the connectivity graph, the connection length as the feature is fed to the mean shift. A small and fixed bandwidth, is used. The number of connection groups as discovered by the mean shift clustering could be different than the initial .
In Fig.9, a sample connectivity graph and individual connection groups, discovered by clustering using the connection length, is depicted.
a  b 
Continuous labelling on the binary masks. For each mask of each ornament, each pixel on each foreground connected component is assigned an initial label, , reflecting whether or not the probability of the pixel belonging to a centroid is higher than that of the pixel belonging to the component boundary. For this purpose, distance from each foreground pixel to the nearest background pixel is computed, and then those which have bigger distance than the half of the maximum of all distances are assigned a positive constant where the others .
After this label initialization, a relaxation is performed. In the relaxation step, the value is increased (decreased) depending on whether the current label at a pixel is less (more) than the average of neighboring labels. Formulawise, relaxation is expressed as follows:
where is the relaxation constant which we take as positive, and the is
where the is computed as the average of . At convergence or after sufficient number iterations, whichever comes first, the continuous labelling of binary masks is achieved as
where is last performed iteration.
3.3 Local connectivity analysis II: from connections to symmetry elements
The connection groups extracted as described above are further analyzed in order to detect node relations. The way the nodes are related gives a hint on various symmetry elements. Recall that the connections with small sizes are favored, since larger connections repeat the same node relations at a larger scale. Thus, the analyses begin with the connection group of the smallest size and continue in ascending order. The analyses are done as following.
Each individual connection group is divided into . Afterwards, each connected graph is further analyzed independent of the other connected graphs in the connection group. Given a connected graph, the following decisions are made.

If a graph is a , the nodes are connected in a closed chain, then the probability of the graph being either equilateral triangle, square or regular hexagon is computed (Fig.10(top row)). The probability is the product of ratios of polygon edges. Thus, if a graph is constructed from the connection of three (four, six) nodes, and the probability computed as a product of its edge ratios is higher than by chance ( , 0.5), the center of the triangle (square, hexagon) is taken as a center of threefold (fourfold, sixfold) rotation.

If the graph is acyclic then the nodes might be related either by twofold rotation (a graph containing only two nodes) or glide reflection (a graph containing nodes connected in a zigzag form) (Fig.10(bottom row)).

If the graph contains only two nodes, their center is taken as the center of twofold rotation. Since twonode connections might occur by accident, the twofold centers are accepted only if the number of nodes involved in such connection in a connection group is more than of entire node number in a mask.

If the graph contains more than three nodes, the polynomial of order one is fit to the given nodes. For a graph of zigzag structure the line should pass through the centers of the edges between two adjacent nodes. In this case, the distances between the centers of the edges of two adjacent nodes and a line are computed. The probability of the graph being of zigzag structure is computed by taking the product of the above distance ratios. If the probability is more than 0.5, then there is a glide reflection axis passing through edge centers. On the contrary, if the nodes themselves (not the edge centers) lie on the fit polynomial, then this is a line, representing translational symmetry.

Detecting implicit node relations. After defining symmetry centers, elimination of repeating symmetry centers is performed. For example, if from early connection group a point on a mask has been marked as a center of fourfold rotation, and the same point is again marked as fourfold rotation center from subsequent connection group, the later one is discarded. This means that a symmetry center of one type is obtained only from one connection group. However, the symmetry center of one type may coincide with the symmetry centers of other types. For example, the symmetry center marked as fourfold rotation center from some connection group, may also be the symmetry center of twofold rotation derived from some other connection group. A point in a mask may represent multiple symmetry centers of the same type, if and only if they are derived from the same connection group. Thus, two threefold rotation centers (double triangle) at the same point might occur only when those are detected from the same connection group. Such repetitions are used to detect implicit node relations. If there is a double triangle at a point, this indicates that there is a sixfold rotation around that point. Similarly, if there is double twofold (triple twofold) rotation center at the point this is an indicator of fourfold (sixfold) rotation around that point if the angle between twopoint lines is (). Double glide reflection axes represent reflection symmetry which is perpendicular to the glide reflection axis. Samples of such connections are shown in Fig.11.
(a)  (b) 
(c)  (d) 
Another case is paired twopoint connections (see Fig.12). They too indicate reflection symmetry. Actual twofold centers lie at the centers of those paired lines. Moreover, they are the indicators of reflection axis passing through twofold rotation centers. The algorithm detects them by computing the minimal distance between twofold rotation centers and connecting nodes that are in this minimal distance from each other. If a graph of maximal degree four is obtained then these are actual twofold rotation centers. However, if twopoint connections occur then these are paired twofold rotation centers and are handled accordingly.
Refinement. Notice that up to now the symmetries for each mask are detected. After these steps, the algorithm yields a mask structure which contains fields for various symmetry centers and their classes. For example, if a mask contains threefold rotation centers, the structure contains their centers and classes. The class of symmetry element is the number of connection group from which it was extracted. Thus, all threefold rotational centers extracted from the same connection group are of the same class. The next step is to collect all symmetries detected on individual masks. This is a necessary step, since each mask may contain only one class of some symmetry type, while a whole pattern contains more classes of that symmetry type. Initially, all symmetries of the same type are collected without considering their classes. Say, the threefold rotation centers extracted from all masks are marked on an ornament. Then for each center, the algorithm defines which classes fall into that particular center. Thus, if threefold rotation centers of the first mask are of class 2, 5 for mask two and 3 for mask three, the centers contain classes 2 and 5 (25), 2 and 3 (23), and, 5 and 3 (53). These numbers define the classes of the symmetry centers and they are classified accordingly into three groups. This example is illustrated in Fig.13. It might be the case that the symmetry center is detected only for one mask, and that center has not been detected for another masks. Then this center forms a group on its own, leading to fourth symmetry class. To eliminate such groups, the algorithm counts the number of centers of each class, sorts them in ascending order, computes the minimal distance between the nodes in one class, and beginning with the largest class propagates its group to the nodes that are of similar distance. This is done iteratively, until no changes occur.
(a)  (b) 
(c)  (d) 
Once all the symmetry types are combined and their classes are determined the maximal order of rotation is defined. If no rotations are observed then the maximal order of rotation is taken as and it is checked whether the pattern contains glide reflections. If no glide reflection observed then the tile is of group, i.e., contains only translational symmetry. After defining the maximal order of rotation, further elimination is done using the symmetry group information. Thus, if the maximal order of rotation is four or six then there can be only one class of twofold rotation centers. If, erroneously, two classes were detected, the class intersecting with the unit cell edges is left, the other type is eliminated. If the maximal order of rotation is six and there are more than one classes of threefold rotations, then the one intersecting with the unit cell centers is left, and the others eliminated.
3.4 Final symmetry group detection
Individual cues in the form of symmetry elements as described in the previous subsection are integrated to yield the final symmetry group decision via a decision tree which we propose for the reduced set of groups, The tree is shown in Fig. 14. Comparing our tree to the classical decision tree in Fig. 5, observe that mirror reflection checks are postponed to be performed at the last stage and their number is significantly reduced. Furthermore, with the exception of twofold rotations case, a mirror reflection check is performed only when other cues indirectly imply it. For example, if the maximal order of rotation is six and glide reflections or paired twofold rotations are detected, then a mirror reflection check is in order. If the probability of mirror reflection is calculated to be more than then the ornament is classified as belonging to group and else to group. The reason has been explained before in the Introduction section: An erroneously detected mirror reflection is less desirable than a missed one. In case of a missed mirror reflection, the fundamental domain will just be twice as big as it really should be, the half of it being the mirror of the other half; hence, the whole pattern can be correctly generated.
There is one exception: The case when two twofold classes are detected. In this case, there are two possibilities.
The first possibility is that the ornament is of group . The second possibility is that the ornament is a pattern formed by all mirror symmetric protiles.
In the latter case, the third twofold center for is missed because it happens to be on the glide reflection axis and cannot be captured from the connection graphs.
Hence, a tile is classified as due to a missed third twofold center.
This is not tolerable because unlike versus or versus , the mirror reflection is not the only distinction between the two groups of and , hence, the fundamental domain is not correct up to a mirror.
For this reason, in the case of two twofold centers,
the mirror reflection check is performed even if it is not implied via indirect cues.
Mirror Reflection. Mirror reflection checks are performed on a single unit cell. Note that, in case of rotational groups of three, four or six fold orders, the unit cells of the regular and reflectional groups are identical, the difference being in the fundamental domain. In the case of twofold rotations, the unit cell of the group is employed. At any rate, given a unit cell, all the objects lying in it are extracted using the masks of the ornament pattern. Then the unit cell is divided into two along the expected reflection axis (based on group information). For each object lying on both sides, the area (), the perimeter (), the distance of the object center to the reflection axis (), the distance of the farthest point on the object to the reflection axis (fdRef) and the point on the reflection axis which is the closest to the object center () are computed. A pair of objects, one from the first part of the unit cell () and the other from the second (), is picked. The probability of the first object being a reflection symmetry of the second object is estimated via the product of the corresponding feature ratios:
In all experiments, .
For each object in the first part, the above probability is computed for all its pairs. The highest score is picked. It indicates with what probability the object has a mirror object on the other side. Then, the mean of all the highest scores in the first part is taken to indicate with what probability the unit cell is symmetrical along the given reflection axis. Note that there might be more than one reflection axes for a given symmetry group. The final probability is the mean value computed for all reflection axes.
Unit Cell. Recall that, in all rotational groups, the corners of the units cells are the points of maximal order of rotation. With the exception of and , the unit cell is readily constructed by connecting the nearest maximal rotation centers of the same class. For the two groups, and , however, the maximal rotational symmetry center appearing on the unit cell corners also appears on the unit cell center. Therefore, for the latter two groups, a point of certain class is chosen, then four closest points to of similar class are selected. Among these four points and are selected such that the length of equals to the length of and are in opposing directions. Then the rest two points and should also be at equal distance to the point and be in opposing directions. If all points are exhausted, the algorithm reports a failure.
If there are no rotations but glide reflections, then the lattice nodes are centers of twopoint connections of the zigzag structure with similar directions. To detect a unit cell, a point () of particular class is chosen as a first node for unit cell. Then two closest points ( and ) to are chosen so that the angle between lines and is less than or equal to and bigger than or equal to . The last point is chosen so that the angle between lines and is equal to the angle between lines and , and the length of lines and are equal to lines and , respectively. Recall that all lattice nodes are of the same class. If no such points detected, then another point is chosen as and the same operations performed according to this point. If all points have been exhausted, then the algorithm reports a failure.
If an ornament pattern has neither rotations nor glide reflections then the unit cell nodes are merely the motif centers.
Fundamental Domain. If the unit cell is successfully constructed, then the fundamental domain extraction is straightforward, recall Fig. 3. For example, the fundamental domain for a ornament is of its unit cell whereas it is for a ornament. If, however, the algorithm reports a failure during unit cell construction, then algorithm returns to the stage before the attempt to construct a unit cell to make use of all the previously collected information on individual symmetries. (Note that failure during unit cell construction may arise due to lack of sufficient translational repetition.) Let us explain the fundamental domain extraction step in case of failure via an example.
Suppose only two centers of sixfold rotation (, ) are found, hence, the unit cell cannot be constructed. Assuming that this two centers are both nodes of the same unit cell the distance between these two points is computed. Since it is known that the unit cell for ornaments with sixfold rotations is composed of two equilateral triangles, there should be a threefold center () which is in a distance from the both points. In order to detect , two circles of radius centering at and are drawn. There will be two points at which this two circles intersect. Any one of them can be selected as . In this manner, three points indicating nodes of a fundamental domain are computed. For the cases when more than two sixfold rotations exist, two closest nodes are selected as and . If, on the other hand, only one sixfold rotation center () is found, then threefold centers are searched. Then the fundamental domain is extracted using the relation of sixfold center and threefold center () that is closest to the sixfold center. A point is selected so that the length () is equal to the length of () and the angle between these two lines is . If no threefold centers are found then twofold centers are searched. Then the twofold center () closest to the sixfold center is used to extract the rest of the fundamental domain. In this case is a point which is equal to and in opposing direction. This new point is actually a sixfold center. Then is found using the first case where two sixfold centers are known. If no other symmetries detected except for one sixfold center then algorithm fails to detect the fundamental domain. Note that it might be the case that two sixfold centers are found in two different locations so that in between sixfold centers are missed by the algorithm. In such cases the first case gives large fundamental domain. If more than one type of symmetries detected so that more than one cases described above hold, then the fundamental domain is computed for all of the cases. The final fundamental domain is the one with the smallest region.
This approach is extended to other symmetry groups using their own properties.
4 Experiments
4.1 Data
We formed a labelled ornament data set produced under different imaging conditions by a variety of ornament artists including the authors using iOrnament tool iornament (). This set is enriched with 14 representative ornament fragments that are equivalent to , ,, , , , , , if color permutations are ignored. The total number of ornaments in the set add up to 100. In forming the data set, we paid attention to cover a variety of styles in terms of brush, color, tone, and motif choice. We further paid attention that half of the ornaments are mimicking Escher’s style with asymmetric interlocking forms and the set contains enough representative elements in each of the groups.
4.2 Parameters
For all ornaments, identical parameter values are used as detailed below.
Gamma correction. is set to .
Clustering. is set to . For the refinement stage, is set to , where is the maximum radius of the connected components.
Connection extraction. The connectivity graph is constructed using iterations with . The bandwidth fed to mean shift for connection group clustering is set to .
4.3 Results
We first present our result on representative ornaments taken from Escher’s collection (Fig. 15). For the first out of ornaments, depicted in Fig. 15 (a) through (l), our method works successfully. In each group, (a) through (l), the first rows show the original input. The second rows show the detected symmetries, the unit cells and the fundamental domains superimposed on the input. The third rows show the fundamental domains cut out automatically from the original patterns. Unit cells (in the forms as previously described in Fig. 3) are shown by red quadrilaterals.
For illustration purposes, the region belonging to the fundamental domain is made lighter while the rest of the pattern is made slightly darker. In case an ornament image is too small to fit a whole unit cell in it, only the fundamental domain is shown. Half of the cases ( out of ) are like this.
Notice that in some cases, (d)(f), the letter of the group names are in red color. This is to indicate that our method in these three cases missed the mirror reflection. For none of these three examples, mirror reflection checks at the last stage are performed because indirect clues did not imply mirror reflection. As a result, the fundamental domains are twice as big; though, the unit cells are correct. Recall that in general mirror symmetry can not cause any problems, since the centers of the maximal order of rotations do not reside on the centers of the respective protiles.
For the famous mariposas pattern (Fig. 15 (a)) only three threefold rotation centers all from different classes are detected. For the second tile (Fig. 15 (h)) only two fourfold rotation centers both from different classes are detected. Yet, for both patterns we obtain enough information to identify their symmetry groups and detect fundamental domains. The results in Fig. 15 (h)(i) show twofold rotation centers that are detected in wrong places. Nevertheless, since the symmetry group depends on the maximal rotation order, incorrect twofold centers do not influence final decision.
For the last two ornaments, respectively in and groups, the collected cues are not sufficient. In each case, only one of the glide axes are detected. Hence, these two cases are inconclusive. The reason for the failure is that it is harder to detect glides in the absence of sufficient repetition. As we demonstrate in later examples, detecting or is possible when there are slightly more samples.
The rest of the results for the remaining tiles are organized as follows. (Note that the result for one ornament had been shown previously in the Introduction.) To save space, samples are placed in the Appendix. They are split into three figures each showing the results for patterns. To be illustrated in this section, illustrative samples from groups of higher order rotations are selected. The remaining are selected from the remaining groups. The patterns with higher order rotations are further organized into two groups. The first one contains the groups with triangular lattice structures (); for these groups the unit cell consists of two equilateral triangles. The other contains the groups with square lattice (). The results are respectively shown in Figs. 16 and 17. Original ornaments are shown in their full sizes whereas the ones depicting the results are cropped in order to make the symmetries, unit cells and fundamental domains visible.
In all samples of the higher order rotation groups, the symmetry groups are correctly identified, except up to mirror reflection in some samples. Because a ornament has two threefold rotation centers, it is automatically classified correctly without a need for mirror reflection check. In our examples, the mirror reflections were implied as a result of detected glides, so the flow in the decision tree proceeded to mirror reflection check; hence, the fundamental domains are correctly identified. The fundamental domain for a ornament is half the size of a one, and that of a ornament is half the size of a one. This is because the other halves can be obtained by mirror reflection. In our examples, the fundamental domains are double the size they should be. This is because the samples are classified as belonging to as a consequence of missed mirror reflections. Nevertheless, using the generation the rules for the group, instead of for the , the original tiles can still be recreated.
(a)  (b)  (c)  (d)  (e)  (f) 
(g)  (h)  (i)  (j)  (k)  (l) 
(m)  (n) 
As for the two samples of (Fig. 17), the first one successfully passed the reflection double check (implied via glides and then checked). But for the second sample, the mirror reflection test is not performed because indirect clues did not indicate its existence. Hence, for this particular ornament of the group, the fundamental domain is double the size that it should be. Similar to the previously discussed cases of ornaments, using the generation the rules for the group, instead of for the , the original pattern can still be recreated.
As for the ornaments, their fundamental domains are accurately identified. In this group of ornaments, we observed that glide reflections are easily detected, while for a human it is hard to perceive this type of symmetry. For an ornament to be classified to group it is enough to detect one class of fourfold rotation centers. Hence, mirror reflection check is unnecessary.
more results on tiles with higher order rotations are given in Appendix Figs. A1 and A2. There are in total ornaments out of with higher order rotations (excluding Escher ornaments), and of them contain mirror reflections. The detected fundamental domains for these ornaments are given in Fig. 18. The fundamental domains of the ornaments for which mirror reflections have not been detected are framed in black boxes. Observe that for ornaments, the mirror reflections are missed. Hence, they are classified as belonging to the corresponding reflectionless groups, and their fundamental domains are double the sizes they should be.
Fig. 19 shows results for the samples of the remaining five symmetry groups with lower order rotations: , , , , . The first five ornaments belong to group. This group of ornaments contain four distinct classes of twofold symmetry centers. The second and the third groups respectively contain ornaments with two and three distinct classes of twofold centers. Both of the groups ( and ) have two distinct classes of glide reflection axes perpendicular to each other. The second sample of group (yellow and purple ornament) shows a case when the third class of the twofold centers is not detected. This is because we employ twopoint connections to indicate any binary connections, i.e., both the twofold rotations and the glide fragments forming zigzag structures. In this particular example, our algorithm detects zigzag structures and defines them as glide reflection axes. During this process, however, our algorithm loses track of the twofold rotations that are also indicated by the same twopoint connections.
In general, we observe that when twofold rotation centers lie on the glide axes and all the protiles are mirror symmetric (as in the case of the present sample), our algorithm loses track of the one of the twofold rotation centers. When one of the twofold rotation centers is lost, the symmetries of the pattern becomes similar to the symmetries of one except for the mirror reflection symmetry: while the group does not have mirror reflections, the group does. Thus, if exactly two twofold rotation centers detected, we always need to perform mirror reflection check. For the present sample, checking for the mirror reflection identifies the correct group. We performed further tests with additional samples as presented in the Appendix Fig. A3. With the help of mirror reflection checks, our algorithm achieves and separation.
Finally, the last row of Fig. 19 illustrates the results for ornaments without rotational symmetries. If two distinct classes of glides which are all parallel to each other are detected, then the ornament is classified as . For the group, the group of pure translation symmetry, the algorithm detects all grids or lines indicating translations.
5 Summary and Conclusion
We have presented a fully automated method to detect the symmetry group and extract fundamental domains of ornaments belonging to symmetry groups. We have focused on ornaments where motifs do not hint the symmetries of the underling tiling, the pattern formed by repeating shapes. As long as an ornament contains sufficient number of motifs (protiles) that are either rotationally asymmetric, or strongly concave, or at least less symmetric than the higher order rotational symmetry of the group, our method works. The asymmetry assumption is not a serious restriction. Even in cases where all motifs are symmetric, as long as they do not all centered at the corners of the translational unit, the method still works. If all motifs are symmetric and centered at the corners of the translational unit, our method can not determine the symmetry group. Nevertheless, it is still possible to extract a translational repetition lattice.
As a proof of concept, to show a range of ornaments for which our method works, we have compiled an ornament database of images. In the set, of the ornaments are images painted by Escher. All of them are classics, such as the famous Mariposas, Angles and Damons and Lizards. The remaining ornament images are constructed via iOrnament software either by the authors or by several iOrnament artists.
Because we do not explicitly check for the existence of mirror symmetry, unless indirectly implied by other clues, we sometimes miss mirror reflections. This causes for groups () to be classified as belonging to respective reflectionless groups of lower symmetry (). Nevertheless, since their fundamental domains (double the sizes they should be) contain mirror reflected copies, recreation of the original patterns using the generation rule of the respective reflectionless groups is possible. Indeed, this forms our motivation for postponing the hard mirror reflection checks till they are implied by indirect clues, such as glide reflections.
Acknowledgements
The work is funded by TUBITAK Grant 114E204.
References
 (1) G. Polya, Über die analogie der kristallsymmetrie in der ebene, Zeitschrift für Kristallographie 60 (1) (1924) 278–282.
 (2) D. Schattschneider, The plane symmetry groups: their recognition and notation, The American Mathematical Monthly 85 (6) (1987) 439–450.
 (3) D. K. Washburn, D. W. Crowe, Symmetries of culture: theory and practice of plane pattern analysis, University of Washington Press, 1988.
 (4) B. Grünbaum, Z. Grünbaum, G. C. Shepard, Symmetry in Moorish and other ornaments, Computers and Mathematics with Applications 12 (34) (1986) 641–653.
 (5) M. Senechal, Color symmetry, Computers and Mathematics with Applications 16 (58) (1988) 545–553.
 (6) H. S. M. Coxeter, Coloured symmetry, in: M. C. Escher: Art and Science, NorthHolland, 1986, pp. 15–33.
 (7) D. Schattschneider, M.C. Escher: Visions of Symmetry, Thames & Hudson, 2004.
 (8) Y. Liu, R. T. Collins, Y. Tsin, Gait sequence analysis using frieze patterns, in: European Conference on Computer Vision, 2002, pp. 657–671.
 (9) H. Y. T. Ngan, G. K. H. Pang, N. H. C. Yung, Motifbased defect detection for patterned fabric, Pattern Recognition 41 (6) (2008) 1878–1894.
 (10) V. Asha, P. Nagabhushan, N. U. Bhajantri, Automatic extraction of textureperiodicity using superposition of distance matching functions and their forward differences, Pattern Recognition Letters 33 (5) (2012) 629–640.
 (11) H. Y. T. Ngan, G. K. H. Pang, N. H. C. Yung, Ellipsoidal decision regions for motifbased patterned fabric defect detection, Pattern Recognition 43 (6) (2010) 2132–2144.
 (12) C. Sun, J. Sherrah, 3D symmetry detection using the extended Gaussian image, IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (2) (1997) 164–169.
 (13) C. Sun, D. Si, Fast reflectional symmetry detection using orientation histograms, RealTime Imaging 5 (1) (1999) 63–74.
 (14) Y. Keller, Y. Shkolnisky, A signal processing approach to symmetry detection, Transactions on Image Processing 15 (8) (2006) 2198–2207.
 (15) G. Loy, J. Eklundh, Detecting symmetry and symmetric constellations of features, in: European Conference on Computer Vision, 2006, pp. 508–521.
 (16) B. Prasad, V. S. N. amd Yegnanarayana, Finding axes of symmetry from potential fields, IEEE Transactions on Image Processing 13 (12) (2004) 1559–1566.
 (17) V. S. N. Prasad, L. S. Davis, Detecting rotational symmetries, in: IEEE International Conference on Computer Vision, 2005, pp. 954–961.
 (18) S. Lee, R. Collins, Y. Liu, Rotation symmetry group detection via frequency analysis of friezeexpansions, in: IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8.
 (19) S. Lee, Y. Liu, Skewed rotation symmetry group detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 32 (9) (2010) 1659–1672.
 (20) J. Liu, Y. Liu, Curved reflection symmetry detection with selfvalidation, in: Asian Conference on Computer Vision, 2010, pp. 102–114.
 (21) S. Lee, Y. Liu, Curved glidereflection symmetry detection, IEEE Transactions on Pattern Analylsis and Machine Intelligence 34 (2) (2012) 266–278.
 (22) J. Liu, Y. Liu, GRASP recurring patterns from a single view, in: Computer Vision and Pattern Recognition, 2013, pp. 2003–2010.
 (23) P. Doubek, J. Matas, Detection of 2D lattice patterns of repetitive elements and their use for image retrieval, Tech. Rep. CTUCMP200916, Department of Cybernetics, Czech Technical University (2009).
 (24) P. Doubek, J. Matas, M. Perdoch, O. Chum, Image matching and retrieval by repetitive patterns, in: International Conference on Pattern Recognition, 2010, pp. 3195–3198.
 (25) A. Torii, J. Sivic, T. Pajdla, M. Okutomi, Visual place recognition with repetitive structures, in: Computer Vision and Pattern Recognition, 2013, pp. 883–890.
 (26) J. Gao, Y. Hu, J. Liu, R. Yang, Unsupervised learning of higherorder structural semantics from images, in: International Conference on Computer Vision, 2009, pp. 2122–2129.
 (27) M. Park, R. T. Collins, Y. Liu, Deformed lattice discovery via efficient meanshift belief propagation, in: European Conference on Computer Vision, 2008, pp. 474–485.
 (28) M. Park, K. Brocklehurst, R. T. Collins, Y. Liu, Deformed lattice detection in realworld images using meanshift belief propagation, IEEE Transactions on Pattern Analysis and Machine Intelligence 31 (10) (2009) 1804–1816.
 (29) J. Han, S. J. Mckenna, Lattice estimation from images of patterns that exhibit translational symmetry, Image and Vision Computing 32 (1) (2014) 64–73.
 (30) Y. Liu, R. Collins, Frieze and wallpaper symmetry groups classification under affine and perspective distortion, Tech. rep., Robotics Institute (1998).
 (31) Y. Liu, R. T. Collins, A computational model for repeated pattern perception using frieze and wallpaper groups, in: Computer Vision and Pattern Recognition, 2000, pp. 537–544.
 (32) Y. Liu, R. Collins, Periodic pattern analysis under affine distortions using wallpaper groups, in: International Workshop on Algebraic Frames for the Perception Action Cycle, SpringerVerlag Belin, 2000, pp. 241 – 250.
 (33) Y. Liu, R. T. Collins, Y. Tsin, A computational model for periodic pattern perception based on frieze and wallpaper groups, IEEE Transactions on Pattern Analysis and Machine Intelligence 26 (3) (2004) 354–371.
 (34) Y. Liu, R. T. Collins, Skewed symmetry groups, in: Computer Society Conference on Computer Vision and Pattern Recognition, 2001, pp. 872–879.
 (35) F. Albert, J. M. Gomis, J. Blasco, J. M. Valiente, N. Aleixos, A new method to analyse mosaics based on Symmetry Group theory applied to Islamic Geometric Patterns, Computer Vision and Image Understanding 130 (2015) 54–70.
 (36) A. ElOuaazizi, A. Nasri, R. Benslimane, A rotation symmetry group detection technique for the characterization of Islamic Rosette Patterns, Pattern Recognition Letters 68 (P1) (2015) 111–117.
 (37) M. AgustiMelchor, A. RodasJorda, J. M. ValienteGonzalez, Computational framework for symmetry classification of repetitive patterns, in: International Conference on Computer Vision Theory and Applications, 2013, pp. 257–270.
 (38) M. AgustiMelchor, J. M. ValienteGonzalez, A. RodasJorda, Lattice extraction based on symmetry analysis, in: International Conference on Computer Vision Theory and Applications, 2008, pp. 396–402.
 (39) V. Adanova, S. Tari, Beyond symmetry groups: a grouping study on Escher’s Euclidean ornaments, Graphical Models 83 (C) (2016) 15–27.
 (40) C. S. Kaplan, D. H. Salesin, Escherization, in: Computer Graphics and Interactive Techniques, 2000, pp. 499–510.
 (41) M. von Gagern, J. RichterGebert, Hyperbolization of Euclidean ornaments, The Electronic Journal of Combinatorics 16 (2) (2009) 1–29.
 (42) J. Dijk, P. W. Verbeek, Lightness filtering in color images with respect to the gamut, in: European Conference on Color in Graphics, Imaging, and Vision, 2006, pp. 330–335.
 (43) K. Fukunaga, L. Hostetler, The estimation of the gradient of a density function, with applications in pattern recognition, IEEE Transactions on Information Theory (1975) 32–40.
 (44) J. RichterGebert, Sciencetotouch, http://www.sciencetotouch.com/en/index.html (2012).
Appendix
The Appendix contains the results for the ornaments that are not included in the main sections of the paper. They are organized according to their symmetry groups and placed in three figures.