M2 branes suspended between adjacent parallel M5 branes lead to light strings, the ‘M-strings’. In this paper we compute the elliptic genus of M-strings, twisted by maximally allowed symmetries that preserve 2 supersymmetry. In a codimension one subspace of parameters this reduces to the elliptic genus of the supersymmetric quiver theory in 2. We contrast the elliptic genus of M-strings with the sigma model on the -fold symmetric product of . For they are the same, but for they are close, but not identical. Instead the elliptic genus of M-strings is the same as the elliptic genus of sigma models on the -fold symmetric product of , but where the right-moving fermions couple to a modification of the tangent bundle. This construction arises from a dual quiver 6 gauge theory with gauge groups. Moreover we compute the elliptic genus of domain walls which separate different numbers of M2 branes on the two sides of the wall.

]Babak Haghighat, §]Amer Iqbal, †]Can Kozçaz, ]Guglielmo Lockhart, ]and
Cumrun Vafa \affiliation[]Jefferson Physical Laboratory, Harvard University, Cambridge, MA 02138, USA


[§]Department of Physics, LUMS School of Science & Engineering, U-Block, D.H.A, Lahore, Pakistan. \affiliation[§]Department of Mathematics, LUMS School of Science & Engineering, U-Block, D.H.A, Lahore, Pakistan.


[†]International School of Advanced Studies (SISSA), via Bonomea 265, 34136 Trieste, Italy and INFN, Sezione di Trieste

1 Introduction

The SCFTs with the maximal amount of supersymmetry in the highest dimension are the (2,0) theories in . Despite the unique status they enjoy, and despite the fact that they have been instrumental in constructing lower dimensional theories, they remain among the least understood theories. This is mainly related to the fact that we do not have a Lagrangian description of these theories. Moreover, if we go slightly away from the conformal point we get a theory of interacting almost tensionless strings. Clearly a deeper understanding of these strings is called for. One goal of the present paper is to take a step in this direction. In particular we focus on the SCFT arising from coincident M5 branes, and study the M2 branes suspended between the M5 branes when we separate them, which leads to strings on their boundaries. We will call these strings ‘M-strings’, as they involve basic M-theory ingredients for their definition.

If we consider two parallel M5 branes, and consider one M2 brane suspended between them clearly the moduli space of the M2 brane is labeled by its transverse position on the M5 brane it ends on, which is a copy of . So at least the IR degrees of freedom on this string should correspond to the supersymmetric sigma model on . Moreover if we consider M2 branes stretched between 2 M5 branes, one would naively expect the IR degrees of freedom to correspond to the choice of points on , modulo the action of the permutation group on the points, i.e. to a supersymmetric sigma model on

This space is singular and one can ask whether the target space is smoothed out at coincident points. If the target space is smoothed out, as in the Hilbert scheme of -points on , then this would give us an effective way to compute at least supersymmetry protected quantities for this theory. However, as argued in a related context in [1] this is not necessarily the case (not even the B-field on the vanishing ’s is turned on as in the orbifold points), and one expects that the relevant theory should be the one corresponding to the singular target space, which is infinitely far away from the smoothed out points. This in particular raises the question of whether at least for BPS quantities one may use the smoothed out target space to perform such computations. A surprising result we find is that this is not possible for . Instead we find a related sigma model with supersymmetry on the smoothed out space (the Hilbert scheme of n-points on ) which has the same elliptic genus as the suspended M2 branes. The right-moving fermions couple, instead of the tangent bundle, to a bundle where is the tautological bundle on the Hilbert scheme.

From the viewpoint of the M2 brane worldvolume theory, ending on an M5 brane corresponds to a boundary condition on the theory [2], as is familiar in the context of D-branes. More generally we will be considering a number of M2 branes suspended on an M5 brane from the left and a number of M2 branes suspended from the right. This can be viewed as a domain wall which separates M2 branes. In addition we need to choose a vacuum for each M2 brane, which in turn is labeled by the partition of for the left-vacuum and of for the right vacuum [3, 4, 5] . Thus the theory living on the domain wall is labeled by . One main computational result of this paper is the supersymmetric partition function of the theory on . More precisely we consider the elliptic genus of this theory, including twisting by maximal allowed symmetries consistent with supersymmetry as we go around the cycles of . In some limit (turning off some chemical potentials corresponding to turning off the ‘mass for the adjoint’) this computation can also be viewed, using a dual type IIB description, as elliptic genus of quiver supersymmetric theories in , which can be computed using the recent works [6, 7]. We check our computations against these results in this limit and find agreement.

The main tool we use is the relation between the refined topological string partition function and the degeneracy of BPS states in 5 [8, 9, 10, 11], which we apply to the M5 brane SCFT compactified on . Apart from the Kaluza-Klein reduction of the 6 mode, the suspended M2 branes wrapped around are the only other states contributing to BPS states, and we are thus able to extract the partition function, or more precisely the elliptic genus of the strings obtained from suspended M2 branes. Reversing this, one can recover the full refined topological string partition function in terms of the elliptic genera of the M-strings. This in turn can be used to compute the index of M5 branes [12, 13].

The organization of this paper is as follows: In section 2 we review the relation between M5 brane CFT and supersymmetric Yang-Mills in 5, and toric realization of it. In section 3 we show how to use this setup to compute the partition function of it on twisted (including modular properties and symmetries) using refined topological strings as well as the instanton calculus. Moreover we explain the relation between the partition function for the case and the BPS degeneracies. We compute this using two dual 5 theories: one in terms of the theory, and the other in terms of a dual quiver theory in 6 dimensions with gauge group. This latter perspective turns out to be particularly important for our purposes. In section 4 we contrast some expectations of BPS degeneracies based on generalities about suspended M2 branes with the actual results we obtain using the topological strings. We interpret our computations and explain what they tell us about M-strings, including their relation to elliptic genus of quiver theories. Furthermore we interpret our results as leading to the partition function of domain walls separating M2 branes. Moreover we discuss the fact that new additional bound states between M-strings arise when we compactify the M5 brane theory on the circle, which cannot be viewed as bound states before compactification. Furthermore, we show that in a particular limit (where the mass term is turned off) the result agrees with that of the dual type IIB description involving the elliptic genus of quiver theories. In section 5 we explain the relation of our results with the computation of the superconformal index for coincident M5 branes (i.e. the partition function on ) as well as their partition function on , which can be viewed as the partition function of a quantum deformation of Toda theories on . In section 6 we conclude by suggesting some directions for future research. Some technical aspects of the computations are discussed in appendices A, B, C, and D.

2 Parallel M5 branes on and and Suspended M2 Branes

In this section we discuss some general aspects of parallel M5 branes including their twisted compactifications on and . The twisted compactification on leads to a theory with the same IR degrees of freedom as in 5 dimensions, where the mass of the adjoint field is given by the twist parameter. The further compactification on the circle can be used to twist the left-over 4 dimensions of the M5 brane. We also discuss general aspects of M2 branes suspended between the parallel M5 branes. Furthermore we discuss various dualities which map this to related systems, and in particular to compactifications of M-theory on elliptically fibered geometries, which we will use in the following section to compute the partition function of M5 branes using the refined topological strings.

2.1 Basics of M-strings

Consider parallel and coincident branes. This is believed to lead to a superconformal theory in six dimensions usually called the theory. The choice for the terminology is because the same system is believed to arise when considering type IIB string theory in the presence of singularities. The latter viewpoint generalizes it to the and versions of the theory.

This theory has as the superconformal group whose bosonic part is


is the global R-symmetry group of this theory and is the double cover of which is the rotation group of the space transverse to the M5 branes.

On the worldvolume of a single M5 brane we have the tensor multiplet of the theory which consists of:
an antisymmetric 2-form such that its field strength is self-dual,
four symplectic Majorana-Weyl fermions in the of
five scalar fields giving the transverse fluctuations of the M5 brane.

If we compactify the six dimensional theory described above on a circle it gives a theory with 16 real supercharges in five dimensions, the Super Yang-Mills in five dimensions. Since we will be discussing the M2 branes suspended between M5 branes let us fix the worldvolume and the transverse directions of the M5/M2 branes. We denote the coordinates of as , , then

The worldvolume of coincident M5 branes has coordinates

The space transverse to the coincident M5 brane worldvolume is and is acted upon by the R-symmetry group . We can pick a direction in and separate the coincident M5 branes along this direction. We choose the coordinate to separate the branes. This breaks the global symmetry to acting on the coordinates . It is important to note that does not act on the M5 brane worldvolume coordinates. For later convenience we denote the position of the M5 branes in the direction as , .

We can now introduce M2 branes ending on M5 branes with the boundary of the M2 brane inside the M5 brane coupling to the 2-form . We can introduce multiple M2 branes for each pair of M5 branes extending in the direction. We consider the worldvolume of M2 branes such that

The worldvolume of an M2 brane suspended between M5 branes

The boundary of the M2 brane given by the coordinates is a string inside the M5 brane, which we call the M-string. The presence of this string breaks the M5 brane worldvolume Lorentz group to , where is the Lorentz group on the string and acts on the space transverse to the string inside the M5 brane.

From our choice of the worldvolume coordinates of the M5/M2 branes and the string it is easy to see that the supersymmetries preserved by the string are given by


where is the 32-component spinor, and are the eleven dimensional Gamma matrices. Since in eleven dimensions the above two conditions imply that


Hence the chirality under , chirality under and chirality under of the preserved supersymmetries on the string are the same. Since the M2/M5 brane configuration breaks of the 32 supersymmetries therefore on the string world sheet we have a supersymmetric theory with . By taking a specific form of the eleven dimensional Gamma matrices it is easy to show that the theory on the string has supersymmetry. It then follows from Eq.(3) that preserved supercharges and where denote the chiral/antichiral spinor of and denote the chiral/anti-chiral spinor of are in the representation,


of . The denote the chirality with respect to .

The above supercharges can be organized in terms of representations of as well and it will be useful for later purposes to do so. Consider a number of coincident M2 branes in with worldvolume along and . Then the transverse space is and the global symmetry of the theory on the M2 branes is given by . Now introducing M5 branes, separated along as before and M2 branes ending on them, breaks to . Notice that the preserved supercharges form a positive chirality spinor of , i.e. they are in . The chirality for is determined by and therefore it follows from Eq.(3) that


and hence preserved supersymmetries form a positive chirality spinor of . If we denote by and the simple roots of then the supercharges are in with highest weight vector 1

The weight vectors for the supercharges are given by


2.2 Compactification on

Next, consider compactifying the M5 branes on a circle. Recall that

The worldvolume of M5 branes has coordinates

and that the M5 branes are separated in the direction. Now consider compactifying to a circle of radius . More generally we can introduce a partial breaking of the supersymmetry by making the transverse to the M5 branes fibered over . We will denote this spanned by by . In particular identifying with coordinates and consider a rotation of the two complex planes as we go around the circle:


The resulting theory in 5 is a mass deformation of the maximally supersymmetric Yang-Mills theory, by addition of a mass term to the adjoint, which we have informally called the ‘ theory in 5’ (borrowing the terminology from the more familiar 4 case). The radius of the is identified with the gauge coupling of the Yang-Mills theory as follows


The 5 theory has charged particles in its spectrum which carry instanton number which is identified with the momentum around the :


From the point of view of the six-dimensional theory these particles arise as M-strings wrapped around the . If we consider M-strings wrapped around and carrying a momentum of units along its BPS mass is given by


where is the separation between the M5 branes which gives the tension of the M-string stretched between and M5 branes.

We can also ask how the twisting by around affects the theory as seen by the M-string wrapped around . The is embedded in the of of the M2 brane theory. It is easy to see that this choice of the leaves the negative chirality supercharges of Eq.(6) invariant but not the positive chirality ones. Hence the resulting theory has broken the supersymmetric theory on the worldsheet in the directions.

As already mentioned we can view the 6 theory as coming from type IIB theory with an singularity. Compactifying this on a circle and using the duality between M-theory and type IIB we can view this as compactification of M-theory on a threefold with geometry . The duality between type IIB and M-theory identifies the Kähler class of with

Moreover the twisting by the mass parameter can be viewed as blowing up [10]. This is the geometric analog of giving mass to the adjoint field in the brane construction [14]. The blow up parameter is identified with

The geometry of the blow-up is a local Calabi-Yau and is given by the periodic toric diagram [15, 10] in Fig. 1 where we have specialized to the case of the theory which corresponds to two M5 branes. There is a dual description of the same system [16] in terms of the web of 5-branes [17]. The picture is the same as the one of the toric diagram, only one has to associate the toric legs with branes of type IIB as is shown in Fig. 1.

Figure 1: The brane and toric geometry. The red line marks mean to identify the toric legs or branes with each other and therefore describe a compactified direction which is associated to the gauge coupling in the gauge theory. The length of the branes is associated to the mass of the theory. Last but not least the separation of the branes maps to the Coulomb branch parameter of the gauge theory.

In the massless case, where one has the maximally supersymmetric gauge theory, the NS5 brane is extended along an subspace of while the D5 branes have the geometry and intersect the NS5 branes transversally such that they have five dimensions in common. Then the gauge theory is living on the intersection and its rank is specified by the number of D5 branes. Note furthermore, that the compactified direction of the D5 branes is perpendicular to the NS5 brane. The gauge theory is then living on the intersection of these branes. Now let us deform the theory by introducing mass as shown in Fig. 1. To simplify matters we will take the gauge group to be for the moment. In this case the Calabi-Yau is the canonical bundle over a surface which is an elliptic fibration over , that is locally we have . The torus arises from the compactified direction of the brane system with size and the size of the is the Coulomb branch parameter of the gauge theory of size that is the separation of the D5-branes in the brane-picture. This is related to the separation between the M5 branes (which is proportional to the tension of the M2 brane string) times :

Moreover, there is yet a third Kähler class coming from a singular elliptic fibre over the discriminant locus. The singular fibre is a degeneration of the into two spheres and thus adds another Kähler class corresponding to the size of one of the ’s. This size determines the mass of the adjoint hypermultiplet in five dimensions, i.e. it is identified with .

2.3 Compactification on

We can also consider a further compactification on another which we take to be the direction. In trying to connect this geometry to topological string [8, 9, 10, 18] or -background [19] we fiber the space-time over this circle. In other words we twist the by the action of as we go around the circle in the direction:

Note that in the unrefined case where to preserve the symmetry we do not need to rotate .

Again we can ask what the suspended M2 brane theory sees if it is wrapped around the directions.

Figure 2: The system of M2 and M5 branes. The M5 branes are depicted in yellow whereas the M2 brane is blue. They intersect at the torus which is depicted in green.

The M2 branes as well as the M5 branes will then be all at a fixed point in and the M5 branes are extended along . Furthermore, the M2 branes will intersect the M5 branes along and appear point-like in . This configuration is shown schematically in Fig. 2. As these points can be separated in it is natural to conceive that the effective worldvolume theory of M2 branes admits a description in terms of the Hilbert scheme on points on as will be described in detail in section 4.1.4.

The weight vector corresponding to the is . For the unrefined case corresponding to the above action leaves the supercharges invariant. However, for it breaks with surviving supercharges corresponding to the weights given by Eq.(6),


In general we will be interested in the compactification on a generic torus with complex structure . In the case where the torus is rectangular can be identified as the ratio of the radii of the circle from six to five and the one from five to four as follows,


Upon further compactification to four dimensions the Kähler parameters get complexified in the type IIA setup. Moreover all the Kähler parameters of M-theory get rescaled by a factor of as we go to the type IIA description,

These are the parameters we will be using, and in particular we get Kähler parameters which can be identified with the gauge theory parameters as follows


Thus from the viewpoint of the original M5 branes, we have compactified on a torus with complex structure , where the A-cycle of is twisted by and the B-cycle of the torus is twisted by . Since we would be ultimately interested in computing the elliptic genus of the M2 branes stretched between the M5 branes and wrapped on and the twistings can be viewed as coupling to background fields, the dependence of the amplitudes for each of the twistings will appear in the combination:

where denote the twist parameters around the two cycles. Thus for the mass term we have

which is equivalent to

and for the we have the twists

This suggests that we can think of all the twistings to be around the B-cycle as long as we use our type IIA parameterization of . For simplicity of notation later in this paper we replace with , when we discuss partition functions. We summarise the geometry of the torus and its relation to the parameters of the gauge theory in Fig. 5.

Figure 5: The torus and its cycles. In (a), the rectangular torus is depicted. While going around the circle with radius one twists with the mass rotation, and along the circle one introduces the rotations. In (b), the same geometry is depicted where we use the holomorphy of the result to move the twisting along the A-cycle by to twisting around the B-cycle by .

Let us now come to the identification of states. From the discussion preceding Eq.(10) it is clear that self-dual string solutions which wrap the whole appear as instanton solutions in the four dimensional gauge theory. There will be also magnetically charged states (which shall be not of relevance here and which we only include for the purpose of completeness of the discussion) which arise from the string which does not wrap the first . Electric-magnetic duality of the 4 theory then corresponds to transformations of the . A string which wraps the times and has Kaluza-Klein momentum then gives rise to a BPS degeneracy which can be counted with the topological string on the elliptic Calabi-Yau. Furthermore, such strings can have non-trivial charge under the rotation induced by . Their degeneracies appear in the free energy of the topological string in the form


The task of the following sections will be to compute these degeneracies in the presence of the rotations and obtain a closed formula for it in terms of the refined topological string partition function. More precisely, the partition function of M-theory in this background is by definition the partition function of the refined topological string on the corresponding Calabi-Yau threefold:


Moreover the degeneracy of BPS states is known to be computed by the topological vertex and its refinement [15, 8, 9, 10, 11], which in this case, as we will discuss in section 4, consists mainly of the suspended M2 branes wrapped on . We thus use this correspondence to compute the twisted elliptic genus of suspended M2 branes.

Special values of parameters

As already discussed, for generic values of the suspended M2 branes lead to a supersymmetric system on . We can ask whether there are any special values of these parameters and in particular what happens to supersymmetry on the M-strings at these special values.

As already noted, in the unrefined limit where the supersymmetry gets enhanced to . We can also ask if there are special values of . For there is supersymmetry enhancement to . The holonomy is


(up to the permutation of the last two factors) and the preserved charges are given by,


The consequence of this enhancement is that the elliptic genus of suspended M2 branes should be a constant independent of the moduli of . There is also another limit in which the partition function simplifies and a different set of BPS states contribute. This limit is given by . In this case the supersymmetry is still so a priori nothing should have simplified, except that the center of mass degree of freedom of the string acquires additional zero modes. This is because in this case the holonomy becomes


and a single M2 brane acquires a fermionic zero mode, due to the direction in the holonomy twist (as we will review below in more detail). We can modify the computation of the elliptic genus in this limit to get a non-zero answer by computing instead a modified index


to absorb the zero mode from this single fermion zero mode and obtain a non-trivial answer even in this limit. This is somewhat similar to what one sees in the context of topologically twisted Yang-Mills in 4 [20] where the theory gives a vanishing partition function due to fermionic zero modes, but stripping off the leads to a non-vanishing partition function for theories.

To summarize, the M-strings enjoy a supersymmetry. If we turn on generic on the supersymmetry is broken to and we would be computing a non-trivial elliptic genus. If we have supersymmetry. If we tune the supersymmetry is enhanced to and the elliptic genus becomes a constant. If the supersymmetry is still but the partition function vanishes due to a fermionic zero mode associated to the ‘center of mass mode’. The fermionic zero mode can be eliminated in this case by insertion of suitable operators leading to a non-trivial function of . We summarise our discussion in the following table.

Values SUSY
Table 1: Enhanced symmetry configurations. The case is a configuration with supersymmetry but an extra fermionic zero mode which leads to the vanishing of the part of the elliptic genus.

2.4 Quiver realization of the suspended M2 branes

There is a dual description of this system [21] which generalizes it to and superconformal theories2. This corresponds to type IIB theory in the presence of singularity. The duality between the -series and M5 branes follows from the fact that singularities in type IIB is dual to NS5 branes for type IIA strings [22]. By lifting the NS5 branes to M-theory we see that this is equivalent to M5 branes where one of the five transverse directions to the 5-brane is compactified on . Therefore, when we consider separated branes, the rotation symmetry is reduced from . Thus this realization has the slight disadvantage that not all the symmetries are manifest. In particular we cannot twist by the mass parameter as we go down on the circle from 6 to 5 dimensions.

The singularity is given by where is one of the discrete subgroups of , which are in one to one correspondence with the Dynkin diagrams, for which is the two dimensional representation. The singularity can be resolved to . The resolution is such that generated by 2-cycles, which are topologically , can be identified with the root lattice of Lie algebra corresponding to such that the intersection number of the 2-cycles is given by the inner product on the root lattice which is determined by the Cartan matrix , i.e. there exists a basis of such that


As we blow down these 2-cycles to zero size we get back the singular space .

Consider type IIB strings propagating on and let , where is the size of blown up 2-cycles. The conformal limit is achieved by taking . For the corresponding Dynkin diagram is that of and the corresponding type IIB theory in the conformal limit gives the superconformal theory of coincident M5 branes. Moving away from the conformal point by turning on corresponds to the separation of the M5 branes along a linear direction as discussed before.

The emergence of conformal theory is signalled by the appearance of tensionless strings. In the M-theory setup this arises by M2 branes ending on M5 branes, and the tension of the resulting string is proportional to the separation of the corresponding M5 branes. Thus each pair of M5 branes leads to a string which become tensionless in the conformal limit. Similarly in the type IIB the strings arise by wrapping D3-branes over holomorphic 2-cycles of the blown-up geometry . Since holomorphic curves satisfy , where is the genus of the curve , and the inner product is given by minus the Cartan matrix it follows that the only holomorphic curves in the geometry are 2-spheres with , i.e. they are in one to one correspondence with the positive roots of . The tension of a string coming from D3 brane wrapped on is given by hence giving rise to strings with tensions for each 2-sphere . Unwrapped D1 branes can also be considered and they would correspond to M2 branes winding along a compactified circle transverse to the M5 brane.

The theory describing M-strings, when they have finite tension, can be deduced using the quiver description [24]. This in particular leads to the affine quiver. If we are interested in the local behaviour of the 6 dimensional CFT, we will be mostly interested in the limit where the transverse circle to the M5 brane is infinitely large where we would be ignoring the D1 brane. One could also consider the opposite limit where the transverse circle shrinks and consider the little string theory [25, 26], where the considerations of this paper will still apply. If we ignore the D1 brane charge, this corresponds to deleting the affine root from the quiver and gives rise to the ordinary quiver. This theory is equivalent to the reduction to two dimensions of the familiar quiver theory in four dimensions. In two dimensions this leads to a supersymmetric quiver theory.

Note however that, as already noted, not all the symmetries of the M5 branes are realized in this setup. This also impacts the symmetries that the M-string sees. In particular the symmetries of the 2d quiver theory (i.e. that of the quiver theory) are given by

where where is diagonally embedded in the two ’s. The Cartan of this can be identified with the rotation of the normal line bundle on the blown up ’s. As already noted we cannot realize the twisting by in this setup. This particular Cartan can be viewed as being in the (or ) direction of holonomy. Thus in the setup of the most general twisting discussed in the previous section, we see that we are in the limit where . Thus a 2-parameter subspace of the 3-parameter elliptic genus should be computable using the elliptic genus of quiver theories. Of course, as noted before, we would need to get rid of zero modes. In the and cases this gives a new way to compute the BPS degeneracies, which is not so simple in the geometric setup.

For concreteness let us focus on the case. Let D3 branes wrap the cycle , which correspond to the simple roots forming a basis of positive root lattice of . Then this theory has gauge group


with bi-fundamental matter between adjacent gauge factors. From the perspective of M-theory, this should be identified with the theory living on a collection of M-strings. For simplicity let us consider the case of the theory. This corresponds to having two M5 branes with M2 branes between them. The theory in this case corresponds to the pure gauge theory [1]. This theory has a Coulomb branch which at least far away from the origin of the Coulomb branch gives rise to the sigma model on , i.e. the -fold symmetric product of . These points in can be identified as the end points of the transverse to the M2 brane in the M5 brane. This also follows from the fact that broken supercharges and give rise to four left moving bosons and four right moving bosons where such that and are the gamma matrices. Modulo the resolution of the singularities when the points coincide, this can also be viewed as the Hilbert scheme of points on . What is the status of the theory when the points coincide is of course critical to the formation of BPS bound states, and therefore the above heuristic picture for is not guaranteed to be correct. In fact we will find later that our computation suggests that this picture is not accurate.

3 Topological partition function of M5 branes

The gauge theories can be geometrically engineered using elliptic Calabi-Yau threefolds. These elliptic Calabi-Yau threefolds, which we will denote by , are given by a deformation of the fibration over . The geometry of is captured by the toric diagram shown in Fig. 6.

Figure 6: The toric diagram of elliptic Calabi-Yau threefold .

The gauge theory partition function can be obtained either from Nekrasov’s instanton calculus or by calculating the topological string partition function of . The topological string partition function can be calculated using the refined topological vertex formalism [11]. The refined topological vertex has a preferred direction which breaks the cyclic symmetry of the topological vertex. For a given toric diagram that engineers a gauge theory we need to pick an orientation for the preferred direction. It was argued that the total amplitude is independent of the choice [27] although the form of the amplitude could have a significantly different looking form3. The choice is not necessarily arbitrary and has, as we will see later, important physical meaning. The preferred direction usually determines the instanton directions. In other words, according to the gluing algorithm of the topological vertex we perform sums of Young diagrams along each internal edge in the toric diagram. All of these sums can be explicitly performed except the ones along the preferred directions. The preferred direction is generally chosen such that the left-over sums match with the instanton expansion of the corresponding gauge theory.

From Fig. 6 it is clear that there are two choices for the preferred direction. One choice of preferred direction is along the vertical which is compactified on a circle and the other choice is along the horizontal. We will calculate the partition function for both these cases.

Before we begin calculating the partition functions we would like to explain our notation which will appear in later sections. We will denote by Greek letters partitions of natural numbers. An empty partition will be denoted by . A non-empty partition is a set of non-negative integers such that . The number of parts of the partition will be denoted by . We will denote by the transpose of the partition . is also a partition such that . For example, if then . The following are few functions on the set of partitions which will be of use later,


As is well known the partition has a two dimensional representation called the Young diagram. A Young diagram corresponding to the partition is obtained by placing a box in the first quadrant with upper left hand coordinate for each . Thus the number of boxes in the column of the Young diagram give . We will not distinguish between the partition and its Young diagram so that makes sense and means the box in the Young diagram with coordinates .

To calculate the topological string partition function we will use the refined topological vertex which is given by


where , is the Schur function labelled by a partition , and


We will also calculate gauge theory partition functions using equivariant instanton calculus where the torus action on is given by . The topological string parameters and are related to the gauge theory parameters and as


3.1 Case 1: Preferred direction along the compactified circle

Let us consider the case of in detail and then we will generalize this to . The toric diagram for the case is shown in Fig. 7 below. The vertical lines in the toric diagram are glued and the preferred direction is along the vertical.

Figure 7: Toric diagram of the geometry giving rise to theory. The preferred direction is taken to be vertical.

The refined topological string partition function in terms of the refined vertex is given by,


where the superscript refers to the number of M5 branes in the construction. Using standard techniques of summing up the Schur symmetric function given in Appendix B we get ()