Toward realistic gauge–Higgs grand unification

Abstract

The $SO(11)$ gauge–Higgs grand unification in the Randall–Sundrum warped space is presented. The 4D Higgs field is identified as the zero mode of the fifth-dimensional component of the gauge potentials, or as the fluctuation mode of the Aharonov–Bohm phase ${\theta }_H$ along the fifth dimension. Fermions are introduced in the bulk in the spinor and vector representations of $SO(11)$⁠. $SO(11)$ is broken to $SO(4)\times SO(6)$ by the orbifold boundary conditions, which is broken to $SU{(2)}_L\times U{(1)}_Y\times SU{(3)}_C$ by a brane scalar. Evaluating the effective potential $V_{\text{eff}}({\theta }_H)$⁠, we show that the electroweak symmetry is dynamically broken to $U{(1)}_{\text{EM}}$⁠. The quark–lepton masses are generated by the Hosotani mechanism and brane interactions, with which the observed mass spectrum is reproduced. Proton decay is forbidden thanks to the new fermion number conservation. It is pointed out that there appear light exotic fermions. The Higgs boson mass is determined with the quark–lepton masses given; however, it turns out to be smaller than the observed value.

1. Introduction

Up to now almost all observational data at low energies are consistent with the standard model (SM) of electroweak interactions. Yet it is not clear whether the Higgs boson discovered in 2012 at LHC is precisely what is introduced in the SM. Detailed study of interactions of the Higgs boson is necessary to pin down its nature. From the theory viewpoint the Higgs boson sector of the SM lacks a principle that governs and regulates the Higgs interactions with itself and other fields, quite in contrast to the gauge sector in which the gauge principle of $SU{(3)}_C\times SU{(2)}_L\times U{(1)}_Y$ completely fixes gauge interactions among gauge fields, quarks, and leptons. Further, the mass of the Higgs scalar boson $m_H$ generally acquires large quantum corrections from much higher energy scales, which have to be canceled by fine-tuning bare masses in a theory. This is called the gauge hierarchy problem.

Many proposals have been made to overcome these problems. Supersymmetric generalization of the SM is among them. There is an alternative scenario of gauge–Higgs unification in which the 4D Higgs boson is identified with a part of the extra-dimensional component of gauge fields defined in higher dimensional spacetime [1–5]. The Higgs boson, which is massless at the tree level, acquires a finite mass at the quantum level, independent of a cutoff scale and regularization scheme. The $SO(5)\times U{(1)}_X$ gauge–Higgs electroweak (EW) unification in the five-dimensional Randall–Sundrum (RS) warped space has been formulated [6–12]. It gives almost the same phenomenology at low energies as the SM, provided that the Aharonov–Bohm (AB) phase ${\theta }_H$ in the fifth dimension is ${\theta }_H\lesssim 0.1$⁠. In particular, cubic couplings of the Higgs boson with other fields, $W$⁠, $Z$⁠, quarks, and leptons, are approximately given by the SM couplings multiplied by $\cos {\theta }_H$ [8,13–16]. The corrections to the decay rates $H\to \gamma \gamma ,Z\gamma$⁠, which take place through one-loop diagrams, turn out finite and small [12,17]. Although infinitely many Kaluza–Klein (KK) excited states of $W$ and top quark contribute, there appears to be miraculous cancelation among their contributions. In the gauge–Higgs unification the production rate of the Higgs boson at LHC is approximately that in the SM times ${\cos }^2{\theta }_H$⁠, and the branching fractions of various Higgs decay modes are nearly the same as in the SM. The cubic and quartic self-interactions of the Higgs boson show deviations from those in the SM, which should be checked in future LHC and ILC experiments. Further, the gauge–Higgs unification predicts the $Z^{\prime }$ bosons, namely the first KK modes of $\gamma$⁠, $Z$⁠, and $Z_R$⁠, around 6 to 8 TeV with broad widths for ${\theta }_H=0.11$ to 0.07, which awaits confirmation at the 14 TeV LHC in the near future [18,19].

With a viable model of gauge–Higgs EW unification at hand, it is natural and necessary to extend it to gauge–Higgs grand unification to incorporate the strong interaction. Since the idea of grand unification was proposed [20], a lot of grand unified theories based on grand unified gauge groups in spaces with four and higher dimensions have been discussed. (See, e.g., Refs. [21–52] for recent works and Refs. [53,54] for reviews.) The mere fact of the charge quantization in the quark–lepton spectrum strongly indicates grand unification. Such an attempt to construct gauge–Higgs grand unification has been made recently: $SO(11)$ gauge–Higgs grand unification in the RS space with fermions in the spinor and vector representations of $SO(11)$ has been proposed [55–57]. The model carries over good features of the $SO(5)\times U{(1)}_X$ EW unification. In this paper we present a detailed analysis of the $SO(11)$ gauge–Higgs grand unification. In particular, we present how to obtain the observed quark–lepton mass spectrum in the combination of the Hosotani mechanism and brane interactions on the Planck brane. It will be shown that proton decay can be forbidden by the new fermion number conservation.

There have been many proposals for gauge–Higgs grand unification in the literature, but they are not completely satisfactory with regard to a realistic spectrum and the symmetry breaking structure [58–63]. In the current model, $SO(11)$ symmetry is broken to $SO(4)\times SO(6)$ by orbifold boundary conditions, which breaks down to $SU{(2)}_L\times U{(1)}_Y\times SU{(3)}_C$ by a brane scalar on the Planck brane. Finally, $SU{(2)}_L\times U{(1)}_Y$ is dynamically broken to $U{(1)}_{\text{EM}}$ by the Hosotani mechanism. The quark–lepton mass spectrum is reproduced. However, unwanted exotic fermions appear. Further elaboration of the scenario is necessary to achieve a completely realistic grand unification model. We note that there have been many advances in gauge–Higgs unification both in electroweak theory and grand unification [64–70]. A mechanism for dynamically selecting orbifold boundary conditions has been explored [71]. The gauge symmetry breaking by the Hosotani mechanism has been examined not only in the continuum theory, but also on the lattice by nonperturbative simulations [72–74].

This paper is organized as follows. In Sect. 2 the $SO(11)$ model is introduced. The symmetry-breaking structure and fermion content are explained in detail. The proton stability is also shown. In Sect. 3 the mass spectrum of gauge fields is determined. In Sect. 4 the mass spectrum of fermion fields is determined. With these results the effective potential $V_{\text{eff}}({\theta }_H)$ is evaluated in Sect. 5. Conclusions and discussion are given in Sect. 6.

2. The $SO(11)$ model

The $SO(11)$ gauge theory is defined in the Randall–Sundrum (RS) space whose metric is given by  

$$d{s}^2=G_{MN}d{x}^{M}d{x}^N=e^{-2\sigma (y)}{\eta }_{\mu \nu }d{x}^{\mu }d{x}^{\nu }+d{y}^2,$$

(2.1)

where $M,N=0,1,2,3,5$⁠, $\mu ,\nu =0,1,2,3$⁠, $y=x^5$⁠, ${\eta }_{\mu \nu }=\text{diag}(-1,+1,+1,+1)$⁠, $\sigma (y)=\sigma (y+2L)=\sigma (-y)$⁠, and $\sigma (y)=ky$ for $0\le y\le L$⁠. The topological structure of the RS space is $S_1/{\mathbb{Z}}_2$⁠. In terms of the conformal coordinate $z=e^{ky}$ (⁠$1\le z\le z_L=e^{kL}$⁠) in the region $0\le y\le L$⁠,  

$$d{s}^2=\frac{1}{z^2}({\eta }_{\mu \nu }d{x}^{\mu }d{x}^{\nu }+\frac{d{z}^2}{k^2}).$$

(2.2)

The bulk region $0<y<L$ (⁠$1<z<z_L$⁠) is anti-de Sitter (AdS) spacetime with a cosmological constant $\Lambda =-6k^2$⁠, which is sandwiched by the Planck brane at $y=0$ (⁠$z=1$⁠) and the TeV brane at $y=L$ (⁠$z=z_L$⁠). The KK mass scale is $m_{\text{KK}}=\pi k/(z_L-1)~\pi k{z}_L^{-1}$ for $z_L\gg 1$⁠.

2.1. Action and boundary conditions

The model consists of $SO(11)$ gauge fields $A_M$⁠, fermion multiplets in the spinor representation ${\Psi }_{32}$ and in the vector representation ${\Psi }_{11}$⁠, and a brane scalar field ${\Phi }_{16}$ [55]. In each generation of quarks and leptons, one ${\Psi }_{32}$ and two ${\Psi }_{11}$s are introduced. ${\Phi }_{16}(x)$⁠, in the spinor representation of $SO(10)$⁠, is defined on the Planck brane.

The bulk part of the action is given by  

$$\begin{array}{c}{S}_{\text{bulk}}={\displaystyle \int d^5}x\sqrt{-\text{det}G}[-\text{tr}\{\frac{1}{4}{F}^{MN}{F}_{MN}+\frac{1}{2\xi }{(f_{\text{gf}})}^2+{\mathcal{L}}_{\text{gh}}\}\\ +{\displaystyle \sum _{a=1}^3\{}{\overline{\Psi }}_{32}^a\mathcal{D}(c_{{\Psi }_{32}^a}){\Psi }_{32}^a+{\overline{\Psi }}_{11}^a\mathcal{D}(c_{{\Psi }_{11}^a}){\Psi }_{11}^a+{\overline{\Psi }}_{11}^{\prime a}\mathcal{D}(c_{{\Psi }_{11}^{\prime a}}){\Psi }_{11}^{\prime a}\},\\ \mathcal{D}(c)={\gamma }^A{e}_A{}^M({\partial }_M+\frac{1}{8}{\omega }_{MBC}[{\gamma }^B,{\gamma }^C]-ig{A}_M)-c{\sigma }^{\prime }(y),\end{array}$$

(2.3)

where $M,N,A,B,C=0,1,2,3,5$⁠. $a=1,2,3$ stands for the generation index, and $c_{{\Psi }_{32}^a}$⁠, $c_{{\Psi }_{11}^a}$⁠, $c_{{\Psi }_{32}^{\prime a}}$ represent bulk mass parameters. We employ the background field method, separating $A_M$ into the classical part $A_M^{\text{c}}$ and the quantum part $A_M^{\text{q}}$⁠: $A_M=A_M^{\text{c}}+A_M^{\text{q}}$⁠. The gauge-fixing function and the associated ghost term are given, in the conformal coordinate, by  

$$\begin{array}{c}{f}_{\text{gf}}=z^2\{{\eta }^{\mu \nu }{\mathcal{D}}_{\mu }^{\text{c}}{A}_{\nu }^{\text{q}}+\xi k^{2}z{\mathcal{D}}_z^{\text{c}}(\frac{1}{z}{A}_z^{\text{q}})\},\\ {\mathcal{L}}_{\text{gh}}=\overline{c}\{{\eta }^{\mu \nu }{\mathcal{D}}_{\mu }^{\text{c}}{\mathcal{D}}_{\nu }+\xi k^{2}z{\mathcal{D}}_z^{\text{c}}\frac{1}{z}{\mathcal{D}}_z\}c,\end{array}$$

(2.4)

where ${\mathcal{D}}_M^{\text{c}}{A}_N^{\text{q}}\equiv {\partial }_M{A}_N^{\text{q}}-i{g}_A[A_M^{\text{c}},A_N^{\text{q}}]$⁠, ${\mathcal{D}}_{M}c\equiv {\partial }_{M}c-ig[A_M,c]$⁠, etc. We adopt the convention $\{{\gamma }^A,{\gamma }^B\}=2{\eta }^{AB}$⁠, ${\eta }^{AB}=\text{diag}(-1,1,1,1,1)$⁠, $G_{MN}=e_{AM}{e}^A{}_N$⁠, and $\overline{\Psi }=i{\Psi }^{†}{\gamma }^0$⁠.

Generators $T_{jk}=-T_{kj}$ of $SO(11)$ are summarized in Appendix A in the vectorial and spinorial representations. We adopt the normalization $A_M=2^{-1/2}{\displaystyle {\sum }_{j<k}{A}_M^{(jk)}}{T}_{jk}$ and $F_{MN}={\partial }_M{A}_N-{\partial }_N{A}_M-ig[A_M,A_N]=2^{-1/2}{\displaystyle {\sum }_{j<k}{F}_{MN}^{(jk)}}{T}_{jk}$⁠. With this normalization the $SU{(2)}_L$ weak gauge-coupling constant is given by $g_w=g/\sqrt{L}$⁠. The orbifold boundary conditions for the gauge fields are given, in the $y$-coordinate, by  

$$\begin{array}{c}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }\\ A_y\end{array}\end{array}\right)(x,y_j-y)=P_j\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }\\ -A_y\end{array}\end{array}\right)(x,y_j+y)P_j^{-1},\\ A_M(x,y+2L)=U{A}_M(x,y)U^{-1},U=P_1{P}_0,\end{array}$$

(2.5)

where $(y_0,y_1)=(0,L)$⁠. The ghost fields, $c$ and $\overline{c}$⁠, satisfy the same boundary conditions as $A_{\mu }$⁠. $P_j=P_j^{†}=P_j^{-1}$ is given in the vectorial representation by  

$$P_0^{\text{vec}}=\text{diag}(I_{10},-I_1),P_1^{\text{vec}}=\text{diag}(I_4,-I_7),$$

(2.6)

and in the spinorial representation by  

$$\begin{array}{c}{P}_0^{\text{sp}}={\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^3=I_{16}\otimes {\sigma }^3,\\ P_1^{\text{sp}}={\sigma }^0\otimes {\sigma }^3\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0=I_2\otimes {\sigma }^3\otimes I_8.\end{array}$$

(2.7)

The $SO(11)$ symmetry is broken down to $SO(10)$ by $P_0$ at $y=0$⁠, and to $SO(4)\times SO(7)$ by $P_1$ at $y=L$⁠. With these two combined, there remains $SO(4)\times SO(6)\simeq SU{(2)}_L\times SU{(2)}_R\times SU(4)$ symmetry, which is further broken to $G_{\text{SM}}=SU{(2)}_L\times SU{(3)}_C\times U{(1)}_Y$ by $\langle {\Phi }_{16}\rangle$ on the Planck brane as described below. $SU{(2)}_L\times U{(1)}_Y$ is dynamically broken to $U{(1)}_{\text{EM}}$ by the Hosotani mechanism. Fermion fields obey the following boundary conditions:  

$$\begin{array}{c}{\Psi }_{32}^a(x,y_j-y)=-{\gamma }^5{P}_j^{\text{sp}}{\Psi }_{32}^a(x,y_j+y),\\ {\Psi }_{11}^a(x,y_j-y)={(-1)}^j{\gamma }^5{P}_j^{\text{vec}}{\Psi }_{11}^a(x,y_j+y),\\ {\Psi }_{11}^{\prime a}(x,y_j-y)={(-1)}^{j+1}{\gamma }^5{P}_j^{\text{vec}}{\Psi }_{11}^{\prime a}(x,y_j+y).\end{array}$$

(2.8)

Eigenstates of ${\gamma }^5$ with ${\gamma }^5=\pm 1$ correspond to right- and left-handed components in four dimensions. For ${\Psi }_{11}^a$ and ${\Psi }_{11}^{\prime a}$ one might impose alternative boundary conditions given by  

$$\begin{array}{c}{\Psi }_{11}^a(x,y_j-y)={\gamma }^5{P}_j^{\text{vec}}{\Psi }_{11}^a(x,y_j+y),\\ {\Psi }_{11}^{\prime a}(x,y_j-y)=-{\gamma }^5{P}_j^{\text{vec}}{\Psi }_{11}^{\prime a}(x,y_j+y).\end{array}$$

(2.9)

It turns out that the model with (2.8) is easier to analyze in reproducing the mass spectrum of quarks and leptons.

On the Planck brane (at $y=0$⁠) the brane scalar field ${\Phi }_{16}$ has an $SO(10)$-invariant action given by  

$$\begin{array}{c}{S}_{{\Phi }_{16}}={\displaystyle \int d^5}x\sqrt{-\text{det}G}\delta (y)\{-{(D_{\mu }{\Phi }_{16})}^{†}{D}^{\mu }{\Phi }_{16}-{\lambda }_{{\Phi }_{16}}{({\Phi }_{16}^{†}{\Phi }_{16}-w^2)}^2\},\\ D_{\mu }{\Phi }_{16}=({\partial }_{\mu }-ig{A}_{\mu }^{SO(10)}){\Phi }_{16}=\{{\partial }_{\mu }-\frac{ig}{\sqrt{2}}{\displaystyle \sum _{j<k}^{10}{A}_{\mu }^{(jk)}}{T}_{jk}^{\text{sp}}\}{\Phi }_{16}.\end{array}$$

(2.10)

${\Phi }_{16}$ develops the VEV. Without loss of generality one can take  

$$\langle {\Phi }_{16}\rangle =\left(\begin{array}{c}\begin{array}{c}{0}_4\\ 0_4\\ v_4\\ 0_4\end{array}\end{array}\right),v_4=\left(\begin{array}{c}\begin{array}{c}0\\ 0\\ 0\\ w\end{array}\end{array}\right).$$

(2.11)

On the Planck brane, $SO(10)$ symmetry is spontaneously broken to $SU(5)$ by $\langle {\Phi }_{16}\rangle \ne 0$⁠. With the orbifold boundary condition $P_1$⁠, $SO(11)$ symmetry is broken to the SM symmetry $G_{\text{SM}}=SU{(3)}_C\times SU{(2)}_L\times U{(1)}_Y$⁠.

To see this more explicitly, we note that mass terms for gauge fields are generated from (2.10) in the form $-g^2\langle {\Phi }_{16}^{†}\rangle {\eta }^{\mu \nu }{A}_{\mu }^{SO(10)}{A}_{\nu }^{SO(10)}\langle {\Phi }_{16}\rangle$⁠. Making use of (A3) and (A4) for $T_{jk}^{\text{sp}}$⁠, one obtains  

(2.12)

In all, 21 components in $SO(10)/SU(5)$ acquire large brane masses by $\langle {\Phi }_{16}\rangle$⁠, which effectively alters the Neumann boundary condition at $y=0$ to the Dirichlet boundary condition for their low-lying modes (⁠$m_n\ll gw/\sqrt{L}$⁠), as will be seen in Sect. 3. It follows that the $SU(5)$ generators are given, up to normalization, by Twelve components of the gauge fields $A_{\mu }$ in class (iv) have no zero modes by the boundary conditions. This leaves $SU{(2)}_L\times SU{(3)}_C\times U{(1)}_Y$ symmetry.

• (i) $SU{(2)}_L$⁠:  

  $$T_L^1=\frac{1}{2}(T_{23}+T_{14}),T_L^2=\frac{1}{2}(T_{31}+T_{24}),T_L^3=\frac{1}{2}(T_{12}+T_{34}),$$
• (ii) $SU{(3)}_C$⁠:  

  $$\begin{array}{l}8\left(\begin{array}{c}\begin{array}{c}{T}_{57}+T_{68}\\ T_{58}-T_{67}\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{T}_{59}+T_{610}\\ T_{510}-T_{69}\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{T}_{79}+T_{810}\\ T_{710}-T_{89}\end{array}\end{array}\right),\\ T_{56}-T_{78},T_{56}+T_{78}-2T_{910},\end{array}$$
• (iii) $U{(1)}_Y$⁠:  

  $$Q_Y=\frac{1}{2}(T_{12}-T_{34})-\frac{1}{3}(T_{56}+T_{78}+T_{910}),$$
• (iv) $SU(5)/SU{(3)}_C\times SU{(2)}_L\times U{(1)}_Y$⁠:  

  $$\begin{array}{l}\left(\begin{array}{c}\begin{array}{c}{T}_{15}+T_{26}\\ T_{16}-T_{25}\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{T}_{17}+T_{28}\\ T_{18}-T_{27}\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{T}_{19}+T_{210}\\ T_{110}-T_{29}\end{array}\end{array}\right),\\ \left(\begin{array}{c}\begin{array}{c}{T}_{35}-T_{46}\\ T_{36}+T_{45}\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{T}_{37}-T_{48}\end{array}\\ T_{38}+T_{47}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{T}_{39}-T_{410}\\ T_{310}+T_{49}\end{array}\end{array}\right).\end{array}$$

  (2.13)

It will be seen later that $SU{(2)}_L\times U{(1)}_Y$ symmetry is dynamically broken to $U{(1)}_{\text{EM}}$ by the Hosotani mechanism. The AB phase ${\theta }_H$ associated with $A_z^{4,11}$ becomes nontrivial so that $A_{\mu }^{34}$ picks up an additional mass term. Consequently, the surviving massless gauge boson, the photon, is given by  

$$\begin{array}{c}{A}_{\mu }^{\text{EM}}=\frac{\sqrt{3}}{2}{A}_{\mu }^{12}-\frac{1}{2}{A}_{\mu }^{0_C},A_{\mu }^{0_C}=\frac{1}{\sqrt{3}}(A_{\mu }^{56}+A_{\mu }^{78}+A_{\mu }^{910}),\\ Q_{\text{EM}}=T_{12}-\frac{1}{3}(T_{56}+T_{78}+T_{910})=T_L^3+T_R^3-\frac{1}{3}(T_{56}+T_{78}+T_{9,10}),\end{array}$$

(2.14)

where $T_R^a$ are generators of $SU{(2)}_R$⁠. The orthogonal component ${\tilde{A}}_{\mu }=\frac{1}{2}{A}_{\mu }^{12}+\frac{\sqrt{3}}{2}{A}_{\mu }^{0_C}$ and $A_{\mu }^{34}$ mix with each other for ${\theta }_H\ne 0$⁠. More rigorous and detailed reasoning is given in Sect. 3 in the twisted gauge. The $U{(1)}_Y$ gauge boson, $B_{\mu }^Y$⁠, is given by  

$$B_{\mu }^Y=\sqrt{\frac{3}{5}}{A}_{\mu }^{3_R}-\sqrt{\frac{2}{5}}{A}_{\mu }^{0_C},A_{\mu }^{3_L,3_R}=\frac{1}{\sqrt{2}}(A_{\mu }^{12}\pm A_{\mu }^{34}).$$

(2.15)

The gauge couplings become  

$$\begin{array}{c}g{A}_{\mu }=g\{A_{\mu }^{3_L}{T}_{3_L}+A_{\mu }^{3_R}{T}_{3_R}+\frac{1}{\sqrt{2}}(A_{\mu }^{56}{T}_{56}+A_{\mu }^{78}{T}_{78}+A_{\mu }^{910}{T}_{910})+\cdots \}\\ =g\{\frac{\sqrt{3}}{2\sqrt{2}}{A}_{\mu }^{\text{EM}}{Q}_{\text{EM}}+\cdots \}\\ =g\{A_{\mu }^{3_L}{T}_{3_L}+\sqrt{\frac{3}{5}}{B}_{\mu }^Y{Q}_Y+\cdots \}.\end{array}$$

(2.16)

In other words, 4D gauge couplings and the Weinberg angle are given, at the grand unification scale, by  

$$g_w=\frac{g}{\sqrt{L}},e=\sqrt{\frac{3}{8}}{g}_w,g_Y=\sqrt{\frac{3}{5}}{g}_w,{\sin }^2{\theta }_W=\frac{g_Y{}^2}{g_w^2+g_Y{}^2}=\frac{3}{8}.$$

(2.17)

The content ${\Psi }_{32}$⁠, ${\Psi }_{11}$⁠, and ${\Psi }_{11^{\prime }}$ is determined with the EM charge given by (2.14). In the spinorial representation,  

$$\begin{array}{c}{Q}_{\text{EM}}=\frac{1}{2}{\sigma }^3\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\\ -\frac{1}{6}{\sigma }^0\otimes \{{\sigma }^3\otimes {\sigma }^3\otimes {\sigma }^0\otimes {\sigma }^0+{\sigma }^0\otimes {\sigma }^3\otimes {\sigma }^3\otimes {\sigma }^0+{\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^3\otimes {\sigma }^3\}.\end{array}$$

(2.18)

We tabulate the content of ${\Psi }_{32}$ in Table 1. We note that zero modes appear only for particles with the same quantum numbers as quarks and leptons in the SM, but not for anything else.

Table 1.

The content of ${\Psi }_{32}$⁠. In the second and third columns, $SU{(5)}_Z=SU(5)\times U{(1)}_Z$ and $G_{227}=SU{(2)}_L\times SU{(2)}_R\times SO(7)$ are shown. In the fourth column, $SO(6)$ in $G_{PS}=SU{(2)}_L\times SU{(2)}_R\times SO(6)$ is shown. In the fifth column $G_{\text{SM}}=SU{(3)}_C\times SU{(2)}_L\times U{(1)}_Y$⁠. Superscripts and subscripts indicate $U{(1)}_Y$ charges. In the last column, parity at $y=0$ and $L$ is given for left-handed components. The right-handed components have opposite parity.

${\Psi }_{11}$ decomposes into an $SO(10)$ vector and a singlet. The former further decomposes into an $SO(4)$ vector ${\psi }_j$ (⁠$j=1,\dots ,4$⁠) and an $SO(6)$ vector ${\psi }_j$ (⁠$j=5,\dots ,10$⁠). An $SO(4)$ vector ${\psi }_j$ (⁠$j=1,\dots ,4$⁠) transforms as $(2,2)$ of $SU{(2)}_L\times SU{(2)}_R$⁠. Under ${\Omega }_L\in SU{(2)}_L$ and ${\Omega }_R\in SU{(2)}_R$⁠,  

$$\begin{array}{c}\hat{\psi }\to {\Omega }_L\hat{\psi }{\Omega }_R^{†},\\ \hat{\psi }=\frac{1}{\sqrt{2}}({\psi }_4+i\overrightarrow{\psi }\overrightarrow{\sigma })\cdot i{\sigma }_2\\ =\frac{1}{\sqrt{2}}\left(\begin{array}{cc}{\psi }_2+i{\psi }_1& -{\psi }_4-i{\psi }_3\\ {\psi }_4-i{\psi }_3& {\psi }_2-i{\psi }_1\end{array}\right)\leftrightarrow \left(\begin{array}{cc}\hat{E}& N\\ \hat{N}& E\end{array}\right).\end{array}$$

(2.19)

An $SO(6)$ vector ${\psi }_j$ (⁠$j=5,\dots ,10$⁠) decomposes into two parts:  

$$\left(\begin{array}{c}\begin{array}{c}{D}_j\\ {\hat{D}}_j\end{array}\end{array}\right)=\frac{1}{\sqrt{2}}({\psi }_{3+2j}\mp i{\psi }_{4+2j})(j=1,2,3).$$

(2.20)

With this notation the contents of ${\Psi }_{11}$ and ${\Psi }_{11}^{\text{'}}$ in (2.8) are summarized in Table 2. Notice that ${\Psi }_{11}$ and ${\Psi }_{11}^{\text{'}}$ have no components carrying the quantum numbers of the $u$ quark. With the boundary conditions (2.8), only $(D_{jR},{\hat{D}}_{jR})$ and $(D_{jL}^{\text{'}},{\hat{D}}_{jL}^{\text{'}})$ have zero modes. If the boundary conditions (2.9) were adopted, then $(N_R,E_R,{\hat{E}}_R,{\hat{N}}_R,S_L)$ and $(N_L^{\text{'}},E_L^{\text{'}},{\hat{E}}_L^{\text{'}},{\hat{N}}_L^{\text{'}},S_R^{\text{'}})$ would have zero modes.

Table 2.

The contents of ${\Psi }_{11}$ and ${\Psi }_{11}^{\text{'}}$ with the boundary conditions (2.8). The same notation is adopted as in Table 1.

${\Psi }_{11}$
$SO(10)$	$SU{(5)}_Z$	$G_{PS}$	$G_{\text{SM}}$	$Q_{\text{EM}}$	name	zero mode	parity (left)
$10$	$5_{+2}$	$(2,2,1)$	${(1,2)}_{+1/2}$	$\begin{array}{c}\begin{array}{c}+1\end{array}\\ 0\end{array}$	$\begin{array}{c}\begin{array}{c}\hat{E}\end{array}\\ \hat{N}\end{array}$		$(-,+)$
	${\overline{5}}_{-2}$	$(2,2,1)$	${(1,2)}_{-1/2}$	$\begin{array}{c}\begin{array}{c}0\end{array}\\ -1\end{array}$	$\begin{array}{c}\begin{array}{c}N\end{array}\\ E\end{array}$		$(-,+)$
	$5_{+2}$	$(1,1,6)$	${(3,1)}_{-1/3}$	$-\frac{1}{3}$	$D_j$	$D_{jR}$	$(-,-)$
	${\overline{5}}_{-2}$	$(1,1,6)$	${(\overline{3},1)}_{+1/3}$	$+\frac{1}{3}$	${\hat{D}}_j$	${\hat{D}}_{jR}$	$(-,-)$
$1$	$1_0$	$(1,1,1)$	${(1,1)}_0$	$0$	$S$		$(+,-)$

${\Psi }_{11}^{\text{'}}$
$SO(10)$	$SU{(5)}_Z$	$G_{PS}$	$G_{\text{SM}}$	$Q_{\text{EM}}$	name	zero mode	parity (left)
$10$	$5_{+2}$	$(2,2,1)$	${(1,2)}_{+1/2}$	$\begin{array}{c}\begin{array}{c}+1\end{array}\\ 0\end{array}$	$\begin{array}{c}\begin{array}{c}{\hat{E}}^{\prime }\end{array}\\ {\hat{N}}^{\prime }\end{array}$		$(+,-)$
	${\overline{5}}_{-2}$	$(2,2,1)$	${(1,2)}_{-1/2}$	$\begin{array}{c}\begin{array}{c}0\end{array}\\ -1\end{array}$	$\begin{array}{c}\begin{array}{c}{N}^{\prime }\end{array}\\ E^{\prime }\end{array}$		$(+,-)$
	$5_{+2}$	$(1,1,6)$	${(3,1)}_{-1/3}$	$-\frac{1}{3}$	$D_j^{\text{'}}$	$D_{jL}^{\text{'}}$	$(+,+)$
	${\overline{5}}_{-2}$	$(1,1,6)$	${(\overline{3},1)}_{+1/3}$	$+\frac{1}{3}$	${\hat{D}}_j^{\text{'}}$	${\hat{D}}_{jL}^{\text{'}}$	$(+,+)$
$1$	$1_0$	$(1,1,1)$	${(1,1)}_0$	$0$	$S^{\prime }$		$(-,+)$

2.2. Brane interactions

In addition to (2.3) and (2.10), there appear brane mass–Yukawa interactions among ${\Psi }_{32}^a$⁠, ${\Psi }_{11}^a$⁠, ${\Psi }_{11}^{\prime a}$⁠, and ${\Phi }_{16}$ on the Planck brane at $y=0$⁠. On the Planck brane the $SO(10)$ local gauge invariance must be manifestly preserved. The ${\Psi }_{32}$ (⁠${\Psi }_{11}$⁠) field decomposes to ${\Psi }_{16}$ and ${\Psi }_{\overline{16}}$ (⁠${\Psi }_{10}$ and ${\Psi }_1$⁠), as indicated in Table 1 (Table 2), under $SO(10)$ transformations. Only fields of even parity at $y=0$ participate in the brane mass–Yukawa interactions. We need to write down $SO(10)$-invariant terms in terms of ${\Psi }_{16L}^a$⁠, ${\Psi }_{\overline{16}R}^a$⁠, ${\Psi }_{10R}^a$⁠, ${\Psi }_{1L}^a$⁠, ${\Psi }_{10L}^{\prime a}$⁠, ${\Psi }_{1R}^{\prime a}$⁠, and ${\Phi }_{16}$⁠. Further, we impose the condition that the action be invariant under a global $U(1)$$\Psi$ fermion number (⁠$N_{\Psi }$⁠) transformation  

$${\Psi }_{32}^a\to e^{i\alpha }{\Psi }_{32}^a,{\Psi }_{11}^a\to e^{i\alpha }{\Psi }_{11}^a,{\Psi }_{11}^{\prime a}\to e^{i\alpha }{\Psi }_{11}^{\prime a}.$$

(2.21)

Six types of brane interactions are allowed:  

 

(2.22)

Here, ${\tilde{\Phi }}_{\overline{16}}=\hat{R}{\Phi }_{16}^{*}$ with $\hat{R}$ defined in (A7) transforms as $\overline{16}$⁠, and we have employed 32-component notation given by  

$${\hat{\Phi }}_{16}=\left(\begin{array}{c}\begin{array}{c}{\Phi }_{16}\\ 0\end{array}\end{array}\right),{\hat{\tilde{\Phi }}}_{\overline{16}}=\left(\begin{array}{c}\begin{array}{c}0\\ {\tilde{\Phi }}_{\overline{16}}\end{array}\end{array}\right),{\hat{\Psi }}_{16}=\left(\begin{array}{c}\begin{array}{c}{\Psi }_{16}\\ 0\end{array}\end{array}\right),{\hat{\Psi }}_{\overline{16}}=\left(\begin{array}{c}\begin{array}{c}0\\ {\Psi }_{\overline{16}}\end{array}\end{array}\right).$$

(2.23)

In general, all coefficients $\kappa$ and $\mu$ in (2.22) have matrix structure in the generation space, which induces flavor mixing. In the present paper we restrict ourselves to diagonal $\kappa$ and $\mu$⁠.

The total action is given by  

$$S=S_{\text{bulk}}+S_{{\Phi }_{16}}+S_{\text{brane}}.$$

(2.24)

2.3. EW Higgs boson

The orbifold boundary condition (2.5) reduces the $SO(11)$ gauge symmetry to $SO(4)\times SO(6)\simeq SU{(2)}_L\times SU{(2)}_R\times SO(6)$⁠. It is easy to see that terms bilinear in fields in the gauge field part of the action $S_{\text{bulk}}$⁠, (2.3), become  

$$\begin{array}{l}{\displaystyle \int d^4}x\frac{dz}{kz}{\displaystyle \sum _{j<k}[}\frac{1}{2}{A}_{\mu }^{(jk)}\{{\eta }^{\mu \nu }(\square +k^2{\mathcal{P}}_4)-(1-\frac{1}{\xi }){\partial }^{\mu }{\partial }^{\nu }\}{A}_{\nu }^{(jk)}\\ +\frac{1}{2}{k}^2{A}_z^{(jk)}(\square +\xi k^2{\mathcal{P}}_z)A_z^{(jk)}+{\overline{c}}^{(jk)}(\square +\xi k^2{\mathcal{P}}_4)c^{(jk)}],\\ \square \equiv {\eta }^{\mu \nu }{\partial }_{\mu }{\partial }_{\nu },{\partial }^{\mu }={\eta }^{\mu \nu }{\partial }_{\nu },{\mathcal{P}}_4\equiv z\frac{\partial }{\partial z}\frac{1}{z}\frac{\partial }{\partial z},{\mathcal{P}}_z\equiv \frac{\partial }{\partial z}z\frac{\partial }{\partial z}\frac{1}{z}.\end{array}$$

(2.25)

For four-dimensional components $A_{\mu }$ of $SO(10)$⁠, gauge fields have additional bilinear terms coming from the brane scalar interaction $S_{{\Phi }_{16}}$⁠, (2.10), with $\langle {\Phi }_{16}\rangle \ne 0$⁠. The parity of $A_{\mu }$ and $A_z$ is summarized in Table 3. For $A_z$⁠, only four components $(2,2,1)$ in $G_{PS}=SU{(2)}_L\times SU{(2)}_R\times SO(6)$ have parity $(+,+)$⁠, and therefore zero modes corresponding to the Higgs doublet in the SM. For $A_{\mu }$⁠, components $(3,1,1)$⁠, $(1,3,1)$⁠, and $(1,1,15)$⁠, corresponding to $SU{(2)}_L$⁠, $SU{(2)}_R$⁠, and $SO(6)$⁠, respectively, have parity $(+,+)$⁠. $SU{(2)}_R\times SO(6)$ symmetry is spontaneously broken to $SU{(3)}_C\times U{(1)}_Y$ by $\langle {\Phi }_{16}\rangle \ne 0$⁠, reducing to the SM gauge symmetry.

Table 3.

Parity of $A_{\mu }$ and $A_z$ classified with the content in $G_{PS}=SU{(2)}_L\times SU{(2)}_R\times SO(6)$⁠.

$G_{PS}$	$A_{\mu }$	$A_z$
$(3,1,1)$	$(+,+)$	$(-,-)$
$(1,3,1)$	$(+,+)$	$(-,-)$
$(1,1,15)$	$(+,+)$	$(-,-)$
$(2,2,6)$	$(+,-)$	$(-,+)$
$(2,2,1)$	$(-,-)$	$(+,+)$
$(1,1,6)$	$(-,+)$	$(+,-)$

Mode functions of $A_z$ in the fifth dimension are determined by  

$${\mathcal{P}}_z{h}_n(z)=-{\lambda }_n^2{h}_n(z),{\displaystyle {\int }_1^{z_L}\frac{kdz}{z}}{h}_n(z)h_{\ell }(z)={\delta }_{n\ell },$$

(2.26)

where boundary conditions at $z=1$ and $z_L$ are given by $(d/dz)(h_n/z)=0$ or $h_n=0$ for parity even or odd fields, respectively. In particular, the zero mode (⁠${\lambda }_0=0$⁠) function is given by  

$$h_0^{(++)}(z)=u_H(z)=\frac{1}{kz}{\tilde{u}}_H(y)=\sqrt{\frac{2}{k(z_L^2-1)}}z$$

(2.27)

for $1\le z\le z_L$ (⁠$0\le y\le L$⁠). The mode function in the $y$-coordinate satisfies ${\tilde{u}}_H(-y)={\tilde{u}}_H(y)={\tilde{u}}_H(y+2L)$⁠. We note that  

$${\displaystyle {\int }_0^{L}d}y{\tilde{u}}_H(y)={\displaystyle {\int }_1^{z_L}d}z{u}_H(z)=\sqrt{\frac{z_L^2-1}{2k}}.$$

(2.28)

The zero modes of $A_z$ are physical degrees of freedom, being unable to be gauged away. They are in $A_z^{a11}(x,z)$ (⁠$a=1,\dots ,4$⁠). In terms of mode functions $\{h_n^{(++)}(z)\}$ for the parity $(+,+)$ boundary condition,  

$$A_z^{a11}(x,z)={\phi }_H^a(x)u_H(z)+{\displaystyle \sum _{n=1}^{\infty }{\phi }_H^{a(n)}}(x)h_n^{(++)}(z)(a=1,\dots ,4),$$

(2.29)

where the four-component real field, ${\phi }_H^a(x)$⁠, plays the role of the EW Higgs doublet field in the SM. It will be shown that ${\phi }_H\equiv {\phi }_H^4$ dynamically develops VEV $\langle {\phi }_H\rangle \ne 0$⁠, breaking the SM symmetry to $U{(1)}_{\text{EM}}$⁠. ${\phi }_H^{1,2,3}$ are absorbed by $W$ and $Z$ bosons.

2.4. AB phase ${\theta }_H$

Under a general gauge transformation $A_M^{\text{'}}=\Omega A_M{\Omega }^{-1}+(i/g)\Omega {\partial }_M{\Omega }^{-1}$⁠, new gauge potentials satisfy new boundary conditions  

$$\begin{array}{c}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{\text{'}}\\ A_y^{\text{'}}\end{array}\end{array}\right)(x,y_j-y)=P_j^{\text{'}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }\\ -A_y\end{array}\end{array}\right)(x,y_j+y)P_j^{\prime -1}+\frac{i}{g}{P}_j^{\text{'}}\left(\begin{array}{c}\begin{array}{c}{\partial }_{\mu }\end{array}\\ -{\partial }_y\end{array}\right)P_j^{\prime -1},\\ P_j^{\text{'}}=\Omega (x,y_j-y)P_j\Omega {(x,y_j+y)}^{-1}.\end{array}$$

(2.30)

The original boundary conditions (2.5) are maintained if and only if $P_j^{\text{'}}=P_j$⁠. Such a class of gauge transformations define the residual gauge invariance [2,60].

There is a class of “large” gauge transformations which transform $A_y$ nontrivially. Consider  

$$\Omega (y;\alpha )=\exp \{-i\frac{g\alpha }{\sqrt{2}}{\displaystyle {\int }_y^{L}d}y{\tilde{u}}_H(y)T_{4,11}\}.$$

(2.31)

For $A_y=\frac{1}{\sqrt{2}}{A}_y^{(4,11)}(x,y)T_{4,11}$⁠,  

$$\begin{array}{l}{A}_y^{\text{'}}=A_y+\frac{\alpha }{\sqrt{2}}{\tilde{u}}_H(y)T_{4,11},\\ {\phi }_H(x)\to {\phi }_H^{\text{'}}(x)={\phi }_H(x)+\alpha .\end{array}$$

(2.32)

The new boundary condition matrices $P_{j^{\prime }}$ are evaluated, with the aid of $\{P_j,T_{4,11}\}=0$⁠, to be $P_j^{\text{'}}=\Omega (y_j-y;\alpha )\Omega (y_j+y;\alpha )P_j$⁠, that is,  

$$\begin{array}{l}{P}_0^{\text{'}}=\Omega (-y;\alpha )\Omega (y;\alpha )P_0=\Omega (0;2\alpha )P_0,\\ P_1^{\text{'}}=\Omega (L-y;\alpha )\Omega (L+y;\alpha )P_1=P_1.\end{array}$$

(2.33)

The boundary conditions are preserved provided $\Omega (0;2\alpha )=1$⁠. As ${(T_{4,11}^{\text{sp}})}^2=\frac{1}{4}{I}_{32}$ in the spinorial representation, the boundary conditions are preserved provided  

$$\frac{g\alpha }{\sqrt{2}}{\displaystyle {\int }_0^{L}d}y{\tilde{u}}_H(y)=\frac{g\alpha }{\sqrt{2}}\sqrt{\frac{z_L^2-1}{2k}}=2\pi n(n=\text{an\hspace{0.5em}integer}).$$

(2.34)

Aharonov–Bohm (AB) phases along the fifth dimension are defined by phases of eigenvalues of  

$$\hat{W}=P\exp \left\{ig{\displaystyle {\int }_{-L}^{L}d}y{A}_y\right\}\cdot P_1{P}_0.$$

(2.35)

They are gauge invariant. $A_y^{4,11}(x,-y)=A_y^{4,11}(x,y)$ and the orthonormality relation (2.26) implies that ${\displaystyle {\int }_1^{z_L}d}z{h}_n^{(++)}(z)=0$ for $n\ne 0$⁠. Hence, for $A_y=(1/\sqrt{2})A_y^{4,11}{T}_{4,11}$⁠,  

$$\hat{W}=\exp \{i\frac{g{\phi }_H}{\sqrt{2}}{\displaystyle {\int }_0^{L}d}y{\tilde{u}}_H(y)2T_{4,11}\}\cdot \left(\begin{array}{ccc}{I}_4& & \\ & -I_6& \\ & & 1\end{array}\right)$$

(2.36)

in the vectorial representation so that the relevant phase ${\hat{\theta }}_H(x)$ is given by  

$$\begin{array}{c}{\hat{\theta }}_H(x)=\frac{g{\phi }_H(x)}{\sqrt{2}}{\displaystyle {\int }_0^{L}d}y{\tilde{u}}_H(y)=\frac{{\phi }_H(x)}{f_H},\\ f_H=\frac{2}{g}\sqrt{\frac{k}{z_L^2-1}}=\frac{2}{g_w}\sqrt{\frac{k}{L(z_L^2-1)}}.\end{array}$$

(2.37)

Under a large gauge transformation satisfying (2.33),  

$${\hat{\theta }}_H(x)\to {\hat{\theta }}_{H^{\prime }}(x)={\hat{\theta }}_H(x)+2\pi n.$$

(2.38)

Denoting ${\theta }_H=\langle {\hat{\theta }}_H(x)\rangle$⁠, one has  

$$A_z^{(4,11)}(x,z)=\{{\theta }_H{f}_H+H(x)\}{u}_H(z)+\cdots .$$

(2.39)

$H(x)$ corresponds to the neutral Higgs boson found at LHC. The value of ${\theta }_H$ is undetermined at the tree level, but is determined dynamically at the one-loop level. The effective potential $V_{\text{eff}}({\theta }_H)$ is evaluated in Sect. 5.

2.5. Twisted gauge

Under the gauge transformation (2.31) ${\hat{\theta }}_H(x)$ is transformed to ${\hat{\theta }}_H^{\text{'}}(x)={\hat{\theta }}_H(x)+(\alpha /f_H)$⁠. In the new gauge given with $\alpha =-{\theta }_H{f}_H$⁠,  

$$\begin{array}{c}\Omega (y)=\exp \{i\frac{g{\theta }_H{f}_H}{\sqrt{2}}{\displaystyle {\int }_y^{L}d}y{\tilde{u}}_H(y)\cdot T_{4,11}\}\\ =\exp \{i\theta (z)T_{4,11}\},\theta (z)={\theta }_H\frac{z_L^2-z^2}{z_L^2-1}\text{\hspace{1em}for}1\le z\le z_L,\end{array}$$

(2.40)

$\langle {\theta }_H^{\text{'}}(x)\rangle =0$⁠. Hereafter, this new gauge is called the twisted gauge, and quantities in the twisted gauge are denoted with tildes [8,75]. According to (2.33), the boundary conditions become  

$${\tilde{P}}_0=\Omega {(0)}^2{P}_0=e^{2i{\theta }_H{T}_{4,11}}{P}_0,{\tilde{P}}_1=P_1.$$

(2.41)

In the vectorial representation,  

$${\tilde{P}}_0^{\text{vec}}=\{\begin{array}{ll}\left(\begin{array}{cc}\cos 2{\theta }_H& -\sin 2{\theta }_H\\ -\sin 2{\theta }_H& -\cos 2{\theta }_H\end{array}\right)\hfill & \text{in\hspace{0.5em}the\hspace{0.5em}4}-\text{11\hspace{0.5em}subspace,}\hfill \\ I_9\hfill & \text{otherwise.}\hfill \end{array}$$

(2.42)

Components of ${\Psi }_{32}$ split into two groups:  

$$\begin{array}{c}\chi =\left(\begin{array}{c}\begin{array}{c}\nu \\ {\nu }^{\prime }\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}e\end{array}\\ e^{\prime }\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{u}_j\end{array}\\ u_j^{\text{'}}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{d}_j\end{array}\\ d_j^{\text{'}}\end{array}\right),\\ \hat{\chi }=\left(\begin{array}{c}\begin{array}{c}\hat{\nu }\\ {\hat{\nu }}^{\prime }\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}\hat{e}\\ {\hat{e}}^{\prime }\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{\hat{u}}_j\\ {\hat{u}}_j^{\text{'}}\end{array}\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{\hat{d}}_j\\ {\hat{d}}_j^{\text{'}}\end{array}\end{array}\right).\end{array}$$

(2.43)

The boundary condition matrix becomes  

$${\tilde{P}}_0^{\text{sp}}=\left(\begin{array}{cc}\cos {\theta }_H& \mp i\sin {\theta }_H\\ \pm i\sin {\theta }_H& -\cos {\theta }_H\end{array}\right)\text{\hspace{1em}for}\{\begin{array}{c}\begin{array}{c}\chi \\ \hat{\chi }\end{array}\end{array}.$$

(2.44)

It turns out very convenient to evaluate $V_{\text{eff}}({\theta }_H)$ in the twisted gauge.

2.6. Proton stability

In the present model of gauge–Higgs grand unification, proton decay is forbidden. Fields of up quark quantum number, $u$ and $u^{\prime }$⁠, are contained only in ${\Psi }_{32}$⁠. Fields of down quark quantum number are in ${\Psi }_{32}$ (⁠$d$ and $d^{\prime }$⁠) and in ${\Psi }_{11}$ and ${\Psi }_{11}^{\text{'}}$ (⁠$D$ and $D^{\prime }$⁠); fields of electron quantum number are in ${\Psi }_{32}$ (⁠$e$ and $e^{\prime }$⁠) and in ${\Psi }_{11}$ and ${\Psi }_{11}^{\text{'}}$ (⁠$E$ and $E^{\prime }$⁠); and fields of neutrino quantum number are in ${\Psi }_{32}$ (⁠$\nu$⁠, $\hat{\nu }$⁠, ${\nu }^{\prime }$⁠, and ${\hat{\nu }}^{\prime }$⁠) and in ${\Psi }_{11}$ and ${\Psi }_{11}^{\text{'}}$ (⁠$N$⁠, $\hat{N}$⁠, $S$⁠, $N^{\prime }$⁠, ${\hat{N}}^{\prime }$⁠, and $S^{\prime }$⁠). The down quark at low energies, for instance, is a linear combination of $d$⁠, $d^{\prime }$⁠, $D$⁠, and $D^{\prime }$⁠. All quarks and leptons have $N_{\Psi }=1$ fermion number. The total action preserves the $N_{\Psi }=1$ fermion number. As the proton has $N_{\Psi }=3$ and the positron has $N_{\Psi }=-1$⁠, the proton decay $p\to {\pi }^0{e}^{+}$⁠, for instance, cannot occur. This is contrasted with the $SU(5)$ or $SO(10)$ GUT in four dimensions in which proton decay inevitably takes place. The $SO(11)$ gauge–Higgs grand unification provides a natural framework of grand unification in which proton decay is forbidden.

One comment is in order. $S$ and $S^{\prime }$ in ${\Psi }_{11}$ and ${\Psi }_{11}^{\text{'}}$ are $SO(10)$ singlets. One could introduce Majorana masses, such as $\overline{S}{S}^c\delta (y)$ on the Planck brane, which break the $N_{\Psi }$ fermion number. They would give rise to Majorana masses for neutrinos, and at the same time could induce proton decay at higher loops.

3. Spectrum of gauge fields

$V_{\text{eff}}({\theta }_H)$ at the one-loop level is determined from the mass spectrum of all fields when $\langle {\hat{\theta }}_H(x)\rangle ={\theta }_H$⁠. It is convenient to determine the spectrum in the twisted gauge in which $\langle {\tilde{\hat{\theta }}}_H(x)\rangle =0$⁠. The nontrivial ${\theta }_H$ dependence is transferred to the boundary conditions. In this section we determine the spectrum of gauge fields.

In the absence of the brane interactions with ${\Phi }_{16}$ in $S_{{\Phi }_{16}}$⁠, (2.10), the boundary conditions for gauge fields are given by  

$$\begin{array}{l}\{\begin{array}{ll}N:\frac{\partial }{\partial z}{A}_{\mu }=0\hfill & \text{for\hspace{0.5em}parity}=+\hfill \\ D:A_{\mu }=0\hfill & \text{for\hspace{0.5em}parity}=-\hfill \end{array}\\ \{\begin{array}{ll}N:\frac{\partial }{\partial z}(\frac{1}{z}{A}_z)=0\hfill & \text{for\hspace{0.5em}parity}=+\hfill \\ D:A_z=0\hfill & \text{for\hspace{0.5em}parity}=-\hfill \end{array}\end{array}$$

(3.1)

at $z=1$ (⁠$y=0$⁠) and $z=z_L$ (⁠$y=L$⁠). In the presence of $S_{{\Phi }_{16}}$⁠, the brane mass terms (2.12) for $A_{\mu }$ are induced, and the Neumann boundary condition $N$ is modified to an effective Dirichlet condition $D_{\text{eff}}$ for low-lying KK modes of the 21 components of $A_{\mu }$ as described below. The boundary conditions for the gauge fields are summarized in Table 4.

Table 4.

A Venn diagram of the gauge group structure is displayed above, and the boundary conditions in each category are summarized in the table below. Here, $G_{\text{SM}}=SU{(3)}_C\times SU{(2)}_L\times U{(1)}_Y$⁠, $G_{PS}=SU{(2)}_L\times SU{(2)}_R\times SO(6)$⁠, and $SU(5)\cap G_{PS}=G_{\text{SM}}$⁠. $N$ and $D$ stand for Neumann and Dirichlet conditions, respectively. $D_{\text{eff}}$ represents the effective Dirichlet condition explained in Sect. 3.1, (i), (ii), (v), (vii), and (viii).

In the twisted gauge, ${\tilde{A}}_M=\Omega (z)A_M\Omega {(z)}^{-1}+\frac{i}{g}\Omega (z){\partial }_M\Omega {(z)}^{-1}$⁠, where $\Omega (z)$ is given by (2.40). One finds  

$$\begin{array}{c}{A}_M^{k4}=\cos \theta (z){\tilde{A}}_M^{k4}-\sin \theta (z){\tilde{A}}_M^{k,11},(k\ne 4),\\ A_M^{k,11}=\sin \theta (z){\tilde{A}}_M^{k4}+\cos \theta (z){\tilde{A}}_M^{k,11},\\ A_z^{4,11}={\tilde{A}}_z^{4,11}-\frac{\sqrt{2}}{g}{\theta }^{\prime }(z)={\tilde{A}}_z^{4,11}+\frac{2\sqrt{2}}{g}{\theta }_H\frac{z}{z_L^2-1}.\end{array}$$

(3.2)

All other components are unchanged. At $z=z_L$⁠, $\theta (z_L)=0$⁠, and $A_M^{k4}$ and $A_M^{k,11}$ always have opposite parity. It follows that ${\tilde{A}}_M$ satisfies the same boundary condition as $A_M$ at $z=z_L$⁠. In the bulk (⁠$1<z<z_L$⁠) the bilinear part of the action is the same as in the free theory in the twisted gauge. Hence, depending on the boundary condition at $z=z_L$⁠, wave functions for ${\tilde{A}}_{\mu }$ and ${\tilde{A}}_z$ are given by  

$$\begin{array}{l}{\tilde{A}}_{\mu }N:C(z;\lambda ),D:S(z;\lambda ),\\ {\tilde{A}}_{z}N:S^{\prime }(z;\lambda ),D:C^{\prime }(z;\lambda ),\end{array}$$

(3.3)

where $C(z;\lambda )$ and $S(z;\lambda )$ are defined in Appendix B.

3.1. $A_{\mu }$ components

3.1.1. (i) $({\tilde{A}}_{\mu }^{a_L},{\tilde{A}}_{\mu }^{a_R},{\tilde{A}}_{\mu }^{a,11})$$(a=1,2)$⁠: $W$ and $W_R$ towers

The original $A_{\mu }^{a_L}$⁠, $A_{\mu }^{a_R}$⁠, and $A_{\mu }^{a,11}$ have parity $(+,+)$⁠, $(+,+)$⁠, and $(-,-)$⁠, respectively. $A_{\mu }^{a_R}$ picks up a brane mass. To find its boundary condition at $z=1$⁠, we need to recall the equation of motion. ${\mathcal{L}}_{\text{branemass}}^{\text{gauge}}$ in (2.12) yields $-(g^2{w}^2/4)\delta (y){(A_{\mu }^{a_R})}^2$⁠. The equation of motion for $A_{\mu }^{a_R}$ in the $y$-coordinate is  

$$\{{\eta }^{\mu \nu }(\square +\frac{\partial }{\partial y}{e}^{-2\sigma (y)}\frac{\partial }{\partial y}-\frac{g^2{w}^2}{2}\delta (y))-(1-\frac{1}{\xi }){\partial }^{\mu }{\partial }^{\nu }\}{A}_{\nu }^{a_R}=\cdots ,$$

(3.4)

where interaction terms on the right-hand side involve neither the total $y$ derivative nor $\delta (y)$⁠. By integrating the equation ${\displaystyle {\int }_{-ϵ}^{ϵ}d}y\cdots$ and taking the limit $ϵ\to 0$⁠, one finds that $\partial A_{\mu }^{a_R}/\partial y{|}_{y=ϵ}=(g^2{w}^2/4)A_{\mu }^{a_R}{|}_{y=0}$⁠. The brane mass gives rise to cusp behavior at $y=0$⁠. The boundary conditions for $(A_{\mu }^{a_L},A_{\mu }^{a_R},A_{\mu }^{a,11})$ (⁠$a=1,2$⁠) at $z=1$ are  

$$\begin{array}{c}\frac{\partial }{\partial z}{A}_{\mu }^{a_L}=0,\\ (\frac{\partial }{\partial z}-\omega )A_{\mu }^{a_R}=0,\omega =\frac{g^2{w}^2}{4k},\\ A_{\mu }^{a,11}=0.\end{array}$$

(3.5)

Here it is understood that $\partial A_{\mu }^{a_R}/\partial z$ is evaluated at $z=1^{+}$⁠.

Expressed in terms of fields in the twisted gauge, (3.5) becomes  

$$\begin{array}{l}{\partial }_z{\tilde{A}}_{\mu }^{23}+{\partial }_z{\tilde{A}}_{\mu }^{14}\cos {\theta }_H-{\partial }_z{\tilde{A}}_{\mu }^{1,11}\sin {\theta }_H=0,\\ ({\partial }_z-\omega ){\tilde{A}}_{\mu }^{23}-({\partial }_z-\omega ){\tilde{A}}_{\mu }^{14}\cos {\theta }_H+({\partial }_z-\omega ){\tilde{A}}_{\mu }^{1,11}\sin {\theta }_H=0,\\ {\tilde{A}}_{\mu }^{14}\sin {\theta }_H+{\tilde{A}}_{\mu }^{1,11}\cos {\theta }_H=0.\end{array}$$

(3.6)

From the boundary conditions at $z_L$⁠, one can set  

$$[{\tilde{A}}_{\mu }^{23},{\tilde{A}}_{\mu }^{14},{\tilde{A}}_{\mu }^{1,11}]=[{\alpha }_{23}C(z;\lambda ),{\alpha }_{14}C(z;\lambda ),{\alpha }_{1,11}S(z;\lambda )]a_{\mu }(x)$$

for each mode. The boundary conditions (3.6) are transformed to  

$$\begin{array}{l}K\left(\begin{array}{c}\begin{array}{c}{\alpha }_{23}\\ {\alpha }_{14}\\ {\alpha }_{1,11}\end{array}\end{array}\right)=0,\\ K=\left(\begin{array}{ccc}{C}^{\prime }& \cos {\theta }_H{C}^{\prime }& -\sin {\theta }_H{S}^{\prime }\\ C^{\prime }-\omega C& -\cos {\theta }_H(C^{\prime }-\omega C)& \sin {\theta }_H(S^{\prime }-\omega S)\\ 0& \sin {\theta }_{H}C& \cos {\theta }_{H}S\end{array}\right),\end{array}$$

(3.7)

where $C^{\prime }=C^{\prime }(1;\lambda )$⁠, etc. The spectrum $\left\{{\lambda }_n\right\}$ is determined by $\det K=0$⁠, or by  

$$2C^{\prime }(S{C}^{\prime }+\lambda {\sin }^2{\theta }_H)-\omega C(2S{C}^{\prime }+\lambda {\sin }^2{\theta }_H)=0.$$

(3.8)

For sufficiently large $w$⁠, the spectrum $\{m_n=k{\lambda }_n\}$ of low-lying KK modes is approximately determined by the second term in (3.8). This approximation for $z_L={10}^5$⁠, for instance, is justified for $\omega >{10}^{-3}$⁠:  

$$\begin{array}{l}W\text{\hspace{0.17em}tower:\hspace{0.5em}}2S(1;\lambda )C^{\prime }(1;\lambda )+\lambda {\sin }^2{\theta }_H=0,\\ W_R\text{\hspace{0.17em}tower:\hspace{0.5em}}C(1;\lambda )=0.\end{array}$$

(3.9)

Asymptotically, the equations determining the spectra of the $W$ and $W_R$ towers become $S{C}^{\prime }+\lambda {\sin }^2{\theta }_H=0$ and $C^{\prime }=0$⁠, respectively. The mass of the $W$ boson, $m_W=m_{W^{(0)}}$⁠, is given by  

$$m_W~\sqrt{\frac{k}{L}}{z}_L^{-1}\sin {\theta }_H=\frac{\sin {\theta }_H}{\pi \sqrt{kL}}{m}_{\text{KK}}.$$

(3.10)

3.1.2. (ii) $({\tilde{A}}_{\mu }^{3_L},{\tilde{C}}_{\mu },{\tilde{B}}_{\mu }^Y,{\tilde{A}}_{\mu }^{3,11})$⁠: $\gamma$⁠, $Z$⁠, and $Z_R$ towers

Here, $C_{\mu }=\sqrt{1/5}(A_{\mu }^{12}-A_{\mu }^{34}+A_{\mu }^{56}+A_{\mu }^{78}+A_{\mu }^{910})=\sqrt{2/5}{A}_{\mu }^{3_R}+\sqrt{3/5}{A}_{\mu }^{0_C}$⁠. The original $A_{\mu }^{3_L}$⁠, $C_{\mu }$⁠, $B_{\mu }^Y$⁠, and $A_{\mu }^{3,11}$ have parity $(+,+)$⁠, $(+,+)$⁠, $(+,+)$⁠, and $(-,-)$⁠, respectively. $C_{\mu }$ picks up a brane mass $-\delta (y)(5g^2{w}^2/8)C_{\mu }^2$⁠, so that the boundary conditions at $z=1$ are  

$$\begin{array}{l}\frac{\partial }{\partial z}{A}_{\mu }^{3_L}=0,\\ (\frac{\partial }{\partial z}-\frac{5}{2}\omega )C_{\mu }=0,\\ A_{\mu }^{3,11}=0,\\ \frac{\partial }{\partial z}{B}_{\mu }^Y=0.\end{array}$$

(3.11)

In the twisted gauge they become  

$$\begin{array}{l}{\partial }_z{\tilde{A}}_{\mu }^{12}+{\partial }_z{\tilde{A}}_{\mu }^{34}\cos {\theta }_H-{\partial }_z{\tilde{A}}_{\mu }^{3,11}\sin {\theta }_H=0,\\ ({\partial }_z-\frac{5}{2}\omega )\{{\tilde{A}}_{\mu }^{12}-{\tilde{A}}_{\mu }^{34}\cos {\theta }_H+{\tilde{A}}_{\mu }^{3,11}\sin {\theta }_H+\sqrt{3}{\tilde{A}}_{\mu }^{0_C}\}=0,\\ {\tilde{A}}_{\mu }^{34}\sin {\theta }_H+{\tilde{A}}_{\mu }^{3,11}\cos {\theta }_H=0,\\ {\partial }_z{\tilde{A}}_{\mu }^{12}-{\partial }_z{\tilde{A}}_{\mu }^{34}\cos {\theta }_H+{\partial }_z{\tilde{A}}_{\mu }^{3,11}\sin {\theta }_H-\frac{2}{\sqrt{3}}{\partial }_z{\tilde{A}}_{\mu }^{0_C}=0.\end{array}$$

(3.12)

Adding the first and second equations, one gets  

$$(2{\partial }_z-\frac{5}{2}\omega ){\tilde{A}}_{\mu }^{12}+\frac{5}{2}\omega ({\tilde{A}}_{\mu }^{34}\cos {\theta }_H-{\tilde{A}}_{\mu }^{3,11}\sin {\theta }_H)+\sqrt{3}({\partial }_z-\frac{5}{2}\omega ){\tilde{A}}_{\mu }^{0_C}=0.$$

Adding the first and fourth equations, one gets  

$${\partial }_z{\tilde{A}}_{\mu }^{12}-\frac{1}{\sqrt{3}}{\partial }_z{\tilde{A}}_{\mu }^{0_C}=0.$$

Writing $[{\tilde{A}}_{\mu }^{12},{\tilde{A}}_{\mu }^{34},{\tilde{A}}_{\mu }^{3,11},{\tilde{A}}_{\mu }^{0_C}]=[{\alpha }_{12}C(z),{\alpha }_{34}C(z),{\alpha }_{3,11}S(z),{\alpha }_{0_C}C(z)]a_{\mu }(x)$⁠, one finds that  

$$\begin{array}{l}K\left(\begin{array}{c}\begin{array}{c}{\alpha }_{12}\\ {\alpha }_{34}\\ {\alpha }_{3,11}\\ {\alpha }_{0_C}\end{array}\end{array}\right)=0.\\ K=\left(\begin{array}{cccc}{C}^{\prime }& \cos {\theta }_H{C}^{\prime }& -\sin {\theta }_H{S}^{\prime }& 0\\ \frac{4}{5}{C}^{\prime }-\omega C& \omega \cos {\theta }_{H}C& -\omega \sin {\theta }_{H}S& \sqrt{3}(\frac{2}{5}{C}^{\prime }-\omega C)\\ 0& \sin {\theta }_{H}C& \cos {\theta }_{H}S& 0\\ C^{\prime }& 0& 0& -\frac{1}{\sqrt{3}}{C}^{\prime }\end{array}\right).\end{array}$$

(3.13)

The spectrum is determined by $\det K=0$⁠:  

$$C^{\prime }\{2C^{\prime }(S{C}^{\prime }+\lambda {\sin }^2{\theta }_H)-\omega C(5S{C}^{\prime }+4\lambda {\sin }^2{\theta }_H)\}=0.$$

(3.14)

For sufficiently large $w$⁠, the spectrum of low-lying KK modes is approximately determined by the second term in (3.14). One finds that  

$$\begin{array}{l}\gamma \text{\hspace{0.17em}tower:\hspace{0.5em}}{C}^{\prime }(1;\lambda )=0,\\ Z\text{\hspace{0.17em}tower:\hspace{0.5em}}5S(1;\lambda )C^{\prime }(1;\lambda )+4\lambda {\sin }^2{\theta }_H=0,\\ Z_R\text{\hspace{0.17em}tower:\hspace{0.5em}}C(1;\lambda )=0.\end{array}$$

(3.15)

The mass of the $Z$ boson, $m_Z=m_{Z^{(0)}}$⁠, is given by  

$$m_Z~\sqrt{\frac{8k}{5L}}{z}_L^{-1}\sin {\theta }_H=\frac{m_W}{\cos {\theta }_W},{\sin }^2{\theta }_W=\frac{3}{8}.$$

(3.16)

3.1.3. (iii) ${\tilde{A}}_{\mu }^{4,11}$⁠: ${\hat{A}}^4$ tower

$A_{\mu }^{4,11}$ obeys $(D,D)$ and there is no zero mode. Its spectrum is determined by  

$${\hat{A}}^4\text{tower:\hspace{1em}}S(1;\lambda )=0.$$

(3.17)

3.1.4. (iv) $SU{(3)}_C$ gluons

The boundary condition is $(N,N)$ so that  

$$\text{gluon\hspace{0.5em}tower:\hspace{1em}}{C}^{\prime }(1;\lambda )=0.$$

(3.18)

3.1.5. (v) $X$-gluons

These are six components given by  

$$\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{57}-A_{\mu }^{68}\\ A_{\mu }^{58}+A_{\mu }^{67}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{59}-A_{\mu }^{610}\\ A_{\mu }^{510}+A_{\mu }^{69}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{79}-A_{\mu }^{810}\\ A_{\mu }^{710}+A_{\mu }^{89}\end{array}\end{array}\right),$$

(3.19)

which originally obey $(N,N)$ boundary conditions. They have brane masses of the form $-\delta (y)(g^2{w}^2/4)A_{\mu }^2$ in (2.12) so that the boundary conditions at $z=1$ become $({\partial }_z-\omega )A_{\mu }=0$⁠. Consequently, the spectrum is determined by $C^{\prime }-\omega C=0$⁠. For the low-lying KK modes,  

$$\text{X-gluon\hspace{0.5em}tower:\hspace{1em}}C(1;\lambda )=0.$$

(3.20)

3.1.6. (vi) $X$-bosons

These are six components given by  

$$\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{15}+A_{\mu }^{26}\\ A_{\mu }^{16}-A_{\mu }^{25}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{17}+A_{\mu }^{28}\\ A_{\mu }^{18}-A_{\mu }^{27}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{19}+A_{\mu }^{210}\\ A_{\mu }^{110}-A_{\mu }^{29}\end{array}\end{array}\right),$$

(3.21)

which originally obey $(N,D)$ boundary conditions. There is no brane mass, and the spectrum is determined by  

$$X\text{-boson\hspace{0.5em}tower:\hspace{1em}}{S}^{\prime }(1;\lambda )=0.$$

(3.22)

3.1.7. (vii) $X^{\prime }$-bosons

These are six components given by  

$$\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{15}-A_{\mu }^{26}\\ A_{\mu }^{16}+A_{\mu }^{25}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{17}-A_{\mu }^{28}\\ A_{\mu }^{18}+A_{\mu }^{27}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{19}-A_{\mu }^{210}\\ A_{\mu }^{110}+A_{\mu }^{29}\end{array}\end{array}\right),$$

(3.23)

which originally obey $(N,D)$ boundary conditions. They have brane masses of the form $-\delta (y)(g^2{w}^2/4)A_{\mu }^2$ in (2.12) so that boundary conditions at $z=1$ become $({\partial }_z-\omega )A_{\mu }=0$⁠. Consequently, the spectrum is determined by $S^{\prime }-\omega S=0$⁠. For the low-lying KK modes,  

$$X^{\prime }\text{-boson\hspace{0.5em}tower:\hspace{1em}}S(1;\lambda )=0.$$

(3.24)

3.1.8. (viii) $Y$⁠, $Y^{\prime }$-bosons

There are three classes:  

$$\begin{array}{l}Y:\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{35}-A_{\mu }^{46}\\ A_{\mu }^{36}+A_{\mu }^{45}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{37}-A_{\mu }^{48}\\ A_{\mu }^{38}+A_{\mu }^{47}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{39}-A_{\mu }^{410}\\ A_{\mu }^{310}+A_{\mu }^{49}\end{array}\end{array}\right),\\ Y^{\prime }:\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{35}+A_{\mu }^{46}\\ A_{\mu }^{36}-A_{\mu }^{45}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{37}+A_{\mu }^{48}\\ A_{\mu }^{38}-A_{\mu }^{47}\end{array}\end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\begin{array}{c}{A}_{\mu }^{39}+A_{\mu }^{410}\\ A_{\mu }^{310}-A_{\mu }^{49}\end{array}\end{array}\right),\\ \hat{Y}:A_{\mu }^{b11}(b=5,\dots ,10).\end{array}$$

(3.25)

The original fields in $Y$⁠, $Y^{\prime }$⁠, and $\hat{Y}$ satisfy $(N,D)$⁠, $(N,D)$⁠, and $(D,N)$ boundary conditions. $A_{\mu }^{4b}$ and $A_{\mu }^{b11}$ (⁠$b=5,\dots ,10$⁠) mix with each other by ${\theta }_H\ne 0$⁠. Fields in $Y^{\prime }$ have brane masses of the form $-\delta (y)(g^2{w}^2/4)A_{\mu }^2$ in (2.12). Boundary conditions at $z=1$ for $(A_{\mu }^{35},A_{\mu }^{46},A_{\mu }^{611})$⁠, for instance, are  

$${\partial }_z(A_{\mu }^{35}-A_{\mu }^{46})=0,({\partial }_z-\omega )(A_{\mu }^{35}+A_{\mu }^{46})=0,A_{\mu }^{611}=0.$$

(3.26)

Writing $[{\tilde{A}}_{\mu }^{35},{\tilde{A}}_{\mu }^{46},{\tilde{A}}_{\mu }^{11,6}]=[{\alpha }_{35}S(z;\lambda ),{\alpha }_{46}S(z;\lambda ),{\alpha }_{11,6}C(z;\lambda )]a_{\mu }(x)$⁠, one finds that  

$$\left(\begin{array}{ccc}{S}^{\prime }& -\cos {\theta }_H{S}^{\prime }& \sin {\theta }_H{C}^{\prime }\\ S^{\prime }-\omega S& \cos {\theta }_H(S^{\prime }-\omega S)& -\sin {\theta }_H(C^{\prime }-\omega C)\\ 0& \sin {\theta }_{H}S& \cos {\theta }_{H}C\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\alpha }_{35}\\ {\alpha }_{46}\\ {\alpha }_{11,6}\end{array}\end{array}\right)=0.$$

(3.27)

The spectrum is determined by  

$$\begin{array}{l}2S^{\prime }(C{S}^{\prime }-\lambda {\sin }^2{\theta }_H)-\omega S(2C{S}^{\prime }-\lambda {\sin }^2{\theta }_H)\\ =2S^{\prime }(S{C}^{\prime }+\lambda {\cos }^2{\theta }_H)-\omega S\{2S{C}^{\prime }+\lambda (1+{\cos }^2{\theta }_H)\}=0.\end{array}$$

(3.28)

For the low-lying modes, one finds  

$$\begin{array}{l}Y\text{\hspace{0.5em}boson\hspace{0.5em}tower:\hspace{0.5em}}2C(1;\lambda )S^{\prime }(1;\lambda )-\lambda {\sin }^2{\theta }_H=0,\\ Y^{\prime }\text{\hspace{0.5em}boson\hspace{0.5em}tower:\hspace{0.5em}}S(1;\lambda )=0.\end{array}$$

(3.29)

3.2. $A_z$ components

Similarly, one can find the spectrum for $A_z$⁠. The evaluation is simpler as $A_z$ does not couple to the brane scalar field ${\Phi }_{16}$⁠.

3.2.1. (i) $A_z^{ab}(1\le a<b\le 3)$ and $A_z^{jk}(5\le j<k\le 10)$

These components satisfy boundary conditions $(D,D)$⁠, so that  

$$C^{\prime }(1;\lambda )=0.$$

(3.30)

3.2.2. (ii) $A_z^{a4},A_z^{a11}(a=1~3)$

Boundary conditions of $(A_z^{a4},A_z^{a11})$ are $(D,D)$ and $(N,N)$⁠, respectively. $A_z^{a4}$ and $A_z^{a11}$ mix with each other by ${\theta }_H$⁠. Writing $[{\tilde{A}}_z^{a4},{\tilde{A}}_z^{a11}]=[{\beta }_{a4}{C}^{\prime }(z;\lambda ),{\beta }_{a11}{S}^{\prime }(z;\lambda )]a_z(x)$⁠, one finds that at $z=1$⁠,  

$$\left(\begin{array}{cc}\cos {\theta }_H{C}^{\prime }& -\sin {\theta }_H{S}^{\prime }\\ \sin {\theta }_{H}C& \cos {\theta }_{H}S\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\beta }_{a4}\\ {\beta }_{a11}\end{array}\end{array}\right)=0.$$

(3.31)

The spectrum is determined by  

$$S(1;\lambda )C^{\prime }(1;\lambda )+\lambda {\sin }^2{\theta }_H=0.$$

(3.32)

3.2.3. (iii) $A_z^{4,11}$⁠: Higgs tower

This obeys $(N,N)$ boundary conditions, so that  

$$\text{Higgs\hspace{0.5em}tower:\hspace{1em}}S(1;\lambda )=0.$$

(3.33)

There is always a zero mode, which will acquire a mass at the one-loop level.

3.2.4. (iv) $A_z^{ak}$$(a=1~3,k=5~10)$

These components obey $(D,N)$ boundary conditions, so that  

$$S^{\prime }(1;\lambda )=0.$$

(3.34)

3.2.5. (v) $A_z^{k4}$⁠, $A_z^{k11}$$(k=5~10)$

$A_z^{k4}$ and $A_z^{k11}$ satisfy $(D,N)$ and $(N,D)$ boundary conditions, respectively. Writing $[{\tilde{A}}_z^{k4},{\tilde{A}}_z^{k11}]=[{\beta }_{a4}{S}^{\prime }(z;\lambda ),{\beta }_{a11}{C}^{\prime }(z;\lambda )]a_z(x)$⁠, one finds that  

$$\left(\begin{array}{cc}\cos {\theta }_H{S}^{\prime }& -\sin {\theta }_H{C}^{\prime }\\ \sin {\theta }_{H}S& \cos {\theta }_{H}C\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\beta }_{k4}\\ {\beta }_{k11}\end{array}\end{array}\right)=0.$$

(3.35)

The spectrum is determined by  

$$S(1;\lambda )C^{\prime }(1;\lambda )+\lambda {\cos }^2{\theta }_H=0.$$

(3.36)

4. Spectrum of fermion fields

We take Dirac matrices ${\gamma }^A$ in the spinor representation in (2.3):  

$${\gamma }^{\mu }=\left(\begin{array}{cc}& {\sigma }^{\mu }\\ {\overline{\sigma }}^{\mu }& \end{array}\right),{\gamma }^5=\left(\begin{array}{cc}{I}_2& \\ & -I_2\end{array}\right),{\sigma }^{\mu }=(I_2,\overrightarrow{\sigma }),{\overline{\sigma }}^{\mu }=(-I_2,\overrightarrow{\sigma }).$$

(4.1)

The fermion action becomes  

$$\begin{array}{l}{\displaystyle \int d^5}x\sqrt{-\det G}\overline{\Psi }\mathcal{D}(c)\Psi \\ ={\displaystyle \int d^4}x{\displaystyle {\int }_1^{z_L}\frac{dz}{k}}\overline{\check{\Psi }}\left[\begin{array}{cc}-k(D_{-}(c)+ig{A}_z)& {\sigma }^{\mu }({\partial }_{\mu }-ig{A}_{\mu })\\ 5pt{\overline{\sigma }}^{\mu }({\partial }_{\mu }-ig{A}_{\mu })& -k(D_{+}(c)-ig{A}_z)\end{array}\right]\check{\Psi },\end{array}$$

(4.2)

where $\check{\Psi }=z^{-2}\Psi$ and $D_{\pm }(c)$ is defined in (B5).

4.1. Brane mass terms

In addition to (2.3), the fermion fields have brane interactions given by $S_{\text{brane}}$ in (2.22). With $\langle {\Phi }_{16}\rangle \ne 0$ in (2.11), $S_{\text{brane}}$ generates fermion mass terms on the Planck brane. As indicated in (2.22), the mass terms have matrix structure in the three generations. In the present paper we restrict ourselves to the case of diagonal mass matrices, and consider each generation of quarks and leptons separately. We shall drop the generation index henceforth. Each interaction Lagrangian ${\mathcal{L}}_j$ (⁠$j=1,\dots ,6$⁠) in (2.22) generates a brane mass ${\mathcal{L}}_j^m$⁠.  

$$\begin{array}{c}{S}_{\text{brane}}^m={\displaystyle \int d^5}x\sqrt{-\det G}\delta (y)\{{\mathcal{L}}_1^m+{\mathcal{L}}_2^m+{\mathcal{L}}_3^m+{\mathcal{L}}_4^m+{\mathcal{L}}_5^m+{\mathcal{L}}_6^m\},\\ {\mathcal{L}}_1^m=-2{\mu }_1({\overline{S}}_R^{\text{'}}{\hat{\nu }}_L+{\overline{\hat{\nu }}}_L{S}_R^{\text{'}}),\\ {\mathcal{L}}_2^m=-2{\mu }_2({\overline{S}}_L{\nu }_R^{\text{'}}+{\overline{{\nu }^{\prime }}}_R{S}_L),\\ {\mathcal{L}}_3^m=-2{\mu }_3\{i({\overline{E}}_R{e}_L-{\overline{e}}_L{E}_R)+i({\overline{N}}_R{\nu }_L-{\overline{\nu }}_L{N}_R)\\ +({\overline{\hat{D}}}_{1R}{\hat{d}}_{1L}+{\overline{\hat{d}}}_{1L}{\hat{D}}_{1R})+({\overline{\hat{D}}}_{2R}{\hat{d}}_{2L}+{\overline{\hat{d}}}_{2L}{\hat{D}}_{2R})+({\overline{\hat{D}}}_{3R}{\hat{d}}_{3L}+{\overline{\hat{d}}}_{3L}{\hat{D}}_{3R})\},\\ {\mathcal{L}}_4^m=-2{\mu }_4\{i({\overline{\hat{E}}}_L^{\text{'}}{\hat{e}}_R^{\text{'}}-{\overline{{\hat{e}}^{\prime }}}_R{\hat{E}}_L^{\text{'}})+i({\overline{\hat{N}}}_L^{\text{'}}{\hat{\nu }}_R^{\text{'}}-{\overline{{\hat{\nu }}^{\prime }}}_R{\hat{N}}_L^{\text{'}})\\ -({\overline{D}}_{1L}^{\text{'}}{d}_{1R}^{\text{'}}+{\overline{d}}_{1R}^{\text{'}}{D}_{1L}^{\text{'}})+({\overline{D}}_{2L}^{\text{'}}{d}_{2R}^{\text{'}}+{\overline{d}}_{2R}^{\text{'}}{D}_{2L}^{\text{'}})-({\overline{D}}_{3L}^{\text{'}}{d}_{3R}^{\text{'}}+{\overline{d}}_{3R}^{\text{'}}{D}_{3L}^{\text{'}})\},\\ {\mathcal{L}}_5^m=-2{\mu }_5({\overline{S}}_R^{\text{'}}{S}_L+{\overline{S}}_L{S}_R^{\text{'}}),\\ {\mathcal{L}}_6^m=-2{\mu }_6\{{\overline{E}}_L^{\text{'}}{E}_R+{\overline{E}}_R{E}_L^{\text{'}}+{\overline{\hat{E}}}_L^{\text{'}}{\hat{E}}_R+{\overline{\hat{E}}}_R{\hat{E}}_L^{\text{'}}\\ +{\overline{N}}_L^{\text{'}}{N}_R+{\overline{N}}_R{N}_L^{\text{'}}+{\overline{\hat{N}}}_L^{\text{'}}{\hat{N}}_R+{\overline{\hat{N}}}_R{\hat{N}}_L^{\text{'}}\\ +{\displaystyle \sum _{j=1}^3(}{\overline{D}}_{jL}^{\text{'}}{D}_{jR}+{\overline{D}}_{jR}{D}_{jL}^{\text{'}}+{\overline{\hat{D}}}_{jL}^{\text{'}}{\hat{D}}_{jR}+{\overline{\hat{D}}}_{jR}{\hat{D}}_{jL}^{\text{'}})\},\end{array}$$

(4.3)

where  

$$\begin{array}{c}2{\mu }_1={\kappa }_{[1,16]}w,\\ 2{\mu }_2={\kappa }_{[1,\overline{16}]}w,\\ 2{\mu }_3=\sqrt{2}{\kappa }_{[10,16]}w,\\ 2{\mu }_4=\sqrt{2}{\kappa }_{[10,\overline{16}]}w,\\ 2{\mu }_5={\mu }_{[1,1]},\\ 2{\mu }_6={\mu }_{[10,10]}.\end{array}$$

(4.4)

All ${\mu }_k$s are taken to be real without loss of generality. Fermions with $Q_{\text{EM}}=\pm \frac{2}{3}$⁠, $u_j$⁠, $u_j^{\text{'}}$⁠, ${\hat{u}}_j$⁠, ${\hat{u}}_j^{\text{'}}$⁠, do not appear in the brane masses in (4.3).

4.2. Quarks and leptons

To derive the mass spectrum for fermions, we note that the components of ${\Psi }_{32}$ in the original and twisted gauges are related by  

$$\begin{array}{c}\chi =\left(\begin{array}{cc}\cos \frac{1}{2}\theta (z)& -i\sin \frac{1}{2}\theta (z)\\ -i\sin \frac{1}{2}\theta (z)& \cos \frac{1}{2}\theta (z)\end{array}\right)\tilde{\chi },\\ \hat{\chi }=\left(\begin{array}{cc}\cos \frac{1}{2}\theta (z)& i\sin \frac{1}{2}\theta (z)\\ i\sin \frac{1}{2}\theta (z)& \cos \frac{1}{2}\theta (z)\end{array}\right)\tilde{\hat{\chi }},\end{array}$$

(4.5)

where $\chi$ and $\hat{\chi }$ are defined in (2.43), and $\theta (z)$ is given in (2.40). In the original gauge with ${\theta }_H$⁠, one has  

$$\begin{array}{c}g{A}_z^{cl}=\frac{g}{\sqrt{2}}{A}_z^{(4,11)}{T}^{4,11}=-{\theta }^{\prime }(z)T^{4,11},\\ T^{4,11}=\{\begin{array}{ll}\frac{1}{2}{\tau }_1\hfill & \text{for\hspace{0.5em}}\chi ,\hfill \\ -\frac{1}{2}{\tau }_1\hfill & \text{for\hspace{0.5em}}\hat{\chi }.\hfill \end{array}\end{array}$$

(4.6)

We denote  

$$\begin{array}{c}{\hat{\mathcal{D}}}^{cl}(c)=\left(\begin{array}{cc}-k{\hat{D}}_{-}(c)& {\sigma }^{\mu }{\partial }_{\mu }\\ {\overline{\sigma }}^{\mu }{\partial }_{\mu }& -k{\hat{D}}_{+}(c)\end{array}\right),\\ {\hat{D}}_{\pm }(c)=\pm (\frac{d}{dz}+i{\theta }^{\prime }(z)T^{4,11})+\frac{c}{z},\\ {\mathcal{D}}_0(c)=\left(\begin{array}{cc}-k{D}_{-}(c)& {\sigma }^{\mu }{\partial }_{\mu }\\ {\overline{\sigma }}^{\mu }{\partial }_{\mu }& -k{D}_{+}(c)\end{array}\right).\end{array}$$

(4.7)

To simplify the notation the bulk mass parameters are denoted as  

$$c_0=c_{{\Psi }_{32}},c_1=c_{{\Psi }_{11}^{\text{'}}},c_2=c_{{\Psi }_{11}}.$$

(4.8)

4.2.1. (i) $Q_{\text{EM}}=+\frac{2}{3}$ : $u_j,u_j^{\text{'}}$

There are no brane mass terms. The boundary conditions are $D_{+}{\check{u}}_{jL}=0$⁠, ${\check{u}}_{jR}=0$⁠, ${\check{u}}_{jL}^{\text{'}}=0$⁠, and $D_{-}{\check{u}}_{jR}^{\text{'}}=0$ at $z=1,z_L$⁠. The equations of motion in the twisted gauge are  

$$\begin{array}{l}-k{D}_{-}(c_0)\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{u}}}_{jR}\end{array}\\ {\tilde{\check{u}}}_{jR}^{\text{'}}\end{array}\right)+{\sigma }^{\mu }{\partial }_{\mu }\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{u}}}_{jL}\end{array}\\ {\tilde{\check{u}}}_{jL}^{\text{'}}\end{array}\right)=0,\\ -k{D}_{+}(c_0)\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{u}}}_{jL}\end{array}\\ {\tilde{\check{u}}}_{jL}^{\text{'}}\end{array}\right)+{\overline{\sigma }}^{\mu }{\partial }_{\mu }\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{u}}}_{jR}\end{array}\\ {\tilde{\check{u}}}_{jR}^{\text{'}}\end{array}\right)=0.\end{array}$$

(4.9)

$({\tilde{\check{u}}}_j,{\tilde{\check{u}}}_j^{\text{'}})$ satisfy the same boundary conditions at $z=z_L$ as $({\check{u}}_j,{\check{u}}_j^{\text{'}})$⁠, so that one can write, for each mode,  

$$\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{u}}}_{jR}\end{array}\\ {\tilde{\check{u}}}_{jR}^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_u{S}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_u^{\text{'}}{C}_R(z;\lambda ,c_0)\end{array}\right)f_R(x),\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{u}}}_{jL}\end{array}\\ {\tilde{\check{u}}}_{jL}^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_u{C}_L(z;\lambda ,c_0)\end{array}\\ {\alpha }_u^{\text{'}}{S}_L(z;\lambda ,c_0)\end{array}\right)f_L(x),$$

(4.10)

where $\overline{\sigma }\partial f_R(x)=k\lambda f_L(x)$ and $\sigma \partial f_L(x)=k\lambda f_R(x)$⁠. Both right- and left-handed modes have the same coefficients ${\alpha }_u$ and ${\alpha }_u^{\text{'}}$ as a result of the equations of motion.

The boundary conditions at $z=1$ for the right-handed components, ${\check{u}}_{jR}=0$ and $D_{-}{\check{u}}_{jR}^{\text{'}}=0$⁠, become  

$$\left(\begin{array}{cc}\cos \frac{1}{2}{\theta }_H{S}_R^{c_0}& -i\sin \frac{1}{2}{\theta }_H{C}_R^{c_0}\\ -i\sin \frac{1}{2}{\theta }_H{C}_L^{c_0}& \cos \frac{1}{2}{\theta }_H{S}_L^{c_0}\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\alpha }_u\end{array}\\ {\alpha }_{u^{\prime }}\end{array}\right)=0,$$

(4.11)

so that the spectrum is determined by  

$$S_L^{c_0}{S}_R^{c_0}+{\sin }^2\frac{1}{2}{\theta }_H=0,$$

(4.12)

where $S_L^c=S_L(1;\lambda ,c)$⁠, etc. The mass of the lowest mode, $m=k\lambda$⁠, is given by  

$$m_u=\{\begin{array}{ll}{\pi }^{-1}\sqrt{1-4c_0^2}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0<\frac{1}{2},\hfill \\ {\pi }^{-1}\sqrt{4c_0^2-1}{z}_L^{-c_0+0.5}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0>\frac{1}{2}.\hfill \end{array}$$

(4.13)

$c_0=c_{{\Psi }_{32}}$ is determined from the up-type quark mass. For the top quark, $c_0<\frac{1}{2}$⁠, whereas for the charm and up quarks, $c_0>\frac{1}{2}$⁠. Note that  

$$\frac{m_t}{m_W}~\frac{\sqrt{kL(1-4c_0^2)}}{2\cos \frac{1}{2}{\theta }_H}.$$

(4.14)

4.2.2. (ii) $Q_{\text{EM}}=-\frac{2}{3}$ : ${\hat{u}}_j,{\hat{u}}_j^{\text{'}}$

There are no brane mass terms. The boundary conditions are $D_{+}{\check{\stackrel{ˆ}{u}}}_{jL}=0$⁠, ${\check{\stackrel{ˆ}{u}}}_{jR}=0$⁠, ${\check{\stackrel{ˆ}{u}}}_{jL}^{\text{'}}=0$⁠, and $D_{-}{\check{\stackrel{ˆ}{u}}}_{jR}^{\text{'}}=0$ at $z=1$⁠, and ${\check{\stackrel{ˆ}{u}}}_{jL}=0$⁠, $D_{-}{\check{\stackrel{ˆ}{u}}}_{jR}=0$⁠, $D_{+}{\check{\stackrel{ˆ}{u}}}_{jL}^{\text{'}}=0$⁠, and ${\check{\stackrel{ˆ}{u}}}_{jR}^{\text{'}}=0$ at $z=z_L$⁠. Wave functions of each mode are given by  

$$\left(\begin{array}{c}\begin{array}{c}{\widetilde{\check{\stackrel{ˆ}{u}}}}_{jR}\end{array}\\ {\widetilde{\check{\stackrel{ˆ}{u}}}}_{jR}^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{\hat{u}}{C}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_{{\hat{u}}^{\prime }}{S}_R(z;\lambda ,c_0)\end{array}\right)f_R(x),\left(\begin{array}{c}\begin{array}{c}{\widetilde{\check{\stackrel{ˆ}{u}}}}_{jL}\end{array}\\ {\widetilde{\check{\stackrel{ˆ}{u}}}}_{jL}^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{\hat{u}}{S}_L(z;\lambda ,c_0)\end{array}\\ {\alpha }_{{\hat{u}}^{\prime }}{C}_L(z;\lambda ,c_0)\end{array}\right)f_L(x).$$

(4.15)

Boundary conditions at $z=1$ lead to  

$$\left(\begin{array}{cc}\cos \frac{1}{2}{\theta }_H{C}_R^{c_0}& i\sin \frac{1}{2}{\theta }_H{S}_R^{c_0}\\ i\sin \frac{1}{2}{\theta }_H{S}_L^{c_0}& \cos \frac{1}{2}{\theta }_H{C}_L^{c_0}\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\alpha }_{\hat{u}}\end{array}\\ {\alpha }_{{\hat{u}}^{\prime }}\end{array}\right)=0,$$

(4.16)

so that the spectrum is determined by  

$$S_L^{c_0}{S}_R^{c_0}+{\cos }^2\frac{1}{2}{\theta }_H=0.$$

(4.17)

The mass of the lowest mode is given by  

$$m_{\hat{u}}=\{\begin{array}{ll}{\pi }^{-1}\sqrt{1-4c_0^2}\cos \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0<\frac{1}{2},\hfill \\ {\pi }^{-1}\sqrt{4c_0^2-1}{z}_L^{-c_0+0.5}\cos \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0>\frac{1}{2}.\hfill \end{array}$$

(4.18)

Note that  

$$\frac{m_{\hat{u}}}{m_u}=\cot \frac{1}{2}{\theta }_H.$$

(4.19)

4.2.3. (iii) $Q_{\text{EM}}=-\frac{1}{3}$ : $d_j,d_j^{\text{'}},D_j,D_j^{\text{'}}$

The equations of motion are  

$$\begin{array}{l}\begin{array}{c}\begin{array}{c}(a)\\ (b)\end{array}\end{array}-k{\hat{D}}_{-}\left(\begin{array}{c}\begin{array}{c}{\check{d}}_{jR}\\ {\check{d}}_{jR}^{\text{'}}\end{array}\end{array}\right)+{\sigma }^{\mu }{\partial }_{\mu }\left(\begin{array}{c}\begin{array}{c}{\check{d}}_{jL}\end{array}\\ {\check{d}}_{jL}^{\text{'}}\end{array}\right)=0,\\ \begin{array}{c}\begin{array}{c}(c)\end{array}\\ (d)\end{array}-k{\hat{D}}_{+}\left(\begin{array}{c}\begin{array}{c}{\check{d}}_{jL}\end{array}\\ {\check{d}}_{jL}^{\text{'}}\end{array}\right)+{\overline{\sigma }}^{\mu }{\partial }_{\mu }\left(\begin{array}{c}\begin{array}{c}{\check{d}}_{jR}\end{array}\\ {\check{d}}_{jR}^{\text{'}}\end{array}\right)=2{\mu }_4\delta (y)\left(\begin{array}{c}\begin{array}{c}0\end{array}\\ {\check{D}}_{jL}^{\text{'}}\end{array}\right),\\ (e)-k{D}_{-}{\check{D}}_{jR}^{\text{'}}+{\sigma }^{\mu }{\partial }_{\mu }{\check{D}}_{jL}^{\text{'}}=2{\mu }_4\delta (y){\check{d}}_{jR}^{\text{'}}+2{\mu }_6\delta (y){\check{D}}_{jR},\\ (f)-k{D}_{+}{\check{D}}_{jL}^{\text{'}}+{\overline{\sigma }}^{\mu }{\partial }_{\mu }{\check{D}}_{jR}^{\text{'}}=0,\\ (g)-k{D}_{-}{\check{D}}_{jR}+{\sigma }^{\mu }{\partial }_{\mu }{\check{D}}_{jL}=0,\\ (h)-k{D}_{+}{\check{D}}_{jL}+{\overline{\sigma }}^{\mu }{\partial }_{\mu }{\check{D}}_{jR}=2{\mu }_6\delta (y){\check{D}}_{jL}^{\text{'}}.\end{array}$$

(4.20)

Here, $D_{+}$ acting on ${\check{d}}_{jL}$⁠, ${\check{D}}_{jL}$⁠, ${\check{D}}_{jL}^{\text{'}}$ means $D_{+}(c_0)$⁠, $D_{+}(c_2)$⁠, $D_{+}(c_1)$⁠, respectively. Brane interactions affect the boundary conditions at $y=0$⁠. ${\check{d}}_{jR}$⁠, ${\check{d}}_{jL}^{\text{'}}$⁠, ${\check{D}}_{jR}^{\text{'}}$⁠, ${\check{D}}_{jL}$ are parity-odd at $y=0$⁠, whereas ${\check{d}}_{jL}$⁠, ${\check{d}}_{jR}^{\text{'}}$⁠, ${\check{D}}_{jL}^{\text{'}}$⁠, ${\check{D}}_{jR}$ are parity-even. Recall that $D_{\pm }(c)$ is parity-odd at $y=0$⁠:  

$$D_{\pm }(c)=\frac{e^{-\sigma (y)}}{k}(\pm \frac{d}{dy}+c{\sigma }^{\prime }(y))=e^{-\sigma (y)}(\pm \frac{1}{k}\frac{d}{dy}+cϵ(y)).$$

(4.21)

Noting that $A_z^{(4,11)}$ is parity-even and integrating over $y$ from $-ϵ$ to $ϵ$ in (4.20), one finds  

$$\begin{array}{l}(a)\Rightarrow 2{\check{d}}_{jR}(x,ϵ)=0,\\ (d)\Rightarrow -2{\check{d}}_{jL}^{\text{'}}(x,ϵ)=2{\mu }_4{\check{D}}_{jL}^{\text{'}}(x,0),\\ (e)\Rightarrow 2{\check{D}}_{jR}^{\text{'}}(x,ϵ)=2{\mu }_4{\check{d}}_{jR}^{\text{'}}(x,0)+2{\mu }_6{\check{D}}_{jR}(x,0),\\ (h)\Rightarrow -2{\check{D}}_{jL}(x,ϵ)=2{\mu }_6{\check{D}}_{jL}^{\text{'}}(x,0).\end{array}$$

(4.22)

For parity-even fields we evaluate the equations (4.20) at $y=ϵ>0$⁠, with the help of (4.22), to find  

$$\begin{array}{l}(c)\Rightarrow {\hat{D}}_{+}{\check{d}}_{jL}=0,\\ (b)\Rightarrow {\hat{D}}_{-}{\check{d}}_{jR}^{\text{'}}+{\mu }_4{D}_{-}(c_1){\check{D}}_{jR}^{\text{'}}=0,\\ (f)\Rightarrow D_{+}{\check{D}}_{jL}^{\text{'}}-{\mu }_4{\hat{D}}_{+}{\check{d}}_{jL}^{\text{'}}-{\mu }_6{D}_{+}{\check{D}}_{jL}=0,\\ (g)\Rightarrow D_{-}{\check{D}}_{jR}+{\mu }_6{D}_{-}{\check{D}}_{jR}^{\text{'}}=0.\end{array}$$

(4.23)

To summarize, the boundary conditions at $z=1^{+}$ (⁠$y=ϵ$⁠) are given by  

$$\begin{array}{ll}\text{(right-handed)}\hfill & \text{(left-handed)}\hfill \\ {\check{d}}_{jR}=0,\hfill & {\hat{D}}_{+}{\check{d}}_{jL}=0,\hfill \\ {\hat{D}}_{-}{\check{d}}_{jR}^{\text{'}}+{\mu }_4{D}_{-}{\check{D}}_{jR}^{\text{'}}=0,\hfill & {\check{d}}_{jL}^{\text{'}}+{\mu }_4{\check{D}}_{jL}^{\text{'}}=0,\hfill \\ {\check{D}}_{jR}^{\text{'}}-{\mu }_4{\check{d}}_{jR}^{\text{'}}-{\mu }_6{\check{D}}_{jR}=0,\hfill & D_{+}{\check{D}}_{jL}^{\text{'}}-{\mu }_4{\hat{D}}_{+}{\check{d}}_{jL}^{\text{'}}-{\mu }_6{D}_{+}{\check{D}}_{jL}=0,\hfill \\ D_{-}{\check{D}}_{jR}+{\mu }_6{D}_{-}{\check{D}}_{jR}^{\text{'}}=0,\hfill & {\check{D}}_{jL}+{\mu }_6{\check{D}}_{jL}^{\text{'}}=0.\hfill \end{array}$$

(4.24)

In the twisted gauge all fields obey free equations in the bulk so that eigenmodes are expressed, with the boundary conditions at the TeV brane taken into account, as  

$$\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{d}}}_{jR}\end{array}\\ {\tilde{\check{d}}}_{jR}^{\text{'}}\\ {\check{D}}_{jR}^{\text{'}}\\ {\check{D}}_{jR}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_d{S}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_{d^{\prime }}{C}_R(z;\lambda ,c_0)\\ {\alpha }_{D^{\prime }}{S}_R(z;\lambda ,c_1)\\ {\alpha }_D{C}_R(z;\lambda ,c_2)\end{array}\right)f_R(x),\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{d}}}_{jL}\end{array}\\ {\tilde{\check{d}}}_{jL}^{\text{'}}\\ {\check{D}}_{jL}^{\text{'}}\\ {\check{D}}_{jL}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_d{C}_L(z;\lambda ,c_0)\end{array}\\ {\alpha }_{d^{\prime }}{S}_L(z;\lambda ,c_0)\\ {\alpha }_{D^{\prime }}{C}_L(z;\lambda ,c_1)\\ {\alpha }_D{S}_L(z;\lambda ,c_2)\end{array}\right)f_L(x).$$

(4.25)

The boundary conditions (4.24) for the right-handed components are converted to  

$$\begin{array}{l}K\left(\begin{array}{c}\begin{array}{c}{\alpha }_d\end{array}\\ {\alpha }_{d^{\prime }}\\ {\alpha }_{D^{\prime }}\\ {\alpha }_D\end{array}\right)=0,\\ K=\left(\begin{array}{cccc}\cos \frac{{\theta }_H}{2}{S}_R^{c_0}& -i\sin \frac{{\theta }_H}{2}{C}_R^{c_0}& 0& 0\\ -i\sin \frac{{\theta }_H}{2}{C}_L^{c_0}& \cos \frac{{\theta }_H}{2}{S}_L^{c_0}& {\mu }_4{C}_L^{c_1}& 0\\ i{\mu }_4\sin \frac{{\theta }_H}{2}{S}_R^{c_0}& -{\mu }_4\cos \frac{{\theta }_H}{2}{C}_R^{c_0}& S_R^{c_1}& -{\mu }_6{C}_R^{c_2}\\ 0& 0& {\mu }_6{C}_L^{c_1}& S_L^{c_2}\end{array}\right),\end{array}$$

(4.26)

where $S_R^c=S_R(1;\lambda ,c)$⁠, etc. The spectrum $\left\{{\lambda }_n\right\}$ is determined from $\det K=0$⁠,  

$$\{{\cos }^2\frac{1}{2}{\theta }_H{S}_L^{c_0}{S}_R^{c_0}+{\sin }^2\frac{1}{2}{\theta }_H{C}_L^{c_0}{C}_R^{c_0}\}\{S_R^{c_1}{S}_L^{c_2}+{\mu }_6^2{C}_L^{c_1}{C}_R^{c_2}\}+{\mu }_4^2{S}_R^{c_0}{C}_R^{c_0}{C}_L^{c_1}{S}_L^{c_2}=0,$$

(4.27)

or, by making use of $C_L{C}_R-S_L{S}_R=1$ one finds that  

$$\{S_L^{c_0}{S}_R^{c_0}+{\sin }^2\frac{1}{2}{\theta }_H\}\{S_R^{c_1}{S}_L^{c_2}+{\mu }_6^2{C}_L^{c_1}{C}_R^{c_2}\}+{\mu }_4^2{S}_R^{c_0}{C}_R^{c_0}{C}_L^{c_1}{S}_L^{c_2}=0.$$

(4.28)

The same result is obtained from the boundary conditions for the left-handed components in (4.24).

For the mode with the lowest mass, the down-type quark, one can suppose that $\lambda z_L\ll 1$⁠, ${\sin }^2\frac{1}{2}{\theta }_H\gg |S_L^{c_0}{S}_R^{c_0}|$⁠, and ${\mu }_6^2{C}_L^{c_1}{C}_R^{c_2}\gg |S_R^{c_1}{S}_L^{c_2}|$⁠, so that  

$$-S_R^{c_0}{S}_L^{c_2}\simeq \frac{{\mu }_6^2{C}_R^{c_2}}{{\mu }_4^2{C}_R^{c_0}}{\sin }^2\frac{1}{2}{\theta }_H.$$

(4.29)

In the first and second generations, $c_j>\frac{1}{2}$⁠, whereas in the third generation, $c_j<\frac{1}{2}$⁠. The mass is given by  

$$m_d=\{\begin{array}{ll}\frac{1}{\pi }\frac{{\mu }_6}{{\mu }_4}\sqrt{(1-2c_0)(1+2c_2)}{z}_L^{c_0-c_2}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_2<\frac{1}{2},\hfill \\ \frac{1}{\pi }\frac{{\mu }_6}{{\mu }_4}\sqrt{(2c_0-1)(1+2c_2)}{z}_L^{-c_2+0.5}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_2>\frac{1}{2}.\hfill \end{array}$$

(4.30)

In either case one finds that  

$$\frac{m_d}{m_u}=\frac{{\mu }_6}{{\mu }_4}\sqrt{\frac{1+2c_2}{1+2c_0}}{z}_L^{c_0-c_2}.$$

(4.31)

4.2.4. (iv) $Q_{\text{EM}}=+\frac{1}{3}$ : ${\hat{d}}_j,{\hat{d}}_j^{\text{'}},{\hat{D}}_j,{\hat{D}}_j^{\text{'}}$

${\hat{D}}_{jR}$ and ${\hat{D}}_{jL}^{\text{'}}$ have zero modes. The equations of motion are  

$$\begin{array}{l}-k{\hat{D}}_{-}\left(\begin{array}{c}\begin{array}{c}{\check{\stackrel{ˆ}{d}}}_{jR}\end{array}\\ {\check{\stackrel{ˆ}{d}}}_{jR}^{\text{'}}\end{array}\right)+{\sigma }^{\mu }{\partial }_{\mu }\left(\begin{array}{c}\begin{array}{c}{\check{\stackrel{ˆ}{d}}}_{jL}\end{array}\\ {\check{\stackrel{ˆ}{d}}}_{jL}^{\text{'}}\end{array}\right)=2{\mu }_3\delta (y)\left(\begin{array}{c}\begin{array}{c}{\check{\stackrel{ˆ}{D}}}_{jR}\end{array}\\ 0\end{array}\right),\\ -k{\hat{D}}_{+}\left(\begin{array}{c}\begin{array}{c}{\check{\stackrel{ˆ}{d}}}_{jL}\end{array}\\ {\check{\stackrel{ˆ}{d}}}_{jL}^{\text{'}}\end{array}\right)+{\overline{\sigma }}^{\mu }{\partial }_{\mu }\left(\begin{array}{c}\begin{array}{c}{\check{\stackrel{ˆ}{d}}}_{jR}\end{array}\\ {\check{\stackrel{ˆ}{d}}}_{jR}^{\text{'}}\end{array}\right)=0,\\ -k{D}_{-}{\check{\stackrel{ˆ}{D}}}_{jR}^{\text{'}}+{\sigma }^{\mu }{\partial }_{\mu }{\check{\stackrel{ˆ}{D}}}_{jL}^{\text{'}}=2{\mu }_6\delta (y){\check{\stackrel{ˆ}{D}}}_{jR},\\ -k{D}_{+}{\check{\stackrel{ˆ}{D}}}_{jL}^{\text{'}}+{\overline{\sigma }}^{\mu }{\partial }_{\mu }{\check{\stackrel{ˆ}{D}}}_{jR}^{\text{'}}=0,\\ -k{D}_{-}{\check{\stackrel{ˆ}{D}}}_{jR}+{\sigma }^{\mu }{\partial }_{\mu }{\check{\stackrel{ˆ}{D}}}_{jL}=0,\\ -k{D}_{+}{\check{\stackrel{ˆ}{D}}}_{jL}+{\overline{\sigma }}^{\mu }{\partial }_{\mu }{\check{\stackrel{ˆ}{D}}}_{jR}=2{\mu }_3\delta (y){\check{\stackrel{ˆ}{d}}}_{jL}+2{\mu }_6\delta (y){\check{\stackrel{ˆ}{D}}}_{jL}^{\text{'}}.\end{array}$$

(4.32)

At the Planck brane (⁠$y=0$⁠), ${\hat{d}}_R$⁠, ${\hat{d}}_L^{\text{'}}$⁠, ${\hat{D}}_R^{\text{'}}$⁠, ${\hat{D}}_L$ are parity-odd, whereas ${\hat{d}}_L$⁠, ${\hat{d}}_R^{\text{'}}$⁠, ${\hat{D}}_L^{\text{'}}$⁠, ${\hat{D}}_R$ are parity-even. The boundary conditions at $y=ϵ$ (⁠$z=1^{+}$⁠) become  

$$\begin{array}{ll}\text{(right-handed)}\hfill & \text{(left-handed)}\hfill \\ {\hat{D}}_{-}{\check{\stackrel{ˆ}{d}}}_{jR}^{\text{'}}=0,\hfill & {\check{\stackrel{ˆ}{d}}}_{jL}^{\text{'}}=0,\hfill \\ {\check{\stackrel{ˆ}{d}}}_{jR}-{\mu }_3{\check{\stackrel{ˆ}{D}}}_{jR}=0,\hfill & {\hat{D}}_{+}{\check{\stackrel{ˆ}{d}}}_{jL}-{\mu }_3{D}_{+}{\check{\stackrel{ˆ}{D}}}_{jL}=0,\hfill \\ D_{-}{\check{\stackrel{ˆ}{D}}}_{jR}+{\mu }_3{\hat{D}}_{-}{\check{\stackrel{ˆ}{d}}}_{jR}+{\mu }_6{D}_{-}{\check{\stackrel{ˆ}{D}}}_{jR}^{\text{'}}=0,\hfill & {\check{\stackrel{ˆ}{D}}}_{jL}+{\mu }_3{\check{\stackrel{ˆ}{d}}}_{jL}+{\mu }_6{\check{\stackrel{ˆ}{D}}}_{jL}^{\text{'}}=0,\hfill \\ {\check{\stackrel{ˆ}{D}}}_{jR}^{\text{'}}-{\mu }_6{\check{\stackrel{ˆ}{D}}}_{jR}=0,\hfill & D_{+}{\check{\stackrel{ˆ}{D}}}_{jL}^{\text{'}}-{\mu }_6{D}_{+}{\check{\stackrel{ˆ}{D}}}_{jL}=0.\hfill \end{array}$$

(4.33)

Eigenmodes are given by  

$$\left(\begin{array}{c}\begin{array}{c}{\widetilde{\check{\stackrel{ˆ}{d}}}}_{jR}^{\text{'}}\end{array}\\ {\widetilde{\check{\stackrel{ˆ}{d}}}}_{jR}\\ {\check{\stackrel{ˆ}{D}}}_{jR}\\ {\check{\stackrel{ˆ}{D}}}_{jR}^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{{\hat{d}}^{\prime }}{S}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_{\hat{d}}{C}_R(z;\lambda ,c_0)\\ {\alpha }_{\hat{D}}{C}_R(z;\lambda ,c_2)\\ {\alpha }_{{\hat{D}}^{\prime }}{S}_R(z;\lambda ,c_1)\end{array}\right)f_R(x),\left(\begin{array}{c}\begin{array}{c}{\widetilde{\check{\stackrel{ˆ}{d}}}}_{jL}^{\text{'}}\end{array}\\ {\widetilde{\check{\stackrel{ˆ}{d}}}}_{jL}\\ {\check{\stackrel{ˆ}{D}}}_{jL}\\ {\check{\stackrel{ˆ}{D}}}_{jL}^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{{\hat{d}}^{\prime }}{C}_L(z;\lambda ,c_0)\end{array}\\ {\alpha }_{\hat{d}}{S}_L(z;\lambda ,c_0)\\ {\alpha }_{\hat{D}}{S}_L(z;\lambda ,c_2)\\ {\alpha }_{{\hat{D}}^{\prime }}{C}_L(z;\lambda ,c_1)\end{array}\right)f_L(x).$$

(4.34)

The boundary conditions (4.33) lead to  

$$\begin{array}{l}K\left(\begin{array}{c}\begin{array}{c}{\alpha }_{{\hat{d}}^{\prime }}\\ {\alpha }_{\hat{d}}\\ {\alpha }_{\hat{D}}\\ {\alpha }_{{\hat{D}}^{\prime }}\end{array}\end{array}\right)=0,\\ K=\left(\begin{array}{cccc}\cos \frac{{\theta }_H}{2}{C}_L^{c_0}& i\sin \frac{{\theta }_H}{2}{S}_L^{c_0}& 0& 0\\ i\sin \frac{{\theta }_H}{2}{S}_R^{c_0}& \cos \frac{{\theta }_H}{2}{C}_R^{c_0}& -{\mu }_3{C}_R^{c_2}& 0\\ i{\mu }_3\sin \frac{{\theta }_H}{2}{C}_L^{c_0}& {\mu }_3\cos \frac{{\theta }_H}{2}{S}_L^{c_0}& S_L^{c_2}& {\mu }_6{C}_L^{c_1}\\ 0& 0& -{\mu }_6{C}_R^{c_2}& S_R^{c_1}\end{array}\right).\end{array}$$

(4.35)

The spectrum is determined by  

$$\{{\cos }^2\frac{1}{2}{\theta }_H{C}_L^{c_0}{C}_R^{c_0}+{\sin }^2\frac{1}{2}{\theta }_H{S}_L^{c_0}{S}_R^{c_0}\}\{S_R^{c_1}{S}_L^{c_2}+{\mu }_6^2{C}_L^{c_1}{C}_R^{c_2}\}+{\mu }_3^2{S}_L^{c_0}{C}_L^{c_0}{S}_R^{c_1}{C}_R^{c_2}=0,$$

(4.36)

or  

$$\{S_L^{c_0}{S}_R^{c_0}+{\cos }^2\frac{1}{2}{\theta }_H\}\{S_R^{c_1}{S}_L^{c_2}+{\mu }_6^2{C}_L^{c_1}{C}_R^{c_2}\}+{\mu }_3^2{S}_L^{c_0}{C}_L^{c_0}{S}_R^{c_1}{C}_R^{c_2}=0.$$

(4.37)

For the mode with the lowest mass,  

$$-S_L^{c_0}{S}_R^{c_1}\simeq \frac{{\mu }_6^2{C}_L^{c_1}}{{\mu }_3^2{C}_L^{c_0}}{\cos }^2\frac{1}{2}{\theta }_H.$$

(4.38)

The mass is given by  

$$m_{\hat{d}}=\{\begin{array}{ll}\frac{1}{\pi }\frac{{\mu }_6}{{\mu }_3}\sqrt{(1-2c_1)(1+2c_0)}{z}_L^{c_1-c_0}\cos \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_1<\frac{1}{2},\hfill \\ \frac{1}{\pi }\frac{{\mu }_6}{{\mu }_3}\sqrt{(2c_0+1)(2c_1-1)}{z}_L^{-c_0+0.5}\cos \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_1>\frac{1}{2}.\hfill \end{array}$$

(4.39)

One finds that  

$$\frac{m_{\hat{d}}}{m_u}=\frac{{\mu }_6}{{\mu }_3}\sqrt{\frac{1-2c_1}{1-2c_0}}\cot \frac{1}{2}{\theta }_H\times \{\begin{array}{ll}{z}_L^{c_1-c_0}\hfill & \text{for\hspace{0.5em}}{c}_0,c_1<\frac{1}{2},\hfill \\ 1\hfill & \text{for}{c}_0,c_1>\frac{1}{2}.\hfill \end{array}$$

(4.40)

4.2.5. (v) $Q_{\text{EM}}=-1$ : $e,e^{\prime },E,E^{\prime }$

The spectrum in the $Q_{\text{EM}}=-1$ sector is found in a similar manner. The boundary conditions at $y=ϵ$ (⁠$z=1^{+}$⁠) are given by  

$$\begin{array}{ll}\text{(right-handed)\hspace{0.5em}}\hfill & \text{(left-handed)}\hfill \\ {\hat{D}}_{-}{\check{e}}_R^{\text{'}}=0,\hfill & {\check{e}}_L^{\text{'}}=0,\hfill \\ {\check{e}}_R+i{\mu }_3{\check{E}}_R=0,\hfill & {\hat{D}}_{+}{\check{e}}_L+i{\mu }_3{D}_{+}{\check{E}}_L=0,\hfill \\ D_{-}{\check{E}}_R+i{\mu }_3{\hat{D}}_{-}{\check{e}}_R+{\mu }_6{D}_{-}{\check{E}}_R^{\text{'}}=0,\hfill & {\check{E}}_L+i{\mu }_3{\check{e}}_L+{\mu }_6{\check{E}}_L^{\text{'}}=0,\hfill \\ {\check{E}}_R^{\text{'}}-{\mu }_6{\check{E}}_R=0,\hfill & D_{+}{\check{E}}_L^{\text{'}}-{\mu }_6{D}_{+}{\check{E}}_L=0,\hfill \end{array}$$

(4.41)

and mode functions in the twisted gauge are given by  

$$\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{e}}}_R^{\text{'}}\end{array}\\ {\tilde{\check{e}}}_R\\ {\check{E}}_R\\ {\check{E}}_R^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{e^{\prime }}{C}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_e{S}_R(z;\lambda ,c_0)\\ {\alpha }_E{S}_R(z;\lambda ,c_2)\\ {\alpha }_{E^{\prime }}{C}_R(z;\lambda ,c_1)\end{array}\right)f_R(x),\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{e}}}_L^{\text{'}}\end{array}\\ {\tilde{\check{e}}}_L\\ {\check{E}}_L\\ {\check{E}}_L^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{e^{\prime }}{S}_L(z;\lambda ,c_0)\end{array}\\ {\alpha }_e{C}_L(z;\lambda ,c_0)\\ {\alpha }_E{C}_L(z;\lambda ,c_2)\\ {\alpha }_{E^{\prime }}{S}_L(z;\lambda ,c_1)\end{array}\right)f_L(x).$$

(4.42)

The boundary conditions in (4.41) lead to  

$$\left(\begin{array}{cccc}\cos \frac{{\theta }_H}{2}{S}_L^{c_0}& -i\sin \frac{{\theta }_H}{2}{C}_L^{c_0}& 0& 0\\ -i\sin \frac{{\theta }_H}{2}{C}_R^{c_0}& \cos \frac{{\theta }_H}{2}{S}_R^{c_0}& i{\mu }_3{S}_R^{c_2}& 0\\ {\mu }_3\sin \frac{{\theta }_H}{2}{S}_L^{c_0}& i{\mu }_3\cos \frac{{\theta }_H}{2}{C}_L^{c_0}& C_L^{c_2}& {\mu }_6{S}_L^{c_1}\\ 0& 0& -{\mu }_6{S}_R^{c_2}& C_R^{c_1}\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\alpha }_{e^{\prime }}\end{array}\\ {\alpha }_e\\ {\alpha }_E\\ {\alpha }_{E^{\prime }}\end{array}\right)=0.$$

(4.43)

Consequently, the spectrum is determined by  

$$\{S_L^{c_0}{S}_R^{c_0}+{\sin }^2\frac{1}{2}{\theta }_H\}\{C_R^{c_1}{C}_L^{c_2}+{\mu }_6^2{S}_L^{c_1}{S}_R^{c_2}\}+{\mu }_3^2{S}_L^{c_0}{C}_L^{c_0}{C}_R^{c_1}{S}_R^{c_2}=0.$$

(4.44)

For the lowest mode, the electron,  

$$-S_L^{c_0}{S}_R^{c_2}\simeq \frac{C_L^{c_2}}{{\mu }_3^2{C}_L^{c_0}}{\sin }^2\frac{1}{2}{\theta }_H,$$

(4.45)

so that  

$$m_e=\{\begin{array}{ll}\frac{1}{\pi }\frac{1}{{\mu }_3}\sqrt{(1-2c_2)(1+2c_0)}{z}_L^{c_2-c_0}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_2<\frac{1}{2},\hfill \\ \frac{1}{\pi }\frac{1}{{\mu }_3}\sqrt{(2c_2-1)(1+2c_0)}{z}_L^{-c_0+0.5}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_2>\frac{1}{2}.\hfill \end{array}$$

(4.46)

One finds that  

$$\frac{m_e}{m_u}=\frac{1}{{\mu }_3}\sqrt{\frac{1-2c_2}{1-2c_0}}\times \{\begin{array}{ll}{z}_L^{c_2-c_0}\hfill & \text{for\hspace{0.5em}}{c}_0,c_2<\frac{1}{2},\hfill \\ 1\hfill & \text{for\hspace{0.5em}}{c}_0,c_2>\frac{1}{2}.\hfill \end{array}$$

(4.47)

4.2.6. (vi) $Q_{\text{EM}}=+1$ : $\hat{e},{\hat{e}}^{\prime },\hat{E},{\hat{E}}^{\prime }$

There are no zero modes. The boundary conditions at $y=ϵ$ (⁠$z=1^{+}$⁠) for right-handed components are given by  

$$\begin{array}{l}{\check{\stackrel{ˆ}{e}}}_R=0,\\ {\hat{D}}_{-}{\check{\stackrel{ˆ}{e}}}_R^{\text{'}}-i{\mu }_4{D}_{-}{\check{\stackrel{ˆ}{E}}}_R^{\text{'}}=0,\\ {\check{\stackrel{ˆ}{E}}}_R^{\text{'}}-i{\mu }_4{\check{\stackrel{ˆ}{e}}}_R^{\text{'}}-{\mu }_6{\check{\stackrel{ˆ}{E}}}_R=0,\\ D_{-}{\check{\stackrel{ˆ}{E}}}_R+{\mu }_6{D}_{-}{\check{\stackrel{ˆ}{E}}}_R^{\text{'}}=0,\end{array}$$

(4.48)

and wave functions in the twisted gauge are given by  

$$\left(\begin{array}{c}\begin{array}{c}{\widetilde{\check{\stackrel{ˆ}{e}}}}_R\end{array}\\ {\widetilde{\check{\stackrel{ˆ}{e}}}}_R^{\text{'}}\\ {\check{\stackrel{ˆ}{E}}}_R^{\text{'}}\\ {\check{\stackrel{ˆ}{E}}}_R\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{\hat{e}}{C}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_{{\hat{e}}^{\prime }}{S}_R(z;\lambda ,c_0)\\ {\alpha }_{{\hat{E}}^{\prime }}{C}_R(z;\lambda ,c_1)\\ {\alpha }_{\hat{E}}{S}_R(z;\lambda ,c_2)\end{array}\right)f_R(x).$$

(4.49)

Expressions for the left-handed components are obtained by simple replacement which would be obvious from the cases for $Q_{\text{EM}}=\pm \frac{1}{3}$⁠, etc. The boundary conditions in (4.48) are converted to  

$$\left(\begin{array}{cccc}\cos \frac{{\theta }_H}{2}{C}_R^{c_0}& i\sin \frac{{\theta }_H}{2}{S}_R^{c_0}& 0& 0\\ i\sin \frac{{\theta }_H}{2}{S}_L^{c_0}& \cos \frac{{\theta }_H}{2}{C}_L^{c_0}& -i{\mu }_4{S}_L^{c_1}& 0\\ {\mu }_4\sin \frac{{\theta }_H}{2}{C}_R^{c_0}& -i{\mu }_4\cos \frac{{\theta }_H}{2}{S}_R^{c_0}& C_R^{c_1}& -{\mu }_6{S}_R^{c_2}\\ 0& 0& {\mu }_6{S}_L^{c_1}& C_L^{c_2}\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\alpha }_{\hat{e}}\end{array}\\ {\alpha }_{{\hat{e}}^{\prime }}\\ {\alpha }_{{\hat{E}}^{\prime }}\\ {\alpha }_{\hat{E}}\end{array}\right)=0.$$

(4.50)

The spectrum is determined by  

$$\{S_L^{c_0}{S}_R^{c_0}+{\cos }^2\frac{1}{2}{\theta }_H\}\{C_R^{c_1}{C}_L^{c_2}+{\mu }_6^2{S}_L^{c_1}{S}_R^{c_2}\}+{\mu }_4^2{S}_R^{c_0}{C}_R^{c_0}{S}_L^{c_1}{C}_L^{c_2}=0.$$

(4.51)

For the lowest mode,  

$$-S_R^{c_0}{S}_L^{c_1}\simeq \frac{C_R^{c_1}}{{\mu }_4^2{C}_R^{c_0}}{\cos }^2\frac{1}{2}{\theta }_H,$$

(4.52)

so that  

$$m_{\hat{e}}=\{\begin{array}{ll}\frac{1}{\pi }\frac{1}{{\mu }_4}\sqrt{(1-2c_0)(1+2c_1)}{z}_L^{c_0-c_1}\cos \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_1<\frac{1}{2},\hfill \\ \frac{1}{\pi }\frac{1}{{\mu }_4}\sqrt{(2c_0-1)(1+2c_1)}{z}_L^{-c_1+0.5}\cos \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_1>\frac{1}{2}.\hfill \end{array}$$

(4.53)

One finds that  

$$\frac{m_{\hat{e}}}{m_u}=\frac{1}{{\mu }_4}\sqrt{\frac{1+2c_1}{1+2c_0}}{z}_L^{c_0-c_1}\cot \frac{1}{2}{\theta }_H$$

(4.54)

both for $c_0,c_1<\frac{1}{2}$ and for $c_0,c_1>\frac{1}{2}$⁠.

4.2.7. (vii) $Q_{\text{EM}}=0$ : $\nu ,{\nu }^{\prime },N,N^{\prime },S,S^{\prime },\hat{\nu },{\hat{\nu }}^{\prime },\hat{N},{\hat{N}}^{\prime }$

Only ${\nu }_L$⁠, ${\nu }_{R^{\prime }}$ have zero modes. In general all these ten components mix with each other. It is convenient to split them into two sets:  

$$\begin{array}{l}\text{Set\hspace{0.5em}1}:\nu ,{\nu }^{\prime },N,N^{\prime },S,\\ \text{Set\hspace{0.5em}2}:\hat{\nu },{\hat{\nu }}^{\prime },\hat{N},{\hat{N}}^{\prime },S^{\prime }.\end{array}$$

The boundary conditions at $y=ϵ$ (⁠$z=1^{+}$⁠) become  

$$\begin{array}{l}{\check{\nu }}_R+i{\mu }_3{\check{N}}_R=0,\\ {\hat{D}}_{-}{\check{\nu }}_R^{\text{'}}+{\mu }_2{D}_{-}{\check{S}}_R=0,\\ {\check{N}}_R^{\text{'}}-{\mu }_6{\check{N}}_R=0,\\ {\check{S}}_R-{\mu }_5{\check{S}}_R^{\text{'}}-{\mu }_2{\check{\nu }}_R^{\text{'}}=0,\\ D_{-}{\check{N}}_R+i{\mu }_3{\hat{D}}_{-}{\check{\nu }}_R+{\mu }_6{D}_{-}{\check{N}}_R^{\text{'}}=0,\end{array}$$

(4.55)

for Set 1, and  

$$\begin{array}{l}{\hat{D}}_{-}{\check{\stackrel{ˆ}{\nu }}}_R^{\text{'}}-i{\mu }_4{D}_{-}{\check{\stackrel{ˆ}{N}}}_R^{\text{'}}=0,\\ {\check{\stackrel{ˆ}{\nu }}}_R-{\mu }_1{\check{S}}_R^{\text{'}}=0,\\ D_{-}{\check{\stackrel{ˆ}{N}}}_R+{\mu }_6{D}_{-}{\check{\stackrel{ˆ}{N}}}_R^{\text{'}}=0,\\ D_{-}{\check{S}}_R^{\text{'}}+{\mu }_5{D}_{-}{\check{S}}_R+{\mu }_1{\hat{D}}_{-}{\check{\nu }}_R=0,\\ {\check{\stackrel{ˆ}{N}}}_R^{\text{'}}-i{\mu }_4{\check{\stackrel{ˆ}{\nu }}}_R^{\text{'}}-{\mu }_6{\check{\stackrel{ˆ}{N}}}_R=0,\end{array}$$

(4.56)

for Set 2. When ${\mu }_5=0$⁠, the two sets of boundary conditions decouple from each other. We set ${\mu }_5=0$ in the following analysis.

Wave functions in the twisted gauge are given by  

$$\left(\begin{array}{c}\begin{array}{c}{\tilde{\check{\nu }}}_R\end{array}\\ {\tilde{\check{\nu }}}_R^{\text{'}}\\ {\check{N}}_R\\ {\check{N}}_R^{\text{'}}\\ {\check{S}}_R\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{\nu }{S}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_{{\nu }^{\prime }}{C}_R(z;\lambda ,c_0)\\ {\alpha }_N{S}_R(z;\lambda ,c_2)\\ {\alpha }_{N^{\prime }}{C}_R(z;\lambda ,c_1)\\ {\alpha }_S{C}_R(z;\lambda ,c_2)\end{array}\right),\left(\begin{array}{c}\begin{array}{c}{\widetilde{\check{\stackrel{ˆ}{\nu }}}}_R^{\text{'}}\end{array}\\ {\widetilde{\check{\stackrel{ˆ}{\nu }}}}_R\\ {\check{\stackrel{ˆ}{N}}}_R^{\text{'}}\\ {\check{\stackrel{ˆ}{N}}}_R\\ {\check{S}}_R^{\text{'}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_{{\hat{\nu }}^{\prime }}{S}_R(z;\lambda ,c_0)\end{array}\\ {\alpha }_{\hat{\nu }}{C}_R(z;\lambda ,c_0)\\ {\alpha }_{{\hat{N}}^{\prime }}{C}_R(z;\lambda ,c_1)\\ {\alpha }_{\hat{N}}{S}_R(z;\lambda ,c_2)\\ {\alpha }_{S^{\prime }}{S}_R(z;\lambda ,c_1)\end{array}\right).$$

(4.57)

The boundary conditions (4.55) and (4.56) for ${\mu }_5=0$ lead to  

$$\begin{array}{l}\left(\begin{array}{ccccc}\cos \frac{{\theta }_H}{2}{S}_R^{c_0}& -i\sin \frac{{\theta }_H}{2}{C}_R^{c_0}& i{\mu }_3{S}_R^{c_2}& 0& 0\\ -i\sin \frac{{\theta }_H}{2}{C}_L^{c_0}& \cos \frac{{\theta }_H}{2}{S}_L^{c_0}& 0& 0& {\mu }_2{S}_L^{c_2}\\ i{\mu }_2\sin \frac{{\theta }_H}{2}{S}_R^{c_0}& -{\mu }_2\cos \frac{{\theta }_H}{2}{C}_R^{c_0}& 0& 0& C_R^{c_2}\\ 0& 0& -{\mu }_6{S}_R^{c_2}& C_R^{c_1}& 0\\ i{\mu }_3\cos \frac{{\theta }_H}{2}{C}_L^{c_0}& {\mu }_3\sin \frac{{\theta }_H}{2}{S}_L^{c_0}& C_L^{c_2}& {\mu }_6{S}_L^{c_1}& 0\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\alpha }_{\nu }\end{array}\\ {\alpha }_{{\nu }^{\prime }}\\ {\alpha }_N\\ {\alpha }_{N^{\prime }}\\ {\alpha }_S\end{array}\right)=0,\\ \left(\begin{array}{ccccc}\cos \frac{{\theta }_H}{2}{C}_L^{c_0}& i\sin \frac{{\theta }_H}{2}{S}_L^{c_0}& -i{\mu }_4{S}_L^{c_1}& 0& 0\\ i\sin \frac{{\theta }_H}{2}{S}_R^{c_0}& \cos \frac{{\theta }_H}{2}{C}_R^{c_0}& 0& 0& -{\mu }_1{S}_R^{c_1}\\ i{\mu }_1\sin \frac{{\theta }_H}{2}{C}_L^{c_0}& {\mu }_1\cos \frac{{\theta }_H}{2}{S}_L^{c_0}& 0& 0& C_L^{c_1}\\ 0& 0& {\mu }_6{S}_L^{c_1}& C_L^{c_2}& 0\\ -i{\mu }_4\cos \frac{{\theta }_H}{2}{S}_R^{c_0}& {\mu }_4\sin \frac{{\theta }_H}{2}{C}_R^{c_0}& C_R^{c_1}& -{\mu }_6{S}_R^{c_2}& 0\end{array}\right)\left(\begin{array}{c}\begin{array}{c}{\alpha }_{{\hat{\nu }}^{\prime }}\end{array}\\ {\alpha }_{\hat{\nu }}\\ {\alpha }_{{\hat{N}}^{\prime }}\\ {\alpha }_{\hat{N}}\\ {\alpha }_{S^{\prime }}\end{array}\right)=0.\end{array}$$

(4.58)

The spectrum for Set 1 is determined by  

$$\begin{array}{l}{\sin }^2\frac{{\theta }_H}{2}\{C_L^{c_0}{C}_R^{c_2}+{\mu }_2^2{S}_R^{c_0}{S}_L^{c_2}\}\{C_R^{c_0}{C}_R^{c_1}{C}_L^{c_2}+{\mu }_3^2{S}_L^{c_0}{C}_R^{c_1}{S}_R^{c_2}+{\mu }_6^2{C}_R^{c_0}{S}_L^{c_1}{S}_R^{c_2}\}\\ +{\cos }^2\frac{{\theta }_H}{2}\{S_L^{c_0}{C}_R^{c_2}+{\mu }_2^2{C}_R^{c_0}{S}_L^{c_2}\}\{S_R^{c_0}{C}_R^{c_1}{C}_L^{c_2}+{\mu }_3^2{C}_L^{c_0}{C}_R^{c_1}{S}_R^{c_2}+{\mu }_6^2{S}_R^{c_0}{S}_L^{c_1}{S}_R^{c_2}\}=0.\end{array}$$

(4.59)

As will be seen shortly, ${\mu }_2$ needs to be very large to have small neutrino masses. Careful evaluation of each term in (4.59) is necessary to find approximate formulas for neutrinos. For the lowest mode with $\lambda z_L\ll 1$⁠,  

$$\begin{array}{l}{C}_L^{c_0}{C}_R^{c_2}+{\mu }_2^2{S}_R^{c_0}{S}_L^{c_2}\simeq z_L^{c_0-c_2}\{1-\frac{{\mu }_2^2{(\lambda z_L)}^2{z}_L^{2(c_2-c_0)}}{(1-2c_0)(1+2c_2)}\},\\ S_L^{c_0}{C}_R^{c_2}+{\mu }_2^2{C}_R^{c_0}{S}_L^{c_2}\simeq -\lambda z_L^{1+c_0-c_2}\{\frac{1}{1+2c_0}+\frac{{\mu }_2^2{z}_L^{2(c_2-c_0)}}{1+2c_2}\},\\ S_R^{c_0}{C}_L^{c_2}+{\mu }_3^2{C}_L^{c_0}{S}_R^{c_2}\simeq \lambda z_L^{1+c_2-c_0}\{\frac{1}{1-2c_0}+\frac{{\mu }_3^2{z}_L^{2(c_0-c_2)}}{1-2c_2}\}\end{array}$$

(4.60)

for $c_0,c_2<\frac{1}{2}$⁠, and  

$$\begin{array}{l}{C}_L^{c_0}{C}_R^{c_2}+{\mu }_2^2{S}_R^{c_0}{S}_L^{c_2}\simeq z_L^{c_0-c_2}\{1-\frac{{\mu }_2^2{(\lambda z_L)}^2{z}_L^{2c_2-1}}{(2c_0-1)(1+2c_2)}\},\\ S_L^{c_0}{C}_R^{c_2}+{\mu }_2^2{C}_R^{c_0}{S}_L^{c_2}\simeq -\lambda z_L^{1+c_0-c_2}\{\frac{1}{1+2c_0}+\frac{{\mu }_2^2{z}_L^{2(c_2-c_0)}}{1+2c_2}\},\\ S_R^{c_0}{C}_L^{c_2}+{\mu }_3^2{C}_L^{c_0}{S}_R^{c_2}\simeq \lambda z_L^{c_0+c_2}\{\frac{1}{2c_0-1}+\frac{{\mu }_3^2}{2c_2-1}\}\end{array}$$

(4.61)

for $c_0,c_2>\frac{1}{2}$⁠. For neutrinos, $\lambda z_L=\pi m_{\nu }/m_{\text{KK}}$⁠. In the third generation, for which we choose $c_0,c_2<\frac{1}{2}$⁠, it is found, a posteriori, that ${\mu }_2\lambda z_L^{1+c_2-c_0}~(m_{\tau }/m_t)\sin \frac{1}{2}{\theta }_H\ll 1$⁠, ${\mu }_2{z}_L^{c_2-c_0}~m_{\tau }/m_{{\nu }_{\tau }}\gg 1$⁠, and ${\mu }_3{z}_L^{c_0-c_2}~m_t/m_{\tau }\gg 1$⁠. In the first and second generations we have $c_0,c_2>\frac{1}{2}$⁠. It is found, a posteriori, that ${\mu }_2\lambda z_L^{0.5+c_2}~(m_e/m_u)\sin \frac{1}{2}{\theta }_H\ll 1$⁠, ${\mu }_2{z}_L^{c_2-c_0}~m_e/m_{{\nu }_e}\gg 1$⁠, and ${\mu }_3^2~{(m_u/m_e)}^2\gg 1$⁠. Hence, in both cases the mass of the lowest mode, the neutrino, is determined approximately by  

$$-S_L^{c_2}{S}_R^{c_2}\simeq \frac{1}{{\mu }_2^2{\mu }_3^2}{\sin }^2\frac{1}{2}{\theta }_H,$$

(4.62)

so that  

$$m_{\nu }=\{\begin{array}{ll}\frac{1}{\pi }\frac{1}{{\mu }_2{\mu }_3}\sqrt{1-4c_2^2}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_2<\frac{1}{2},\hfill \\ \frac{1}{\pi }\frac{1}{{\mu }_2{\mu }_3}\sqrt{4c_2^2-1}{z}_L^{-c_2+0.5}\sin \frac{1}{2}{\theta }_H{m}_{\text{KK}}\hfill & \text{for\hspace{0.5em}}{c}_2>\frac{1}{2}.\hfill \end{array}$$

(4.63)

One finds that  

$$\frac{m_{\nu }}{m_u}=\frac{1}{{\mu }_2{\mu }_3}\sqrt{\frac{1-4c_2^2}{1-4c_0^2}}\times \{\begin{array}{ll}1\hfill & \text{for\hspace{0.5em}}{c}_0,c_2<\frac{1}{2},\hfill \\ z_L^{c_0-c_2}\hfill & \text{for\hspace{0.5em}}{c}_0,c_2>\frac{1}{2}.\hfill \end{array}$$

(4.64)

We note that  

$$\frac{m_{\nu }}{m_e}=\frac{1}{{\mu }_2}\sqrt{\frac{1+2c_2}{1+2c_0}}{z}_L^{c_0-c_2}$$

(4.65)

for $c_0,c_2<\frac{1}{2}$ and for $c_0,c_2>\frac{1}{2}$⁠.

The spectrum for Set 2 is determined by  

$$\begin{array}{l}{\sin }^2\frac{{\theta }_H}{2}\{S_R^{c_0}{C}_L^{c_1}+{\mu }_1^2{C}_L^{c_0}{S}_R^{c_1}\}\{S_L^{c_0}{C}_L^{c_2}{C}_R^{c_1}+{\mu }_4^2{C}_R^{c_0}{C}_L^{c_2}{S}_L^{c_1}+{\mu }_6^2{S}_L^{c_0}{S}_R^{c_2}{S}_L^{c_1}\}\\ +{\cos }^2\frac{{\theta }_H}{2}\{C_R^{c_0}{C}_L^{c_1}+{\mu }_1^2{S}_L^{c_0}{S}_R^{c_1}\}\{C_L^{c_0}{C}_L^{c_2}{C}_R^{c_1}+{\mu }_4^2{S}_R^{c_0}{C}_L^{c_2}{S}_L^{c_1}+{\mu }_6^2{C}_L^{c_0}{S}_R^{c_2}{S}_L^{c_1}\}=0.\end{array}$$

(4.66)

The mass of the lowest mode is approximately given by  

$$m_{\hat{\nu }}=\{\begin{array}{ll}\frac{{\pi }^{-1}{m}_{\text{KK}}\cot \frac{1}{2}{\theta }_H}{\sqrt{(\frac{1}{1-2c_0}+\frac{{\mu }_1^2{z}_L^{2(c_0-c_1)}}{1-2c_1})(\frac{1}{1+2c_0}+\frac{{\mu }_4^2{z}_L^{2(c_1-c_0)}}{1+2c_1})}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_1<\frac{1}{2},\hfill \\ \frac{{\pi }^{-1}{m}_{\text{KK}}\cot \frac{1}{2}{\theta }_H}{\sqrt{(\frac{1}{2c_0-1}+\frac{{\mu }_1^2}{2c_1-1})(\frac{z_L^{2c_0}}{1+2c_0}+\frac{{\mu }_4^2{z}_L^{2c_1}}{1+2c_1})}}\hfill & \text{for\hspace{0.5em}}{c}_0,c_1>\frac{1}{2}.\hfill \end{array}$$

(4.67)

4.3. Exotic particles

In each generation one can reproduce the mass spectrum of quarks and leptons at the unification scale by adjusting the parameters $c_0$⁠, $c_1$⁠, $c_2$⁠, ${\mu }_2$⁠, ${\mu }_3$⁠, ${\mu }_4/{\mu }_6$ in (4.13), (4.30), (4.46), and (4.63). There are more than enough parameters.

However, there also appear new particles below the KK scale as shown in (4.18) in the $Q_{\text{EM}}=-\frac{2}{3}$ sector, in (4.39) in the $Q_{\text{EM}}=+\frac{1}{3}$ sector, in (4.53) in the $Q_{\text{EM}}=+1$ sector, and in (4.67) in the $Q_{\text{EM}}=0$ sector. In particular, the exotic particle in the $Q_{\text{EM}}=-\frac{2}{3}$ sector causes a severe problem. As shown in (4.19), the ratio of $m_{\hat{u}}$ to $m_u$ is solely determined by ${\theta }_H$⁠. Phenomenologically, ${\theta }_H<0.1$⁠. It will be seen in the next section that with reasonable parameters it is not possible to get a minimum of the effective potential $V_{\text{eff}}({\theta }_H)$ at very small ${\theta }_H$⁠. It is unavoidable to have unwanted light $\hat{u}$ particles in the first and second generations.

5. Effective potential

In this section, we evaluate the Higgs effective potential $V_{\text{eff}}({\theta }_H)$ by using the mass spectrum formulas of $SO(11)$ gauge bosons and fermions. The contributions to the effective potential from the quark–lepton multiplets in the first and second generations are negligibly small in the RS space, and can be ignored. In numerical evaluation, we use the mass parameters and gauge-coupling constants listed in Ref. [76].

The one-loop effective potential from each KK tower is given by [9,75,77]  

$$V_{\text{eff}}({\theta }_H)=\pm \frac{1}{2}{\displaystyle \int \frac{d^{4}p}{{(2\pi )}^4}}{\displaystyle \sum _n\text{ln}}\left(p^2+m_n{({\theta }_H)}^2\right)=\pm I\left[Q(q);f({\theta }_H)\right],$$

(5.1)

where $\{m_n({\theta }_H)\}$ is the mass spectrum of the KK tower and we take the $+$ sign for bosons and $-$ sign for fermions. $I[Q;f]$ is given by  

$$I[Q(q);f({\theta }_H)]:=\frac{{(k{z}_L^{-1})}^4}{{(4\pi )}^2}{\displaystyle {\int }_0^{\infty }d}q{q}^3\ln [1+Q(q)f({\theta }_H)],$$

(5.2)

where $Q(q)=\tilde{Q}(iq{z}_L^{-1})$ when the mass spectrum (⁠$m_n=k{\lambda }_n$⁠) is determined by $\tilde{Q}({\lambda }_n)=0$⁠. For example, when a mass spectrum is determined by the equation $A({\lambda }_n)+B({\lambda }_n)f({\theta }_H)=0$⁠, we rewrite the equation as $1+\tilde{Q}({\lambda }_n)f({\theta }_H)=0$ where $\tilde{Q}(\lambda )=A(\lambda )/B(\lambda )$⁠. The first and second derivatives of $I\left[Q(q);f({\theta }_H)\right]$ with respect to ${\theta }_H$ are given by  

(5.3)

where $f^{(n)}({\theta }_H):={\partial }^{n}f({\theta }_H)/\partial {\theta }_H^n$⁠.

The evaluation of the total effective potential $V_{\text{eff}}({\theta }_H)=V_{\text{eff}}^{\text{gauge}}({\theta }_H)+V_{\text{eff}}^{\text{fermion}}({\theta }_H)$ is straightforward. We are interested in the ${\theta }_H$-dependent part of $V_{\text{eff}}$⁠, to which only KK towers with ${\theta }_H$-dependent spectra contribute. $V_{\text{eff}}^{\text{gauge}}({\theta }_H)$ in the $\xi =1$ gauge is decomposed as  

$$V_{\text{eff}}^{\text{gauge}}({\theta }_H)=V_{\text{eff}}^{W^{\pm }}+V_{\text{eff}}^Z+V_{\text{eff}}^Y+V_{\text{eff}}^{A_z^{a4},A_z^{a11}}+V_{\text{eff}}^{A_z^{k4},A_z^{k11}}.$$

(5.4)

The equations determining the spectra are given by (3.9) for the $W^{\pm }$ tower, (3.15) for the $Z$ tower, (3.29) for the $Y$ boson tower, (3.36) for the $A_z^{a4},A_z^{a11}$$(a=1,2,3)$ towers, and (3.32) for the $A_z^{k4},A_z^{k11}$$(k=5,\dots ,10)$ towers. It has been confirmed that the use of the approximate formula (3.9) in place of the exact formula (3.8), for instance, is numerically justified. One finds that  

$$\begin{array}{c}{V}_{\text{eff}}^{W^{\pm }}({\theta }_H)=4I\left[\frac{1}{2}{Q}_0\left(q,\frac{1}{2}\right);{\sin }^2{\theta }_H\right],\\ V_{\text{eff}}^Z({\theta }_H)=2I\left[Q_0\left(q,\frac{1}{2}\right);{\sin }^2{\theta }_H\right],\\ V_{\text{eff}}^Y({\theta }_H)=12I\left[\frac{1}{2}{Q}_0\left(q,\frac{1}{2}\right);1+{\cos }^2{\theta }_H\right],\\ V_{\text{eff}}^{A_z^{a4},A_z^{a11}}({\theta }_H)=3I\left[Q_0\left(q,\frac{1}{2}\right);{\sin }^2{\theta }_H\right],\\ V_{\text{eff}}^{A_z^{k4},A_z^{k11}}({\theta }_H)=6I\left[Q_0\left(q,\frac{1}{2}\right);{\cos }^2{\theta }_H\right].\end{array}$$

(5.5)

Here,  

$$\begin{array}{l}{Q}_0\left(q,c\right)=\frac{z_L}{q^2}\frac{1}{{\hat{F}}_c^{++}(q){\hat{F}}_c^{--}(q)},\\ {\hat{F}}_c^{\pm \pm }(q)={\hat{F}}_{c\pm \frac{1}{2},c\pm \frac{1}{2}}(q{z}_L^{-1},q),\\ {\hat{F}}_{\alpha ,\beta }(u,v)=I_{\alpha }(u)K_{\beta }(v)-e^{-i(\alpha -\beta )\pi }{K}_{\alpha }(u)I_{\beta }(v),\end{array}$$

(5.6)

where $I_{\alpha }(u)$ and $K_{\alpha }(u)$ are modified Bessel functions.

The fermion part $V_{\text{eff}}^{\text{fermion}}({\theta }_H)$ is evaluated in a similar manner. Following the classification based on $Q_{\text{EM}}$ in the previous section, we decompose $V_{\text{eff}}^{\text{fermion}}$ into eight parts:  

$$\begin{array}{l}{V}_{\text{eff}}^{\text{fermion}}({\theta }_H;c_0,c_1,c_2;{\mu }_k)\\ =V_{\text{eff}}^{(\text{i})}+V_{\text{eff}}^{(\text{ii})}+V_{\text{eff}}^{(\text{iii})}+V_{\text{eff}}^{(\text{iv})}+V_{\text{eff}}^{(\text{v})}+V_{\text{eff}}^{(\text{vi})}+V_{\text{eff}}^{(\text{vii-1})}+V_{\text{eff}}^{(\text{vii-2})}.\end{array}$$

(5.7)

$V_{\text{eff}}^{\text{fermion}}$ depends on the three bulk mass parameters (⁠$c_j$⁠) and brane interaction mass parameters (⁠${\mu }_k$⁠) in the third generation. We set ${\mu }_5=0$ as before. The equations determining the mass spectra are (i) (4.12) for the $Q_{\text{EM}}=+\frac{2}{3}$ (⁠$u$-type) quarks, (ii) (4.17) for the $Q_{\text{EM}}=-\frac{2}{3}$ (⁠$\hat{u}$-type) quarks, (iii) (4.28) for the $Q_{\text{EM}}=-\frac{1}{3}$ (⁠$d$-type) quarks, (iv) (4.37) for the $Q_{\text{EM}}=+\frac{1}{3}$ (⁠$\hat{d}$-type) quarks, (v) (4.44) for the $Q_{\text{EM}}=-1$ (⁠$e$-type) leptons, (vi) (4.51) for the $Q_{\text{EM}}=+1$ (⁠$\hat{e}$-type) leptons, (vii-1) (4.59) for the $Q_{\text{EM}}=0$ (⁠$\nu$-type) leptons, and (vii-2) (4.66) for the $Q_{\text{EM}}=0$ (⁠$\hat{\nu }$-type) leptons. $V_{\text{eff}}^{\text{F}(\text{i})}$⁠, $V_{\text{eff}}^{\text{F}(\text{ii})}$⁠, etc. are given by  

$$\begin{array}{c}{V}_{\text{eff}}^{(\text{i})}({\theta }_H)=-4I\left[Q_0\left(q,c_0\right);{\sin }^2\frac{1}{2}{\theta }_H\right],\\ V_{\text{eff}}^{(\text{ii})}({\theta }_H)=-4I\left[Q_0\left(q,c_0\right);{\cos }^2\frac{1}{2}{\theta }_H\right],\\ V_{\text{eff}}^{(\text{iii})}({\theta }_H)=-4I\left[Q_{(\text{iii})}(q,c_0,c_1,c_2,{\mu }_4,{\mu }_6);{\sin }^2\frac{1}{2}{\theta }_H\right],\\ V_{\text{eff}}^{(\text{iv})}({\theta }_H)=-4I\left[Q_{(\text{iv})}(q,c_0,c_1,c_2,{\mu }_3,{\mu }_6);{\cos }^2\frac{1}{2}{\theta }_H\right],\\ V_{\text{eff}}^{(\text{v})}({\theta }_H)=-4I\left[Q_{(\text{v})}(q,c_0,c_1,c_2,{\mu }_3,{\mu }_6);{\sin }^2\frac{1}{2}{\theta }_H\right],\\ V_{\text{eff}}^{(\text{vi})}({\theta }_H)=-4I\left[Q_{(\text{v})}(q,c_0,c_1,c_2,{\mu }_4,{\mu }_6);{\cos }^2\frac{1}{2}{\theta }_H\right],\\ V_{\text{eff}}^{(\text{vii01})}({\theta }_H)=-4I\left[Q_{(\text{vii01})}(q,c_0,c_1,c_2,{\mu }_2,{\mu }_3,{\mu }_6);{\cos }^2\frac{1}{2}{\theta }_H\right],\\ V_{\text{eff}}^{(\text{vii02})}({\theta }_H)=-4I\left[Q_{(\text{vii02})}(q,c_0,c_1,c_2,{\mu }_1,{\mu }_4,{\mu }_6);{\sin }^2\frac{1}{2}{\theta }_H\right],\end{array}$$

(5.8)

where  

(5.9)

and ${\hat{F}}_c^{\pm \pm }={\hat{F}}_c^{\pm \pm }(q)$⁠.

The Higgs mass $m_H({\theta }_H={\theta }_H^{\text{min}})$ is determined by  

$$m_H^2({\theta }_H)={\frac{1}{f_H^2}\frac{d^2{V}_{\text{eff}}({\theta }_H)}{d{\theta }_H^2}|}_{{\theta }_H={\theta }_1},$$

(5.10)

where $f_H$ is given by (2.37), or by  

$$f_H=\sqrt{\frac{{\sin }^2{\theta }_W}{\pi {\alpha }_{em}}\frac{k^2}{(z_L^2-1)\text{log}(z_L)}}.$$

(5.11)

Here, ${\alpha }_{em}$ is the fine-structure constant, and ${\theta }_W$ is the Weinberg angle.

In the following we give example calculations for the effective potential and show a result for the Higgs mass. As we remarked before, the current $SO(11)$ model necessarily contains light exotic particles, and therefore is not completely realistic. With this in mind, we do not insist on reproducing all of the observed values of the masses of the SM gauge bosons, Higgs boson, quarks, and leptons. Further, the GUT relation leads to ${\sin }^2{\theta }_W=\frac{3}{8}$⁠. The renormalization group equation effect must be taken into account to compare it with the observed value at low energies.

From the mass relations (4.31), (4.47), and (4.65) applied to the third generation, one finds the following constraints for the brane mass parameters $({\mu }_2,{\mu }_3,{\mu }_4,{\mu }_6)$⁠:  

$$\begin{array}{c}{\mu }_2\simeq \sqrt{\frac{1+2c_2}{1+2c_0}}{z}_L^{c_0-c_2}\frac{m_{\tau }}{m_{{\nu }_{\tau }}},\\ {\mu }_3\simeq \sqrt{\frac{1-2c_2}{1-2c_0}}{z}_L^{c_2-c_0}\frac{m_t}{m_{\tau }},\\ \frac{{\mu }_6}{{\mu }_4}\simeq \sqrt{\frac{1+2c_0}{1+2c_2}}{z}_L^{c_2-c_0}\frac{m_b}{m_t}.\end{array}$$

(5.12)

For $c_0=c_2$⁠, these constraints lead typically to ${\mu }_2>O({10}^{10})$⁠, ${\mu }_3~100$⁠, ${\mu }_4\simeq 40{\mu }_6$⁠, and ${\mu }_6=O(1)$⁠. No constraint appears for ${\mu }_1$⁠. It should be noted that the brane parameters ${\mu }_k$ sensitively depend on the bulk mass parameters $c_0$ and $c_2$⁠, as demonstrated below.

To find a consistent set of parameters we use the following procedure: Depending on the initial values of $c_2$⁠, ${\mu }_1$⁠, and ${\mu }_4$⁠, one may not find a consistent solution at step 4. In particular, we could not find consistent solutions with ${\theta }_H^{\text{min}}~0.1$ for $c_0=c_2$⁠. Judicious choice of appropriate values for $c_2$⁠, ${\mu }_1$⁠, and ${\mu }_4$ is necessary.

• (0) We fix $z_L=e^{kL}$ and pick ${\theta }_H={\theta }_H^{\text{min}}$⁠.

• (1) We suppose that the minimum of $V_{\text{eff}}({\theta }_H)$ is located at ${\theta }_H={\theta }_H^{\text{min}}$⁠. Equation (3.15) determines the spectrum $\left\{{\lambda }_{Z^{(n)}}\right\}$ of the $Z$ tower. By using the zero mode mass $m_{Z^{(0)}}$ and the observed $Z$ boson mass $m_Z^{\text{obs}}$⁠, $k=m_{Z^{(0)}}/{\lambda }_{Z^{(0)}}=m_Z^{\text{obs}}/{\lambda }_{Z^{(0)}}$ is determined. The KK mass scale is also determined by $m_{\text{KK}}=\pi k/(z_L-1)\simeq \pi k{z}_L^{-1}$⁠.

• (2) The observed top quark mass $m_t^{\text{obs}}$ determines the bulk mass parameter of the third generation $SO(11)$ spinor fermion $c_0$ through (4.13).

• (3) We choose a sample value of the bulk mass parameter of $SO(11)$ vector fermion $c_2$⁠. Then the brane mass parameters ${\mu }_2$⁠, ${\mu }_3$⁠, and ${\mu }_6/{\mu }_4$ are determined by (5.12). For ${\mu }_1$ and ${\mu }_4$ we have taken sample values ${\mu }_1=1$ and ${\mu }_4=0.5$⁠.

• (4) At this stage $V_{\text{eff}}({\theta }_H)$ is determined, once the bulk mass parameters of $SO(11)$ vector fermions $c_1$ is given. $c_1$ is fixed by demanding that $V_{\text{eff}}({\theta }_H)$ has a global minimum at ${\theta }_H={\theta }_H^{\text{min}}$⁠.  

  $$0={\frac{d}{d{\theta }_H}{V}_{\text{eff}}({\theta }_H)|}_{{\theta }_H={\theta }_H^{\text{min}}}.$$

  (5.13)

• (5) By using the above values of the bulk and brane parameters, we obtain the Higgs boson mass $m_H$ by using (5.10).

We give a sample calculation for the effective potential $V_{\text{eff}}({\theta }_H)$ and $m_H({\theta }_H)$⁠. Let us take, as a set of input parameters, $z_L={10}^{10}$⁠, ${\theta }_H=0.10$⁠, ${\alpha }_{em}^{-1}(m_Z)=127.916\pm 0.015$⁠, ${\sin }^2{\theta }_W(m_Z)=0.23116\pm 0.00013$⁠, $m_t=173.1\pm 1.22$GeV, $m_b=4.18$GeV, $m_{\tau }=1.776$GeV, and $m_{{\nu }_{\tau }}=0.1$eV. For $m_b$ and $m_{\tau }$⁠, we use the central values. The value of $m_{{\nu }_{\tau }}$ is a reference value for our calculation. $m_{\text{KK}}$⁠, $k$⁠, and $f_H$ are determined to be  

$$m_{\text{KK}}=1.088\times {10}^4\text{\hspace{0.5em}GeV},k=3.464\times {10}^{13}\text{\hspace{0.5em}GeV},f_H=2216\text{\hspace{0.5em}GeV}.$$

(5.14)

For $m_t=165.0,170.0,175.0$GeV, consistent sets of the bulk and brane parameters are tabulated in Table 5. The Higgs boson mass $m_H$ is the output. In the current model it comes out in the range $50\text{\hspace{0.5em}GeV}<m_H<55\text{\hspace{0.5em}GeV}$⁠, smaller than the observed value $m_H\simeq 125$GeV. Even if one takes slightly different values for $c_2$⁠, ${\mu }_1$⁠, and ${\mu }_4$⁠, the value of $m_H$ does not change very much.

Table 5.

Consistent parameter sets for ${\theta }_H=0.10$⁠, $z_L={10}^{10}$⁠, $({\mu }_1,{\mu }_4)=(1.0,0.50)$ for various values of $m_t$⁠. The Higgs boson mass $m_H$ is the output.

Top quark	Bulk parameters			Brane parameters			Higgs
$m_t$[GeV]	$c_0$	$c_1$	$c_2$	${\mu }_2$	${\mu }_3$	${\mu }_6$	$m_H$[GeV]
165.0	0.3696	0.4286	0.2970	$9.05\times {10}^{10}$	21.8	0.00249	50.96
170.0	0.3559	0.4293	0.3120	$5.20\times {10}^{10}$	36.8	0.00420	51.77
175.0	0.3496	0.4286	0.3270	$2.95\times {10}^{10}$	62.8	0.00719	53.52

The effective potential for $m_t=170$GeV is displayed in Fig. 1. The global minimum is located at ${\theta }_H=0.10$⁠, and the EW symmetry breaking takes place. In Figs. 2 and 3, the contributions of gauge fields and fermions are plotted separately.

Fig. 1.

The effective potential $V_{\text{eff}}({\theta }_H)$ for ${\theta }_H^{\text{min}}=0.10$⁠, $z_L={10}^{10}$⁠, and $m_t=170$GeV. $V_{\text{eff}}(\theta )/[{(k{z}_L^{-1})}^4/{(4\pi )}^2]$ has been plotted. The bulk mass parameters are given by $(c_0=0.3599,c_1=0.4293,c_2=0.3120)$⁠. The bottom figure shows the behavior near the minimum. The blue solid, the green dashed, and the red short-dashed lines show the effective potential containing the contributions from all the $SO(11)$ bulk gauge boson and fermions, only $SO(11)$ bulk gauge boson, and only $SO(11)$ bulk fermions, respectively.

Fig. 2.

Contributions of gauge fields to $V_{\text{eff}}({\theta }_H)$⁠. The input parameters are the same as in Fig. 1. The green solid line represents all gauge field contributions for the effective potential, which is the same as the green dashed in Fig. 1. The red dashed line is the (i) $W^{\pm }$ contribution, the purple short-dashed line is the (ii) $Z$ contribution, the orange dashed line is the (viii) $Y$ contribution, the blue dashed line is the (ii) $A_z^{a4},A_z^{a11}(a=1,2,3)$ contribution, and the brown short-dashed line is the (v) $A_z^{k4},A_z^{k11}(k=5,\dots ,10)$ contribution.

Fig. 3.

Contributions of fermions to $V_{\text{eff}}({\theta }_H)$⁠. The parameter set is the same as in Figure 1. The red solid line is the total fermion contribution for effective potentials, which is the same as the red dashed line in Figure 1. The green dashed line is the (i) $Q_{\text{EM}}=+\frac{2}{3}$ fermion contribution, the cyan short-dashed line is the (ii) $Q_{\text{EM}}=-\frac{2}{3}$ fermion contribution, the brown dashed line is the (iii) $Q_{\text{EM}}=-\frac{1}{3}$ fermion contribution, the pink short-dashed line is the (iv) $Q_{\text{EM}}=+\frac{1}{3}$ fermion contribution, the blue dashed line is the (v) $Q_{\text{EM}}=-1$ fermion contribution, the gray short-dashed line is the (vi) $Q_{\text{EM}}=+1$ fermion contribution, the orange dashed line is the (vii-1) $Q_{\text{EM}}=0$ fermion contribution, and the magenta short-dashed line is (vii-2) $Q_{\text{EM}}=0$ fermion contribution.

It can be seen in Figure 2 that the contributions from (v) $A_z^{k4},A_z^{k11}$ (⁠$k=5,\dots ,10$⁠) and (viii) $Y$ bosons dominate over the others in the gauge field sector. In the fermion sector there appears to be cancelation among contributions from various components. It can be seen in Figure 3 that the contribution of (i) the top quark is almost canceled by that of (ii) the $\hat{u}$-type $\hat{t}$ fermion. The bottom quark and $\hat{d}$-type $\hat{b}$ fermion contributions are not canceled out, but each contribution is small. The tau lepton and $\hat{e}$-type $\hat{\tau }$ fermion contributions are not canceled out, but each contribution is small. The four contributions from $b$⁠, $\hat{b}$⁠, $\tau$⁠, and $\hat{\tau }$ add up to almost zero. The contribution from neutral fermions is appreciable in the current model.

In the previous section we observed that there appear light exotic fermions that should not exist in reality. In this section we have observed that there appear cancelations among the contributions to $V_{\text{eff}}({\theta }_H)$ from fermions and their corresponding exotics. These two seem to be related, and the too-light Higgs boson mass $m_H$ is inferred to be a result of those cancelations.

6. Conclusion and discussions

In the present paper we have explored $SO(11)$ gauge–Higgs grand unification in the RS space. $SO(11)$ gauge symmetry is broken to $SO(4)\times SO(6)$ symmetry by the orbifold boundary conditions, which is spontaneously broken to $SU{(2)}_L\times U{(1)}_Y\times SU{(3)}_C$ by the brane scalar ${\Phi }_{16}$ on the Planck brane. The EW $SU{(2)}_L\times U{(1)}_Y$ symmetry is dynamically broken to $U{(1)}_{\text{EM}}$ by the Hosotani mechanism. The Higgs boson appears as the four-dimensional fluctuation mode of the AB phase in the fifth dimension, or the zero mode of $A_y$⁠. Thus the gauge–Higgs unification is achieved.

Quark–lepton fermion multiplets are introduced in ${\Psi }_{32}$⁠, ${\Psi }_{11}$⁠, and ${\Psi }_{11}^{\text{'}}$ in each generation. Unlike the $SO(5)\times U{(1)}_X$ gauge–Higgs EW unification, one need not introduce brane fermions on the Planck brane. The quark–lepton masses are generated by the Hosotani mechanism with ${\theta }_H\ne 0$⁠, supplemented with the brane interactions on the Planck brane. We have demonstrated that the quark–lepton mass spectrum can be reproduced by adjusting the parameters of the brane interactions.

One of the interesting features of the model is that proton decay is forbidden, in sharp contrast to the GUT models in four dimensions. The quark–lepton number $N_{\Psi }$ is conserved by the gauge interactions and brane interactions.

In the current model, however, there appear light exotic fermions associated with $\hat{u}$-type, $\hat{d}$-type, and $\hat{e}$-type fermions, which contradicts observation. The Higgs boson mass $m_H$⁠, which is predicted in the current gauge–Higgs grand unification, turns out too small. The small $m_H$ is a result of the partial cancelation among the contributions of the quark–lepton component and the exotic fermion component to the effective potential $V_{\text{eff}}({\theta }_H)$⁠. In other words the exotic fermion problem and the small $m_H$ problem seem to be related to each other. The model needs improvement in this regard. We hope to report how to cure these problems in the near future.

Acknowledgements

This work was supported in part by the Japan Society for the Promotion of Science, Grants-in-Aid for Scientific Research Nos. 23104009 and 15K05052.

Funding

Open Access funding: SCOAP³.

Appendix A. $SO(11)$⁠, $SO(10)$⁠, and $SO(4)$

The generators of $SO(11)$⁠, $T_{jk}=-T_{kj}=T_{jk}^{†}$ (⁠$j,k=1,\dots ,11$⁠), satisfy the algebra  

$$[T_{ij},T_{kl}]=i({\delta }_{ik}{T}_{jl}-{\delta }_{il}{T}_{jk}+{\delta }_{jl}{T}_{ik}-{\delta }_{jk}{T}_{il}).$$

(A1)

In the adjoint representation,  

$$\begin{array}{l}{(T_{ij})}_{pq}=-i({\delta }_{ip}{\delta }_{jq}-{\delta }_{iq}{\delta }_{jp}),\\ \text{Tr}{T}_{jk}{T}_{lm}=2({\delta }_{jl}{\delta }_{km}-{\delta }_{jm}{\delta }_{kl}),\text{Tr}{(T_{jk})}^2=2.\end{array}$$

(A2)

As a basis of $SO(11)$ Clifford algebra, it is convenient to adopt  

$$\begin{array}{l}\{{\Gamma }_j,{\Gamma }_k\}=2{\delta }_{jk}{I}_{32},\\ {\Gamma }_1={\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1,\\ {\Gamma }_2={\sigma }^2\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1,\\ {\Gamma }_3={\sigma }^3\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1,\\ {\Gamma }_4={\sigma }^0\otimes {\sigma }^2\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1,\\ {\Gamma }_5={\sigma }^0\otimes {\sigma }^3\otimes {\sigma }^1\otimes {\sigma }^1\otimes {\sigma }^1,\\ {\Gamma }_6={\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^2\otimes {\sigma }^1\otimes {\sigma }^1,\\ {\Gamma }_7={\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^3\otimes {\sigma }^1\otimes {\sigma }^1,\\ {\Gamma }_8={\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^2\otimes {\sigma }^1,\\ {\Gamma }_9={\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^3\otimes {\sigma }^1,\\ {\Gamma }_{10}={\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^2,\\ {\Gamma }_{11}={\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^3=-i{\Gamma }_1\cdots {\Gamma }_{10},\end{array}$$

(A3)

where ${\sigma }^0=I_2$ and $\left\{{\sigma }^k\right\}$ are Pauli matrices. In terms of ${\Gamma }_j$ the $SO(11)$ generators in the spinorial representation are given by  

$$\begin{array}{l}{T}_{jk}=-\frac{i}{4}[{\Gamma }_j,{\Gamma }_k]\left(=-\frac{i}{2}{\Gamma }_j{\Gamma }_k\text{\hspace{1em}for\hspace{0.5em}}j\ne k\right),\\ {(T_{jk})}^2=\frac{1}{4}{I}_{32},\text{\hspace{1em}Tr}{(T_{jk})}^2=8.\end{array}$$

(A4)

The orbifold boundary conditions $P_0,P_1$ in (2.6) and (2.7) break $SO(11)$ to $SO(4)\times SO(6)$⁠. The generators of the corresponding $SO(4)\simeq SU{(2)}_L\times SU{(2)}_R$ in the spinorial representation are given by  

$$\begin{array}{c}{\overrightarrow{T}}_L=\frac{1}{2}\left(\begin{array}{c}\begin{array}{c}{T}_{23}+T_{14}\\ T_{31}+T_{24}\\ T_{12}+T_{34}\end{array}\end{array}\right)=\frac{1}{2}\overrightarrow{\sigma }\otimes \left(\begin{array}{c}1\\ 0\end{array}\right)\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0,\\ {\overrightarrow{T}}_R=\frac{1}{2}\left(\begin{array}{c}\begin{array}{c}{T}_{23}-T_{14}\end{array}\\ T_{31}-T_{24}\\ T_{12}-T_{34}\end{array}\right)=\frac{1}{2}\overrightarrow{\sigma }\otimes \left(\begin{array}{c}0\\ 1\end{array}\right)\otimes {\sigma }^0\otimes {\sigma }^0\otimes {\sigma }^0.\end{array}$$

(A5)

The orbifold boundary condition $P_0$ at the Planck brane reduces $SO(11)$ to $SO(10)$⁠, whose generators are given by $T_{jk}$ (⁠$j,k=1,\dots ,10$⁠). In the representation (A3), those generators become block-diagonal $T_{jk}^{SO(10)}=[\cdots ]\otimes ({\sigma }^0\text{or}{\sigma }^3)$ so that a spinor 32 of $SO(11)$ splits into $16\oplus \overline{16}$ of $SO(10)$⁠:  

$${\Psi }_{32}=\left(\begin{array}{c}\begin{array}{c}{\Psi }_{16}\\ {\Psi }_{\overline{16}}\end{array}\end{array}\right).$$

(A6)

With (A3) one finds that  

$$\begin{array}{l}{\Gamma }_j^{*}={(-1)}^{j+1}{\Gamma }_j,\\ R{\Gamma }_{j}R={(-1)}^j{\Gamma }_j,R{\Gamma }_j^{*}R=-{\Gamma }_j,\\ R{T}_{jk}^{*}R=-T_{jk},\\ R={\Gamma }_2{\Gamma }_4{\Gamma }_6{\Gamma }_8{\Gamma }_{10}=R^{†}=R^{-1}\\ =-{\sigma }^2\otimes {\sigma }^3\otimes {\sigma }^2\otimes {\sigma }^3\otimes {\sigma }^2\\ \equiv -\hat{R}\otimes {\sigma }^2.\end{array}$$

(A7)

It follows that for an $SO(11)$ spinor ${\Psi }_{32}$⁠, the $R$-transformed one also transforms as 32:  

$$\begin{array}{c}{\tilde{\Psi }}_{32}\equiv iR{\Psi }_{32}^{*},\\ {\Psi }_{32}^{\text{'}}=(1+\frac{i}{2}{ϵ}_{jk}{T}_{jk}){\Psi }_{32}\Rightarrow {\tilde{\Psi }}_{32}^{\text{'}}=(1+\frac{i}{2}{ϵ}_{jk}{T}_{jk}){\tilde{\Psi }}_{32}.\end{array}$$

(A8)

Its $SO(10)$ content is given by  

$${\tilde{\Psi }}_{32}=\left(\begin{array}{c}\begin{array}{c}{\tilde{\Psi }}_{16}\end{array}\\ {\tilde{\Psi }}_{\overline{16}}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}-\hat{R}{\Psi }_{\overline{16}}^{*}\end{array}\\ +\hat{R}{\Psi }_{16}^{*}\end{array}\right).$$

(A9)

Appendix B. Basis functions in RS space

Mode functions of various fields in the RS spacetime are expressed in terms of Bessel functions. We define, for gauge fields,  

$$\begin{array}{l}C(z;\lambda )=\frac{\pi }{2}\lambda z{z}_L{F}_{1,0}(\lambda z,\lambda z_L),C^{\prime }(z;\lambda )=\frac{\pi }{2}{\lambda }^{2}z{z}_L{F}_{0,0}(\lambda z,\lambda z_L),\\ S(z;\lambda )=-\frac{\pi }{2}\lambda z{F}_{1,1}(\lambda z,\lambda z_L),S^{\prime }(z;\lambda )=-\frac{\pi }{2}{\lambda }^{2}z{F}_{0,1}(\lambda z,\lambda z_L),\end{array}$$

(B1)

where $F_{\alpha ,\beta }(u,v)=J_{\alpha }(u)Y_{\beta }(v)-Y_{\alpha }(u)J_{\beta }(v)$⁠. They satisfy  

$$\begin{array}{l}z\frac{d}{dz}\frac{1}{z}\frac{d}{dz}\left(\begin{array}{c}\begin{array}{c}C(z;\lambda )\end{array}\\ S(z;\lambda )\end{array}\right)=-{\lambda }^2\left(\begin{array}{c}\begin{array}{c}C(z;\lambda )\end{array}\\ S(z;\lambda )\end{array}\right),\\ C(z_L;\lambda )=z_L,C^{\prime }(z_L;\lambda )=0,S(z_L;\lambda )=0,S^{\prime }(z_L;\lambda )=\lambda ,\\ C{S}^{\prime }-S{C}^{\prime }=\lambda z.\end{array}$$

(B2)

It follows that  

$$\begin{array}{l}\frac{d}{dz}z\frac{d}{dz}\frac{1}{z}\left(\begin{array}{c}\begin{array}{c}{C}^{\prime }(z;\lambda )\end{array}\\ S^{\prime }(z;\lambda )\end{array}\right)=-{\lambda }^2\left(\begin{array}{c}\begin{array}{c}{C}^{\prime }(z;\lambda )\end{array}\\ S^{\prime }(z;\lambda )\end{array}\right),\\ \frac{d}{dz}\{\frac{1}{z}{S}^{\prime }(z;\lambda )\}{|}_{z=z_L}=0.\end{array}$$

(B3)

For fermions with a bulk mass parameter $c$ we define  

$$\begin{array}{l}\left(\begin{array}{c}\begin{array}{c}{C}_L\\ S_L\end{array}\end{array}\right)(z;\lambda ,c)=\pm \frac{\pi }{2}\lambda \sqrt{z{z}_L}{F}_{c+\frac{1}{2},c\mp \frac{1}{2}}(\lambda z,\lambda z_L),\\ \left(\begin{array}{c}\begin{array}{c}{C}_R\end{array}\\ S_R\end{array}\right)(z;\lambda ,c)=\mp \frac{\pi }{2}\lambda \sqrt{z{z}_L}{F}_{c-\frac{1}{2},c\pm \frac{1}{2}}(\lambda z,\lambda z_L),\end{array}$$

(B4)

which satisfy  

$$\begin{array}{l}{D}_{+}(c)\left(\begin{array}{c}\begin{array}{c}{C}_L\end{array}\\ S_L\end{array}\right)=\lambda \left(\begin{array}{c}\begin{array}{c}{S}_R\\ C_R\end{array}\end{array}\right),D_{-}(c)\left(\begin{array}{c}\begin{array}{c}{C}_R\\ S_R\end{array}\end{array}\right)=\lambda \left(\begin{array}{c}\begin{array}{c}{S}_L\end{array}\\ C_L\end{array}\right),\\ D_{\pm }(c)=\pm \frac{d}{dz}+\frac{c}{z},\\ C_R=C_L=1,S_R=S_L=0,\text{at}z=z_L,\\ C_L{C}_R-S_L{S}_R=1.\end{array}$$

(B5)

We note that for $\lambda z_L\ll 1$ and $c\ge 0$⁠,  

$$\begin{array}{l}C(1;\lambda )~z_L\cdot \{1+O({\lambda }^2{z}_L^2)\},\\ C^{\prime }(1;\lambda )~{\lambda }^2{z}_L\ln z_L\cdot \{1+O({\lambda }^2{z}_L^2)\},\\ S(1;\lambda )~-\frac{1}{2}\lambda z_L\cdot \{1+O({\lambda }^2{z}_L^2)\},\\ S^{\prime }(1;\lambda )~\lambda z_L^{-1}\cdot \{1+O({\lambda }^2{z}_L^2)\},\\ C_L(1;\lambda ,c)~z_L^c\cdot \{1+O({\lambda }^2{z}_L^2)\},\\ C_R(1;\lambda ,c)~z_L^{-c}\cdot \{1+O({\lambda }^2{z}_L^2)\},\\ S_L(1;\lambda ,c)~-\frac{\lambda z_L^{c+1}}{2(c+\frac{1}{2})}\cdot \{1+O({\lambda }^2{z}_L^2)\},\\ S_R(1;\lambda ,c)~\{\begin{array}{ll}\frac{\lambda z_L^c}{2(c-\frac{1}{2})}\cdot \{1+O({\lambda }^2{z}_L^2)\}\hfill & \text{for\hspace{0.5em}}c>\frac{1}{2},\hfill \\ \lambda z_L^{1/2}\ln z_L\hfill & \text{for\hspace{0.5em}}c=\frac{1}{2},\hfill \\ \frac{\lambda z_L^{1-c}}{2(\frac{1}{2}-c)}\cdot \{1+O({\lambda }^2{z}_L^2)\}\hfill & \text{for\hspace{0.5em}}c<\frac{1}{2}.\hfill \end{array}\end{array}$$

(B6)

In particular,  

$$S_L(1;\lambda ,c)S_R(1;\lambda ,c)~\{\begin{array}{ll}-\frac{{\lambda }^2{z}_L^{2c+1}}{4c^2-1}\cdot \{1+O({\lambda }^2{z}_L^2)\}\hfill & \text{for\hspace{0.5em}}c>\frac{1}{2},\hfill \\ -\frac{1}{2}{\lambda }^2{z}_L^2\ln z_L\hfill & \text{for\hspace{0.5em}}c=\frac{1}{2},\hfill \\ -\frac{{\lambda }^2{z}_L^2}{1-4c^2}\cdot \{1+O({\lambda }^2{z}_L^2)\}\hfill & \text{for\hspace{0.5em}}c<\frac{1}{2}.\hfill \end{array}$$

(B7)