Modular S 3 -invariant flavor model in SU(5) grand unified theory

................................................................................................................... We present a ﬂavor model with S 3 modular invariance in the framework of SU(5) grand uniﬁed theory (GUT). The S 3 modular forms of weights 2 and 4 give the quark and lepton mass matrices with a common complex parameter, the modulus τ . The GUT relation of down-type quarks and charged leptons is imposed by the vacuum expectation value (VEV) of the adjoint 24-dimensional Higgs multiplet in addition to the VEVs of 5 and ¯ 5 Higgs multiplets of SU(5). The observed Cabibbo–Kobayashi–Maskawa and Pontecorvo–Maki–Nakagawa–Sakata mixing parameters as well as the mass eigenvalues are reproduced properly. We discuss the leptonic charge–parity phase and the effective mass of the neutrinoless double beta decay with the sum of neutrino masses.


Introduction
The standard model (SM) was well established by the discovery of the Higgs boson. The SM, however, does not answer a fundamental question about the origin of flavor structure. In order to understand this, many works have addressed the discrete groups for flavors. The S 3 group was used in early models of quark masses and mixing angles [1,2]. This group was also studied to explain the large mixing angle [3] in the oscillation of atmospheric neutrinos [4]. After the discovery of the neutrino oscillations, the discrete symmetries of flavors have been developed to reproduce the observed lepton mixing angles [5][6][7][8][9][10][11][12][13].
Superstring theory with certain compactifications can lead to non-Abelian discrete flavor symmetries. (See, e.g., Refs. [14][15][16][17][18][19][20].) The torus and orbifold compactifications have the modular symmetry of the modulus parameter. The flavors of both quarks and leptons transform non-trivially under the modular transformation [21][22][23][24][25][26][27]. In this sense, the modular symmetry is a non-Abelian discrete flavor symmetry. Yukawa and other couplings depend on the moduli parameters in four-dimensional lowenergy effective field theory derived from superstring theory. Each coupling therefore transforms non-trivially under the modular symmetry, which is an important difference from the conventional flavor symmetries.
The modular group includes S 3 , A 4 , S 4 , and A 5 as its finite subgroups [28].An attractive flavor model has been put forward based on the 3 A 4 modular group [29]. This work stimulates model building based on 2 S 3 [30], 4 S 4 [31], and 5 A 5 [32]. Phenomenological discussions of neutrino PTEP 2020, 053B05 T. Kobayashi et al. flavor mixing have been presented based on the A 4 [33,34], S 4 [35], and A 5 [36] modular groups. In particular, comprehensive analysis of the A 4 modular group has provided a distinct prediction of the neutrino mixing angles and the charge-parity (CP) violating phase [34]. Applications of the modular symmetry have begun to develop in quark and lepton flavors. The A 4 modular symmetry has also been applied to the SU(5) grand unified theory (GUT) of quarks and leptons [37], while the residual symmetry of the A 4 modular symmetry has been investigated phenomenologically [38]. The modular forms for (96) and (384) have also been constructed [39], and the extension of the traditional flavor group has been discussed with modular symmetries [40]. Moreover, multiple modular symmetries are proposed as the origin of flavor [41]. The modular invariance has also been studied combined with generalized CP symmetries for theories of flavors [42]. The quark mass matrix has been discussed in the S 3 and A 4 modular symmetries as well [43,44]. Besides the mass matrices of quarks and leptons, related topics such as baryon number violation [43], dark matter [45], radiatively induced neutrino masses [46], and the modular symmetry anomaly [47] have been discussed.
Among these, the unification of quark and lepton flavors based on the modular symmetry is an important work from the standpoint of quark-lepton unification [37,48] since the modulus τ is common to both quarks and leptons. In this paper we construct an S 3 flavor model with modular invariance in the framework of SU(5) GUT and discuss the Dirac CP violating phases in both quark and lepton sectors as well as the neutrino masses and mixing, the effective neutrino mass of the neutrinoless double beta decay, and Majorana CP violating phases. We consider a six-dimensional compact space X 6 in addition to our four-dimensional spacetime in superstring theory. Suppose that the six-dimensional compact space has some constituent spaces and that they include a twodimensional compact space X 2 . Note that X 2 can have geometrical symmetry such as the modular symmetry. Quark mixing and lepton mixing are explained by a single flavor symmetry originating from X 2 . The modular forms for the quark and lepton sectors are the same and determined by a common value of τ in our setup. The other four-dimensional part of X 6 may contribute to an overall factor of the Yukawa couplings, but not to their ratios.
We assume the S 3 modular symmetry for flavors of quarks and leptons since it is the minimal non-Abelian discrete symmetry. Furthermore, we assume SU(5) GUT as a first step to building a realistic flavor model with modular invariance for both quarks and leptons. It is emphasized that the vacuum expectation value (VEV) of the 24-dimensional adjoint Higgs multiplet H 24 creates a difference between the mass eigenvalues of down-type quarks and charged leptons. Our mass matrices reproduce the observed Cabibbo-Kobayashi-Maskawa (CKM) and Pontecorvo-Maki-Nakagawa-Sakata (PMNS) parameters successfully. We predict the leptonic CP violation phase and the effective mass of the neutrinoless double beta decay versus the sum of neutrino masses, respectively. This paper is organized as follows. In Sect. 2 we present our SU(5) GUT model with the finite modular symmetry 2 S 3 . In Sect. 3, we present numerical analyses of our model. Section 4 is devoted to a summary. Appendix A shows the modular forms of S 3 briefly, and Appendix B presents relevant parameters in the lepton flavor mixing.   (5), S 3 , and weight for superfields and modular forms. The subscript i of F i and T i denotes the ith family.

Quark and lepton mass matrices in SU(5) GUT
where the subscripts 1, 2, 3 denote the quark colors, the superscript c denotes CP-conjugated fermions, and the flavor indices are omitted. In addition, we introduce the right-handed neutrinos N c i (i = 1, 2, 3), which are SU(5) singlets. We present the charge assignments of superfields for the SU(5) gauge group, S 3 flavor symmetry, and modular weights in Table 1, where the subscript i of F i and T i denotes the ith family. An adjoint representation of scalars H 24 breaks the SU(5) gauge symmetry and leads to the mass differences among quarks and charged leptons. The electroweak breaking of the SM is realized by a 5 (5) of Higgs, H 5 (H5), which also contribute to the fermion mass matrices. These Higgs multiplets are listed in Table 1, which also presents the modular forms of weights 2 and 4 that we use.
For Yukawa interactions, the S 3 modular invariant superpotential is written as where the three terms of the right-hand side lead to the mass terms of up-type quarks, down-type quarks, and charged leptons and neutrinos, respectively. The up-type quark mass matrix is derived from w 10 , which is explicitly given as: where α 1,2 , β , k 1,2,3 , and γ are dimensionless complex constants. Here, denotes the cut-off scale around the SU(5) energy scale. We set H 24 / = 0.3. Thus, the next-order corrections of H 24 2 / 2 are O(0.1). We neglect their effect because the experimental values of masses and mixing angles for the quarks and leptons include errors of O(10%). We focus on the parameter regions |k i | = [0, 1.5] in the following numerical analysis. By using the S 3 tensor product of doublets in Appendix A, the mass matrix of up-type quarks is given in terms of the modular forms Y 1 (τ ) and Y 2 (τ ) of Appendix A as PTEP 2020, 053B05 T. Kobayashi et al. where the argument τ of the modular forms is omitted, and the parameters are redefined as follows: where v u is the VEV for the doublet component H u of H 5 . This mass matrix was investigated in our previous work [43]. Suppose the neutrinos to be Majorana particles, which are realized by the seesaw mechanism. Then, the neutrino mass matrix is derived from the superpotential w ν : The additional term w ν is the contribution to the right-handed Majorana mass terms from the dimension-five operators, where the f i are arbitrary coefficients. Here, we take the diagonal basis of N c i N c i . After integrating out N c i (i = 1-3) fields, the Majorana left-handed neutrino mass matrix is therefore given as follows: where the parameters a 0 , a 1 , a 2 , b, and c are redefined as and The superpotentials for the down-type quarks and charged leptons are written as where α 1,2 , β 1,2 , k 1,2,3,4 , and γ are dimensionless complex constants. We focus on the parameter region |k i | = [0, 1.5] in the following numerical analysis. We can construct a mass matrix for the down-type quarks and charged leptons: where we have introduced a new parameter α 0 defined as and v d is the VEV of the doublet component of H5. We can obtain a successful mass matrix for the down-type quarks: where we have redefined some parameters as in Eq. (5).
The quark mass matrices in Eqs. (4) and (14) can reproduce the observed CKM mixing matrix elements and quark mass ratios at the GUT scale [49,50]. Indeed, we have obtained successful uptype and down-type quark mass matrices with hierarchical flavor structure which are completely consistent with the observed masses and CKM parameters [43].
Let us discuss the charged lepton mass matrix, which is possibly related to the down-type quark mass matrix, by using the SU(5) GUT relation. We rewrite the coefficients in the down-type quark mass matrix elements in terms of the sum of contributions from VEVs of H5 and H 24 as follows: 24 13 , c d 31 = c 5 31 + c 24 31 , c d 33 = c 5 33 + c 24 33 , (15) where we have the following relations for the parameters of Eq. (12): Let us give the Clebsch-Gordan (CG) factor C, which is derived by the ratio of VEVs for the charged lepton sector and down-type quark sector: since H 24 takes the VEV as H 24 ∝ diag[2, 2, 2, −3, −3]. The charged lepton mass matrix is therefore obtained in terms of the elements of the down-type quark mass matrix and the coefficient C by transposing the down-type quark mass matrix: as well as the proper hierarchy of quark and charged lepton masses. We use the above parameters for the prediction of the neutrino sector in the next section.

Numerical results
We have obtained parameter regions that reproduce the observed fermion mass ratios and CKM mixing parameters. Our results are consistent with the experimental results for quark mass ratios and charged lepton mass ratios at the GUT scale within the 1 σ range [49,50]. 1 In the following PTEP 2020, 053B05 T. Kobayashi et al.
subsections we present predictions in the neutrino sector and discuss the correlation between the CKM and PMNS mixing parameters. Since we have separated the parameters of the down-type quarks such as ε, c , c 13 , c 31 , and c 33 into two terms as defined in Eq. (15), we can scan the parameters in the charged lepton mass matrix of Eq. (18) by using the successful parameter sets of the down-type quark sector. A typical sample is presented in Eq. (20). The parameters of the neutrino mass matrix of Eq. (8) have been scanned in the region of 0 < |a 0 | < 2, 0 < |a 2 | < 50, and 0 < |b| < 15 in c = 1 units, while phases have been scanned in [−π, π]. We present a sample point satisfying the recent neutrino oscillation experimental data [51,52] as well as the fermion mass ratio and CKM mixing parameters at the GUT scale. 2

Neutrino phenomenology
In our numerical study, we have set H 24 / = 0.3 with H 24 2 × 10 16 GeV. Then, we have H 24 2 / 6 × 10 15 GeV, which is related to the right-handed neutrino mass in Eq. (10). Therefore, we take m Ni 10 15 GeV by choosing relevant values form i and f i . 3 Then, the neutrino Yukawa couplings are found to be at most 1.3 by inputting the experimental data. Thus, our setup is reasonably accepted in the neutrino phenomenology if m Ni 10 15 GeV is taken. We also discuss proton decay in this setup later.
Our lepton mass matrices of Eqs. (8) and (18) reproduce the experimental result of neutrino mass squared differences and the three mixing angles within 3 σ range [51,52]. The following results are constrained by the cosmological bound of the sum of three light neutrino masses m i , which is m i < 0.12 eV [55,56]. First, we show two sample parameter sets leading to successful results, which are completely consistent with the observed CKM and PMNS matrices. We obtain a prediction for the normal hierarchy (NH) of neutrino masses from the parameter set of Eq. (20) and the following parameters: 4 in which cv 2 u /m N 3 is a typical neutrino mass scale. If we take the right-handed neutrino mass m N 3 to be smaller than 10 14 GeV, c is less than 0.1. We obtain the three lepton mixing angles θ 12 , θ 23 , and θ 13 , the Dirac CP violating phase δ CP , neutrino masses, the effective mass of the neutrinoless double beta decay m ee , and the Majorana phases α 21 2 We have neglected the renormalization corrections for the neutrino masses and mixing parameters although the numerical analysis should be presented at GUT scale. A numerical estimation of the quantum corrections in Ref. [53] showed that the corrections are negligible as long as the neutrino mass scale is smaller than 200 meV and tan β ≤ 10. See also Refs. [33,54]. 3 We may consider f i 1/6 or the cancellation due to phases ofm i and f i . 4 Note that a 2 /c and b/c are larger than O(1), but they are parameters and the couplings a 2 Y (4) 2 and bY (2) themselves are smaller than 1.
Let us discuss our prediction of the leptonic CP phase, the effective mass of the neutrinoless double beta decay with the sum of neutrino masses. We show the allowed region in the plane of the sum of neutrino masses m i and δ CP in Fig. 1, where the cyan points and red points denote the predictions for the NH and IH cases, respectively. For NH, the predicted Dirac CP violating phase is allowed in the whole range of [−180 • , 180 • ] while m i is larger than 75 meV. In the case of IH, δ CP is predicted in the region of ±[50 • , 130 • ]. in particular, it is around ±90 • near the lower bound of our prediction of the sum of neutrino masses, 100 meV. It is noted that the lightest neutrino mass is larger than m 3 = 1.61 meV for the IH case. The future development of neutrino oscillation experiments or cosmological analysis for the neutrino mass is therefore expected to test our model.
We also show a prediction of m ee in the neutrinoless double beta decay in Fig. 2. The predicted region of the effective mass is about 10 < m ee < 30 meV for NH and 47 < m ee < 50 meV for IH. If the neutrinos are Majorana particles, the experiments for the neutrinoless double beta decay may test this model in the future.

Common modulus τ in quarks and leptons
The modulus τ is a key parameter in unifying quark and lepton flavors. We show the allowed region of the modulus τ in Fig. 3    The quark and lepton mass matrices should have the common modulus τ . Inputting τ obtained in the quark sector as well as other parameters of the quarks, we have obtained the allowed region of τ which satisfies the observed 1 σ range of the charged lepton mass ratios at the GUT scale and 3 σ range of the PMNS parameters. It is noted that there are no clear correlations between CKM and PMNS mixing parameters because of the large number of free parameters embedded in our model.
For the NH case, we plot the allowed region of τ in Fig. 4, where τ for the quark sector is also shown. Both regions almost overlap. However, for the IH case, the allowed region of τ is different from that in the case of quarks only, as seen in Fig. 5. Note that the allowed region is clearly reduced compared with the one for quarks only.
Thus, we obtain the restricted common τ in spite of the many free parameters of our model.

Proton decay
We give a brief comment on proton decay. An SU(5) GUT model includes the color-triplet Higgs multiplets, which can lead to proton decay [58][59][60][61][62]. The color-triplet Higgs multiplets induce the 9/15 PTEP 2020, 053B05 T. Kobayashi et al.  effective superpotential as well as QQQL/M H c including the quark doublet superfields Q, where M H c is the mass of the color-triplet Higgs multiplets, f u i and f d are Yukawa couplings of up-sector and down-sector quarks, and φ i are their phases. 5 The above operator leads to the proton decay p → K + ν through higgsino exchange. The factors f u 3 f d V * 1 with = 1, 2 are crucial to estimating the proton lifetime, because the couplings among u R , d R (s R ), right-handed stop, and right-handed stau are important in this PTEP 2020, 053B05 T. Kobayashi et al. process. Then, the proton lifetime is given as [62] τ P 4 × 10 35 whereÂ R is the renormalization factor and M S is the sfermion mass scale. The proton lifetime is longer than the observed lower bound of 10 33 yrs [57] for the case of M H c 2 × 10 16 GeV if M S ≥ 10 TeV and tan β ≤ 3. Since our numerical results for the quark/lepton mass matrices are changed only within a few percent in the range of tan β = 3-10, as stated in footnote 1, M S = 10 TeV is the minimal one that is consistent with our numerical results for quark/lepton flavor mixing to protect the proton decay. 6 We may have additional contributions to the effective potential in Eq. (28), because our cut-off scale is slightly higher than the GUT scale, H 24 / = 0.3. For example, the following term is allowed by the symmetries:

Summary and discussions
We have presented a flavor model with the S 3 modular invariance in the framework of SU(5) GUT and discussed the CKM and PMNS mixing parameters of both quark and lepton sectors. We have considered a six-dimensional compact space X 6 in addition to our four-dimensional space-time and supposed that the six-dimensional compact space has some constituent parts that include a twodimensional compact space X 2 . Then, the quarks and leptons have the same modular symmetry S 3 and the same value of τ in our setup. We note that our model does not require any gauge singlet scalars such as flavons. The difference between the mass eigenvalues of down-type quarks and charged leptons is realized by the 24-dimensional adjoint Higgs multiplet H 24 .
The setup of our model is reasonably accepted in the neutrino phenomenology if the right-handed neutrino masses are taken to be around 10 15 GeV. Their favored ranges are fairly limited; masses larger than 10 15 GeV are basically disfavored by the perturbativity of the neutrinoYukawa couplings, whereas lower masses require a more significant suppression in the dimension-five operators, or more severe cancellation between the operators and the bare mass terms.
We have analyzed our model numerically and found parameter regions which are consistent with both the observed CKM and PMNS mixing parameters for both NH and IH cases. The predicted Dirac CP violating phase is allowed in the whole range [−180 • , 180 • ] for the NH case. The sum of neutrino PTEP 2020, 053B05 T. Kobayashi et al. masses is larger than 75 meV. For IH, it is predicted in the region of ±[50 • , 130 • ]. In particular, it is around ±90 • near the lower bound of our prediction of the sum of neutrino masses, 100 meV. It is expected to test our model by astronomical observation for the neutrino mass constraint, as well as precise observation for the Dirac CP violating phase.
We have also predicted the effective mass in the neutrinoless double beta decay m ee , which is 10 < m ee < 30 [meV] for NH and 47 < m ee < 50 [meV] for IH. The development of experiments for the neutrinoless double beta decay is also expected to test our model. It is also noted that the proton lifetime is sufficiently long compared with the observed lower bound of 10 33 yrs.
Since our model has a large number of free parameters, distinct correlations between the CKM and PMNS mixing parameters are not found. However, the common value of the modulus τ is clearly obtained by imposing the experimental data for the CKM and PMNS mixing parameters as well as the quark and lepton masses. If we can build a flavor model with a small number of free parameters in a specific GUT framework, it is expected to find correlations between the CKM and PMNS matrices.