The Intrinsic Alignment of Galaxy Clusters and Impact of Projection Effects

Galaxy clusters, being the most massive objects in the Universe, exhibit the strongest alignment with the large-scale structure. However, mis-identification of members due to projection effects from the large scale structure can occur. We studied the impact of projection effects on the measurement of the intrinsic alignment of galaxy clusters, using galaxy cluster mock catalogs. Our findings showed that projection effects result in a decrease of the large scale intrinsic alignment signal of the cluster and produce a bump at $r_p\sim 1h^{-1}/Mpc$, most likely due to interlopers and missed member galaxies. This decrease in signal explains the observed similar alignment strength between bright central galaxies and clusters in the SDSS redMaPPer cluster catalog. The projection effect and cluster intrinsic alignment signal are coupled, with clusters having lower fractions of missing members or having higher fraction of interlopers exhibiting higher alignment signals in their projected shapes. We aim to use these findings to determine the impact of projection effects on galaxy cluster cosmology in future studies.


INTRODUCTION
Galaxy clusters are a major probe of dark energy (Weinberg et al. 2013).Their abundance and time evolution are sensitive to the growth of structure in the Universe, since they form from rare highest peaks of the initial density field.Cluster cosmology is a major science of many surveys, including Hyper Suprime-Cam (HSC) survey 1 , the Dark Energy Survey (DES) 2 , the Kilo Degree Survey (KiDS) 3 , the Rubin Observatory Legacy Survey of Space and Time (LSST) 4 , Euclid 5 , and the Nancy Grace Roman Telescope 6 .
Cluster shapes are triaxial, originating from the anisotropic matter field and accretion.As a result, cluster shapes are expected to align with the matter field, i.e. intrinsic alignment (IA) (see review papers by Joachimi et al. 2015;Troxel & Ishak 2015;Kirk et al. 2015;Kiessling et al. 2015).IA are distinct from the alignments of galaxy shapes that originate from gravitational lensing by foreground attractors.The IA signal has been observed for massive red galaxies (Okumura et al. 2009;Singh et al. 2015), but no clear detection has been claimed for blue galaxies (Mandelbaum et al. 2011;Yao et al. 2020).The alignment of galaxy clusters have also been detected (Smargon et al. 2012).van Uitert & Joachimi (2017) studied the cluster shape -density correlation using redMaPPer clusters from Sloan Digital Sky Survey-Data Release 8 (SDSS DR8), finding a higher IA amplitude of galaxy clusters than luminous red galaxies (LRGs).As clusters are the most massive bound structures, studies on cluster shapes offer the unique opportunity to yield insight into dark mat-ter halo shapes (Evans & Bridle 2009;Oguri et al. 2010;Shin et al. 2018;Gonzalez et al. 2022).
However, the IA amplitude of galaxy clusters are found to be lower than predictions from numerical N -body simulations based on Λ cold dark matter (ΛCDM) cosmology.Smargon et al. (2012) discussed various systematic observational uncertainties that may have caused this discrepancy, including photometric redshift error, cluster centroiding error, uncertainty in cluster shape estimation using a limited subsample of galaxy members, and inclusion of spherical clusters.However, one of the major systematics for optically identified clusters, the so-called "projection effect", has not been properly discussed for measurement of IA for galaxy clusters.
Projection effects refer to the fact that interloper galaxies along the line-of-sight (LOS) are mistakenly identified as members of galaxy clusters (van Haarlem et al. 1997;Cohn et al. 2007).This is a major systematics for optical clusters whose mass proxy is a number of member galaxies (called richness).It can also boost cluster lensing and clustering signals on large scales, since clusters with a filamentary structure aligned with the LOS direction are preferentially identified by optical cluster finders, which typically detect clusters using red galaxy overdensities in photometric catalogs (Osato et al. 2018;Sunayama et al. 2020;Sunayama 2022).To obtain unbiased cosmological constraints using galaxy clusters, the projection effect has to be corrected or modelled accurately (To et al. 2021;Park et al. 2023;Costanzi et al. 2019).
In this work, we will study the impact of projection effects on measurements of cluster IA with the aim to understand the measured IA of the most massive objects.We also search for new perspectives on projection effects and possible ways to mitigate the impacts on cluster observables.We found that the projection effects can largely explain the lower signal of observed cluster IA compared to that of simulated dark matter halos.
The structure of the paper is organized as follows.In Section 2, we introduce our methodology for measuring the correlation function and modeling the signals.In Section 3, we introduce the observational data and mock simulation used in this paper.The results on measured IA in observation and mocks -including the impact of projection effects -are presented in Section 4 and Section 5.In Section 6, we summarize our results.

METHODOLOGY -LINEAR ALIGNMENT MODEL
In this section we briefly describe the leading theory of IA, i.e. the linear alignment model (Catelan et al. 2001;Hirata & Seljak 2004), and then define the model to use for the comparison with the IA measurements of the redMaPPer clusters.
The linear alignment model predicts that the intrinsic shape of dark matter halos, and galaxy clusters in this paper, is determined by the gravitational tidal field at the time of formation of the halo or galaxy cluster.That is, the intrinsic "shear", which characterizes the shape of galaxy cluster, is given as where Φp is the primordial gravitational field and C1 is a constant.Here we take the (x, y) coordinates to be on the 2D plane perpendicular to the LOS direction.Throughout this paper, we employ a distant observer approximation, and in the above equation we take the LOS direction to be along the z-axis direction.
In this paper, we consider the cross-correlation between the IA shear of galaxy clusters and the galaxy density field.For the latter, we will use the spectroscopic sample of galaxies in the measurement.We can define the coordinate-independent cross-correlation function as with γ+ being defined as Here ℜ denotes a notation to take the real part of the cluster shear, r ≡ x − x ′ , and ϕr is the angle measured from the first coordinate axis to the projected separation vector rp on the sky plane perpendicular to the LOS direction.Since we can measure only the projected shape of each cluster and the positions of clusters and galaxies are modulated by redshift-space distortion (RSD) (Kaiser 1987), the 3D crosscorrelation function is generally given as a function of the 3D separation vector r = (r ∥ , rp), where r ∥ is the component parallel to the LOS direction and rp is the 2D separation vector perpendicular to the LOS.Following the formulation in Kurita & Takada (2022) (also see Kurita & Takada 2023) and as derived in Appendix A, it is convenient to use the multipole moments of the crosscorrelation function using the associated Legendre polynomials with m = 2, denoted as L 2 ℓ : where µr is the cosine angle between r and the line-of-sight direction and ξ g+ is the ℓ-th order multipole moment.Note that the multipole index ℓ starts from 2 (ℓ = 2, 3, . . . ) and , and so forth.The multipole moments ξ (ℓ) g+ can also be expressed in terms of the cross power spectrum using where gE (k) is the corresponding multipole moments of the IA cross power spectrum PgE(k).
Assuming the linear alignment model (Eq. 1) and the linear Kaiser RSD, the cross-power spectrum is given as where bK is the linear shape bias parameter (Schmidt et al. 2014;Kurita et al. 2021;Akitsu et al. 2021), bg is the linear bias parameter of the density sample, β ≡ f (z)/bg, f is the logarithmic of linear growth rate, and µ k is the cosine angle between k and the LOS direction.In In ΛCDM cosmology/Universe, for a wide range of redshifts, f (z) ∼ Ωm(z) 0.55 .In the above equation, we used the nonlinear matter power spectrum, P NL mm , including the effect of nonlinear structure formation, which is the so-called nonlinear alignment model (NLA) (Bridle & King 2007).Also note that we assumed the linear Kaiser RSD factor (1+βµ 2 ), but we will below consider the projected correlation function to minimize the RSD contribution.The shape bias parameter bK is related to the IA amplitude parameter AIA that is often used in the literature as where D(z) is the linear growth factor and we take C1ρcrit = 0.0134 following the convention (Joachimi et al. 2011).
Throughout this paper we focus on AIA to discuss the IA amplitude of redMaPPer clusters.Using Eq. ( 6), the multipole moments of the crosscorrelation function can be found, as derived in Appendix A, as and zero otherwise.The multipole moments of the matter two-point correlation function is defined similarly to Eq. ( 5) using P NL mm .When there is no RSD effect, only the lowest order moment (ℓ = 2) carries all the IA cross-correlation information, which can be realized by the use of the associated Legendre polynomials (Kurita & Takada 2022).
In this paper we consider the projected IA cross-correlation function defined as We adopt Πmax = 100 h −1 Mpc as our fiducial choice.
To estimate the linear bias parameter of the density sample, bg, we model the galaxy clustering signal using where fcorr(rp, z) is Kaiser correction factor given by (van den Bosch et al. 2013), ξ lin gg (rp, r ∥ , z) and ξ lin gg (r ≡ r 2 p + r 2 ∥ , z) here are the linear two-point galaxy correlation function in redshift space and real space, respectively, where ξ lin gg (r, z) = b 2 g ξ lin mm (r, z) and the linear galaxy correlation function in redshift space is ∥ is the real space separation, µ = r ∥ /s, and P 2l (x) is the lth Legendre polynomial.ξ0, ξ2, and ξ4 are given by where To compute the model predictions of the projected IA cross correlation (Eq.9), We assume the ΛCDM cosmology with ΩDM = 0.236, Ω b = 0.046, ΩΛ = 0.718, ns = 0.9646, σ8 = 0.817, h = 0.7 (WMAP9 cosmology, Hinshaw et al. 2013).For the nonlinear matter power spectrum, we employ Halofit7 for the ΛCDM model (Takahashi et al. 2012).We vary the linear bias parameters bg and bK (equivalently AIA) and estimate the best-fit values by comparing the model predictions with the measurements for the ΛCDM model.

BOSS DR12 LOWZ Galaxies
We use SDSS-III BOSS DR12 LOWZ galaxies with spectroscopic redshifts in the range of 0.1 ≤ z ≤ 0.33 as a biased tracer of the matter field.This is due to their significant overlap with redMaPPer clusters.The LOWZ sample consists of luminous red galaxies at z < 0.4, selected from the SDSS DR8 imaging data and observed spectroscopically in the BOSS survey.The sample is roughly volume limited in the redshift range 0.16 < z < 0.36 and has a mean number density of ∼ 3 × 10 −4 h 3 Mpc −3 .We utilize the largescale structure catalogues8 for BOSS (Anderson et al. 2012;Rykoff et al. 2016).Table 1 provides an overview of the properties of the density sample.The final density sample contains 239, 904 galaxies.We apply a weighting scheme to sample, using w = wFKP × wtot, where wtot = wsys × (wcp + wnoz − 1) for density data and w = wFKP for density random.

redMaPPer Cluster
We use galaxy clusters identified with redMaPPer algorithm (Rozo & Rykoff 2014;Rykoff et al. 2014) on SDSS DR8 photometry data (Aihara et al. 2011), over an area of about 10, 000 deg 2 .The redMaPPer algorithm finds optical clusters via identifying overdensity of red sequence galaxies.We use the publicly available version, v6.3.For each cluster, the algorithm provides potential brightest central galaxy (BCG) candidates, cluster richness λ which is the sum-up of pmem over all candidates members, photometric redshift z λ , and spectroscopic redshift zspec if available.pmem gives the membership probability of each galaxy belonging to a cluster in the redMaPPer catalog.We choose the galaxies with the highest pcen as BCGs.In this paper we use galaxy clusters that have available zspec, and select clusters with 20 ≤ λ ≤ 200 and 0.1 ≤ zspec ≤ 0.33.We further divide the sample into sub-samples with 20 ≤ λ < 30, 30 ≤ λ < 40, 40 ≤ λ < 55, 55 ≤ λ < 200, in order to study the richness dependence of AIA.The statistical properties of the redMaPPer clusters are summarized in Table 1.
We use the public random catalog of redMaPPer cluster, which includes cluster positions, redshift, richness λ and weight.The weighted z and λ distributions are the same as in the data.We apply the same z and λ cuts in the random catalog for each cluster sample.

Cluster shape characterization -BCG versus member galaxy distribution
We quantify the shape of each redMaPPer cluster by two ways: the shape of BCGs, and the distribution of the member galaxies relative to BCGs.The BCG shape can be obtained by cross matching with SDSS DR8 shear catalog (Reyes et al. 2012).4, 325 clusters have BCG shape measurement, out of 6, 345 selected clusters with 20 ≤ λ ≤ 200.Alternatively, we follow the method in (van Uitert & Joachimi 2017) to quantify the cluster shape using member galaxy positions with respective to the BCG.Using all cluster members with pmem > 0.2, the second moments of the projected shape are given as where i, j ∈ 1, 2. The ellipcitity components are then defined as The "shear" of cluster shape is estimated as γ1,2 = ϵ1,2/(2R), where R ≡ 1 − ⟨ϵ 2 i ⟩ is the shear responsivity (Bernstein & Jarvis 2002).

Correlation Function Estimator
For the BOSS LOWZ sample and the specp-z matched redMaPPer cluster, we measure the auto-correlation function of LOWZ galaxies, ξgg(r), and the projected IA crosscorrelation function between the LOWZ galaxy and the redMaPPer cluster shapes, ξg+(r).
We use a generalized Landy-Szalay estimator (Landy & Szalay 1993) for estimating the correlation functions: where S+ is the shape field for the cluster sample, D is the density field for the LOWZ galaxy sample, and RS and RD are random points corresponding to shape sample and density sample, respectively.S+ is the +-component of cluster shear with respect to the vector r ≡ x − x ′ connecting the cluster position and the LOWZ galaxy or the density random point (see Eq. 3).
For the IA cross-correlation, we consider the projected correlation function: We compare the measured wg+ with the theory prediction (Eq.9).

redMaPPer Cluster Mock
To study the impact of projection effects on IA of galaxy clusters, we use the cluster mock catalog constructed in Sunayama & More (2019) (see also Sunayama et al. 2020 andSunayama 2022).Here we briefly summarize the mock construction procedures, and refer the readers to Sunayama & More (2019) for more detailed information.
To construct the cluster mock, N -body simulations from Nishimichi et al. (2019) are used, which were performed with 2048 3 particles in a comoving cubic box with side length of 1h −1 Gpc.The simulations adopt the P lanck Cosmology (Planck Collaboration et al. 2016).The particle mass is 1.02 × 10 10 h −1 M⊙.Halos are identified using Rockstar halo finder (Behroozi et al. 2013), and M200m is adopted for halo mass, which is the total mass within R200m.R200m is the radius within which the mean density is 200 times the mean mass density ρm.For our purpose, we use the simulation snapshot and halo catalogs at z = 0.25, which is the mean redshift of the redMaPPer clusters.We have 19 realizations of N -body simulation and cluster mock.
Mock galaxies are populated into halos with mass M200m > 10 12 h −1 M⊙ using halo occupation distribution (HOD) prescription (Zheng et al. 2005).The HOD parameters are chosen to match with the abundance and lensing measurements of the redMaPPer clusters.Instead of distributing the satellite galaxies using Navarro-Frenk-White profile (Navarro et al. 1997), the satellites are populated using the positions of randomly selected member particles in each halo.As a result, the satellites distribution within the halo traces the non-spherical halo shape, which is also used as one of the validation tests in Appendix B.
The photometric redshift uncertainty, which is the main source of the projection effects, is modeled by assuming a specific projection length, dproj.In this work, we use the mock with dproj = 60h −1 Mpc.The cluster finder which mimics the redMaPPer algorithm (Rozo & Rykoff 2014;Rykoff et al. 2014) is then run on the red-sequence mock galaxies, producing the mock cluster catalog that includes the true richness λtrue, the observed richness λ obs , and the membership probability pmem.The galaxy in the most massive halo in each identified cluster is considered as the central galaxy of the cluster.The optical radial cut that scales with the richness, Rc(λ) = R0(100/λ) β , is applied the same way as in observation when running the redMaPPer algorithm in mock, where R0 = 1.0h −1 Mpc and β = 0.2.
Similar as in observation, we divide the mock cluster sample into subsamples with various richness bins, using both λ obs and λtrue.We use halos with M200m > 10 12 h −1 M⊙ as density tracers, δ h , of the matter field, where The properties of the selected cluster samples are shown in Table 1.The cluster bias increases with the richness, which is consistent with the fact that halo/cluster mass increases with richness.Sunayama et al. (2020) presented the halo mass distribution of the mock clusters in different richness bins (divided by both λ obs and λtrue), showing that mass distributions for the "mock observe" sample is more extended than the "mock true" sample because of the projection effects, also the peak mass shifts towards higher masses from finite aperture effects in higher richness bins.This explains the higher cluster bias for "mock observe" sample shown in Table 1.

Cluster shape characterization
For each galaxy cluster in the mock, we calculate the observed cluster shape γ obs using the redMaPPer member galaxies with pmem > 0.2, using Eq. ( 17).Unlike observation, mock cluster catalogs provide the true positions of the satellite galaxies as well as the dark matter particles.So, we can calculate the intrinsic cluster shear γtrue using satellite galaxy positions and γDM using DM particles distributions (see Appendix B for details of the calculation).The IA signal measured from γtrue agrees with that from γDM very well (see Appendix B).So in the following, we take mock clusters selected using λtrue and shape calculated using γtrue as the "mock true" sample, while mock cluster selected using λ obs and γ obs as the "mock observe" sample.
We use TreeCorr (Jarvis et al. 2004) to compute the correlation functions.We measured the signal as a function of transverse comoving separation in 25 logarithmic bins between 0.1 and 200h −1 Mpc.We take Πmax = 100h −1 Mpc and 20 linear bins for r ∥ ∈ [−100, 100]h −1 Mpc.To estimate the covariance matrix, we divide the redMaPPer Cluster sample into 50 jackknife regions of approximately equal area on the sky, and compute the cross-correlation function by excluding one region each time (Norberg et al. 2009).For the mock cluster sample, we divide the simulation box into 64 sub-boxes of equal volume for jackknife covariance matrix estimation.
We restricted the analysis to mildly non-linear scales of rp > 6h −1 Mpc.The size of the jackknife patch is 14 deg., which roughly corresponds to 70h −1 Mpc at z = 0.1.So we take 70h −1 Mpc as the maximum scale in the fitting.

IA of redMaPPer Clusters in SDSS
The measured cross correlation functions of the galaxy density field and the cluster shape field are shown in Figure 1.
Here we used the cluster shapes measured using positions of the member galaxies relative to the BCG in each cluster.We obtain a clear detection of IA signal in all richness bins, meaning that cluster shapes have correlations with the surrounding large-scale structures.
The IA amplitude, AIA, is obtained by fitting NLA model to the measurement, as introduced in Section 2. However, AIA is degenerate with bias parameter bg of the galaxy density sample.We obtain bg = 1.73 ± 0.05 by measuring and fitting the projected clustering signal of LOWZ galaxies to the model (Eq.10), as shown in Figure 2. We have good fits of the model prediction, with reduced χ 2 value of 1.02.Our result for the LOWZ galaxy bias is consistent with the previous measurement, bg=1.77± 0.04, in Singh et al. (2015).We ascribe the slight difference to the different redshift range, where they used 0.16 < z < 0.36 compared to our range, 0.10 ≤ z ≤ 0.33.
The IA amplitude of each subsample can be found in Table 1.The NLA model gives a good fit to the measured wg+ in the fitting range of 6 < rp < 70h −1 Mpc for each cluster sample.However, at small scales, the model predictions are much lower than the measured signal.The IA amplitude, AIA, does not show a clear dependence on cluster richness.This contradicts with the results found from the shapes of halos in simulations (Kurita et al. 2021); they found that AIA in- creases with halo mass.We found this is mainly caused by the projection effects, as we will discuss in Section 4.3 in detail.

Tests for systematics
In Figure 3 we study potential systematic effects in our IA measurements.The upper panel shows the measured correlation function between the cross-component of the cluster shape, γ×, and the galaxy density field, wg×, for the sample with 20 ≤ λ < 200.This cross correlation should be vanishing due to parity symmetry if the measurements is not affected by an unknown systematic effect.We also show the IA cross-correlation function, wg+, measured by integrating the original 3D IA correlation function only over the large line-of-sight separation, 150 < |Π| < 500h −1 Mpc.This crosscorrelation is expected to have a very small signal, if the redshift of clusters is accurate or if there is no significant contamination of fake clusters due to the projection effect.The measured wg+ for the large |Π| separation shows a very small signal.Hence we conclude that our measurements are not affected by the ×-component or the fake clusters.
There are other potential systematic effects that affect our IA measurements.These include photometric redshift errors, errors in cluster shape estimation arising due to a limited number of member galaxies, miscentering effect, contamination of merging clusters, and incompleteness of cluster sample or selection function.van Uitert & Joachimi (2017) presented the tests of above systematic effects for the redMaPPer cluster sample, and showed that the most significant systematic effect arises from photo-z errors for the cluster sample.Since we use only the clusters that have spectroscopic redshifts, we conclude that our IA measurements are not affected by the photo-z errors.
However, we below show that the projection effect due to large-scale structure surrounding the redMaPPer clusters causes a systematic contamination to the IA measurements.

IA of Clusters in Mock -Impact of Projection Effect
In Figure 4 we study the impact of the projection effect on the IA correlation functions using the mock catalog of redMaP-Per clusters.To do this, we compare the IA correlation functions for clusters using the true or "observed" richness (λtrue or λ obs ) and/or the true or "observed" shape estimates (γtrue or γ obs ), where the observed quantities are affected by the projection effect.The figure shows that the IA correlation function using the observed quantities (λ obs and γ obs ) displays about factor of 2 smaller amplitudes than that for non-contaminated clusters (λtrue and γtrue).The solid orange curve shows the result when using the clusters for λ obs and γtrue, which show almost similar amplitudes to that for the non-contaminated clusters (λtrue and γtrue).The comparison tells that the smaller amplitude for the case of (λ obs , γ obs ) is caused mainly by the projection effect on the shape measurement (γ obs against γtrue).The AIA values estimated from w h+ for the different samples are given in Table 1. Figure 4 only shows the result for the cluster sample with 20 ≤ λ < 200, the measurement and fitting results for other richness bins are shown in Appendix C.
When comparing the solid and dashed lines in Figure 4, we notice the existence of a bump in w h+ around rp ∼ 1h −1 Mpc for the case with projection effects.Here 1 h −1 Mpc roughly corresponds to the aperture size used in the redMaPPer cluster finder (Rykoff et al. 2014).We will show later that this specific imprint of projection effects is likely caused by the non-member interlopers, which are however identified as cluster members by the redMaPPer method, and the real member galaxies that are missed by the cluster finder.

ftrue and fmiss
As we have found, the projection effect impacts the shape estimation of clusters.There are two effects: one is caused by including interlopers (non-member galaxies) in the cluster members, and the other is caused by missing real member galaxies, when estimating the cluster shape.To study how these two effects cause a contamination to the IA correlation function, we define the following quantities: , which is the true member fraction of identified members in each cluster.This quantity is the same as that used in Sunayama et al. ( 2020  .The IA correlation functions measured from the mock cluster catalogs.Shown is the ratio of the IA correlation function using the observed shape (γ obs ) to that of the true shape (γtrue), for a subsample of the mock clusters with 20 ≤ λ obs < 200.Left panel: the ratio w h+ (γ obs )/w h+ (γtrue) for subsamples with ftrue ≤ 0.75 and ftrue > 0.75, respectively, where ftrue is the fraction of true members among the cluster members identified by the redMaPPer finder in each cluster.Right: the ratio for subsamples divided by f miss ≤ 0.1 and f miss > 0.1, where f miss is the fraction of true members missed by the finder in each cluster.
• fmiss = 1.− ntrue,mem(< Rc)/λtrue, which is the fraction of true members missed in the membership identification in each cluster.
Here p true mem,i is the membership probability of the i-th true member galaxy identified by the redMaPPer finder, Rc is the cluster radius used in the redMaPPer finder, and ntrue,mem is the number of true member galaxies among all redMaPPer member galaxies.Note 0 < ftrue ≤ 1 by definition, and ftrue = 1 means that the redMaPPer finder-identified member galaxies are true member galaxies that belong to the cluster, and no interlopers contaminate the true membership (however, all the true members are not necessarily identified).On the other hand, a low ftrue indicates a higher contamination fraction of interlopers.fmiss informs how many true member galaxies are not identified as member galaxies by the cluster finder.
In Figure 5, we show the ratio of w h+ (γ obs ) versus w h+ (γtrue) for samples with low ftrue (fmiss) and high ftrue (fmiss) separately.If the ratio between w h+ (γ obs ) and w h+ (γtrue) is close to 1 for a sub-sample, it means the measured cluster shape/IA are less affected by the projection effects.On contrary, if the ratio deviates from unity more, it means the projection effect is making the measured shape/IA deviates from the underlying true signals.Figure 5 shows that the impact on large-scale IA signal of projection effects is weaker for clusters with high ftrue and low fmiss, compared to the clusters with low ftrue and high fmiss.The amplitude of the bump at rp ∼ 1h −1 Mpc is significantly decreased for samples with higher ftrue and higher fmiss.As shown in Figure 4, the bump only appears when the projection effect is included in the mock, i.e. for w h+ (γ obs ).

Coupling between cluster IA and projection effects
Cluster IA and projection effects are coupled with each other.In Figure 6, we compare the IA signal of low ftrue (fmiss) and high ftrue (fmiss) sub-samples.The large-scale IA amplitude is higher when fmiss or ftrue is higher, for both γ obs and γtrue.The coupling between cluster IA and projection effects are illustrated by the cartoons shown in Figure 7.
For clusters with their major axis (orientation) perpendicular to the LOS direction, the measured IA is higher, since their projected shapes appear more elliptical and we measured the cross correlation between the projected shapes and the density field.These clusters also tend to have LSS structures, such as filaments, that are perpendicular to the LOS direction.The missed member galaxy fraction fmiss is higher, since the projected member galaxies distribution is more dispersed; and the contamination from interlopers is lower, since there are less galaxies outside the cluster along the LOS, thus ftrue is higher.In contrary, for clusters with their major axis along the LOS direction: the measured IA is lower, and it is more likely to have LSS structures along the LOS; they are less likely to miss galaxy members (lower fmiss) since they are concentrated in the inner region; the contamination from interlopers along the LOS is higher (lower ftrue).In both cases, the outer region of the cluster is affected more, since the member number density decreases with the distance from the cluster center.This likely explains the existence of the bump at rp ∼ 1h −1 Mpc, which is also the typical cluster boundary.The above picture is supported by Figure D.1 in Appendix D, where we show that clusters with lower ftrue and lower fmiss tend to have their major axis parallel with the LOS direction.In summary, the above picture explains the coupling between cluster IA and fmiss, ftrue.

Dependence on Cluster Richness
The impact of the projection effects on cluster IA is independent of the cluster richness, as shown in Figure 8.The ratios of w h+ (obs) with projection effects versus w h+ (true) without projection effects at scales of 6 < rp < 70h −1 Mpc is roughly constant and doesn't depend on the richness of the clusters.
In Figure 9, we plot the measured AIA versus cluster mean .Cartoons illustrating the couplings between galaxy cluster IA and the projection effects, for clusters with their major axis perpendicular to the LOS direction (upper) and along the LOS direction (lower).For clusters with their major axis perpendicular to the LOS, the measured IA signal using projected cluster shape is stronger, the contamination fraction from interlopers is lower (i.e. higher ftrue), and the missed galaxy member fraction is higher (i.e. higher f miss ); for clusters with their major axis parallel with the LOS direction, the measured IA signal is lower, the contamination from interlopers is more severe (i.e.lower ftrue), and the missed member fraction is lower (i.e.lower f miss ).
richness for redMaPPer clusters in observation, and clusters in the mock with w h+ (true) (filled squares) and w h+ (obs) (open squares).The AIA from observation agree with results using w h+ (obs) from mock pretty well, indicating that our mock construction and inclusion of the projection effects is quite reasonable.A weak increase of AIA with respect to cluster richness can be seen for clusters free of projection effects.However, such dependence can not be seen once projection effects are included.we further derived the AIA -halo mass w h+ (obs) here is calculated using λ obs and γ obs , and w h+ (true) here is calculated using λtrue and γtrue.The typical 1σ scatter of the ratio among realizations is shown in the upper left corner of the plot.
relation for galaxy clusters and compared it with the prediction from N-body simulation, which is shown in Appendix E.

DISCUSSION
5.1 Cluster IA using BCG shape versus member galaxy positions The IA of BCGs are shown in Figure 10.BCGs show a similar IA amplitude as the clusters that they lie in, indicating the good alignment of BCG orientations with respect to the member galaxies distribution of clusters.If we assume that the member galaxy distributions trace well the dark matter halo shapes, then the results in Figure 10  good alignment between BCG and dark matter halos.However, previous studies of Okumura et al. (2009) showed that central LRGs are not perfectly aligned with the dark matter halos, with a misalignment angle of ∼ 35deg.Recent work by Xu et al. (2023) further showed that misalignment angles are likely to be mass dependent.Nevertheless, the good alignment shown in Figure 10 seems to be in contradiction with expectations from previous studies.We found this is mainly caused by the projection effects on the observed IA of redMaPPer clusters, which decreases the measured cluster IA signal using member galaxy positions.If the impact on cluster IA is uncorrected, the inferred misalignment angle between BCGs and clusters are smaller than they should have been.

SUMMARY
We measured the IA of galaxy clusters by cross correlating the shapes of redMaPPer clusters with the LOWZ galaxie at 0.1 ≤ z ≤ 0.33.We detected a positive IA signal, indicating that clusters point towards the density field.We also divide the samples into four richness samples, enabling us to study the dependence on cluster richness.
We investigated the impact of projection effects on the measured IA of clusters using mock cluster catalogues.The inclusion of the projection effects decrease the measured IA signal by a factor of ∼ 2.5, which is almost independent of the cluster richness.The projection effects predominantly impact the measured cluster shapes, including interlopers that are not members of the clusters and missing true members.Consequently, projection effects lead to a smaller observed misalignment angle between BCG and clusters than the underlying one.
In our study, we discovered a correlation between cluster IA and projection effects.Clusters oriented parallel to the LOS are less likely to have undetected members and more likely to have interlopers, and their projected shapes are less elliptical and exhibit weaker alignment signals.This can be attributed to their likely location within a filamentary structure along the LOS direction.Conversely, clusters oriented perpendicular to the LOS direction display a more elliptical projected shape and a stronger IA signal, they also tend to have a higher fraction of missed cluster members and a lower fraction of interlopers.
The measured IA strength, AIA, in the cluster mock with projection effects agrees well with observation.The observed AIA in both real data and mock observe clusters barely depends on cluster richness, while a weak dependence on richness does exist if we can correctly identify the true cluster members without any contamination.
Our work showed that IA measurements of galaxy clusters can be improved by identifying interlopers and by including the true member galaxies in the outer region, leading to a much higher signal-to-noise detection of cluster IA.High signal-to-noise detection of cluster IA is crucial for applying IA as a novel cosmological probe.With more and more incoming spectroscopic data, we expect to suppress (or reduce) the impact of projection effects significantly.We will leave the efforts on removing projection effects for galaxy clusters to the future work.JP20H05861, JP21H01081, and JP22K03634, and by Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.The authors thank the Yukawa Institute for Theoretical Physics at Kyoto University.Discussions during the YITP workshop YITP-W-22-16 on "New Frontiers in Cosmology with the Intrinsic Alignments of Galaxies" were useful to complete this work.J. Shi and T. Kurita also thank Lorentz Center and the organizers of the hol-IA workshop: a holistic approach to galaxy intrinsic alignments held from 13 to 17 March 2023.
where xni, xnj(i, j = 1, 2) are the positions of nth satellite galaxy with respect to the centre of cluster, and Ng is the total number of satellite galaxies used for the calculation; • redMaPPer identified member galaxy distribution (RM Mem), Iij is calculated using Eq.B2, except that we use member galaxies identified by the redMaPPer cluster finder; • redMaPPer identified redMaPPer members that truly belong to the clusters (RM True Mem), also using Eq.B2.
Figure B.1 showed that, w h+ measured using γ (DM) shows the strongest signal, and satellite distributions trace the DM distribution rather well, showing only a slightly weaker IA signal, as shown by the blue line.This is expected since the satellite galaxies are populated following the dark matter distribution.IA measured using redMaPPer identified member galaxy distribution γ (RM Mem) show the lowest signal, with a bump at rp ∼ 0.8h −1 Mpc.If interlopers are removed for the shape calculation, the bump disappears and the IA signal increases a little bit, shown by the green line.However, the IA signal is still much lower than the one measured using DM and satellite galaxy distribution, indicating that another factor, i.e. the satellites that are missed by redMaPPer algorithm, is also responsible for decreasing the IA signal.

Figure 1 .
Figure1.The LOWZ galaxy-cluster shape correlation, w g+ , of clusters in the redshift range of 0.1 ≤ z ≤ 0.33.Different panels show the result for various richness bins.The dots are measurement from data, and the colored solid lines are the fitting using NLA in the range of 6h −1 Mpc < rp < 70h −1 Mpc.The gray solid line shows the fitting result for the cluster sample with 20 ≤ λ < 200, just to guide the eyes.

Figure 2 .
Figure2.The galaxy-galaxy correlation, wgg, of the LOWZ sample in the redshift range of 0.1 ≤ z ≤ 0.33.The blue dots are measurement from data, and the black solid line is the fitting using linear model with non-linear matter power spectrum in the range of 6h −1 Mpc < rp < 70h −1 Mpc.The gray solid line shows the SDSS fibre collision scale at z = 0.33.

Figure 3 .λ
Figure3.Tests of systematic effects in the IA measurement of galaxy clusters with 20 ≤ λ < 200.Upper panel: cross correlation between LOWZ galaxies and redMaPPer cluster shape γ × , the signal is consistent with 0. Lower: measurement of w g+ within 150 < |Π| < 500h −1 Mpc, the signal is also consistent with null signal.

Figure 4 .
Figure4.The IA correlation functions measured for galaxy clusters in the mock simulations, selected using 20 ≤ λ obs < 200 (blue) and 20 ≤ λtrue < 200 (orange), respectively.The solid lines show the measurements using γtrue, i.e., satellites distributions within dark matter halos, while the dashed line shows the measurement using γ obs , i.e., member galaxies identified by redMaPPer algorithm.The lines here show the median values among 19 realizations, and the error bars are the 1σ dispersion.

Figure 6 .
Figure 6.Similar to the previous figure, but we show the IA correlation functions, instead of the ratio.The solid lines are the IA correlation functions measured using γtrue, and the dotted lines are those measured using γ obs

Figure 8 .
Figure8.The ratio of the observed IA signal versus the true IA signal of galaxy clusters in various richness bins in the mock.w h+ (obs) here is calculated using λ obs and γ obs , and w h+ (true) here is calculated using λtrue and γtrue.The typical 1σ scatter of the ratio among realizations is shown in the upper left corner of the plot.

Figure 9 .
Figure 9.The A IA versus richness λ relation for clusters in observation and mock catalog.The blue open circles show the results from fitting the observed w g+ (rp) of redMaPPer clusters, the orange open squares are results from mock observe samples using λ obs and γ obs .The orange filled squares are results from mock true samples using λtrue and γtrue.

FigureFigure D. 1 .
Figure C.1 shows the IA of clusters in the mock in various richness bins and the corresponding NLA fitting results.The IA signal of mock true samples are obtained by selecting clusters using λtrue and measuring shapes γtrue using satellites

Figure E. 1 .
Figure E.1.IA as a function of halo mass and redshift.The lines with error bars are calculated from dark matter halos in N-body simulation as done in Kurita et al. (2021).The dots are results using SDSS DR8 redMaPPer catalog.

Table 1 .
Summary of the sample properties.For the samples in mock observe catalogue, we select clusters based on observed richness λ obs with γ obs ; while for mock true, we use true richness λtrue with γtrue.Ng and N clus are number of galaxies and clusters in the samples separately.⟨λ⟩ is the mean richness parameter of the sample.ϵ 2 RMS = ⟨ϵ 2 i ⟩ is the RMS ellipticity.bg and b clus are bias of the samples.The error bars of b clus in the mock indicate the 1σ scatter among the 19 mock realizations.A IA is the IA strength parameter obtained from fitting with NLA model.The error bars of A IA indicate the 1σ scatter among the 19 mock realizations.
BCG, A IA = 11.47 ± 3.87 Mem Pos, A IA = 16.25 ± 4.10Figure10.The LOWZ galaxy-BCG shape (blue) and LOWZ galaxy-cluster shape (orange) correlation, w g+ , of clusters with available BCG shapes in the redshift range of 0.1 ≤ z ≤ 0.33.The blue and orange dots measurement using BCG shapes and member galaxy positions, γ obs , separately.The solid lines are the fitting results using NLA in the range of 6h −1 Mpc < rp < 70h −1 Mpc.