Cross-linguistic authorship attribution and gender profiling. Machine translation as a method for bridging the language gap

Abstract

This study explores the feasibility of cross-linguistic authorship attribution and the author’s gender identification using Machine Translation (MT). Computational stylistics experiments were conducted on a Greek blog corpus translated into English using Google’s Neural MT. A Random Forest algorithm was employed for authorship and gender profiling, using different feature groups [Author’s Multilevel N-gram Profiles, quantitative linguistics (QL), and cross-lingual word embeddings (CLWE)] in both original and translated texts. Results indicate that MT is a viable method for converting a multilingual corpus into one language for authorship attribution and gender profiling research, with considerable accuracy when training and testing datasets use identical language. In the pure cross-linguistic scenario, higher accuracies than the baselines were obtained using CLWE and QL features.

1. Introduction

Computational Stylistics has grown rapidly in recent years, with authorship attribution and author profiling being among its most common applications. By using vectors of measurements that capture stylistic variation and training Machine Learning algorithms on large corpora, we can reliably predict authors’ identity and traits such as gender, age, and personality. Supervised classification methods exhibit excellent performance in closed-class problems (e.g. David et al. 2016; Posadas-Durán et al. 2017). Advanced document representations, such as word embeddings, have enhanced stylometric approaches with semantic information and higher-level language structure patterns (e.g. Franco-Salvador et al. 2017). These features, paired with state-of-the-art Deep Learning approaches (e.g. Veenhoven et al. 2018), have made Computational Stylistics a promising and evolving research area in Natural Language Processing (NLP).

However, challenges remain, such as the need for a decent amount of textual data from each author to develop reliable quantitative profiles of their idiolect (Luyckx and Daelemans 2008; Zheng and Jin 2022), the largely unsolved open-class authorship problems and the performance reduction caused by mismatches between training and testing datasets in cross-topic and cross-genre authorship attribution.

The toughest experiment is cross-linguistic authorship attribution and author profiling, where texts are written in a non-native language of the author. This problem challenges the central idea of stylometry—that each author has a unique way of using language that is uniquely related to their identity (the “stylome” hypothesis). In this article, we assess computational stylistic approaches in texts translated into another language using Machine Translation (MT), introducing additional bias as the stylometric signal must survive both language conversion and natural language generation.

The remainder of the article is organized as follows: In Section 2, an overview of the cross-linguistic research on author profiling and authorship attribution will be provided. The fundamental methodological decisions we made for this study—including the corpus we created, the features we employed, and the algorithm we tested—are detailed in Section 3. Next, Section 4 evaluates and reports on the outcomes of our experiments. Finally, our key conclusions are outlined in Section 5, including future work.

2. Previous work

2.1 Author profiling

Author profiling, the task of analyzing text to uncover various traits of its author, is based on the theoretical notion of sociolect and the “stylome hypothesis.” Sociolect involves both passive and active engagement with communicative practices, influencing and reflecting socio-psychological traits that can influence language production and, reversely, that can be identified by observing the associated quantitative linguistic patterns. The stylome hypothesis dates back to the mid-2000s, when van Halteren et al. (2005) compared it to the genome, suggesting that an author’s stylometric profile is unique.

Many studies in author profiling have explored various demographic characteristics such as gender (Aravantinou et al. 2015; Veenhoven et al. 2018), age (Rangel and Rosso 2013; Lundeqvist and Svensson 2017), personality (Smynor 2015; McCollister 2016), political affiliation (Tumasjan et al. 2010; David et al. 2016), and even sexual orientation (Loh, Soo, and Xing 2016). In this study, we are focusing on deducing the author’s gender, as systematic differences in linguistic production between men and women have long been observed in nearly all aspects of linguistic levels (Mikros 2013a).

Gender identification has been the focus of many studies since the early 2000s, with Koppel, Argamon, and Shimoni (2002) pioneering the use of stylometric features for this purpose. Their analysis using the British National Corpus revealed gender-specific usage of text features, indicating the interaction between the author’s gender and the text genre. Subsequent studies have employed various document representations and machine learning algorithms for automatic gender identification. Document representations include Bag-of-Words (Bamman, Eisenstein, and Schnoebelen 2014), n-gram models (Potamianos and Jelinek 1998; Mikros and Perifanos 2015), stylometric features (Mikros 2020), and word embeddings (Franco-Salvador et al. 2017), whereas machine learning algorithms range from Logistic Regression (Nowson 2006) and Support Vector Machines (David et al. 2016; Stamatatos et al. 2018), to Random Forest (RF; Xu and Jelinek 2004) and Deep Learning models such as long short-term memory (LSTM; Veenhoven et al. 2018; Alroobaea et al. 2020) and convolutional neural networks (CNNs; Schaetti 2017). These experiments have been effective across various text genres, including social media and blogs, demonstrating significant differences in linguistic usage between male and female authors.

Recent studies (Lee 2019; Mikros 2020) have extended author profiling to translated texts, exploring whether they conform to standard stylometric patterns or exhibit specific peculiarities that constitute a different variety, a separate “dialect” within a language commonly referred to as “third code” (Frawley 1985) or “translationese” (Gellerstam 1986). Baroni and Bernardini (2005) and Bernardini and Baroni (2005) have shown that although a general “translationese” is identified, strong influences from the source language have also been noted. Most research on translationese and translation universals has been conducted using specific linguistic units of analysis to operationalize concepts such as interference, simplification, and normalization. Since “translationese” is a code that incorporates extra-linguistic information, we believe it is reasonable to extend the scope of the research question and form a research hypothesis about how the author’s gender is reflected in the (machine-)translated text. Translations are, after all, language products and are expected to acquire the specific linguistic features typical of a given author (idiolect) and particular social variables describing the text’s author (sociolect). It is important to note the distinction between translator style, which refers to the unique style of an individual translator, and translation style, which emerges as a result of the translation process itself (Saldanha 2011). The focus of this study is on the former, investigating how the translation process influences the reflection of the author’s gender in the translated text.

2.2 Authorship attribution

Authorship attribution, on the other hand, employs computational techniques to determine a text’s author based on their unique writing style (Neal et al. 2017; Sari 2018). There are three primary forms of authorship attribution: author verification, closed-set and open-set attribution (Barlas and Stamatatos 2020). In author verification, a single author is considered, whereas closed-set and open-set involve multiple candidate authors, with the latter allowing for an unknown author. Posadas-Durán et al. (2017) achieved over 98 per cent accuracy in binary author verification using word n-grams and Doc2vec. Other methods range from analyzing lexical, syntactical, and grammatical features (Argamon and Levitan 2005; van Halteren et al. 2005; Zhao and Zobel 2005; Grieve 2007; Wu, Zhang, and Wu 2021; Zheng and Jin 2022) to only focusing on function or content words (Kestemont 2014; Boumparis, n.d.).

Deep Learning, particularly CNNs, has shown promising results in authorship attribution (Zhang, Zhao, and LeCun 2015; Ruder, Ghaffari, and Breslin 2016; Shrestha et al. 2017). Character-level features often outperform word sequences, a finding also supported by Shrestha et al. (2017). Pretrained language models like BERT, enhanced with additional dense layers and softmax activation, have achieved accuracy levels up to 5 per cent above other approaches (Fabien et al. 2020). According to Zheng and Jin (2022), the effectiveness of Deep Learning is attributed to its autonomous feature engineering and hierarchical, complex network structure. However, in open-set attribution, where the author is not in the candidate list, linear classifiers achieve near-perfect accuracy but require more robust approaches for larger candidate pools (Badirli et al. 2021).

Bogdanova and Lazaridou (2014) introduced the task of Cross-Language Authorship Attribution. They investigated author identification in texts translated into different languages than the one in the labeled data. They reported 95 per cent and 93 per cent accuracy for English and Spanish, respectively, using linear SVM with Bag-of-Words. MT reduced the accuracy, although high-level features pairing with k-Nearest Neighbors scored promising results up to 95 per cent.

One admittedly deciding factor in authorship attribution tasks is the data size per candidate author, with many studies measuring the impact of the corpus size (Jin and Murakami 2007; Luyckx and Daelemans 2008; Zheng and Jin 2022). Argamon and Levitan (2005) predicted the author’s identity and nationality in novels with an average of 10,000 words per book, with 99 per cent and 93.5 per cent accuracy, respectively. For shorter texts like Tweets, RF generally yields the best results (Tanaka and Jin 2014; Rao, Raju, and Kumar 2017), guiding our choice of methodology in our series of experiments discussed in the following section.

3. Research methodology

3.1 Research aims

Our study focuses on the impact of MT on authorship attribution and gender identification tasks. Particularly, we want to investigate whether a machine-translated text preserves its stylometric profile and whether author profiling methods can be used reliably across a machine-translated text. The motivation behind this research aim is to validate cross-linguistic stylometric analysis using MT as a potential method to overcome specific language barriers and work with texts in one language regardless of their source language. If the target language of this methodology is English, then we will be able to leverage the vast array of the available linguistic resources and tools existing for English and apply state-of-the-art NLP methods that are yet to be utilized in other languages. To explore the research mentioned above, we set two more specific research objectives. In particular:

• Compare different feature groups (Author’s Multilevel N-gram Profiles—AMNP, lexical diversity features—quantitative linguistics (QL), cross-lingual word embeddings—CLWE) in the authorship attribution and the gender identification task and investigate their relative accuracy in both the source and the target language datasets.

• Explore whether cross-linguistic authorship attribution and authors’ gender profiling are feasible when the training dataset uses the source language and the testing dataset is the translated language.

3.2 Corpus

To explore the above research objectives, we have conducted a small-scale experiment focused on texts written in Modern Greek and translated into English using Google’s Neural MT (NMT). We first compiled a Greek blog corpus to be used as our source language dataset. To this end, we harvested the Greek blogosphere and manually collected 100 Greek blogs equally divided into fifty male and fifty female bloggers. The bloggers’ gender was determined based on their own descriptions on their profile pages and short bios.

In this study, we opted to use Google’s NMT due to its widespread availability, robust performance across a wide range of languages, and its neural architecture, which has been shown to produce more fluent and accurate translations compared to statistical MT systems (Wu et al., 2016). However, we acknowledge that other MT tools, such as DeepL or EU’s e-Translation, may yield different results, and future research could explore the impact of various MT systems on cross-linguistic authorship attribution and gender profiling tasks.

The compiled corpus was translated into English using the Google Translate API. Due to the API’s 5,000-character quota limit, we needed to write a Python script that splits the texts that are longer than this pre-defined threshold into chunks prior to sending the request to the API and getting back the translated text as a response. To minimize mistranslations, we took whole sentences into account. Namely, we split at the very last punctuation mark of the sentences whose total number of characters would not exceed the threshold if added to the total number of characters up to that point. As soon as the API sends the response back containing the machine-translated text, the script would reconnect the split texts into a single target file.

The descriptive statistics for both the source and the target language corpora can be found in Table 1.

Table 1.

Descriptive statistics for the source (Greek—EL) and target (English—EN) language corpus.

	Source (EL)				Target (EN)
	N words	SD	min	max	N words	SD	min	max
Female	899,099	295	10	4,653	918,189	315	10	5,167
Male	873,628	396	10	7,921	907,862	410	10	7,749
Total	1,772,727	349	10	7,921	1,826,051	366	10	7,749

The descriptive statistics reported for both corpora show that the translation process did not affect in any way the quantitative profile of the texts written by males and females. All quantitative gender-based differences have been preserved across all metrics (text size [N words], standard deviation of the text size [SD], minimum text size [min], and maximum text size [max]).

3.3 Feature group comparison

We studied the impact of several features commonly used in authorship profiling tasks to get a better understanding of which dimensions of the stylometric profile are affected during the MT process. We grouped the features into three distinct categories:

3.3.1 AMNP

AMNP, introduced by Mikros and Perifanos (2013), is a highly effective document representation for authorship attribution and author profiling tasks (Mikros 2013a, 2013b; Mikros and Perifanos 2015; Mikros 2018). It is based on increasing n-gram sizes of character and word units, ensuring coverage of different linguistic levels. The theoretical motivation behind AMNP comes from the Prague School of Linguistics and the concept of “double articulation” (Nöth 1995: 238), which states that language is divided into two layers: meaningful units (morphemes) and minimal functional units (phonemes). The combination of these layers produces grammatically correct linguistic production. Similarly, to capture the multi-level manifestation of stylistic traits, features present at various linguistic levels must be detected and combined to accurately represent an author’s style.

Here, we extracted the 1,000 most frequent characters and word n-grams with n = 2 and 3, resulting in a total vector of 4,000 features. The resulting vector is the AMNP, a document representation that simultaneously captures both character and word sequences. We calculated all features’ normalized frequency to prevent introducing text length bias into subsequent calculations.

3.3.2 Lexical diversity (QL)

Lexical diversity indices, inspired by QL, recognize the lexicon as the basis of human core language ability (Long and Richards 2007) and the main source of language variation (Bates and Goodman 1997 in Treffers-Daller 2011: 149). These measures capture the vocabulary size using mathematical approaches applied to types or lexemes in a text (Read 2000: 200).

Being complex constructs, they approach deeper aspects of language production, with lexical knowledge as the most important component of variability in language ability (Treffers-Daller 2011: 149). They have been used to diagnose cognitive disorders like Alzheimer’s (van Velzen, Nanetti, and de Deyn 2014), Parkinson’s (Ellis, Holt, and West 2015), SLI (Owen and Leonard 2002), and aphasia (Fergadiotis, Wright, and West 2013), indicating their correlation with cognitive aspects of language production.

Lexical richness indices have also been used to assess bilingual speakers’ language production (Daller, van Hout, and Treffers‐Daller 2003; Treffers-Daller 2011; Treffers-Daller and Korybski 2015), playing an important role in detecting language dominance and code-switching. We included a list of features calculated using specialized software for quantitative linguistic indices (Kubát, Matlach, and Čech 2014) and custom scripts:

• h-point: This index represents the “bisector” point of the rank ∼ frequency distribution at which rank = frequency. Jorge E. Hirsch (2005) initially proposed it for scientometrics, and Popescu (2007) introduced it into linguistics and developed it further (Popescu, Best, and Altmann 2007). Using the definition mentioned above, the h-point splits the vocabulary into two essential parts, namely into a class of magnitude h of frequent function words (synsemantics) and a much larger class of content words (autosemantics) with size $V–h$ which are not so frequent but constitute the better part of the text’s vocabulary (Popescu et al. 2009: 19).

• Entropy (H): The term has been used across many scientific disciplines with different meanings, mainly to quantitatively define the diversity or the uncertainty of a system. In this article, entropy is calculated using Shannon’s formula on the word frequencies of the corpus (Oakes 1998: 59). As a result, texts with extensive vocabularies and low frequencies produce high entropy, while texts with controlled vocabularies and formulaic or systematic word usage exhibit lower entropy.

• Yule’s characteristic K: A measure of vocabulary “richness” based on the work of Yule (1944). The index measures the lexical Repeat Rate (RR) and has been found to be sufficiently robust regardless of the text size from which it is calculated (Tweedie and Baayen 1998).

• Writer’s view: An index proposed by Popescu and Altmann (2007) that is connected to the golden ratio (φ ≈ 1.618). It is defined as the angle that is formed between the word frequency ∼ rank distribution end and its top, as seen from the h-point (Popescu et al. 2009: 26). Commenting on its name, they claim that it is “baptized in this way because one can imagine the writer ‘sitting’ at this point and controlling the equilibrium between autosemantics and synsemantics.”

• R1: An index of vocabulary richness proposed by Popescu et al. (2009: 29–34), which is based on the h-point and the cumulative relative frequencies up to the h-point.

• RR: The RR shows a text’s degree of vocabulary concentration. In other words, this indicator measures vocabulary richness inversely: the higher the RR, the less vocabulary diverse the text. The resulting values of RR are in the interval <1/V; 1> where V is the number of lexical types in the text (Kubát, Matlach, and Čech 2014).

• Relative RR of McIntosh (RRmc): This is a normalized index of RR so that it takes values in the interval [0, 1], originally proposed by McIntosh (1967). The main difference with the RR is that RRmc takes values in the closed interval [0, 1], while the RR from Kubát, Matlach, and Čech (2014) cannot take the value of 0.

• Curve Length (L): A vocabulary richness index based on the rank ∼ frequency distribution curve, defined as the sum of Euclidean distances between all points on the curve (Kubát, Matlach, and Čech 2014).

• Curve length R Index (R): A vocabulary richness index derived from the curve length (L). It is defined as the ratio of the curve length above the h-point to the whole curve length (Kubát, Matlach, and Čech 2014).

• Adjusted Modulus (A): A frequency structure indicator that is supposed to be independent of text length (Popescu et al. 2010).

• Gini Coefficient (G): A measure of statistical dispersion originally developed for econometric analysis based on the Lorenz curve. It can be used to measure vocabulary richness by considering the rank ∼ frequency distribution by reversing the rank order (Popescu et al. 2009).

3.3.3 CLWE

Word embeddings are a popular representation of a text’s vocabulary. Words or phrases from the lexicon are translated into vectors of real numbers and can subsequently be used in language modeling and feature learning. These numerical vectors can capture text representations in an n-dimensional space, where words with the same meaning are represented similarly. This indicates that two comparable words are located close to each other in vector space and have similar vector representations. Therefore, the objective of creating a word embedding space is to record some form of relationship in that space, be it a relationship based on meaning, morphology, context, or another type of link. Since language produced by different genders exhibits systematic differences across all levels of its linguistic organization (Mikros 2013b), we foresee that word embeddings can be used effectively for the gender identification task. In fact, several studies have reported satisfactory results in various author profiling tasks (Bayot and Gonçalves 2016; Bojanowski et al. 2017; Dias and Paraboni 2018; López-Santillán et al. 2020).

Since, in our study, we are using texts in their source language (Greek) and their translations in a target language (English), we decided to explore how CLWE could be used as a language-neutral approach to the features used for training the classification algorithm both in the authorship attribution and the author profiling tasks. CLWE are a type of word representation that extends the concept of word embeddings to multiple languages. They represent translation-equivalent words from two or more languages close to each other in a cross-lingual space. This allows for comparing and transferring semantic information across different languages.

We created CLWE using monolingual mapping, a method to create a shared semantic space for words from different languages using pre-trained monolingual word embeddings on large corpora within each language independently. To obtain the pretrained word embeddings, we utilized fastText¹ (Bojanowski et al. 2017) for both Greek and English. FastText was chosen for its ability to capture subword information, which is particularly useful for morphologically rich languages like Greek. Additionally, fastText embeddings have been shown to perform well in various downstream NLP tasks (Joulin et al. 2017). Nevertheless, we recognize that other language-agnostic multilingual sentence embeddings, such as LaSER (Artetxe and Schwenk 2019) or LaBERT (Feng et al. 2022), could potentially impact the results, and future research could investigate the performance of these alternative embeddings in cross-linguistic stylometric tasks.

Since these embeddings capture the semantic and syntactic properties of words within their respective languages, the next step was to learn a transformation that aligns the monolingual embedding spaces, effectively mapping them into a shared cross-lingual space. In our case, the alignment was achieved through a linear transformation (Orthogonal Procrustes), which preserves the monolingual properties while aligning translation-equivalent words, finding an orthogonal transformation (a matrix) that most closely maps one set of vectors to another. This process required a bilingual dictionary which was used to supervise the mapping and was provided by the MUSE library (Lample et al. 2018).

For each text, we narrowed down our vocabulary to the 1,000 most frequent words (MFWs) since these carry the most robust stylometric information (Zhao and Zobel 2005; Argamon et al. 2007; García and Martín 2007). We averaged the word embeddings calculated across these 1,000 MFWs, resulting in a document representation where each text vector contains 300-word embeddings representing the mean of the corpus’s 1,000 MFWs.

3.4 Machine learning approach

We trained an RF classifier (Breiman 2001) for authorship attribution and gender profiling, choosing it over more recent deep learning algorithms for its stability and robustness. Our focus is on understanding how MT impacts stylometric analysis, without necessarily achieving state-of-the-art classification. RF, an ensemble of Decision Trees, excels at handling large datasets with many variables. It introduces randomness by bootstrapping data samples and selecting random feature subsets for node splitting, balancing variance and bias. This methodology enhances the general accuracy of the model.

All statistical models developed as part of this study were evaluated using 10-fold Cross-Validation (with 90 per cent to 10 per cent train-test split), and the accuracies reported here represent the mean of the accuracies obtained in each fold. Moreover, all other evaluation metrics (precision, recall, F1) were calculated using macro-averaging since we were interested in understanding the performance of the different models equally across each class. Since the feature space in the AMNP datasets was sparse, we eliminated all features that showed a variance close to zero using the two following rules:

• the percentage of unique values was less than 20 per cent;

• the ratio of the most frequent to the second most frequent value was greater than 20.

Removing the features with near-zero variance shrank the number of employed features and reduced them by 54 per cent for the Modern Greek dataset (from 4,000 to 1,838) and 58.6 per cent for the English dataset (from 4,000 to 1,656 features). Moreover, to normalize the different scales employed in the three mentioned feature groups, we standardized the data to z-scores with $M=0$ and $SD=1$ before training the RF model.

4. Results

The classification experiments are organized in two different sections according to the task at hand—authorship attribution and gender profiling, respectively. In each section, we report the evaluation metrics (accuracy, recall, precision, and F1-score) for each feature group in four distinct experimental setups:

• Baseline accuracy: the classification’s expected accuracy if performed randomly.

• Greek dataset: the obtained cross-validated averaged evaluation metrics when the classification was performed on the Greek dataset.

• English dataset: the obtained cross-validated averaged evaluation metrics when the classification was performed on the English dataset.

• Cross-linguistic classification: the obtained cross-validated averaged evaluation metrics when the RF was trained using the Greek dataset, and the obtained model was used to make predictions on the English one. This is a simulation of the actual cross-linguistic scenario in which we create models of authors and their gender in one language and then use these models to detect their identity and gender after their texts have been translated into another language—here, English.

4.1 Authorship attribution experiments

The results obtained in the authorship attribution task are reported in Tables 2–4 below (bold values represent the highest metrics obtained in the specific task):

Table 2.

Experiment 1a: Authorship attribution using the AMNP feature group.

Experimental setups	Accuracy	Recall	Precision	F1
Cross-linguistic (EL: training ⇒ EN: test)	0.01	0.01	0.01	0.01
English	0.32	0.31	0.27	0.27
Greek	0.35	0.34	0.32	0.30
Baseline	0.01	–	–	–

Bold values represent the highest metrics obtained in the specific task.

Table 3.

Experiment 1b: Authorship attribution using the QL feature group.

Experimental Setups	Accuracy	Recall	Precision	F1
Cross-linguistic (EL: training ⇒ EN: test)	0.04	0.04	0.05	0.03
English	0.11	0.11	0.09	0.09
Greek	0.12	0.12	0.11	0.10
Baseline	0.01	–	–	–

Table 4.

Experiment 1c: Authorship attribution using the CLWE feature group.

Experimental setups	Accuracy	Recall	Precision	F1
Cross-linguistic (EL: training ⇒ EN: test)	0.05	0.05	0.20	0.05
English	0.27	0.27	0.23	0.23
Greek	0.30	0.30	0.26	0.25
Baseline	0.01	–	–	–

Bold values represent the highest metrics obtained in the specific task.

The results reported in the authorship attribution task can be summarized in the following points:

• The highest-scoring feature group in both the monolingual Greek and the English data is the AMNP, followed closely by the CLWE. This is an interesting finding since AMNP contains six times more features (1,838 in the Greek and 1,656 in the English dataset) than CLWE (300 features in either dataset). This confirms previous research findings (Hoenen 2017; Barlas and Stamatatos 2020; Kumar, Gowtham, and Chakraborty 2022), all of whom report excellent authorship attribution results using various forms of word embeddings.

• In the cross-linguistic authorship attribution experiment, the most effective feature group was CLWE, which achieved results five times more accurate than the baseline, scoring 0.05 compared to 0.01. The second most successful feature set was the Quantitative Linguistics (QL), which outperformed the baseline by four times, achieving a score of 0.04. Both CLWE and QL demonstrate strong potential for effective cross-linguistic authorship attribution. Notably, QL’s accuracy is particularly impressive given its small size of only thirteen features, in contrast to CLWE’s 300 features. QL’s high performance aligns with previous findings by Juola, Mikros, and Vinsick (2019a, 2019b), who highlighted significant correlations between Greek, Spanish, and English QL indices among Greek texts. While the AMNP feature group shows good performance in monolingual datasets, its performance in the cross-linguistic setting only matched the baseline. This outcome was expected, as the n-grams in the source language differ from those in the target.

To better understand the dynamic interaction of the variables involved in the authorship attribution experiments reported above, we statistically evaluated the accuracy results further. Before conducting the analysis, we examined the assumptions of the two-way analysis of variance (ANOVA). Levene’s test for homogeneity of variances yielded a non-statistically significant result (⁠$F\mathrm{ }=1.013$⁠, $\text{df}1=5$⁠, $\text{df}2=174$⁠, $P=.411$⁠), indicating that the assumption of equal variances across groups was met. Additionally, the Shapiro–Wilk normality test was performed to assess the normality of the data. The results showed that the accuracy scores for each feature group (AMNP: $\text{Statistic}=0.977$⁠, $\text{df}=60$⁠, $P=.315$⁠; CLWE: $\text{Statistic}=0.987$⁠, $\text{df}=60$⁠, $P=.749$⁠; QL: $\mathrm{Statistic}=0.968$⁠, $\mathrm{df}=60$⁠, $P=.114$⁠) were not significantly different from a normal distribution, satisfying the assumption for normality.

Having confirmed that the assumptions were met, we proceeded with the two-way ANOVA, using the classification accuracy of each fold in the conducted experiments as the dependent variable, and the feature group used and the language employed as the independent variables. We fitted the model of each feature group to 30-folds to create enough data points so that the statistical analysis achieves sufficient statistical power (Bausell and Li 2002). The analysis is restricted to the performance recorded for the two monolingual experiments since, in the cross-linguistic task, we did not employ cross-validation, and the trained Greek model performed a prediction on the English dataset. The interaction plot of the variables involved is shown below (Fig. 1):

Figure 1

Interaction plot of the language and the feature group used in the classification accuracy of the authorship attribution.

The two-way ANOVA revealed a statistically significant interaction in all the main effects but not in the interaction of the independent variables. The main effects analysis showed that both the Feature Group (⁠$F(2, 174)=857$⁠, $P<.001$⁠) and the Language (⁠$F(1, 174)=42$⁠, $P<.001$⁠) did have a statistically significant effect on the classification accuracy. However, the effect size (partial ${\eta }^2$⁠) of the two main effects is considerably different. The Feature Group exhibits a big effect size (0.90), while the Language shows a considerably lower one (0.19).

To gain a clearer insight into the specific differences among the features, we employed Tukey’s post-hoc test for multiple comparisons across the three distinct feature groups. This method is particularly suitable as it effectively handles pairwise comparisons within datasets that contain an equal number of subjects in each group, as noted by Stoll (2017). All the differences between all feature groups were found to be statistically significant, indicating that each feature group was very different from the others in terms of accuracy performance. Overall, we can conclude that all the feature groups employed in this study perform differently from each other but are similar in both languages involved.

Thus, the accuracy of authorship attribution is better in the source language, indicating that MT degrades the stylometric information during generation. However, judging by the relatively small effect size mentioned above, this effect is not massive, and with the appropriate experimentation with different classification algorithms and feature groups, it still can be considered a viable option for cross-linguistic authorship attribution.

4.2 Gender profiling experiments

We performed the gender profiling classifications by following the same methodology described in Section 4.1. We used the same four experimental setups as in the authorship attribution task, and the results are reported in Table 5–7, respectively (bold values represent the highest metrics obtained in the specific task):

Table 5.

Experiment 2a: Gender profiling using the AMNP feature group.

Experimental setups	Accuracy	Recall	Precision	F1
Cross-linguistic (EL: training ⇒ EN: test)	0.50	0.57	0.50	0.54
English	0.67	0.68	0.66	0.67
Greek	0.69	0.70	0.69	0.69
Baseline	0.50	–	–	–

Table 6.

Experiment 2b: Gender profiling using the QL feature group.

Experimental setups	Accuracy	Recall	Precision	F1
Cross-linguistic (EL: training ⇒ EN: test)	0.53	0.53	0.56	0.46
English	0.60	0.63	0.60	0.62
Greek	0.56	0.58	0.57	0.57
Baseline	0.50	–	–	–

Table 7.

Experiment 2c: Gender profiling using the CLWE feature group.

Experimental setups	Accuracy	Recall	Precision	F1
Cross-linguistic (EL: training ⇒ EN: test)	0.54	0.47	0.54	0.50
English	0.69	0.71	0.68	0.69
Greek	0.69	0.72	0.68	0.70
Baseline	0.50	–	–	–

Bold values represent the highest metrics obtained in the specific task.

From the results above, we can conclude the following:

• Word embeddings are the best-performing feature group for gender profiling, followed by AMNP in the monolingual corpora. This confirms the superiority of word embeddings as a feature group in a wide range of computational stylistics tasks, as already mentioned above.

• In the cross-linguistic experiment, both CLWE and QL performed nearly equally (0.54 and 0.53 correspondingly), indicating that both feature groups can convey cross-linguistic stylometric information. AMNP, on the other hand, scored on par with the baseline, as was expected due to the difference in the n-grams involved in the source and target language.

To more systematically analyze the above observations, we performed a two-way ANOVA using as dependent variable the gender classification accuracy and as independent variables the language and the feature group used in the gender profiling experiments. Again, here, the analysis involves only the separate gender classification experiments that were conducted in each monolingual dataset. We followed the same procedure as in the authorship attribution analysis and tested the homogeneity of variances and the normality assumption. Levene’s test yielded a non-statistically significant result (⁠$F=0.360$⁠, $\text{df}1=5$⁠, $\text{df}2=174$⁠, $P=.875$⁠), indicating that the assumption of equal variances across groups was met. Additionally, the Shapiro–Wilk normality test was performed to assess the normality of the data. The results showed that the accuracy scores for each feature group (AMNP: $\text{Statistic}=0.975$⁠, $\text{df}=60$⁠, $P=.266$⁠; CLWE: $\text{Statistic}=0.986$⁠, $\text{df}=60$⁠, $P=.730$⁠; QL: $\text{Statistic}=0.982$⁠, $\text{df}=60$⁠, $P=.526$⁠) were not significantly different from a normal distribution. The interaction plot is shown in Fig. 2:

Figure 2

Interaction plot of the language and the feature group used in the classification accuracy of the gender profiling.

The two-way ANOVA revealed that the main effect of the feature group was found to be statistically significant (⁠$F(2, 174)=122.7$⁠, $P<0.001$⁠) with a considerable effect size (partial ${\eta }^2=0.58$⁠). We further analysed this main effect with Tukey’s post-hoc test which grouped the AMNP and the CLWE feature groups together separating them from the QL since these two feature groups achieve nearly identical classification accuracies.

Furthermore, the analysis showed a statistically significant interaction between the Feature Group and Language (⁠$F(2, 174)=3.2$⁠, $P<.05$⁠). When using the AMNP feature group, gender profiling is more accurate for the Greek dataset, a trend that shifts when employing the QL features, which appear to perform better with the English dataset.

Contrary to the results observed in the authorship attribution experiments, in gender profiling, MT does not appear to significantly reduce classification accuracy. This suggests that the stylometric information pertinent to gender is more resilient during language switching.

5. Conclusions

This study investigated whether using MT to translate original documents into another language can be a reliable strategy for cross-linguistic authorship attribution and gender profiling tasks. We used a large Greek blog corpus translated automatically into English and ran experiments using three different types of features: AMNP, CLWE, and QL.

In the monolingual corpora experiments for authorship attribution, AMNP was efficient in both languages, confirming its superiority in current stylometric research. CLWE also proved highly effective in both source and target languages. Authorship attribution tends to be more precise in the source language, suggesting that MT slightly impairs stylometric characteristics. However, the modest effect size suggests it is manageable, and using translated texts for cross-linguistic authorship attribution can still be considered feasible with appropriate methodological adjustments.

In cross-linguistic authorship attribution with training data in the source language and testing data in the target language, CLWE achieved accuracy quintuple that of the baseline (0.05 versus 0.01), followed closely by QL with fourfold accuracy (0.04 versus 0.01). QL’s effectiveness is remarkable, given its minimal feature count (13) compared to CLWE (1,000). QL indices approach a stable trait of idiolect above the specific language used, with lexical “richness” patterns transferred across linguistic codes and likely associated with higher-level intellectual functions. However, their impact on authorship attribution is limited, and more research is needed to identify stylometric features that associate more efficiently with authors’ identities across languages.

Gender profiling utilizes distinct feature groups, primarily relying on CLWE’s rich semantic representation to uncover diverse writing styles of male and female authors, including differences in word choice, thematic emphasis, and underlying syntactic patterns. CLWE can identify tendencies like more emotive and descriptive language in female authors versus assertive or direct language in male authors and distinguish genders in subject matter and narrative style.

The study found that using MT to convert a multilingual corpus to one language for authorship attribution and gender profiling is feasible with comparable accuracies. Gender profiling accuracy was equal across languages, while authorship attribution was slightly better in the source language with a minor effect on the language condition.

In the scenario of cross-linguistic authorship attribution and gender profiling with trained models and testing data in different languages, CLWE and QL feature groups offered better-than-chance probability in predicting the author’s identity and gender. Although accuracies were lower than in the monolingual experiments, they provide a foundation for more comprehensive research in this area.

However, it is important to note that both Greek and English are Indo-European languages with relatively similar structures and properties. The results obtained in this study may not be generalizable to languages from other language families with significantly different characteristics. Furthermore, when considering languages with vastly different structures, such as polysynthetic languages where a single word may correspond to multiple words in English, the effectiveness of our approach may be limited. In such cases, the translation process could significantly alter the word counts and other linguistic features, potentially affecting the performance of stylometric analyses. The applicability of word embeddings in these scenarios remains an open question that calls for further research efforts.

Moreover, it is important to acknowledge that the tools we chose for obtaining the translations and the multilingual sentence embeddings may have influenced the results of our study. While both Google’s NMT and fastText were selected for their strong performance and wide applicability, it is possible that using alternative MT tools, such as e-Translation or DeepL, or different language-agnostic multilingual sentence embeddings, like LaSER or LaBERT, could lead to varying results.

Future research should explore the impact of these choices on cross-linguistic authorship attribution and gender profiling tasks to better understand the robustness and generalizability of our findings. Additionally, comparing the performance of various MT tools and multilingual embeddings could provide helpful insights into the optimal setup for cross-linguistic stylometric analyses. Moreover, a wider variety of text genres and an enriched set of language pairs and machine learning algorithms would provide generalization validity in our conclusions. We plan to investigate further the properties of lexical diversity features in cross-linguistic stylometry and experiment with novel approaches that capture more dynamic aspects of text production in the future.

Acknowledgement

Open Access funding provided by the Qatar National Library.

Authors’ contributions

George Mikros (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing—original draft, and Writing—review & editing) and Dimitris Boumparis (Data curation, Formal analysis, Investigation, Methodology, Resources, Software, Validation, Writing—original draft, and Writing—review & editing)

Footnotes

1

See https://fasttext.cc/.

Author notes

Work done during Master’s at University of Antwerp.