7 Between Humans and Machines: Judicial Interpretation of the Automated Decision-Making Practices in the EU

In the European Union (EU), recent judicial rulings have provided more precise interpretations of automated decision-making (ADM) practices. The right not to be subject to automated decisions, as described in Article 22 of the General Data Protection Regulation (GDPR),¹ was brought to the Court of Justice of the European Union (CJEU) on 16 March 2023, making the first instance of such consideration.² The case concerns credit scoring used in Germany, known as ‘Schufa’, and whether credit scoring can be considered an automated decision. Additionally, another significant case related to Article 22 of the GDPR came out in the Netherlands. The Court of Appeal in Amsterdam (Gerechtshof Amsterdam) found that several automated processes, including assigning rides, calculating prices, rating drivers, calculating ‘fraud probability scores’, and deactivating drivers’ accounts in response to suspicions of fraud on Uber’s and Ola platforms, are considered as automated decisions.³

In light of these contemporary examples within the EU, this chapter aims to systematize the ADM practices and explores the role of judicial interpretation in defining these activities and involvement of humans in decision-making processes. The chapter is divided into three sections focusing on machines, humans, and courts. It begins by exploring the concrete uses of ADM in the EU (section B. Machines). From there, it delves into how these systems are currently being utilized by taking into account two official reports published by the EU institutions. The objective is to provide nuanced insights into the current applications of ADM systems, avoiding overly broad generalizations. Following this analysis, the research identifies the socio-technical quality of ADM practices and argues how this quality necessitates meaningful human participation in decision-making processes. The chapter then examines the human-centric provisions of the relevant EU legal instruments surrounding ADM systems and targeting human participation (section C. Humans). Finally, it examines four judicial cases which surfaced public and private contexts in the Netherlands, Italy, and Germany (section D. Courts). In conclusion, the chapter identifies three key aspects of judicial interpretation regarding ADM practices: (i) epistemic; (ii) substantial, encompassing socio-technical and legal dimensions; and (iii) methodological. It argues that these aspects prove the pivotal role of judicial interpretation in comprehending the technical aspects of automation and ensuring meaningful human participation in decision-making processes.

In the artificial intelligence (AI) age, the decision-making landscape is undergoing a profound transformation. Significant decisions about modern life are increasingly delegated from human hands to algorithmic machines.⁴ Algorithms are ‘a series of instructions that instruct a software package to take a dataset and learn a model or discover some underlying pattern’.⁵ An ADM system ‘augments or replaces human decision-making by using computational processes to produce answers to questions either as discrete classifications or continuous scores’.⁶ Such decision-making has been implemented in complex areas involving public and private contexts, including social benefits, migration and border control, and loan or mortgage applications. As such systems become more prevalent in modern life, it is important to consider the complexities they introduce to decision-making and to examine their social and legal impacts thoroughly.

However, as noted by an Italian court in 2019, ADM systems possess the quality of ‘multidisciplinary characterization’ (caratterizzazione multidisciplinare), requiring not only legal, but technical, computer, and statistical skills.⁷ This situation makes it even more difficult to understand the complexities posed by such systems. Therefore, legal scholars resort to analogical thinking and explore the similarities first between the new digital technology in question and the previous ones. However, the use of analogy in those examples has demonstrated that it does not sufficiently meet the nature of what a particular technology is, and thus misses many of its unique features.⁸ It is important to note that incorrect conceptualizations of technologies which often rest on the use of analogy can lead to incorrect normative results in the legal sphere, as also recognized by a recent judgment of the District Court of the Hague (Rechtbank Den Haag), which argues that if we base our knowledge of technologies on properties and use terms such as ‘self-learning’, and make wrong analogies between the human person and a new technology, we are then unable ‘to properly justify actions and to properly substantiate decisions’ in an administrative system.⁹

Considering this issue, this chapter takes into account two official reports published by the EU institutions which provide empirical research on the current uses of automated systems used by the public sector: ‘Getting the Future Right: Artificial Intelligence and Fundamental Rights’ published by the Fundamental Rights Agency in 2020¹⁰ (hereafter Report I) and ‘AI Watch Artificial Intelligence in Public Services: Overview of the Use and Impact of AI in Public Services in the EU’ published by the Joint Research Centre in 2020 (hereafter Report II).¹¹ The purpose is to be concrete on the current uses of ADM systems and to avoid examining the systems at stake on an overgeneralized fashion. Considering the two reports, the most critical examples used in the public sector are observed in the fields of social benefits and biometrics.

The Organisation for Economic C-operation and Development (OECD) defines social benefits as ‘current transfers received by households intended to provide for the needs that arise from certain events or circumstances, for example, sickness, unemployment, retirement, housing, education or family circumstances’.¹² The Report I underlines that automated systems used in public administration include areas such as benefit calculations, fraud prevention and detection, eligibility assessments, and risk scoring.¹³ The purpose of governments is to enhance the efficiency of decision-making on these issues. In the context of social benefits, the Report I explains two important areas where ADM systems are used for decisions—housing and unemployment benefits.¹⁴ In these areas, rule-based decision-making is applied, defined based on ‘if–then rules’.¹⁵ For instance, a person will be eligible for a certain income if she/he has an income below a certain threshold.¹⁶  Table 7.1 sketches three artificial intelligence (AI) practices in the field of social benefits, defining their purposes, data sources, and ‘black-box’ aspects.¹⁷

In this field, these systems include processing the applications first and deciding on these applications second. Due to the complexity of the systems used, the techniques on decision-making should be considered as the ‘black-box’ aspects of the ADM practice at stake, meaning that such systems are producing results without clear or understandable explanations of how the results have been reached.²³ This quality is particularly significant because, as the Hague Court highlighted, citizens could neither anticipate the intrusion into their private life nor can they guard themselves against it.²⁴

All three practices are developed as rule-based decision-making. In the last practice, robotic process automation (RPA) has been used in the municipality of Trelleborg, Sweden, since 2016, noted by Report II.²⁵ Despite in-depth conversations in major media outlets, neither the public officials or the company which developed the system have provided satisfactory answers regarding how the system works, nor how it makes decisions.²⁶ Furthermore, such a system makes decisions on the basis of its data sources. It is therefore necessary to take into consideration the fact that the data source may not provide sufficient, necessary, or even correct information. This critical observation emphasizes the importance of human intervention in these areas.

In the case of biometrical systems, two fields are prominent in public use: predictive policing and migration and border control management. Table 7.2 sketches the two use cases in this area based on Report I²⁷ and Report II,²⁸ defining their purposes, data sources, and ‘black-box’ aspects.

Both Report I and Report II do not provide any specific use case about the AI-driven ADM systems used for migration and border control management. However, newer AI techniques are being developed to control borders and to provide a decision support system for border authorities.³¹ Moreover, this area is particularly significant as all the three systems of biometrics mentioned in the table, biometric identification, categorization, and emotion recognition systems, can be used in this area.³² According to a recent empirical study, the existing uses of digital technologies across European immigration and asylum systems include forecasting tools, processing of short- and long-term residency and citizenship applications, risk assessment and triaging systems, speech recognition, distribution of welfare benefits, matching tools, mobile phone data extraction, and electronic monitoring.³³ Due to such a rich variety of advanced technological systems, this field is called a ‘human laboratory’,³⁴ where people are used as ‘test subjects’ for such systems.

As a case study, the ‘iBorderCtrl’ project funded under EU Horizon 2020 over a thirty-six-month period can be considered in this regard. The main objective of this project is ‘to enable faster and thorough border control for third country nationals crossing the land borders of EU Member States (MS), with technologies that adopt the future development of the Schengen Border Management’.³⁵ The project brings together many technologies including biometric verification, automated deception detection, document authentication, and risk assessments in one system.³⁶ The project team evaluated these technologies with the border control officers’ assistance from Hungary, Latvia, and Greece, who were the three end-users of the project.³⁷ However, the project has received severe criticism from digital rights journalists,³⁸ scholars,³⁹ and civil society organizations.⁴⁰ They have highlighted concerns regarding the technology’s accuracy as well as issues related to bias, discrimination, privacy, due process, and procedural fairness.

Furthermore, in 2018, ‘Homo Digitalis’, an organization focusing on the protection of digital rights in Greece and a member of European Digital Rights, filed a petition to the Greek Parliament regarding the pilot implementation of the iBorderCtrl project on the Greek border.⁴¹ They underlined the concerns regarding the lack of transparency and trust in the actual capabilities of the AI systems employed in the project. The petition also underscored the high risk of discrimination against individuals based on specific categories of personal data.⁴² However, the ‘black-box’ aspects of the project have not been unlocked. The project was filed before the CJEU to disclose information about the ethics reports and the legal assessments regarding the technological system concerned and how it works.⁴³ The General Court argued that the public interest justifies the disclosure of the relevant documents:

there is a public interest in participating in an informed, open, and democratic debate regarding the question, whether control technologies, as the one mentioned, are desirable, if they should be funded via public money, and that this public interest must be duly respected.⁴⁴

However, the Court arguably concluded that the public interest in disclosing information should begin only after the completion of research.⁴⁵ It is also important to note that in the case of an ADM use with law enforcement purposes such as prevention, detection, or prosecution of criminal offences, that ADM is subject to the Law Enforcement Directive⁴⁶ (lex specialis) which provides lower standards compared to the General Data Protection Regulation (GDPR) in terms of transparency and data protection rights.⁴⁷

The use cases examined above have proved that ADM systems are more than just technological systems. They are social systems that mediate social institutions and structures.⁴⁸ They are used in different social, public, and human services. In particular, AI-driven ADM systems for deciding on social benefits, making risk assessments and producing risk scores of individuals, and identifying, categorizing, and detecting individuals and their behaviours or their emotions⁴⁹ clearly affect and form social structures and institutions.

In the field of philosophy of technology, the social aspect of AI-driven ADM systems is identified through the notion of relational ethics.⁵⁰ According to the relational understanding of AI, AI is able to recognize the interaction between people and technology, and how complex infrastructures are affected by society and by human behaviours. In other words, AI-driven ADM systems have an impact on people, interpersonal interactions, and society as a whole because they are able to recognize these social components of their environment. Therefore, the notion of relational ethics proposes that AI should be considered as a socio-technical system that is much more than an automation technique. In this regard, it suggests an investigation within the dynamics of the situation in which the decision is taken to see what is ‘right’.⁵¹

The relational understanding of AI helps separate the technical and social aspects of AI, although the two are closely intertwined. While the technical aspect is related to the black-box aspects or decision trees of an AI system, thereby presenting an epistemic problem, the social aspect is related to the data it collects. Indeed, the data possesses a social dimension, given that it originates from societal sources. The AI systems examined above obtain social data, such as social expressions, behaviours, events, emotional reactions, and common patterns or mistakes. This overview underscores a crucial aspect of AI systems: namely, individuals are not only subjected to their own data regarding their own actions and choices but also to the aggregated data that is collected from similarly situated individuals and weaves social contexts of individuals. In the context of AI systems used in the field of migration and border control, this social context is widely built, including different publics coming from different countries.

In other words, the judging framework of AI-driven ADM systems is built on the basis of actions and behaviours not attributable to a single individual. This situation means that the outcome of an ADM system examined above will never be a personal decision but a social one that encompasses not only a single social community but diverse communities. Therefore, it is necessary to protect not only individual interests but also collective interests.⁵² This situation proves that such systems have the potential to produce significant consequences for individuals, minorities, and society in general. They are particularly sensitive in terms of the protection of fundamental rights and can pose life-changing social consequences.⁵³ Therefore, it is imperative to engage human input in decision-making processes to avert such outcomes. This situation necessitates a critical assessment of the human-centric provisions of the relevant EU legal instruments to understand whether they adequately facilitate human participation in decision-making processes.

European legal instruments on digital technologies acknowledge the importance of the human factor in the era of automation.⁵⁴ Despite being in its early stages, European digital legal framework is strongly committed to ensuring that humans play an active role in decision-making. This human-centric approach sets Europe apart on a global scale, distinguishing it from the United States and China.⁵⁵ While the United States is adopting a market-driven approach and China is promoting a state-driven approach, the EU is pursuing a rights-driven human-centric approach.⁵⁶ On 26 January 2022, the Commission also published the European Declaration on Digital Rights and Principles for the Digital Decade defining the European position on digital transition as ‘putting people at the centre’ which emphasizes that rights and freedoms should be duly respected online—just as they are offline.⁵⁷

The reflection forms of this perspective and its connection with ADM systems can be found both in the GDPR and the draft AI Act. Under Article 22 of the GDPR,⁵⁸ data subjects have the right not to subjected to decisions with legal and ‘significant effects’ ‘based solely on automated decision-making’ or profiling.⁵⁹ This means that the GDPR prohibits in general automated decision-making that does not involve meaningful human intervention. It only allows such decision-making in specific circumstances, according the conditions set in the second paragraph: (i) if it is necessary for contractual aims, (ii) if it is authorized by Union or Member State law, or (iii) if it is based on the data subject’s explicit consent. When automated decisions are exceptionally allowed in one of these described circumstances, the data controller shall implement safeguarding measures for the data subject, such as the right to be informed, the right to obtain human intervention, and the right to challenge the decision.⁶⁰ Furthermore, the GDPR also limits the use of sensitive data in ADM systems to mitigate potential discriminatory effects. Processing such kind of data⁶¹ is only permissible with the explicit consent of the data subject or a substantial public interest.⁶²

However, legal scholarship has underlined that the wording of Article 22 leaves an extensive room for interpretation, and that interpretation plays a key role in clarifying its scope.⁶³ In particular, the question of whether there is a meaningful human intervention in an ADM process that can circumvent the prohibition defined in Article 22 can only be understood on a case-by-case basis.⁶⁴

While this situation poses a critical challenge for judges in terms of setting clear and consistent interpretations, the very fact that their interpretations hold decisive authority reveals the significance of judicial interpretation on this issue.

Another critical legal instrument of the EU, the draft AI Act,⁶⁵ also mandates ‘human oversight’ requirements to ensure that fundamental rights of individuals are protected:⁶⁶

Human oversight shall aim at preventing or minimising the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular when such risks persist notwithstanding the application of other requirements set out in this Chapter.⁶⁷

Moreover, in the fourth paragraph of this provision, the draft AI Act recognizes individual autonomy by authorizing the human person who is responsible for human oversight to ‘decide, in any particular situation, not to use the high-risk AI system or otherwise disregard, override or reverse the output’.⁶⁸

However, the normative power of the Article is not sufficient to achieve this purpose, as it does not consider the difference between AI-driven ADM systems and human beings in terms of ‘cognition’.⁶⁹ It is obvious that humans are not capable of examining the whole entire data mining process nor validating the outputs of AI systems in a meaningful way.⁷⁰ The human person might only detect obvious failures. This situation also makes it difficult to detect human gender bias replicated in the ADM system.⁷¹ Therefore, human oversight provisions alone cannot be considered as an effective resort for the fundamental rights challenges that ADM systems pose. Alone, they can neither legitimate the use of the ADM system nor the decisions that the system takes. In fact, the system cannot be considered a legitimate device in a democratic society unless its use is proven legal, necessary, and proportionate.⁷²

However, the human oversight requirement is still crucial and can be advanced as one of the ways for humanizing the digital government. Indeed, the proposal warns the human person who is responsible for the oversight to be ‘aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system’.⁷³ This tendency refers to ‘undue deference to automated systems by human actors that disregard contradictory information from other sources or do not (thoroughly) search for additional information’,⁷⁴ known as ‘automation bias’.⁷⁵ It is promising that the proposal is aware of this tendency.⁷⁶ Still, other aspects must be considered. In future iterations, the AI Act should also consider that the excessive digitalization of government services and the automation of decision-making might exclude the unique human feature of forgiving trivial mistakes, such as misspelling names or dates when filling out benefit applications or tax returns.⁷⁷ Overall, a rigorous approach must consider that the system itself might have bias, the human person who is reviewing the system might also have bias (‘automation bias’), and the individual who is subject to the system might make trivial mistakes due to, for instance, digital illiteracy or ignorance. The current version of the proposal is far from addressing this standard as it does not sufficiently consider the first and last points presented here. However, in the following section, the chapter explains that judicial interpretation plays a critical role in providing guidance for such concerns.

In this section, the chapter focuses on four leading judgments on the concrete ADM practices that have been observed in the EU Member States. The purpose is not to discuss all the ADM-related cases observed in the EU but rather shed light on the role of courts in clarifying the socio-technical and legal aspects of automation and the human factor in decision-making processes. In this regard, the section examines the SyRI, Buona Scuola, Schufa, and Uber cases, respectively, which surfaced public and private contexts in the Netherlands, Italy, and Germany.

The SyRI case from the Netherlands stands out as one of the most significant tech-related cases in the world.⁷⁸ It provides a clear example of the global trend towards the digitalization of the welfare state and the legal concerns surrounding it. As the UN Special Rapporteur on extreme poverty and human rights noted in 2019, the ‘welfare state is gradually disappearing behind a webpage and an algorithm, with significant implications for those living in poverty’.⁷⁹

The Dutch government is the first government in the EU to have applied AI-driven digital welfare technologies and, as a result, to have violated the rights of individuals. On the 5 February 2020, the District Court of the Hague (Rechtbank Den Haag) ruled that the use of the SyRI algorithm system (‘System Risk Indication’), a digital welfare fraud detection system applied by the Dutch government, violated Article 8 of the European Convention of Human Rights (ECHR),⁸⁰ which guarantees the right to respect for private and family life, home and correspondence.⁸¹

According to the Dutch Legislator, SyRI was a technical infrastructure linking and analysing data anonymously, with the ability to generate risk reports that address legal or natural persons considered ‘worthy of investigating with regard to possible fraud, unlawful use and non-compliance with legislation’.⁸² Certain bodies of the Dutch government applied this algorithm in collaboration by exchanging data to identify the perpetrators of related abuses.

The claimants, several human rights activists and non-governmental organizations, stated that the national legislation on SyRI does not have sufficient safeguards for the protection of private life, with the result that its binding effect was deemed invalid. The District Court of the Hague reviewed the algorithm’s legislation and the usage of the algorithm by the Dutch government mainly based on Article 8 of the ECHR, the Charter of Fundamental Rights of the European Union (CFR), and the principles established in the GDPR, particularly the principle of transparency, the principle of purpose limitation, and the principle of data minimization. In this context, the Court analysed the ‘extent and seriousness of the interference’ with Article 8 of the ECHR based on the SyRI Legislation and the information about the algorithm provided by the State.⁸³

To identify the scope of interference, the Court focused mainly on the functioning of the algorithm. The Court concluded that the SyRI legislation did not provide sufficient information about the functioning of the system particularly related to the risk models consisting of risk indicators, risk analysis methods applied in the system, and the generation of the decision trees. Therefore, the Court found a violation under Article 8 of the ECHR on the basis of lacking information about the system in the SyRI legislation.⁸⁴ The main problem, which led to a violation of the Convention, is that the function of the SyRI has remained opaque in the legislation. The problem of lack of transparency in the legislation also illustrates the concentration of private power behind the system.⁸⁵

The claimants also referred to Article 22 of the GDPR. They argued that ‘the submission of a risk report . . . can be considered a decision with legal effect, or at least a decision that affects the data subjects significantly in another way, and that this decision is taken on the basis of automated individual decision-making within the meaning of Article 22 GDPR, which is prohibited’.⁸⁶ The Court agreed with the claimants that a risk report had a ‘“significant effect” on the private life of the person to whom the risk report pertains’.⁸⁷ However, it noted that such a risk report did not have the legal effect. The Court did not ‘give an opinion on whether the exact definition of automated individual decision-making in the GDPR and, insofar as this is the case, one or more of the exceptions to the prohibition in the GDPR have been met. That is irrelevant in the context of the review by the court whether the SyRI legislation meets the requirements of Article 8 ECHR.’⁸⁸ To conclude, the Dutch Court in its SyRI judgment clarified that legislation on ADM systems should articulate the functioning of ADM systems in clear terms.

Another significant judicial case on ADM systems has been observed in Italy. The ADM practice was concerned with using a teacher placement algorithm (known as ‘algoritmo della buona scuola’, ‘good-school algorithm’ in English),⁸⁹ which sparked extensive public debate and prompted public administrative decisions in 2019.⁹⁰ In this case, the Italian Ministry of Education used software to make efficient and swift decisions on the placements of newly selected teachers and process the mobility requests of already employed teachers. According to the Mobility Rankings 2016, the algorithm made structural mistakes by assigning thousands of teachers⁹¹ to incorrect professional placements in practice.⁹² Furthermore, according to the Algorithm Watch, the system automictically compelled some teachers with autistic children to relocate from the southern region of Calabria to Prato, in the northern region of Tuscany.⁹³

Two critical judgements on this issue have offered significant legal interpretations making concrete the principle of transparency and the human factor. First, in April 2019, the Italian Council of State (Consiglio di Stato) found that ‘the use of “robotic” procedures cannot justify circumventing the principles that shape our legal system and regulate the conduct of administrative activities’.⁹⁴ In this regard, the algorithm, which has a legal value, must comply with the general principles of administrative activity, such as transparency, reasonableness, and proportionality.⁹⁵ Furthermore, the Council of State interpreted the transparency principle that requires ‘the full knowability of any rules expressed in a language other than the judicial one’.⁹⁶ This ‘full knowability’ (‘piena conoscibilità’) includes the decision-making procedure and the relevant data of that system in order to verify whether the outcomes of the ‘robotic procedure’ comply with the legal requirements.⁹⁷ It is crucial to emphasize that the Council of State does not advocate for the complete disclosure of the system’s code in question. Instead, it calls for a clear explanation of its ‘technical formula’ that both judges and citizens can comprehend.⁹⁸

In September 2019, the second key judgment came on this issue from the Administrative Court of Lazio (Tribunale Amministrativo Regionale del Lazio). The Court focused on the human factor and pointed out that human judgment is irreplaceable, and automation may only play ‘a merely auxiliary and instrumental role’, rather than taking a ‘dominant or surrogate’ position within the administrative process:⁹⁹

informatics procedures, even when they reach their highest level of precision and even perfection, they can never fully replace, truly supplant, the cognitive, inquisitive, and judgmental activities that only an inquiry entrusted to a physical person is capable of performing.¹⁰⁰

According to the Lazio Court, this interpretation is in line with the Italian Constitution and Article 6 of the ECHR,¹⁰¹ which prevents a ‘deleterious Orwellian perspective’ where the decision-making is entirely handed over to machines.¹⁰² While the Lazio Court did not concretize Article 6 of the ECHR concrete in the present case, the ‘principle of good governance’ is considered in the case law of the Strasbourg Court:

the principle of ‘good governance’ requires that where an issue in the general interest is at stake it is incumbent on the public authorities to act in good time, in an appropriate manner and with utmost consistency.¹⁰³

To conclude, considering these arguments, the Lazio Court underlined that algorithms should serve as supporting tools in public decision-making rather than assuming a primary role.

The right not to be subject to automated decisions was considered for the first time in the Schufa case before the CJEU on 16 March 2023. The Schufa is a private German credit information agency responsible for evaluating the trustworthiness of customers seeking any contractual relationship including loans, mortgages, or house rentals through profiling their financial behaviours.¹⁰⁴ Based on that profiling, Schufa issues a certificate with a score and provides a positive or negative result about the person applied.¹⁰⁵ However, the company offers no reasonable or understandable reasoning about its evaluation. In other words, it does not disclose how the score is calculated.¹⁰⁶

In 2018, an applicant who received a negative score requested Schufa to provide additional information about the negative result. Considering the underlying logic of their automated system as commercial and industrial secrecy, Schufa provided only the basic functioning of its automated system. The applicant waited for information about Schufa’s profiling for two years despite filing a complaint with the German Data Protection Authority. Subsequently, the applicant appealed the decision before the Administrative Court of Wiesbaden (Verwaltungsgericht Wiesbaden). In October 2021, the Wiesbaden Administrative Court (‘referring court’) stayed the administrative proceedings and referred to the CJEU two questions regarding the interpretation of Article 22 of the GDPR, the right not to be subject to automated decision-making which is granting data subjects the right ‘not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her’.¹⁰⁷ Hence, the German Administrative Court initiated the first example of such consideration before the CJEU.

The first matter focuses on clarifying the financial activity conducted by Schufa and questions whether credit scoring is an automated decision. Before analysing the matter, however, the Advocate General (AG) first emphasizes the distinctive character of Article 22(1) and states that that provision ‘establishes a general prohibition on decision of the kind described’ rather than a right to be invoked by the data subject.¹⁰⁸ In terms of interpretating Article 22 GDPR the AG suggests that:

[t]‌he automated establishment of a probability value concerning the ability of a data subject to service a loan in the future constitutes a decision based solely on automated processing, including profiling, which produces legal effects concerning the data subject or similarly significantly affects him or her, where that value, determined by means of personal data of the data subject, is transmitted by the controller to a third-party controller and the latter, in accordance with consistent practice, draws strongly on that value for its decision on the establishment, implementation or termination of a contractual relationship with the data subject.¹⁰⁹

In this regard, the AG argues that the scoring is considered as profiling within the meaning of Article 4(4) of the GDPR since the procedure in question ‘uses personal data to evaluate certain aspects concerning their economic situation, reliability and probably behaviour’.¹¹⁰ Secondly, the AG argues that the refusal of a credit has both legal and significant effects on the data subject since the data subject can no longer benefit from a contractual relationship with the financial institution and is affected significantly from a financial point of view.¹¹¹ This means that the action in question may have an impact that is not only legal but also economic and social.¹¹²

Thirdly, the AG questions what is the relevant ‘decision’ in the case at issue and underlines that in the decision-making process, there are multiple phases such as profiling, the establishment of the score, and the actual decision on the grant of credit.¹¹³ He then highlights that the scoring by Schufa is a ‘decision’ within the meaning of Article 22(1) of the GDPR since it ‘tends to predetermine the financial institution’s decision to grant or refuse the credit to the data subject, such that this position must be considered only to have purely formal character in the process’.¹¹⁴ According to the AG, the crucial factor is the effect that the ‘decision’ has on the data subject.¹¹⁵ Considering that a negative score alone may produce negative impact on data subjects by restricting their freedoms and stigmatizing them in society, it makes sense to qualify that score as a ‘decision’ when a financial institution gives it paramount importance in the decision-making process.¹¹⁶

He concludes that in such circumstances, ‘credit applicants are affected from the stage of the evaluation of their creditworthiness . . . not only at the final stage of the refusal to grant credit, where the financial institutions is merely applying the result of that evaluation to the specific case’.¹¹⁷ It is also worth noting that the referring court states similar aspect: ‘experience from the data protection supervision carried out by the authorities shows that the score plays the decisive role in the granting of loans’.¹¹⁸

He further considers the purpose of the EU Legislator through Article 22, which is to protect the rights of data subjects, and states that a restrictive interpretation of that provision would create a gap in legal protection where data subjects cannot exercise their rights and freedoms, particularly described in Articles 15(1)(h), 16, and 17 of the GDPR.¹¹⁹

Furthermore, the AG clarifies the content of Article 15(1)(h) regarding the obligation to provide ‘information about logic involved’. He states that this information covers the calculation method used by a credit information agency unless there are no conflicting interests that are worthy of protection such as the right to protection of intellectual property under Article 17(2) of the CFR.¹²⁰ In light of joint reading of Recitals 58 and 63 and Article 12(1) of the GDPR, the AG concludes that ‘the obligation to provide ‘meaningful information about the logic involved’ must be understood to include sufficiently detailed explanations of the method used to calculate the score and the reasons for a certain results’.¹²¹ Moreover, considering the complexity of algorithms, the AG emphasizes that the principle of transparent information and communication in Article 12 of the GDPR does not establish any obligation for the controller to disclose the algorithm since there is no benefit of commutating a complex formula without providing a necessary explanation.¹²²

Finally, to reply to the second question posed by the referring court, the AG explains that Article 6(1) and Article 22 do not prevent domestic legislation from profiling as long as it falls outside the scope of Article 22 of the GDPR. However, in this case, the national court must comply with the requirements outlined in Article 6 of the GDPR which includes relying on an appropriate legal basis. The AG Opinion holds notable importance as it marks the initial judicial interpretation of the legal term ‘automated decision’, clarifying that if an algorithm predominantly influences decision-making, the activity of that algorithm qualifies as an ‘automated decision’ within the meaning of Article 22 of the GDPR.

On 4 April 2023, the Court of Appeal in Amsterdam (Gerechtshof Amsterdam) found that several automated processes including assigning rides, calculating prices, rating drivers, calculating ‘fraud probability scores’, and deactivating drivers’ accounts in response to suspicions of fraud on Uber’s and Ola platforms are considered as automated decisions in three judgments.¹²³ In particular, the Court’s judgment on the deactivation decisions taken against Uber drivers has been crucial with regard to ADM systems and human participation. In this case, the Dutch Court has argued whether the deactivation decision taken against Uber drivers, which means they can no longer work through Uber, are automated decisions.¹²⁴

First, the Court has considered the privacy statement of the company, which confirms that Uber makes an ‘automated decision’ when deactivating users ‘who are identified as having engaged in fraud’.¹²⁵ Secondly, Uber has explained that Uber’s Risk Team relies on software to automatically detect various fraudulent activities, such as when a driver repeatedly cancels rides within a short time period, which may suggest ‘cancellation fraud’.¹²⁶ According to the Court of Appeal, this example has showed that Uber ‘involves automated processing of personal data of drivers whereby certain personal aspects of them are evaluated on the basis of that data, with the intention of analysing or predicting their job performance, reliability and behaviour. As such, this processing meets the definition of profiling as contained in Article 4(4) of the GDPR.’¹²⁷ Thirdly, the Court has considered that the deactivation decisions addressed to drivers are worded in a very general manner without mentioning any concrete conduct that forms the basis of decisions.¹²⁸ Furthermore, the Court found that the limited human intervention in Uber’s automated decisions to dismiss workers was not ‘much more than a purely symbolic act’ considering also the fact that the Risk Team of the company is based in Kraków, Poland.¹²⁹ In other words, the Dutch Court clarified that human intervention should have a meaningful contribution to the decision-making process rather than a simply symbolic participation. Table 7.3 sketches all four cases examined in this chapter and illustrates the key legal provisions and their findings.

Similarly to the bricolage activity, judicial interpretation involves navigating a complex landscape of normativities that may not always appear seamlessly fused or unified.¹³¹ When judges engage in interpreting laws, regulations, and legal principles, they often encounter a mosaic of norms and precedents, each with its own distinct nuances and interpretations. This process provides for making concrete the relevant legal norms and clarifies their rules.¹³² The integration of algorithms into public and private decision-making processes and the mosaic landscape of Article 22 of the GDPR covering both public and private decision-makers have cascaded the complexity of this task, necessitating an intricate interplay between technological and legal components.

The judicial cases examined in this chapter have evinced that judicial interpretation is highly crucial for understanding both the socio-technical, and legal aspects of automation and the human factor in decision-making processes. Ultimately, the chapter has identified three aspects of judicial interpretation on ADM practices: (i) epistemic, (ii) substantial including socio-technical and legal aspects, and iii) methodological.

From an epistemic point of view, in all four cases judges struggled to describe the functioning and the purpose of the algorithm at stake. They tried to understand where, how, and for what purpose ADM systems have been used by the relevant public or private actors rather than understanding technical or computer science-related features of a particular system. In this sense, their judicial interpretation had started from defining the digital system at stake, namely defining the functioning of fraud detection, credit scoring, teacher placement, and dismissing workers, in the cases examined in this chapter.

From a substantial point of view, the courts have clarified socio-technical and legal aspects of automation and the human factor in decision-making-processes. On the socio-technical side, the courts have demonstrated that it is necessary to have clear legislations on the functioning of ADM practices to ensure that the system at stake is explainable to human persons. In the SyRI case, the Dutch court has found that the relevant legislation did not provide sufficient information about the functioning of the fraud detection system particularly related to the risk models consisting of risk indicators, risk analysis methods, and the generation of the decision trees. It is also worth noting that understanding the basic functioning of the algorithm has become highly significant to determine the ‘extent and seriousness’ of individual rights interferences by ADM systems.

The Italian courts have also highlighted similar concerns in the Buona Scuola case regarding the teaching placement algorithm. The Italian Council of State has underlined the need for a clear explanation regarding the ‘technical formula’ of a particular ADM system to ensure the general principles of an administrative activity, such as transparency, reasonableness, and proportionality. On the same issue, the Lazio Court focused on the human factor and underlined that automation can only play an auxiliary role in decision-making rather than a dominant position. In this sense, both the SyRI and the Buona Scuola cases have clarified that (i) legislation should articulate the functioning of an algorithm in human-readable terms rather than complex computer codes, and (ii) algorithms should serve as supporting tools in public decision-making rather than assuming a primary role. The two arguments prove that in both cases involving public uses of ADM systems, the courts have largely emphasized public law principles while assessing the ADM system at stake such as the principles of legality and transparency.

From a legal perspective, the AG in the Schufa case and the Dutch Court in the Uber case have focused on the meaning of the ‘automated decision’ and the human intervention measure. In the Schufa case, the AG has clarified that ‘credit scoring’ is an ‘automated decision’ within the meaning of Article 22 of the GDPR when a financial institution gives it paramount importance in the decision-making process. In the Uber case, the Dutch Court has underlined that the deactivation decisions addressed to drivers are worded in a general manner without mentioning any concrete conduct that forms the basis of decisions and the limited human intervention in Uber’s automated decisions to dismiss workers was not ‘much more than a purely symbolic act’. In this sense, both aspects have clarified that (i) if an algorithm plays a primary role in decision-making, the activity of that algorithm is considered as ‘automated decision’ within the meaning of Article 22 of the GDPR, and (ii) the human intervention should have a meaningful contribution to the decision-making process rather than a simply symbolic participation. Both arguments demonstrate that in instances involving private uses of ADM systems, the courts have predominantly focused on elucidating how the ADM system in question was employed in decision-making by private entities.

From a methodological point of view, the judges’ inquiry regarding automation and meaningful human participation has presented an interactional legal ground where the relevant normative actors interact with one another. The courts have considered the normative aspect of automation, the relevant provisions of the ECHR, the GDPR, and the relevant domestic legislation to clarify the roles of automation and humans in decision-making processes. In this sense, their reasoning has not been limited to scope of national legislation, but also included supranational and international legal provisions.

Ultimately, the three aspects of judicial interpretation, epistemic, substantial, and methodological, have proved the pivotal role that judges play in comprehending automation and ensuring meaningful human participation in decision-making processes. In doing so, judges have not only narrowed the divide between machines and humans but also law and digital society.