Has the GDPR hype affected users’ reaction to cookie disclaimers?

For many years, cookies have been widely used by websites, storing information about users’ behaviour. While enabling additional functionality and potentially improving user experience, cookies can be a threat to users’ privacy, especially cookies used by third parties for data analysis. Websites providers are legally required to inform users about cookie use by displaying a so-called cookie disclaimer. We conducted a survey study in 2017 to investigate how users perceive this dis-claimer and whether it affects their actual behaviour. We found that while most participants had negative feelings towards the disclaimer, the disclaimer text had no signiﬁcant effect on their decision to leave the website. Since the extensive media coverage of data protection issues that accompanied the EU General Data Protection Regulation (GDPR) entry into force in May 2018 may have sensitized users to privacy protection, we conducted a follow-up study in December 2018. Our results suggest that users did not change their attitude towards cookie use in favour of privacy protection, but got even more accustomed to the use of cookies, also by third parties. Moreover, many users seem to have misconceptions regarding cookie use. We discuss the implications of our results for the users’ right to make an informed decision about their privacy.


Introduction 1
Since 1994, cookies have been commonly used on websites.Originally introduced in order to remember stateful information on the websites and with this provide better user experience and additional functionality, the usage of cookies has since evolved to include data collection from the user.Such data collection can threaten the users' privacy.As the EU Data protection directive [1] prescribes informing the users regarding the use of cookies on the website, service providers include a corresponding disclaimer on their website (see Fig. 1).
Yet, research from related domains [2][3][4] shows that privacy and security notices are often ineffective in their purpose.As such, they often fail to provide the necessary information to the users in an understandable way, in order to enable the users to make an informed decision.Furthermore, these notices often fail to empower the user, not providing them with meaningful choices and measures for protecting their privacy.As the result, users often ignore the notices not perceiving them as useful, or make decisions based on them without being aware of the consequences.
The goal of our work is to study the effect of the cookie disclaimer as a privacy notice on users.As such, in order to see whether the disclaimer succeeds in informing the users and empowering them in making decisions regarding their privacy, we consider the following research questions: 1. How do the users perceive the usage of cookies by the website provider when confronted with the disclaimer? 2. How do the users react to the displayed disclaimer? 3. Which factors influence the decisions of users regarding their surfing behaviour, when confronted with the cookie disclaimer?
In order to answer these questions, we have first conducted an explorative study in form of an online survey with 150 participants.We provide both qualitative and quantitative analysis of the data collected within the study.The results of our study conclude, that a large number of the participants considered the cookie disclaimer as a nuisance in their surfing rather than useful means for providing information about the cookie usage.The study furthermore revealed that the text of the disclaimer did not play a significant role in users' decision.Instead, more important factors were the reputation of the website and the type of service it provides.At the same time, many participants claimed to have privacy concerns regarding cookies.Specifically, concerns were raised regarding the lack of transparency on how the data collected via cookies is used by the service provider.
As the original study was conducted in 2017, we also ran a study in December 2018 to investigate whether users' attitude towards and reactions to the cookie disclaimer changed due to the introduction of the EU General Data Protection Regulation (GDPR) [5] and the accompanying media coverage of the privacy topic, both replicating parts of the study from 2017 and extending the original study by including several additional follow-up questions.Our study concludes that the GDPR did not lead to users being more concerned about their privacy due to cookie disclaimers, nor did it lead to users rejecting the cookie collection more often.
The article is structured as follows.In the 'background' section we provide the background information for our article.Next we provide an overview of different groups of cookie disclaimers that were commonly used on websites at the time our studies were conducted.In the next section, we describe the studies we performed, followed by the description of the results from the original study.The following section describes the results of the follow-up study as well as their comparison to the original study.We discuss our results and their possible implications followed by conclusion of the article.

Background
Cookies are small text files that are stored by the browser upon visiting a website.In this way, the website provider can store information on the users' computer, such as their login data, that can be accessed next time the user visits the website.As such, use of cookies provides advantage to the users, enabling certain functionality of the website.Additionally, the website provider can use cookies in order to collect information about the users and their behaviour on the website.This information, in particular, can be used for creating user profiles in order to personalize the advertisements shown to the users.
One distinguishes between two types of cookies: session cookies, and so-called persistent cookies.Session cookies, used, for example, in order to store the items in a user's shopping cart in an online store, are deleted as soon as the browser is closed.Persistent cookies, on the other hand, remain on a user's computer until they are explicitly deleted by the user, and can specifically be used for analysing the behaviour of the user.Such cookies, in particular, can be stored by both website providers as well as third parties, such as ad networks-the so-called third-party cookies.

Groups of cookie disclaimers
At the time the original study was performed (in 2017), there were no uniform prescription regarding which information should be provided in the cookie disclaimer in the EU law [1], aside from the bare statement that cookies are used.As such, a variety of such disclaimers have been used on the websites.In order to determine which disclaimers are most commonly used, we have studied the 50 most popular websites in Germany (according to the Alexa rating 2 ).The disclaimers on these websites can be classified into five groups, depending on the content the disclaimer provides.
Group 1: The first group includes the disclaimers that provide only the minimal required information, namely, that the website  uses cookies.A disclaimer from such a group can be seen, for example, on the Amazon website (see Fig. 2).
Group 2: The second group includes the disclaimers that mention that cookies are used in order to improve the services provided by the website.An example of such a disclaimer can be seen on the Paypal website (see Fig. 3).
Group 3: The disclaimers in the third group mention that the cookies are being used for analysis purpose, often also mentioning personalized adds or other services enabled by such an analysis.A disclaimer from this group can be seen on the Microsoft website (see Fig. 4).
Group 4: The disclaimers in the fourth group are characterized by their mentioning of third parties, i.e. partners of the website provider, that the cookies are used by.Similar to the disclaimers from Groups 2 and 3, the disclaimers from Group 4 may include mentioning of personalized ads or service improvements.An example for a disclaimer from this group is displayed on the Twitter page, see Fig. 5.
Group 5: The fifth group includes the disclaimers that furthermore refer to the external use of cookies by mentioning that the cookies are also used outside of the website the user visits.Similar to the disclaimers from Groups 2, 3 and 4, service improvements are often mentioned.Such a disclaimer is displayed on the Facebook website (see Fig. 6).

Study design
This section describes the original and the follow-up studies.The original study, also reported in [6], was conducted in December 2017, before the introduction of the GDPR in the EU in May 2018.The follow-up study was conducted in December 2018.Both studies together therefore allowed us to compare users' perception of cookie disclaimers before and after the introduction of the GDPR, since users may have changed their opinion about data protection in general or more specifically regarding cookies due to the extensive media coverage of this topic that accompanied the introduction of the GDPR.Below, we describe in detail how the studies were designed and what parts of the study differed between the original and the follow-up study.
We used the Clickworker crowdsourcing platform 3 for both studies.The participants were recruited from Germany and received 1, 80 Euro as their compensation.The studies each consisted of four parts, the first three of which were the same in both the original and the follow-up studies, and the fourth part was modified.The first part of the studies, the 'general part', consisted of general questions about participants' experience with the cookie disclaimer.As such, they were asked whether they remembered encountering cookie disclaimers while surfing.They were then asked how they felt seeing the disclaimers they encountered, and how they reacted upon it.They were furthermore asked whether their feelings or reactions have changed with time, or whether they differed depending on the website or the platform.This part was implemented similarly in both studies.
The second part of the studies, the 'disclaimer-specific part', has been designed in order to consider the differences between various kinds of cookie disclaimers.For this purpose, we considered disclaimers of different kinds that can be found on websites.According to our findings described in 'Groups of cookie disclaimers' section, we provided five disclaimers, one from each group of disclaimers described in the section.The disclaimers are provided at Table 1.Each participant has been randomly assigned to one of the groups.The participants then were given a text of a disclaimer from the corresponding group and asked what their thoughts would be upon seeing the disclaimer.They were then asked whether the disclaimer would lead to them leaving the website and asked to explain their answer.The next question asked the participants to evaluate on a scale from 0 to 100, how likely would it be for them to look for additional information regarding use of cookies, e.g. by clicking on the link provided in the disclaimer.Finally, the participants were asked in which cases the disclaimer would lead to them either leaving or staying on the website.
For the third part, the 'disclaimer ranking part', all participants were shown the disclaimers from all five groups and asked to rank them depending on how likely a particular disclaimer would lead the participants to leaving the website.The participants were then asked to explain their ranking.
The fourth part differed between the two studies.In the fourth part of the original study, six additional disclaimers, which were composed specifically for the study have been evaluated.Since we did not find much differences in the participants' evaluations of the new disclaimers (see the description of the results in [6]) and we further did not expect these evaluations to change due to the introduction of the GDPR, we decided to replace this part with another set of questions for the follow-up study.In the fourth part of the follow-up study, the 'mental model part', we thus showed participants a screenshot of a disclaimer that offered different possibilities for the user to respond: accept, decline, close the window, get further information, or open the dialogue to manage one's privacy settings.Participants were then asked to explain (in free text) what they thought would happen if they clicked on either of the different answer options, or ignored the disclaimer and continued surfing on the website.They were further asked about their opinion about a cookie disclaimer that needs to be answered by accepting or declining it before the website can be accessed.At the end of both studies, demographic data was collected, including gender, age, profession and experience in IT-security in years.

Original study results
In this section we provide the results of our original study evaluation.In this, for the sake of brevity we omit the findings from the part of the original study that was not involved in the follow-up study and the comparison, referring to the description of these findings in [6].
For the qualitative evaluation, an open coding approach has been applied for the answers of the original study.Note, as some of the coding was revisited after analysing the follow-up study (see 6, we adjusted the coding of the original study accordingly, so the results may differ from those reported in [6] in this respect).The answers of both studies have been coded by two of the paper authors.We provide quotes from the participants' answers in order to illustrate our findings.

Demographics
A total of 150 persons participated in the study, of them 73 females, 75 males and 2 participants who did not specify their gender.The age distribution among the participants is provided in Table 2.A majority of the participants (106 out of 150) claimed to have no ITsecurity experience, while the remaining 46 participants had between 1 and 28 years of experience, with a median of 5 years.

Users' perception of cookies
In order to answer the first research question, we coded the answers both from the first and second part of the survey, that is, from the general part and the disclaimer-specific part.We considered the answers to the following questions: • General part: [For the disclaimers the participants recalled encountering] What thoughts or feelings did you have while reading the disclaimer?• Disclaimer-specific part: [For one of the disclaimers G1-G5] What thoughts or feelings did you have while reading the disclaimer?• Disclaimer-specific part: [After asking, whether the disclaimer motivate the participant to leave the website?]Please explain your answer.
The answers from the participants over the study can be categorized into the following categories: 'disturbance, privacy concern, habituation, misconceptions, lack of information'.We elaborate on the categories below, providing examples as the quotes from the participants 4 .

Disturbance
A large number of the participants claimed to be annoyed by the cookie disclaimer, as they considered it a disturbance in their surfing: 'As these messages appear constantly, I find them to be disruptive and annoying.'

Privacy concerns
Another common theme was the concern of the users regarding their privacy: 'I feel myself observed.' These concerns have been mentioned in a variety of ways, ranging from abstract feeling of uneasiness ('As I read it the first time, I had a bad feeling') to participants naming concrete consequences for their privacy ('I do not want to be recognized anywhere or get fitted ads').

Factual information
Some participants described the implications of the cookie disclaimer, e.g., that their data are being collected, without expressing concerns about this fact: 'They are collecting information.'

Habituation and fatigue
Due to prominence of cookie disclaimers, many participants claimed to being used to it and not to pay much attention to the disclaimer.As such, many participants reacted in a neutral way to the disclaimer: 'It does not bother me, since cookies are a common tool.'At the same time, for some of the participants, getting used to seeing the disclaimer on the websites resulted in feelings of futility.As such, as they felt that there is no way to avoid it, they admitted to being

G4
This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us and by our partners (the so-called third parties cookies) in order to improve our service for you.

G5
This website uses cookies.By continuing to use the website you consent to the use of cookies on and off our website.
Cookies are used by us in order to improve our service for you.
The underlined text simulates the link to further information provided within the disclaimer.resigned in their attempts to act against it: 'As this is the case with so many websites, I don't have much thoughts anymore regarding these cookies.[. ..]One feels somewhat helpless, but I seldom have this feeling and it is not so strong.When it comes to privacy protection in the internet (where cookies also belong), I've rather resigned myself.'Similar feelings of resignation were expressed by some of the participants who ignored the disclaimer because they did not feel like they have any choice in accepting or declining the use of cookies (if they wanted to continue using the website), considering the disclaimer itself therefore useless: 'One cannot decline the cookies, therefore I find that the message does not make much sense'.

Lack of information
The answers from the participants revealed that many of them feel uncertainty as to what consequences the cookies have for their privacy.A common theme was that the participants expressed the need for more detailed information on what consequences cookie use can have for them, what data is collected and how it is used, and that the lack of such information made them feel nervous: 'It is unpleasant to me, as I do not know exactly what it means to allow cookies, and what consequences it has for me.'Some admitted being unfamiliar with the concept of cookies all together: 'Frankly speaking, I don't know exactly what cookies do.' Another theme was that the participants were unaware at what point exactly they do consent to the use of cookies: 'And I ask myself, whether cookies are set after the point when I click OK, or already earlier.' Consequently, some of the participants questioned the idea of informed consent that the disclaimer theoretically aims to provide: 'The problem [with the cookie disclaimer] is that one does not have to actively give consent, but instead consents passively by using the website.Many users will not, however, read the disclaimer correctly, or maybe they will simply overlook it.In this case there is no real informed consent given.'

Misconceptions
Aside from participants, who were aware that they lack information regarding cookie use, a number of participants had misconceptions regarding what cookies are and what the consequences of cookie use are.As such, some of them were concerned about risks that are usually not connected with cookies: 'Maybe I have a feeling that I am attacked by a virus.' Others were unaware of possible implications of cookies: 'I would not know why I should leave the website, it is not a forbidden website.'

Users' reactions to cookie disclaimers
In order to answer the second research question, we considered answers to the following questions from the general part and the disclaimer-specific part: • General part: How did you react to the disclaimer?For example, did you leave the website, got additional information . ..? • Disclaimer-specific part: What thoughts or feelings did you have while reading the disclaimer?• Disclaimer-specific part: Will the disclaimer move you to leaving the website?Please explain your answer.• Disclaimer-specific part: How likely is that you get further information by clicking on a link in the disclaimer (scale from 0 to 100)?
The answers are analysed both qualitatively via open coding, and quantitatively.The actions of the users upon encountering the disclaimer have been classified into the following categories: 'ignore, accept, deny, get informed, apply countermeasures'.We elaborate on the categories below.

Ignore
A large part of the participants claimed to ignore the disclaimer, considering it a disturbance in their surfing rather than information they should pay attention to.Other times, the participants claimed that they clicked the disclaimer away, so that they can continue surfing: 'Sometimes the displayed window irritates me and I click on close, but sometimes I leave it open.'

Accept
Other participants claimed that they decided to accept the cookies by clicking OK on the disclaimer: 'Mostly I click "OK", what else is there to do?' While many who answered did not provide any explanation why they chose to accept the use of cookies, others elaborated that they did not see any harm in cookies and considered them useful for the functionality of the website: 'Every website does this.As a programmer I know that cookies are necessary for many functions of modern websites'.
Other participants admitted accepting the cookies as otherwise they would not be able to use the website they need: 'When I visit a website, that it is mostly because I can get information or other benefits from it.So I don't leave the website'.
A similar theme has been mentioned by other participants, who chose to accept cookies as a trade-off for using the website: 'This is simply the deal online: using websites in exchange of some sort of payment (infos).' Deny Many of the participants were unhappy with the cookie use and chose different ways to deny it.

Not explicitly accepting.
A number of the participants had the perception that the cookies will not be used as long as they do not explicitly agree to it: 'I tried to click it away without accepting the cookies or ignore it when the disclaimer was not in the way.'Note that while this is true for opt-in methods, such perception might be false in many cases, as opt-out methods are commonly used.
Using countermeasures.A number of participants claimed to apply specific countermeasures that minimize the impact from cookies while still allowing to visit the website.In particular, such countermeasures as using a different browser ('I left the website or used a different browser'), deleting ('I've never left the website, but deleted the cookies after using the website') or blocking the cookies ('One can block the cookies and automatically delete them with the right add-ons') have been mentioned.
Leaving the website.Other participants claimed to prevent cookie use by leaving the website that displays the disclaimer: 'Left the website, I never accepted.' When asked directly (in the disclaimer-specific part) whether they would leave the website if they saw a disclaimer, more than half (58%) of all the participants that participated in the original study answered that they would not leave the website based upon the disclaimer, while 19% answered that they would leave and 23% were not sure (see.Fig. 7).In order to investigate the differences between the disclaimers in groups G1-G5, the answers were compared using the chi-square test (recall, that the participants in the disclaimer-specific section were randomly divided in five groups and shown a disclaimer from the corresponding group).The test did not reveal any significant differences between the groups (v 2 ¼ 4:41, P ¼ 0.82), indicating that all the disclaimers in the study had a similar effect on the participants' decision to continue using the website.

Get informed
Some of the participants mentioned in their answers, that they would try to get additional information, either from the website itself ('I clicked on "learn more"') or from external sources ('I first googled the term "cookies"') that would help them make a decision.
Yet, when answering a direct question on how likely it is that they would click on the link in the disclaimer and get additional information (on a scale 0 to 100), most of the participants in the original study indicated the likelihood to be low.As such, half of all the 150 participants considered the likelihood of them getting additional information to be less than 13%, while only 25% of them considered the likelihood to be higher than 40% (see Fig. 8).In order to compare the disclaimers in groups G1-G5, a Kruskal-Wallis test was conducted.The test did not reveal significant difference between the responses among the different groups (H ¼ 2.51, P ¼ 0.64), indicating that the different disclaimers had similar likelihood of moving participants towards getting further information.

Factors in users' decisions on cookies
We answer the third research question by studying the factors that influence the decisions of users (i.e. which reaction they choose from  the ones outlined above) after reading the cookie disclaimer.In this, we evaluate the answers to the following questions in our study: • General part: If you saw the disclaimer more than once, were your thoughts, feelings and reactions different?put the disclaimer that would most likely lead you to leave the website on the first place, put the disclaimer whereby it is least likely that you leave the website because of the disclaimer on the fifth place.Please explain your answer.
From the participants' answers we were able to distinguish between two types of factors: 'disclaimer-based' and 'website-based'.These factors are elaborated on below.

Disclaimer-specific factors
We first outline the factors related to the displayed disclaimer itself.
Design of the disclaimer.Some participants mentioned that their decision depends on the design of the disclaimer itself.As such, disclaimers that were too prominent or blocking large parts of the website contents were considered a large nuisance, hence, would lead to users leaving the website: 'It depends on whether I can ignore the disclaimer or not.Often the message is placed so unfortunately, that one cannot use certain menus.In this case I would leave [the website].' Text of the disclaimer.The purpose of the disclaimer-specific part, the disclaimer-ranking part and the new disclaimers part of our study was to determine, to which extent the text provided by the disclaimer influences the users' decisions.However, as mentioned in 5.3.3 and as seen in Figure 7., there have been no significant differences between the groups of participants who saw one of the disclaimers and were asked whether they would leave the website reading this disclaimer.
On the other hand, the disclaimer ranking part revealed additional insights with regards to whether the participants themselves perceived a significant difference between the disclaimers.As such, the participants' rankings of the disclaimers G1-G5 were analysed using the Friedman test, which indicated significant differences between the rankings (v 2 r ¼ 147:98, P < 0.001).In order to elaborate on these differences, pairwise comparisons between the disclaimers were conducted using the Nemenyi test.As such, most of the participants preferred the disclaimers from groups G1 ('This website uses cookies.By continuing to use the website you consent to the use of cookies.') and G2 ('This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us in order to improve our service for you.')followed by the disclaimer from G3 ('This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us for analysis in order to improve our service for you'), while the disclaimers from groups G4 ('This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us and by our partners (the so-called third parties cookies) in order to improve our service for you') and G5 ('This website uses cookies.By continuing to use the website you consent to the use of cookies on and off our website.Cookies are used by us in order to improve our service for you.') received the worst rating.An overview of the rankings is provided in Fig. 9, and the significance of differences between individual disclaimers is provided in Table 3.
When asked to explain their answers, the following themes emerged in the disclaimer ranking part.
No difference Many of the participants claimed not to see any difference between the displayed disclaimers in the disclaimer ranking part: 'I never read through these disclaimers and click them away, these disclaimers lead me neither to leave nor to stay on the website.' External use of cookies: Not surprisingly, when asked to compare the different groups of disclaimers, many participants had a particularly negative reaction to the disclaimers that mentioned use of cookies by either another entity or another website than the one the user interacts with (G4 and G5): 'I am against the processing of data by third parties and off the website.If it happens within the website, it seems OK to me.' Analysis: Some of the participants were put off by the mentioning of analysis (G3), considering it a threat to their privacy: '"Analysis" points explicitly that my data will be stored, this is dangerous.'Service improvements: The promise of the disclaimer to use cookies in order to improve the provided services (G2) was seen as positive by some of the participants: 'In order to improve our service, it sounds at least positive.' Length of text: The participants voiced different preferences regarding the length of the text in the disclaimer.As such, some preferred the disclaimer that provides only the bare minimal information (G1), considering longer explanations to be suspicious: 'I perceive all the additional information as excuses and dishonesties.Therefore I would rather accept it, when simply the inevitable is pointed at.' On the other hand, lack of any explanation was perceived by some of the participants as negative, lacking in transparency: 'The more information is given in the disclaimer, the more probable it is that I stay on the website, since I feel well informed then.' Intuitive decision: A number of participants, on the other hand, did not provide any concrete explanation of their ranking, referring to their intuitive feeling: 'Pure gut feeling.'

Website-specific factors
When asked in the disclaimer-specific section, in which cases the users would leave or stay on the website, the factors mentioned by the participants were related to a specific website that displays the disclaimer.These factors include the type of the service the website provides, as well as general characteristics of the website.Specific services.Some participants mentioned specific types of services as an example of the website they would either allow or deny use of cookies.These services include, in particular, online banking, social networks, video streaming, email and news.Note, that each type of these services was mentioned both as an example of the website the participant would stay on, as well as the example of the website the participant would leave.This was particularly prominent for the websites that were dealing with sensitive data such as online banking.As such, some users claimed they would stay on the website, as it was important for them to be able to use the service (e.g. to access their emails or to make a bank transaction): 'If I have to complete some task, for example, with emails and online banking.' Others, on the other hand, would leave the website if it used cookies, as they were concerned about possible implications towards their privacy: 'Online banking, shopping . . .all situations that have something to do with my privacy.'Service-independent characteristics.Instead of mentioning specific services, many of the participants named the following characteristics of the website that would lead them to either leave or stay on it.
Importance of website contents: Not surprisingly, a large number of users mentioned that their decision whether or not to leave the website depends on how important the contents of the website are to them: 'It depends on the website, how urgent I need it.' Trustworthiness of the website: Another factor in deciding whether to leave the website, mentioned by many of the participants has been the trustworthiness of the website.While some referred to a general feeling of uneasiness ('If something seems odd to me'), some mentioned specific concerns that the website is going to misuse the data collected with cookies ('If I have a feeling that my data is not secure').
Sensitivity of input data: Several users mentioned the type of data the website seems to collect as a factor in deciding whether they would leave the website: 'As long as it is evident that personalised data is collected.' In particular, some of the users referred to the data they input on the website: 'If it is a website where one inputs sensitive data.' Familiarity with the website: Finally, an important factor in deciding whether to continue using the website, named by many users, is the familiarity that the users have with the website: 'The more known the source is, the more likely I will stay.'

Quantitative evaluation
We first describe our evaluation of the quantitative questions in the follow-up study that were also present in the original study, namely, (1) a 'yes/no/don't know' question on whether a particular disclaimer would lead the participant to leaving the website, and (2) a numerical question (scale 0 to 100) on how likely the participant thinks they would get additional information about the usage of cookies by clicking on the link in the disclaimer and (3) a question where the participants were asked to rank the disclaimers G1-G5 in terms of how likely the disclaimer would lead them to leaving the website.In order to investigate possible changes since the GDPR entry into force, we furthermore compare these results with the corresponding results of the original study.

Intention on leaving the website
When the participants in the follow-up study were asked in the disclaimer-specific part, whether they would leave the website, 67% of all the participants answered that they would stay on the website, while 17% answered that they would leave the website, and 15% were not sure (see Fig. ? ?).As in the original study, no significant difference between the groups has been identified (using chi-square test, v 2 ¼ 8:16, P ¼ 0.41).
A comparison between the original and the follow-up studies (calculated for the participants overall) furthermore did not reveal significant differences between the studies (using chi-square test, v 2 ¼ 2:98, P ¼ 0.22).

Intention on getting further information
As in the original study, the participants were asked in the disclaimer-specific part, how likely is it that they would get additional information about the usage of cookies by clicking on the link in the disclaimer.Out of 146 participants who answered this question, half of the participants indicated such likelihood lower that 15%, and only 25% of them considered the likelihood to be higher than 43% (see Fig. ? ?).As in the original study, no significant differences between the groups have been identified (Kruskal-Wallis test, H ¼ 0.86, P ¼ 0.93).
A comparison between the original and the follow-up study (calculated for the overall participants) furthermore did not reveal any differences between studies (Kruskal-Wallis test, H ¼ 0.06, P ¼ 0.79).

Disclaimer ranking
Similar to the original study, the follow-up study did not reveal any differences in the disclaimer-specific part of the study between the groups G1 and G5 in the answers to the questions, whether the participants would leave the website if they see the corresponding disclaimer, and how likely is it that they would get additional information by clicking on the link in the disclaimer.For further insights into possible influence of the text of the disclaimer, similar to the original study, the answers of the participants in the disclaimer-ranking part were analysed using the Friedman test.The test revealed significant differences between the rankings of the disclaimers (v 2 r ¼ 83:37, P < 0.001).The Nemenyi post-hoc test furthermore revealed that the participants preferred the disclaimers from group G3 ('This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us for analysis in order to improve our service for you'), while ranking the disclaimers from groups G4 and G5 the lowest.An overview of the rankings is provided in Fig. 9, and the significance of differences between individual disclaimers is provided in Table 4.
The comparison of mean rankings of the groups between the original and the follow-up studies is shown in Fig. 10.The ANOVA comparison has revealed significant differences between the rankings of the disclaimer G1 ('This website uses cookies.By continuing to use the website you consent to the use of cookies'), which was more preferred among the participants in the original study, and G4 ('This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us and by our partners (the so-called third parties cookies) in order to improve our service for you'), which was more preferred among the participants in the follow-up study (P ¼ 0.02 for both groups).For the disclaimers from the rest of the groups, no significant differences have been identified (P > 0.05).

Frequency of codes
In order to further study the mental models and decision making regarding cookies and their potential changes since the GDPR entry into force, we evaluated the frequency of codes (see 'Original study results' section) mentioned by the participants in the original and the follow-up study.Our goal thereby was to (1) study the perceptions of the participants in terms of thoughts and feelings the cookie disclaimer affects in them, (2) the reactions of the participants to the disclaimers, and (3) the factors that influence the participants' decisions regarding cookies.In this, we study the codes to the questions in both of the studies that address these issues directly.
For comparing the codes, we applied a closed coding approach for the follow-up study and used the categories identified in the original study to code the open answers of the follow-up study.For some questions, participants provided responses that did not match the categories identified in the original study.We developed new categories for these cases.In contrast to the original study, we also decided to distinguish between answers relating to privacy topics but emphasizing different aspects (e.g.expressing privacy concerns, feeling forced to accept the cookie disclaimer, merely stating that data are collected without expressing concerns).As mentioned in 'Original study results' section, the changes in the description of the original study compared to the reporting in [6] reflects these adjustments.

Perceptions
For studying the perceptions of cookie disclaimers among the users, we evaluated the codes in the following questions: • General part: [For the disclaimers the participants recalled encountering] What thoughts or feelings did you have while reading the disclaimer?• Disclaimer-specific part: [For one of the disclaimers G1-G5] What thoughts or feelings did you have while reading the disclaimer?
The following codes have been compared: • Disturbance: The participant mentions being irritated by the code, e.g.seeing it as disturbance to the surfing experience.• Privacy concerns: The participant expresses concerns over their privacy.
• Factual information: The participant mentions factual information such as data collection, yet without expressing negative feelings about it.An overview of the code frequencies is shown in Fig. 11.The ANOVA comparison reveals significant differences in the codes 'disturbance' (F ¼ 4.53, P ¼ 0.03), 'privacy concerns' (F ¼ 13.33, P < 0.001) and 'habituation' (F ¼ 8.25, P ¼ 0.004), but not in any other categories (P > 0.05).

Reactions
We study the codes from the answers to the following question: • General part: How did you react to the disclaimer?For example, did you leave the website, got additional information . ..?
Table 4. Comparison of disclaimer rankings in the follow-up study 'þ' indicates significant difference between the groups (P < 0.05), 'À' indicates lack of significance.Fig. 12 shows a comparison between the codes in the original and the follow-up study.The codes thereby are (note, more detailed explanation and the examples of particular codes can be found in 'Original study results' section): • Ignore: The participant mentions either ignoring the disclaimer and continuing surfing the website, or clicks the disclaimer away.• Accept: The participant explicitly mentions agreeing to accept cookies.
• Deny: The participant mentions either explicitly denying or refusing to accept cookies.• Apply countermeasures: The participant mentions a particular countermeasure to the collection of cookies, such as deleting the cookies automatically, or using an ad blocker.• Get informed: The participant mentions getting additional information about cookie usage.
The ANOVA comparison did not reveal any significant differences in the number of codes between the two studies (P > 0.05).

Decision factors
We first look at the factors mentioned by the participants in the disclaimer-specific part.Namely, we consider the following questions: • Disclaimer-specific part: In what situation will the disclaimer move you to stay on the website?
• Disclaimer-specific part: In what situation will the disclaimer move you to leave the website?
The codes that were identified as factors influencing the participants' decision are as follows: • Design of the disclaimer: The participant mentions the look and feel of the disclaimer, e.g.whether it looks distracting or large enough to conceal the important website contents.• Specific services: The participant mentions a specific service provided by the website (e.g.online banking).• Trustworthiness of the website: The participant explains their reason to stay or leave the website based upon how trustworthy the website is perceived.• Familiarity with the website: The participant mentions staying on the websites that are familiar to them.• Importance of the contents: The participant mentions staying on the websites which content is important to them.• Input of sensitive data: The participant mentions leaving the website with the cookie disclaimer if sensitive data is provided by the participant.
The frequency of the codes in both the original and the followup study is provided in Fig. 13.The comparison revealed significant differences in the frequencies of the codes 'Trustworthiness of the website' (F ¼ 4.33, P ¼ 0.04) and 'importance of the contents' (F ¼ 4.16, P ¼ 0.04), and no significant differences for the rest of the codes (P > 0.05).
We furthermore looked at the factors from the text of the disclaimer that the participants mentioned when asked to explain their rankings in the disclaimer ranking part.The codes are as follows: • No difference: The participant says that they did not see any differences between the presented disclaimers.• External use of cookies: The participant expresses concerns over the disclaimer that mentions either third parties or use of cookies outside of the website.• Analysis: The participant is concerned about possible analysis of their data with cookies.
• Short text is preferred: The participant mentions their preference for shorter texts, either because they are easier to read or because they find longer explanations suspicious.
• Long text preferred: The participant expresses their preference for disclaimers with longer and more detailed explanations.• Service improvements: The participant mentions the possible service or functionality improvements that the cookies are potentially used for.• Intuitive: The participant admits that their ranking is based upon intuition and feeling rather than on any rationalizations.
The frequency of the codes is depicted on Fig. 14.The ANOVA test did not reveal any significant differences between the original and the follow-up study (P > 0.05).

Mental model of cookie disclaimer
To investigate people's mental model of a typical cookie disclaimer, we showed our participants a screenshot of a disclaimer that offered different possibilities for the user to respond.Participants were asked to explain what they thought would happen if they clicked on either of the different answer options, or ignored the disclaimer and continued surfing on the website.We then asked them to evaluate the concept of a cookie disclaimer that needs to be answered to before one can access the website.

Accept
Many participants (55%) explained that if they clicked on 'accept', they accepted the use of cookies and/or the disclaimer, and, consequently cookies would be used.Some participants also described that their data would be collected (15%), the disclaimer would disappear (11%), or they would be redirected to the actual website (6%).However, some participants (13%) thought that nothing at all would happen if they clicked on 'accept'.

Decline
A large number of participants (37%) associated clicking on 'decline' with denying the use of cookies and/or the cookie disclaimer.Yet a few participants who stated to deny the use of cookies by clicking on this button still expressed privacy concerns (3%): 'There should be no cookies, but that's not for sure.'Only very few (1%) believed that cookies would be used anyway.Many participants (37%) also thought that they would not be able to use the website if they clicked on 'decline'.Likewise, some participants (10%) thought that the website could not be used fully.A few participants thought that nothing would happen (5%), that the disclaimer would disappear (5%) or that the disclaimer would reappear (2%).

Show further information
Many participants (49%) simply described that they would receive further information if they clicked on this button: 'Further information opens and can be read.'Some participants specified what information they would receive and named either information on cookies (26%), a general privacy disclaimer (8%), legal information (2%), or the general terms and conditions of the website (3%).A few participants (6%) merely described that a new window would open.

Manage privacy settings
A large number of participants (51%) believed that if they clicked on 'manage privacy settings' they would be redirected to a site where they could adjust their general privacy settings, and some participants (14%) thought that they would be redirected to a site where they could manage their settings regarding cookie use.Others (20%) expected to be shown further information about cookies or the website's privacy policy without having the opportunity to change their privacy settings ('Boring information to read'), whereas a few participants thought that nothing would happen (1%), they would accept the use of cookies (1%), or simply stated that a new window would open (3%).

Close the disclaimer
When asked about what was going to happen if they clicked at the 'X' at the top right side of the disclaimer, many participants (35%) stated that the disclaimer would disappear: 'It closes the notification.'Nearly as much participants thought that this would imply denying the use of cookies (12%) as accepting it (16%).Some participants also believed that they would not be able to use the website (11%), the disclaimer would reappear (10%), or that nothing would happen at all (12%).

Ignore the disclaimer
Participants were undecided about what was going to happen if they ignored the disclaimer and continued surfing on the website.Some participants (25%) believed that nothing would happen, whereas others thought they would be accepting (21%) or denying (11%) the use of cookies.Some participants explained that the disclaimer would not disappear or reappear (14%), and some participants were worried that they would not be able to use the website (17%).

Answer to disclaimer required before website can be accessed
We also asked our participants about their opinion of a disclaimer that needed to be answered to before they could access the website.Most participants did not like this idea, and called it annoying (40%), paternalistic (4%), unnecessary (3%), or simply a bad idea (25%).Still, some participants reported to like the concept (23%) or said that it would be okay since they are already used to it (4%): 'You already have to do this, thus nothing is going to change.' We furthermore asked the users, which one of the options shown on the disclaimer, that is, (1) an X in the right upper corner, (2) an 'accept' button, (3) a 'deny' button, (4) a link to the cookie settings, (5) a link to get more information about the usage of cookies, they would rather choose (note, it was possible to select more than one option).An overview of the choices made by the participants is depicted on Fig. 15.

Discussion
The purpose of the cookie disclaimer has been to provide clear and understandable information to the users regarding cookie use.However, as evidenced from responses in our studies, users often click the disclaimer away without paying attention, or ignore it.Some claim not to understand what the disclaimer is saying, are suspicious due to perceived lack of transparency (e.g.not being able to tell how the collected data will be used by the service provider), are not aware of possible privacy-related consequences of cookie use or have other misconceptions regarding what the collection of cookies means to them.Hence, the disclaimer often fails its purpose of informing the users.Moreover, prescribing an opt-in solution might not alleviate the issue.As long as users do not read the disclaimer and try to click it, it is possible that in trying to get rid of the disclaimer they click on the 'agree' button, without realizing the consequences for their privacy.
The results of our follow-up study further suggest that the increased media coverage of this topic that accompanied the GDPR entry into force in May 2018 did not change users' attitude towards cookies.Although we did not find statistical differences between the original and the follow-up study, even more participants reported to accept cookie disclaimers in the follow-up study compared to the original study, whereas less participants said that they would leave the website if they were confronted with a cookie disclaimer.Also, significantly more participants said that they felt disturbed by the disclaimer, while, at the same time, more participants were used to seeing the disclaimer and less participants were concerned about their privacy.In line with this, significantly less participants of the follow-up study stated that their decision to leave or stay on the website depended on how important the content of the website and how trustworthy the website was.Hence, it seems that even more users now tend to accept cookie disclaimers blindly to get rid of it, which may be an unintended side-effect of the increasing use of cookie disclaimers on websites due to the introduction of the GDPR.Furthermore, users seemed to have different mental models regarding the answer options of the cookie disclaimer.Some participants believed that they could deny the use of cookies if they ignored or closed the disclaimer, whereas others associated this with accepting the disclaimer.Likewise, some participants thought that ignoring the cookie disclaimer would be similar to accepting the use of cookies, whereas others thought that they could deny the use of cookies if they refuse to answer the disclaimer.Many participants also thought that they would not be able to use the website if they do not accept the disclaimer.
The results of the study imply, that superficial measures to inform users about data collection, presented without meaningful options or understandable information about consequences of data collection do not help users in making informed decisions.In fact, such measures can bring more harm than good in disincentivizing the users from taking measures to protect their privacy, as the users feel more overwhelmed with the amount of decisions they have to make and feeling more convinced about the futility of privacy protection.Moreover, the wide-spread existence of so-called dark patterns (see e.g.[7,8]) further prevents the users from giving consent as required by the GDPR-that is, 'freely given, specific, informed and unambiguous'. 5This raises a question as to which extent the user's consent can be at all relied on as a basis for data collection-an issue already discussed by some scholars, see e.g.[9].Yet, a better implementation of the disclaimers, with clearer explanations on which data is being collected and for which purpose (possibly also highlighting unexpected data collection practices, as suggested in [4]) and more prominent UI controls for denying the collection, might come a long way towards better privacy protection.
Both of our studies did not reveal any significant effects of the disclaimers' text on users' behaviour when participants were shown different disclaimer texts and asked whether they would leave a website with this disclaimer.Note that every participant only saw one disclaimer text in this part of the study.However, participants were also asked to rank all different disclaimer texts.The results of this ranking suggest that users are most skeptical when the disclaimer text mentions third parties or the external use of cookies.However, as significantly less participants in the follow-up study showed a negative reaction towards the use of cookies by thirdparties compared to the original study, users might get more and more accepting towards this practice as they get used to reading this phrase on actual cookie disclaimers.
These results imply that it is not sufficient to adjust the text of the cookie disclaimer to allow users to make an informed decision about the handling of their data, as they not only tend to accept more privacy-invasive practices once they get used to it, but also react similar to different disclaimer texts if they are only shown one at a time, which is the normal case when surfing on the Internet.Therefore, structural measures such as appropriate legislation are needed to ensure that companies are prevented from employing practices that endanger the privacy of their users.On a technical level, measures that enable more control for the users can be helpful, such as letting users configure their privacy settings directly in the browser and providing a notification if a website requires cookie settings that differ from those that have been specified by the user beforehand.

Recommendations
Based on our findings and their discussion, a number of recommendations can be deduced for consumers, developers, policy/law makers.

Consumers
The following explanations could be given (e.g. by consumer agencies) to costumers while the first one is for those who want to keep using the one browser they are used to.The second one is for those willing to use two different browsers: one for webpages with account and one for webpages without accounts, i.e. for surfing through the Internet.
• Cookie banners often include two ore more buttons to deal with cookies: It is highly recommended to look for the most privacyfriendly option.Often the highlighted one is not the most privacy-friendly one.Furthermore, sometimes, it is required to go deeper in the settings.Furthermore, consider leaving the websites that have disclaimers that make it hard to reject storage of cookies • Configure the browser used for surfing in a way that tracking is blocked, for example, (i) by blocking third-party cookies, (ii) by using the private mode by default, or (iii) by deleting cookies when closing the browser.Having such a setting lets consumers relax with the cookie banners as it is not that important anymore to read their message and find out which option is the privacyfriendly one.

Developers
For developers of webpages or web applications, we have the following recommendations: • Think whether you need to collect data from the users (i.e.apply privacy by design).Unless you see a clear purpose for which this data could be useful for you, consider not collecting it in the first place (data minimization).• Ensure that it is easy for users to choose the most privacyfriendly option in the disclaimer (privacy by default).This includes that the user should neither require more clicks to reject data collection, nor should the UI itself provide any nudging to the user (e.g. by making the 'accept'-option more prominent and visible than the 'reject'-option).[10]).• When designing the aforementioned recommendations and guidelines, involve the users in order to better understand their decision making, as well as developers to evaluate the understandability of the guidelines.For this purpose, empirical studies can be used.

Related work
The users' perceptions and mental models of cookies and other tracking tools, as well as of factors influencing users' decisions, have been investigated in a number of studies.As such, the study by Ha et al. in [11] using focus groups has revealed a number of misconceptions among the users regarding the use of cookies and its purpose.Similar to our results, the study furthermore revealed the feelings of resignation among the participants regarding their privacy protection.Studies by McDonald et al. [12,13], in form of interviews and online surveys, further revealed lack of awareness and misconceptions prevalent among the users regarding cookies.Shirazi et al. furthermore revealed a number of misconceptions, including lack of awareness and feeling of resignation, regarding web tracking and countermeasures against it in their interviews [14].Similarly, the prevalence of misconceptions regarding cookies and online tracking emerged from the study by Ur et al. [15] conducted in the form of interviews.The study furthermore investigated the factors influencing users' decision to share data with advertisement companies, demonstrating that the users were more likely to share data with companies they were familiar with (e.g.Google) than with companies they did not know.Chanchary et al. [16] conducted an online study that investigated factors that influenced the users' decision to share data with advertising companies.Their results have demonstrated, that the level of control over the collected data that the service providers enable the users has only a moderate effect on users' decision, while other factors such as general privacy attitudes of the users and the frequency of their visits to the website, play a more significant role.Further studies focused on privacy notices in the context of online tracking.Leon et al. [17] conducted an online survey, studying factors that influence the participants' willingness to share data with online advertisers, requiring the users to read the privacy policies provided on the website of a health services provider.The study revealed that the privacy policies had a larger effect on users' decisions than the trustworthiness of the website.As the text of the cookie disclaimer in our study did not have a significant influence on users' decisions, the effect of privacy notices in different forms (i.e. as a disclaimer or as much more detailed privacy policy description) is to be investigated more closely.Miyazaki [18] investigated the effect of disclose about cookie use towards users' attitude towards cookies in several user studies.The results of the studies have shown that users are less likely to have a negative reaction to cookies if the website provided a prior disclosure.While many participants in our study still expressed a negative reaction to the cookie disclaimer, either perceiving it as a nuisance or considering cookie use a threat to their privacy, we did not compare their reactions to cookie use without a notifying disclaimer (in [18], instead of the disclosure for the website, the control group received a notification from the browser as soon as cookies were set).
Most recently, several studies have furthermore focused on user studies on cookie disclaimers [7,8,19].While the studies have shown diverse attitudes towards cookies and data collection through them, they have confirmed the prevalence of the so-called dark patterns, that is, UI elements designed to nudge the users into agreeing to data collection, and that these dark patterns were indeed effective in influencing user's decisions.
A study by Santos et al. [20] furthermore reviews the use of cookie disclaimers from both legal perspective and technical perspective, coming up with a list of requirements to ensure legal compliance of the disclaimers.They furthermore conclude that a fully automated verification of compliance via technical means is not possible, and that user studies are required to verify the fulfillment of some requirements.

Conclusion
Cookies allow websites to implement advanced functionalities and improve their service by personalizing the interaction for the user.However, this comes along with the collection of their data and users thus have to weigh the benefits of cookie use against the partial loss of their privacy when deciding about whether to accept the use of cookies.The current law requires websites to inform users about the use of cookies, which is usually done by showing a cookie disclaimer.
A survey study with 150 participants in December 2017 showed that most participants had negative feelings towards such a disclaimer, but the text of the disclaimer had no significant influence on their decision to leave the website.We were interested in whether users changed their perception towards cookies due to the extensive coverage of the data protection topic in the media that accompanied the introduction of the GDPR in May 2018.To this end, we conducted a follow-up study in December 2018.Our results suggest that users did not change their attitude towards cookie use in favor of privacy protection.Indeed, even less users reported to have privacy concerns, while, at the same time, more users were habituated to and annoyed by the cookie disclaimer.In line with this, more users tend to accept the cookie disclaimer, whereas in the 2017 study, more participants depended this decision on the content and trustworthiness of the respective website.It thus seems that instead of empowering users, the GDPR may have driven even more users to blindly accept the use of cookies to get rid of the increasingly used cookie disclaimers.
We further found that users have misconceptions regarding the use of cookies as well as differ in their mental models of the various answer options usually provided by the cookie disclaimer.Consequently, the currently used opt-out solution has to be replaced by other approaches, such as allowing the users to configure their cookie preferences independent of the controls provided on a specific website as well as requiring companies to provide clearer information and controls for the users whose data they are planning to collect.
Limitations and future work: While our study was conducted shortly after the GDPR entry into force, it would be interesting to study its long-term effects in additional studies.A further point of investigation would be to look at cross-cultural effects: as such, our study was conducted in Germany, which already had strong data protection regulations prior to the GDPR.As the difference in security and privacy awareness have been shown to exist between European countries in other contexts, e.g.smart homes and smart health systems [21], such differences might be present in the perception of privacy in context of the GDPR and the cookie disclaimers as well.

Figure 1 .
Figure 1.An example of a cookie disclaimer on a mobile version of a website.

Figure 8 .
Figure 8. Answers to the question how likely the participants considered getting additional information (scale 0 to 100) in the original study (left) and in the follow-up study (right) as a box plot.The values are provided for each individual group (G1-G5), as well as for the participants overall.

Figure 7 .
Figure 7. Answers to the question whether the participants would leave the website upon seeing the disclaimer in the original study (left) and in the follow-up study (right).The values (as percentages of participants choosing each answer option) are provided for each individual group (G1-G5), as well as for the participants overall.
• Disclaimer-specific part: In what situation will the disclaimer move you to stay on the website?• Disclaimer-specific part: In what situation will the disclaimer move you to leave the website?• Disclaimer-ranking part: Please sort the disclaimers as follows:

Figure 9 .
Figure 9. Numbers of participants who placed a disclaimer from the corresponding group (G1-G5) in each rank in the original study (left) and in the follow-up study (right).

Figure 10 .
Figure 10.Mean ranks of the disclaimers in groups G1-G5.

Figure 12 .
Figure 12.Number of participants mentioning each reaction code in original and follow-up studies.

Figure 13 .
Figure 13.Number of participants mentioning each code for the factors from the disclaimer-specific part in original and follow-up studies.Figure 11.Number of participants mentioning each code in original and follow-up studies.

Figure 11 .
Figure 13.Number of participants mentioning each code for the factors from the disclaimer-specific part in original and follow-up studies.Figure 11.Number of participants mentioning each code in original and follow-up studies.

Figure 14 .
Figure 14.Number of participants mentioning each code for the factors from the disclaimer-ranking part in original and follow-up studies.

Figure 15 .
Figure 15.Answers to the question how the participants would react to a disclaimer given a variety of options (as a percentage of total number of selected options).

Table 1 .
Different groups of cookie disclaimers and examples of websites where they can be encountered By continuing to use the website you consent to the use of cookies.G2This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us in order to improve our service for you.G3This website uses cookies.By continuing to use the website you consent to the use of cookies.Cookies are used by us for analysis in order to improve our service for you.

Table 2 .
Age distribution

Table 3 .
Comparison of disclaimer rankings in the original study The participant expresses getting used to seeing the disclaimer, either having neutral feelings towards it, or being resigned about being able to do something about it.The participant mentions not having sufficient information about the use of cookies or wanting to know more about how the cookies are used.The participant voices a misconception about what cookies are and how they are used.

•
Provide information about specific kinds of cookies you collect in readable and understandable form, including what kind of user data they allow to collect.As a continuation of the previous item, the information on what disclaimer designs are compliant should be clearly communicated to the developers and enforced in practice.This includes ensuring that service providers do not rely on dark patterns, which technically seem to provide the users with an option to reject data collection, but at the same time nudge them into consenting to it via deceptive UI patterns.Furthermore, developers should have access to guidelines clearly describing the type of disclaimers that are appropriate to get user consent, if possible, supplemented with examples and counter-examples of such designs (see e.g. the recommendations from the Danish data protection authorities • Make it clear, what kind of disclaimer designs are compliant with privacy regulations.While, as we mentioned above, there is an indication that consent dialogues and disclaimers have to be usable and understandable to the data subjects, more concrete definitions of what this means in practice would be useful.•