Learning the value of experience

This scientific commentary refers to ‘Impaired value-based decision-making in Parkinson’s disease apathy’ by Gilmour et al. (https://doi.org/10.1093/brain/awae025).

Why can’t I be bothered? Answering this question for patients and their families has been the subject of intense interest across the basic and clinical neurosciences. Apathy matters—it is associated with significantly reduced quality of life for patients and is frequently one of the most difficult facets of disease for family members to manage, in Parkinson’s disease as well as in other brain conditions.¹ This is hardly surprising, given that ultimately our motivation is what drives the interactions with the world that build up the tapestry of our lives. Apathy, however, is challenging to understand, not least because there are likely many mechanisms that can lead to a final phenotype of reduced goal-directed behaviour, and our methodologies to detect them—be they clinical, behavioural or physiological—are still evolving.

A defining feature of goal-directed behaviour is the concept of an action-outcome representation. This is the idea that, as an agent chooses what behaviour to pursue, they actively represent both the potential rewarding outcomes of the options and the costs (e.g. effort, time) that would be incurred to obtain them. The field to date has tended to focus on whether this reward and cost information is represented differently in people with apathy at the time they choose whether or not to pursue behaviours.^2,3 However, there are other phases of goal-directed behaviour that may be just as important, particularly when considering how behaviour may slowly shift across time towards an amotivational state. One crucial question has been whether people with apathy learn differently from their experiences—with even small biases in this system over time having the potential to mediate the change from a motivated to amotivated state. In this issue of Brain, Gilmour and colleagues⁴ take an important step towards answering this question by reporting the results of a combined behavioural and imaging experiment that investigated whether learning about the outcomes of choices is systematically altered in people with apathy in the context of Parkinson’s disease.

The study of learning has a long and distinguished history in neuroscience and psychology. A central consideration for the field has been how organisms update the values and costs of options in their environment based on past experience.^5,6 One approach has been the use of ‘bandit’ tasks, in which participants must try to maximize earnings across an experiment by choosing between virtual slot machines, which each have different payout attributes that change across the experiment. Optimal behaviour requires both learning from the outcomes of choices, and striking a balance between exploiting the perceived current best machine and exploring the other machines to see if their payout characteristics have changed.⁷ Crucially, computational models can be used to describe the behaviour of participants with respect to the rate they learn from reward outcomes, or how they balance exploring and exploiting machines.

In the current study, raw (i.e. unmodelled) performance differed as a function of apathy in Parkinson’s disease (Fig. 1A). Patients with apathy were less accurate than those without at choosing the best option, and consequently across the whole experiment received significantly less reward. Although worse performance in the apathetic group could theoretically be driven by other less specific factors (such as poorer understanding of, or engagement in, the task), control analyses suggested these were not major confounds. Instead, this result seemed to be driven by a fundamental difference in how people with apathy learnt about the relationship between choices and outcomes.

Figure 1

Bandit task to probe learning and exploration in Parkinson’s disease apathy. (A) At the time of the trial shown (dotted line), the bottom right (green) bandit is the highest paying bandit. Prior choice history (and learning from this) determines knowledge of the other bandits. On this trial, choosing the bottom right (green) option is generally classified as an ‘exploit’ choice within the computational framework. (B) Blood oxygen level-dependent signals in healthy controls and in participants with Parkinson’s disease (PD) with or without apathy associated with (left) receiving feedback about the outcome of a choice, and (right) making an ‘exploratory’ choice. Adapted from Gilmour et al.⁴ HC = healthy control subjects.

But what drives this difference? Gilmour and colleagues⁴ adopted an established computational model of learning to classify each choice as ‘exploitative’ (choosing the most valuable option), a ‘random exploratory’ choice (choosing an option independent of represented values) or a ‘directed exploratory’ choice (choosing an option whose current value was most uncertain, and from which most information could be gained).⁷ Relative to the non-apathetic Parkinson’s disease group, the apathetic group demonstrated more random exploratory choices (and fewer exploitative choices), but had similar levels of directed exploration. The authors then hypothesized that such a behavioural bias could be explained by two distinct shifts in outcome valuation—a breakdown in encoding what actually happened after a choice, or alternatively (assuming this outcome was in fact validly represented) a shift in the influence of this signal on subsequent behaviour. They used task-based functional MRI (fMRI) to try and distinguish between these possibilities.

To explore the first possibility, they looked for brain regions where changes in blood oxygen level-dependent (BOLD) signal correlated with reward payout on a trial-by-trial basis. In healthy controls, the key region with this pattern of activity was the ventromedial prefrontal cortex, which replicates previous work. However, neither of the two Parkinson’s disease groups showed activity that was significantly related to payout. The authors report several post hoc analyses in the Parkinson’s disease group limited to the ventromedial prefrontal cortex region identified in controls, but find no strong evidence that alterations in payout signalling could explain apathetic behaviour. To investigate the second possibility the authors contrasted brain activity associated with explore versus exploit decisions. When making exploratory choices, there was greater activation in thalamic/midbrain areas in the non-apathetic Parkinson’s disease group relative to the apathetic group, which the authors argue may represent a compensatory mechanism that protects against development of apathy in Parkinson’s disease (Fig. 1B).

This paper tackles a complex question using an ambitious combination of a sensitive behavioural paradigm, computational models of learning, and task-based fMRI in a clinical population with and without apathy. The results of the study are certainly intriguing, and deserve both recognition and scrutiny. For example, although the explore vs exploit contrast provides an indication of differences in brain activity at the time of the decision, it may not directly address the question of whether apathy occurs because of altered learning. An analysis of prediction error activity (the difference between expected and actual rewards) could in theory probe this possibility, although interestingly there was not a strong prediction error fMRI signal in the group with Parkinson’s disease.

Similarly, the greater thalamic activation in the non-apathetic Parkinson’s disease group during exploratory choices is intriguing, but challenging to interpret. Behaviourally, the non-apathetic Parkinson’s disease and control groups performed very similarly, and it is difficult to ascribe changes in BOLD activity to the absence of any significant difference in performance. The higher BOLD activity in the non-apathetic Parkinson’s disease group relative to the other two groups was not specifically linked to any single model parameter, and could potentially be driven by factors other than the tendency to explore per se. Furthermore, the significant correlation between apathy severity and activity in a right thalamic cluster during explore choices is not surprising, given that this cluster was one of several identified from a group comparison between apathetic and non-apathetic groups.⁸ Of course, the question of why a non-apathetic patient group should exhibit greater brain activation than both an apathetic patient group and healthy controls remains an interesting one. In particular, future work is needed to confirm whether such a phenomenon is indeed mediated by compensatory responses, or instead by reduced efficiency of processes that support task performance.⁹

It is interesting that apathy—the archetypal non-exploratory state—was associated with increased exploratory behaviour in this study. As the authors suggest, one possibility is that this could reflect a less precise representation of the action-outcome relationship, rather than altered learning per se.¹⁰ A more definitive interpretation of this result may also benefit from considering behaviour in more naturalistic paradigms, such as those used in the foraging literature. These considerations emphasize the challenges in extrapolating tightly controlled behavioural paradigms to the real world—challenges that to a degree can be resolved by triangulating behavioural parameters, clinical phenotype and neural measures.

Finally, this study highlights several clinically relevant considerations. For example, the question arises as to whether biased learning could be most relevant in the earlier development of apathy, and it would be worthwhile investigating whether the pattern of change reported here could predict future apathy in people with Parkinson’s disease. In addition, the authors excluded people with significant cognitive impairment or depression, given their intention to focus on ‘pure’ apathy, as well as concerns about noise in task performance. Of course, apathy often co-occurs with cognitive dysfunction and mood disturbance, therefore the extent to which these data generalize to people who develop apathy in these other contexts remains to be shown.

Overall, Gilmour and colleagues⁴ demonstrate the neural substrates implicated in disrupted motivated behaviour in people with Parkinson’s disease. The imaging results are probably best viewed as hypothesis-generating, rather than providing strong confirmation of a specific theory, but the clear apathy-associated difference in thalamo-cortical activations at the time of switching, along with the alterations in explore/exploit choices, are both signals that deserve further exploration.

Competing interests

The authors report no competing interests.

Funding

T.C. is supported by the Australian Research Council (FT220100294).