Abstract¶
Financial planning tools can be a double edge sword for consumers as most tools of today lack sufficient theoretical background or are used for commercial purposes. To tackle the shortcomings of today’s financial planning tools, we delve into a Machine Learning subfield called Reinforcement Learning. Reinforcement Learning has seen great advancement in the past decade and is becoming a comprehensive field in which financial tools can be better tailored for the consumer. More specifically, the recent coöperation between reinforcement learning and planning domains like optimal control enable algorithms to not just learn the environment, but also to be able to plan ahead. Given the fact that these algorithms can be widely generic and can thus be employed in a tailored financial environment for the consumer, makes the Reinforcement Learning literature an interesting field for future financial planning tools. Inspired by this idea, the general theory of Reinforcement learning is introduced together with the most fundamental algorithms. Furthermore, a closer look will be given at the dimensions of a Reinforcement Learning algorithm. Thereafter, the current challenges in Reinforcement Learning are discussed together with function approximation, which tackles the biggest hurdle in Reinforcement Learning called the Curse of Dimensionality. Next, the G-learning algorithm and the Deep Backward Stochastic Differential Method are introduced which have implications for financial planning tools. Finally, these two algorithms are applied in a financial setting.
Introduction¶
The financial decisions that consumers need to make in their present lifetime, become increasingly more complex. A good example of this phenomenon is the shift from defined benefits to defined contributions in which consumers take on greater individual responsibility and risks. The evolution in the abstruseness of financial products has become challenging for consumers who possess low financial knowledge and limiting numeracy skills [BFH17]. Combined with uncertainty about the future, the consumer is necessitated to be more aware of his financial well-being than ever before. Looking back into the past, Porteba et al, [PVW11] conducted an examination of preparedness in retirement for Children of Depression, War Baby, and the Early Baby Boomer in the Health and Retirement Study and Asset and Health Dynamics Among the Oldest Old cohorts. They found that 46.1 percent die with less than 10 000 dollars. With this amount of assets, they would not have the capacity to pay for unexpected events and one might wonder if it is adequate asset levels for retirement. Furthermore, saving behavior has not kept pace with the increasing life expectation and the expected prolonged lifespan of the coming generations is unprecedented [Her11]. All these elements give a painstakingly clear picture that having a vital understanding of one’s financial situation has become one of the greatest challenges in life.
To combat these difficulties, consumers require additional undertakings in planning for their future prosperity. One of the approaches to tackle this issue, is by using financial planning tools. These tools give the consumer the capability to estimate complex intertemporal calculations [BDTS20]. They also enhance financial behavior, increase household wealth accumulation and they are a complement to other planning aid like a financial advisor [BFH17]. Although financial planning tools can greatly benefit consumers, they can also be a double-edged sword. More specifically, when consumers are misinformed about the capabilities of the tool, or when the design of the tool is inadequate, the consumer can be given sub-optimal advice or even misleading advice [DMBE18]. Insufficiencies in design can arise when not all essential input variables are included, not all risks are considered, and when accuracy is sacrificed for the ease of use [BDTS20]. On top of that, there are wide variations in results because of the various methodology and assumptions used in the models [DMBE18]. For example, assumptions based on inflation and the use of different financial products have a large impact on the results. On the side of the consumer, the possibility of misunderstanding the implications of the results due to a lack of financial knowledge, is a matter of great concern in the eyes of financial educators [BDTS20]. Clarifying the results is therefore an essential part of making models operational. To improve upon these deficiencies, Dorman et al., [DMBE18] found that when the models handle additional theoretical variables, the accuracy will improve. Besides, they found that the consumer requires unique solutions that better capture their financial situation. Meaning planning tools need to be more flexible. They should be able to operate in different financial settings and have the ability to look at the impact of changes in input variables. To address the variability in results and the adaptability of models to different settings, this paper will look at reinforcement learning techniques in an intertemporal setting. Reinforcement Learning enables an increase in the flexibility of the model while keeping fundamental theoretical aspects like Optimal Control Theory at its core. For the remainder of the paper, the general theory of Reinforcement Learning (RL) will first be introduced. Then, some challenges are discussed together with Deep Reinforcement Learning. Next, the possible implications of RL for financial planning are considered. Finally, two financial applications are reviewed.