In the last course of our specialization, Overview of Advanced Methods of Reinforcement Learning in Finance, we will take a deeper look into topics discussed in our third course, Reinforcement Learning in Finance. In particular, we will talk about links between Reinforcement Learning, option pricing and physics, implications of Inverse Reinforcement Learning for modeling market impact and price dynamics, and perception-action cycles in Reinforcement Learning. Finally, we will overview trending and potential applications of Reinforcement Learning for high frequency trading, cryptocurrencies, peer-to-peer lending, and more.
Langevin Equation and Fokker-Planck Equations
Loading...
来自 New York University Tandon School of Engineering 的课程
Overview of Advanced Methods of Reinforcement Learning in Finance
3 个评分
New York University Tandon School of Engineering
3 个评分
课程 4（共 4 门，Specialization Machine Learning and Reinforcement Learning in Finance）
从本节课中
Perception - Beyond Reinforcement Learning
与讲师见面
Igor Halperin
Now after we spoke a bit about classical mechanics,
let's come back to the Langevin equation that we discussed before.
We have discussed the Langevin equation,
how it was initially produced and how it generalizes
the Ito diffusion that is more conventionally used in finance.
We also saw how
the stochastic differential equation arising in our formulation turns out to be
a particular type of the Langevin equation with
a non-linear drift with potential metastability.
Now the Langevin equation is a path-wise equation for a particle.
Is driven by a particular realization of a noise term, a longer path.
But for some problems this formulation is not the most convenient one and
instead a probabilistic description of a system is preferred.
This can be done starting with
the Langevin equation and this transition leads to another famous differential equation.
This time it's a partial differential equation or
PDE known as the Fokker-Planck equation.
In finance it's also sometimes called the forward Kolmogorov equation.
This equation is equation for you to shown on this slide.
It describes the probability for a particle or
a system of particles or any other stochastic system to
be in state x at time t given that it
was at some other state x not at previous time t not.
So in this equation where the time derivative of the probability density p of x,
and in the right hand side we have two derivatives terms.
The d term contains the derivatives for the potential u prime and
the second diffusion term is
a second derivative term that depends on the volatility function sigma of x.
In our case, sigma of x equals x that is we have a proportional volatility function.
This equation should be solved under
an initial condition given by a delta function shown on the bottom of the slide.
Now, if you want to understand how
the Fokker-Planck equation is derived from the Langevin equation,
here is a sketch for you.
Let's consider some function f of x of
the state variable x and consider it's expected value.
We will denote this average by the left and right brackets.
So, one way to compute the time derivative of this expectation is to
take the time derivative inside the brackets and then use Ito's formula.
This gives us the first line in the first equation on this slide.
On the other hand, we can also replace the brackets by
explicit integration with the density p of x,
and this gives us the second line in this expression.
On the other hand, we can compute the same quantity
differently by recalling that the mean of
the function is given by it's integral with density
p. Because f of x does not depend on time explicitly,
where we differentiate such integral,
the time derivatives applies only to the distribution P of X.
This gives us equation 43 on this slide.
Now, if we integrate by parts twice in the first expression we obtain an equation for 44.
So, we obtain two different expressions for
the same quantity which we can now write as equation 45 here.
But because the function f of x that we used in this derivation was arbitrary,
the equation should apply to it's integrants and not just to the integrals or more sites.
This produces the Fokker-Planck equation.
Now, the Fokker-Planck equation is
a second-order partial differential equation and it
should be supplemented by boundary conditions,
in addition to an initial condition.
The most interesting boundary is the boundary at zero.
This is the default boundary as we already discussed earlier.
If the stock price touches this point it stops moving.
This follows from the form of the SD for
our model because both the gif and the diffusion terms vanish at this point.
This is called a natural boundary in the classification of differential equations.
If the sign of the derivative of the drift potential is negative at this point,
it's called an exit boundary.
Now if we look at the GBM model,
the point x equals zero is also a natural boundary for this diffusion,
for the same reason that both the drift and diffusion vanish at this point.
But a big difference between
these two cases is that in the GBM model the boundary is unattainable.
This is something that we already discussed in the last week.
If we agree to call default events of a stock price when the price drops to zero
then such events are incompatible with the GBM dynamics creating all sorts of problems.
But with their non-linear diffusion such as one obtained with our framework,
such events are perfectly possible.
In other words, a natural boundary at zero is
attainable in our model and it becomes an absorbing boundary.
Once a store gets there it will stay there forever.
In other words, we do not have to enforce a correct behavior of the model by imposing,
absorbing, boundary conditions at zero by hands.
The system knows that once it reaches this point it will stay there forever.
This is called absorption at zero,
but without enforcing specific absorbing boundary conditions at this point.
So, what might happen on this graph is that a particle initially to the right of
the potential barrier denoted as the rightmost to that point
here can go over the barrier and as a result of thermal fluctuations.
Then it stays in the left well as the red point on the left of this graph.
Now, when the particles,
they are touching the barrier at zero becomes
much more plausible event because there is no more barrier to cross.
This scenario can therefore describe stocks that are in nearly free fall to bankruptcy.
Still there is a chance for such stock to go over the barrier back to
health but such double jump would be a very unlikely event.
Now, instead of analyzing things in the x space which is
a space of market capitalization or of a stock price,
we can switch to log price y equals to log of x.
The main reason why such change of variables makes sense is
because it changes the multiplicative noise into additive noise.
Additive noise is much easier to work with because it
affects classical dynamics much less than a multiplicative noise.
So, if we change the variable in
the Fokker-Planck equation and then in terms of the log price y,
the equation now takes the form shown here in equation 47.
There are two major changes that relative to the original formulation of the equation.
First, their diffusion term now has a constant volatility.
Second, the drift potential now goes unbounded to
the negative infinity and becomes negative infinity over there.
This means that in the log space a fall to the center can be
seen as an infinite escape to negative infinity.
Because we change to log price the absolute zero is now at
infinity where the new potential also has a singularity there.
But this approach is convenient because we can approach this singularity,
actually, as close as we want using the white space.
Finally, I want to discuss what happens in
a Fokker-Planck equation in a zero noise limit.
If we take it to the log-space and set sigma equal to zero there we get equation 48.
Now, you can use this equation to compute
the time derivative of u using the chain rule as shown in equation 49,
and this shows that this derivative is always negative unless
the system reaches the point where the derivative vanishes.
But note the interesting fact here that this equation only seems to require that
the derivative vanishes vanishes but not necessarily whether it's a minimum or a maximum.
To settle on the right minimum we actually need noise in the system.
