In the last course of our specialization, Overview of Advanced Methods of Reinforcement Learning in Finance, we will take a deeper look into topics discussed in our third course, Reinforcement Learning in Finance. In particular, we will talk about links between Reinforcement Learning, option pricing and physics, implications of Inverse Reinforcement Learning for modeling market impact and price dynamics, and perception-action cycles in Reinforcement Learning. Finally, we will overview trending and potential applications of Reinforcement Learning for high frequency trading, cryptocurrencies, peer-to-peer lending, and more.
Classical Dynamics
Loading...
来自 New York University Tandon School of Engineering 的课程
Overview of Advanced Methods of Reinforcement Learning in Finance
23 个评分
New York University Tandon School of Engineering
23 个评分
课程 4（共 4 门，Specialization Machine Learning and Reinforcement Learning in Finance）
从本节课中
Perception - Beyond Reinforcement Learning
与讲师见面
Igor Halperin
Now, after we spoke about [inaudible] ,
let's take a bit of a step back and discuss what
happens when there is no noise at all in the system.
Let's look again at our model that I repeat here in equation 17.
Now, if we recall our discussion of the [inaudible] equation,
we can look at the j_th term in this equation,
and identify it with the minus the unity for the potential.
Therefore, we directly read off the deep potential U of x that corresponds to our model.
It's given here in equation 18 and it has the form of a fourth degree polynomial in x.
This is the potential function of a quartic oscillator,
because of the fourth power of the potential.
Now, for various specific cases,
these potential reduces to some of the most famous problems in physics.
It has three parameters Theta,
Kappa, and g. Respectively,
you can set any of these to zero and you can
also change signs of parameters in this potential.
For example, if we take Theta negative and set Kappa and g to zero,
we get the potential of a harmonic oscillator.
If we take G to zero,
we get the verticals potential and
if we take kappa to zero and leave two parameters theta and g,
we can get two interesting cases.
One of them is when Theta is negative and g is positive.
This describes an harmonic oscillator with quadratic nonlinearity.
Another case is obtained when Theta is positive and g is positive.
This potential is related to one of
the most interesting phenomena in physics that is called spontaneous symmetry breaking.
We will talk about this phenomenon a bit later.
For now, let me show you some examples,
but before doing that though,
I want to show you an equivalent form of
the same quartic potential that uses a different parameterization.
This is shown in equation 19 and instead of parameters Kappa and g,
here we have two other parameters, a and b.
These parameters specify positions of zero of the potential.
As you can see directly from the formula itself.
You can also, easily find the relation between these two sets
of parameters that is shown in equation 20 here.
Now, let's go back to examples.
In this first example,
I show you a potential that is obtained if we
take Theta less than zero and parameters a and b positive.
The potential has two minima,
the minimum at x equals zero,
is a local minimum where the minimum at approximately 0.5 is a global minimum.
In between these two minima,
there is a maximum at approximately X equals 1.3.
It's interesting that, if we keep the same combination of
parameters Theta negative and ab positive,
but change the variable to log Price Y equals to log x instead of the price x,
then the change of variable is not quite mechanical because this potential and
the one in the x space is based on the actual [inaudible] equation,
that we will discuss later.
But, I want to show you this out now because we will use it later again.
So, if we switch to log price instead of the price,
which we can do if you want to model non-negative prices.
Now, the potential in the new variable goes to negative infinity.
Actually, linearly for large negative values of y as y itself goes to
negative infinity and this means that the fall to the center in the original log-space,
can be equivalently described as an escape to
a negative infinity under a force that becomes asymptotic really linear in the log-space.
Now, another type of dynamics is obtained if we take
Theta positive and a and b also positive.
In this case, a typical potential is shown in this graph.
Here, I temporarily return to the original x-space instead of the log-price space y.
For such potential, we have a metastable state at x equals zero.
If a particle is initially placed somewhere near this point
and released it can escape so after some time over the barrier to the left,
or the barrier to the right.
If it crosses the left barrier,
the particle can escape to a negative infinity,
but if it first crosses the write barrier,
it can escape to positive infinity.
Finally, one more interesting scenario is obtained if we take
Theta larger than zero and a negative and b positive.
In this case, you will also have a potential similar to what is shown here.
In this case, the point x equals zero would be
unstable point rather than a stable, or metastable point.
It's unstable in this case because it's a maximum rather than a minimum.
Therefore, even a tiny perturbation completely knocks out
the particle that is initially put at x equals zero in such potential.
If we don't allow this particle to go to negative value,
we can put a fraction,
or absorbing boundary condition at zero.
The particle can then move to the right from the point x equals zero and initially,
it's potential will look like inverted harmonic oscillator potential of the GBM model.
So, initially, the motion looks like an exponential expansion,
but such exponential expansion does not continue indefinitely.
Once the particle approaches the global minimum point around x
[inaudible] slows down until it settles at this global minimum.
Now, this scenario may bring in one's mind several possible analogies.
On the physics side,
it reminds some theories of cosmology called inflationary models.
It's also related to very beautiful physics,
of the so-called Higgs Bosons.
If we come back to finance.
Then, as we already said it reminds us the inflationary behavior of the GBM model.
Imagine, for example, that we are mostly
interested in small values of the stock price around
zero and take a linear approximation to a general nonlinear function around this point.
In this case, if we use multiplicative noise that is,
the diffusion term that is linear in x,
we will obtain the GBM model as a result.
So, if we are unaware that
this linear dynamics is only an approximation to nonlinear dynamics,
then we might expect that a period
of unbounded exponential growth will continue indefinitely.
But, this picture shows that,
if the full model is nonlinear with the true potential as in this graph,
then this will just be an illusion after a while and
exponential expansion will slow down and
the system will achieve a non-trivial steady-state.
