How to Optimize a Forex Expert Advisor on Historical Data

Hello, forex traders! On the pages of the blog, we have already discussed quote preparation and expert advisor testing, and now it is time to talk about expert advisor optimization.  Optimization has both opponents and supporters, with opponents being more numerous.

Why does this happen? The process of expert advisor optimization is quite multifaceted; to optimize an expert advisor correctly, you need some knowledge that is unavailable to a beginner due to a lack of experience. Fuel is added to the fire by the abundance of various information on the internet, often incorrect or distorted. That is exactly why proponents of optimization have so many opponents: people simply do not know how to use it. In this lesson, I will explain how to optimize an expert advisor properly and, I hope, save some beginners a couple of deposits.

It is no secret that manual trading systems become outdated over time and stop bringing the profit they did in the past. At the same time, old losing strategies suddenly start performing well. The reason is the cyclical nature of the market, when some trading conditions are replaced by others. The same thing happens with expert advisors. Market conditions stop fitting the strategy built into the advisor's algorithm, and it begins to lose money. So what should be done in such a situation, simply delete the advisor and forget about it? Fortunately, in this case optimization comes to our aid. So what is it? In essence, it is simply adjusting the advisor's parameters to current market conditions, correcting the strategy, and adapting it to changed conditions. Just as traders adjust their manual trading systems to the current market, algo traders adjust their expert advisors. Changes and adaptation are an integral part of the trading process. Whoever fails to change in time is left behind; such is a trader's life.

So, we have determined that optimization is still an important and even necessary part of trading with expert advisors. Besides, let me repeat, you already know how to download quotes, install expert advisors into the terminal, and test them, and you understand what "sets" or set files are. Now it is time to open the terminal and run an optimization. When I talked about expert advisor testing, I told you about three testing models and their features. I recommend optimizing expert advisors using the "every tick" model. This is the most accurate model, and the probability that you will do something incorrectly becomes lower. Let me give you an example of testing an expert advisor with all three models to compare the final results, so that you can clearly see for yourself that my words are true:

"Open prices only" model

"Control points" model

"Every tick" model

So, I think now no one will have any questions about why it is preferable to optimize using the "every tick" model. Notice how strongly the first option differs from the second and third. Results with the "control points" model may not differ too much from the results with the "every tick" model. Only in that case is optimization by control points acceptable for the sake of saving time. Therefore, you should first run the expert advisor tests in all three modes and, after comparing the results, make a decision.

The "Optimized parameter" option lets you select the main output parameter by which each run will be evaluated, namely:

You may also notice the checkbox next to the genetic algorithm. If you remove the checkmark, the tester will run through absolutely every possible combination of expert advisor parameters. In that case, optimization will most likely take about 100500 years. Fortunately, the terminal has a built-in capability to search for optimal parameters using a genetic algorithm, which makes it possible to perform optimization in just a few hours or days. In principle, for now this is all you need to know, because that checkbox is a topic for an entirely separate article.

Expert advisor optimization is generally carried out the same way as testing, with money management turned off and with a 0.1 lot. To do this, you need to find the corresponding block in the advisor parameters and set a fixed lot of 0.1. The table in the input parameters tab contains 4 columns: the parameter itself, its current value, the initial value for optimization, the step, and the final value for optimization. What does all this mean? For example, we want to select the stop loss that is optimal for the expert advisor over a certain period. To do this, we set the initial stop value (start), say, 10 points. We set the final value, for example 60, because with a stop larger than 60 there is nothing to do intraday. You can set even a million, but these values should be chosen wisely, otherwise it will greatly increase the time spent on optimization. And finally, the step. If we specify a step of 10, for example, we get the following sweep of the selected parameter: 10, 20, 30, 40, 50, 60. Here too, it is worth approaching the matter logically; there is no point in setting a step of 1 or a step of 10 (5). A step of 2 will be quite suitable, which will also save resources.

The more parameters you test at one time, the longer the optimization will take. But there are situations when there are so many parameters that the terminal refuses to run the optimization and reports this in the log. In that case, it is necessary to divide all parameters into 4 groups: parameters that strongly affect the result, those that affect it moderately and weakly, and those that do not affect it at all. The degree of influence can be determined by a trial optimization of a single parameter taken separately. Naturally, the parameters that strongly affect the results should be optimized first, and then all the rest in order of importance.

This tab is also designed to save optimization time. Here you can set your own rules for filtering out results, even at the optimization stage itself. For example, limit the maximum continuous number of losing trades to four, and the maximum drawdown to 10 percent. Then only the optimization results that satisfy these parameters will be displayed.

In principle, this is the main question when optimizing an expert advisor, and whether you make some money or lose it will depend on the competent choice of this segment. It is exactly this point that is the source of such a large number of fierce opponents of optimization and of working with expert advisors in general.

Beginner's approach

So, here is the approach used by many beginners. A short available history period is taken, often no longer than a couple of months so that the wait is not long, and the "start" button is pressed. After completion, the pass that produced the most "bucks" is selected. That's it, the set is installed on a live account and the beginner prepares a money bag, often also boasting about this "holy grail." And then, of course, the account is blown.

Popular approach

This approach is the most common among non-beginners. Two history sections are chosen, an optimization section and a forward-test section. At the same time, the optimization section is located before the forward-test section, with no gaps in days. As a rule, the first two thirds of the selected history section are chosen for optimization, and the remaining one third is allocated to the forward test. The best options are selected on the optimization section, and good settings are then chosen on the forward period, which the advisor has not yet "seen." The choice of the history section is left to the trader's discretion. At the same time, the larger the section, the better the settings are adapted to various unexpected market events, the longer it will earn with the same settings, and the later the sets will become outdated. But at the same time, the expert advisor's overall profit will be smaller. The shorter the optimization period, the more the settings are adapted to a specific market period and specific trading conditions, but the greater its efficiency under those conditions and the greater the profit. You can optimize once a week, or once every five years, whichever suits you better. But there is one drawback in traders' efforts to find optimal parameters for a short section: you never know for sure when the settings will become outdated. You may guess right with the set and the expert advisor will trade profitably all the following week, but it may also happen that the nature of the market changes on Monday and the expert advisor will drain the account all week. Personally, this lottery does not inspire me somehow, and when optimizing I do not strive to chase maximum efficiency.  Instead, I select sets "for years."

In addition, there is an opinion that looking further back than three years makes no sense. I cannot dispute this statement with facts, but I still choose an optimization period of no less than 6 years, with a forward-test segment of no less than two years. It makes me feel calmer.

In general, chasing the trend has a right to exist, especially if you are a pro at it and you really manage to predict in time when your settings will stop working.

Voodoo approach

I have often come across such a voodoo approach on the internet, presented as an approach for real pros. The history segment is divided into two equal parts. Optimization is carried out separately on each of them, and 10-20 variants of successful settings are saved. Then the settings from the first and second segments are compared, and those that are approximately similar are accepted as optimal. This is complete nonsense, takes away a wagonload of time, and carries no meaningful value at all. Using this voodoo method, you will kill a bunch of hours on nonsense and ruin your eyesight completely.

My approach

The goal of the approach is to find universal settings that in the long term will ensure stable profitability regardless of changes in the nature of the market, volatility, or the global trend, settings that will not become outdated in a week, a month, or a year. At the same time, unfortunately, not every expert advisor is able to pass my tests.

So, let us assume we have a 15-year piece of history, no less than 10, say from 2000 to 2015. We divide this piece into the following periods: 2000-2003 is our backward-test piece, 2003-2012 is the optimization period, and 2012-2015 is the forward test. After optimization, we conduct forward testing as usual, selecting the 10-20 most successful sets. After that, we run the selected sets on the backward-test section. The results should be similar to those obtained in the forward test. Those sets that withstand the test remain for further comparison. Next, we run a test on the remaining sets across the entire piece of history and choose the one whose results are better than the others. In the end, one most well-adapted settings set remains.
How should sets be selected at the first stage, the forward test? Very simply: the most important thing for us at this stage is the appearance of the balance curve. Ideally, it should be a straight line going from the lower left corner to the upper right corner. At the same time, there is no point in looking through all the best sets in a row, as they are often almost identical. It is worth selecting only those among the best sets that differ in the number of trades.

So, we have obtained the cherished set files for our expert advisor. At the same time, it is still too early to put the expert advisor on a live account. It is time to check our sets on a demo account. In principle, 20-30 trades on one pair will definitely be enough to understand whether the set has worked out. In addition, it makes sense to check whether the trades on the demo coincide with the trades for the same period in the tester. To do this, they run a test and compare the readings. If the trades match at least approximately, then everything is fine. There is no need to expect pip-for-pip and second-for-second matches, and it is also not scary if some trades are missing. The overall picture, the overall similarity, is what matters. In real conditions, the operation of the expert advisor will always differ slightly from the test: sometimes because of slippage, sometimes the advisor did not enter because the spread was too high, sometimes because of requotes, or something else. But the picture should certainly not differ radically. If you see a picture in the test that is completely unlike the live one, then optimizing such an expert advisor is useless: no matter how beautiful a set you choose, the expert advisor will still trade differently.

Today you learned the basic principles of optimizing expert advisors. Nevertheless, there are still many different tricks that I could not talk about within the scope of one article. And yet the knowledge you gained today will be quite enough to optimize an expert advisor that works on periods from H1 and above in such a way that it will bring you profit for many years. Optimize expert advisors correctly, and then, perhaps, algo trading will become a somewhat more attractive activity in the eyes of traders.

Forum topic

Best regards, Dmitry aka Silentspec
TradeLikeaPro.ru