How to Solve the Problem of Expert Advisor Over-Optimization

Hello, fellow forex traders!

Quite often you can come across sad stories from a novice algo trader about how he found excellent set files for an expert advisor, but after putting them to work on a real account, suffered losses. And it seemed he did everything right, even carried out forward testing, yet still lost money. Why does this happen? How does it turn out that settings with excellent test results "bleed" when working in real time? A situation like this is called over-optimization or parameter fitting among traders. That is what we will talk about today, and how to avoid it as well.

Over-optimization, or fitting, is improperly performed optimization. From a technical point of view, excessive optimization is finding such system parameters that fit the historical data on which the optimization was carried out very well, but are completely ineffective on current and future market data. In simple terms, it means finding settings that make a lot of money on history, but drain in real time. Over-optimization can be identified only by the results that the expert advisor gives in real trading. More precisely, by the mismatch between those results and the test results.

Nothing could be easier than describing the real symptoms of an over-optimized trading system: real account losses. Of course, all trading systems have losses, but they also have wins, and that is exactly how these systems grow their owners' accounts. But when those losses become very large over a short period of time, clearly unlike what happened with the system on historical data, then you can say for sure that the sets were overthought.

At the same time, a fitted system will not necessarily start draining immediately after installation; there is a good chance that the system will first still make a profit for some short period of time, and only after that will a series of uninterrupted losses begin.

Another way to determine system fitting is to compare the average annual return over the forward period with the real one. It should be roughly the same, and if it is not, then you are dealing with over-optimization. Nevertheless, if the settings set passed the forward test, then it is most likely workable. A workable set may well be a little fitted, when small discrepancies arise between the real and test results, but it can still remain profitable for a long time.

As I already said, the reasons lie in violating optimization rules. From practice I can name six examples of such violations:

Too many restrictions lead to incorrect results. If the rules of a trading system track too much price data or if there are too few trades compared to the number of rules, then the optimization results may be erroneous. Simply put, the more complex the system is, the more rules and filters it has, the greater the probability of fitting. There is a reason everyone who can be bothered recommends using as few filters as possible when creating your own trading system, ideally two or three.

Roughly speaking, for example, only a period of a trending market was included in the optimization segment, while the summer flat period was left out. In addition, the market often changes from year to year, and sometimes seemingly out of nowhere. This may be connected with certain global processes taking place inside the countries whose currencies form the pair we are considering. The character of a currency pair can sometimes change beyond recognition, to the point that strategies created for these pairs stop working altogether.

Often the developer of a system initially builds the possibility of over-optimization into the system. Here are a couple of such possibilities:

- Sequentially adding rules to observe increases in efficiency and excluding rules that do not work. For example, adding a stochastic, then adding RSI, then removing some other indicator.

- Testing many variants of one and the same rule. For example, entering the overbought/oversold zone and exiting it.

For the purpose of obtaining maximum efficiency from the expert advisor, the above techniques are quite justified, but you should be very careful when fine-tuning the expert.

I have already told you what standard error is and why it is needed. The more trades were completed, the lower the probability of error. Trading speed differs for all systems: some scalpers will rack up 300 trades on one pair in a month, while some will not even reach 50 trades in a year. Therefore, it is also important to choose the optimization period based on the "temperament" of the expert advisor.

Although an incorrect choice of set is not a direct cause of over-optimization, it is still a common cause of failures. First of all, it is necessary to analyze the set's trades. Profits and losses should be distributed as smoothly and evenly as possible, the smoother and more even the better. In no case should there be single trades bringing in 30% of all profit. Losing trades should also be distributed evenly.

When optimizing two parameters, we get an optimization chart like this:

The deeper the green color, the more profitable the settings are. It is always worth choosing from those sets of parameters that cluster together, from those parameters that are surrounded by parameters of the same colors. In that case, with minor market changes, the profitability of the expert advisor will not change much.

This is a special case of an incorrectly chosen segment for optimization. In this case, by coincidence, in the selected optimization segment the market has conditions that are as suitable as possible for the expert advisor. As a result, excellent sets are obtained that start losing in real time. Or conversely, a period with the worst market situation for the expert advisor is chosen, for example, a period of prolonged flat for a trend-following bot. The optimization produces a weak result, and the robot is undeservedly shelved in the archive.

There may be two variants here. The first is when all settings are optimized with a very small step that is impractical for optimization. The second is when normal profitable settings are fine-tuned, again with a small step, in order to squeeze out the maximum. The second variant is normal, but the first leads to over-optimization simply because the tester will drive all settings into narrow ranges of operability, after leaving which the expert advisor will begin to lose. And it will definitely leave them as soon as the current market changes even a little.

As you can see, half of the optimization errors are connected with an incorrect algorithm for carrying it out, and the other half with a small data sample. Try not to rush and carefully follow the technology of expert advisor optimization. Never begrudge a chunk of history for optimization and do not forget about the forward test (it is preferable to remember the backward test as well), and then, I hope, you will avoid optimization traps and your algo trading will be profitable and pleasant.

Sincerely, Dmitry aka Silentspec TradeLikeaPro.ru