run dynamic backtest. #529

rorymac · 2022-01-30T04:10:48Z

rorymac
Jan 30, 2022

Hello

I am trying to run the dynamic backtest shown on Rob's blog : https://qoppac.blogspot.com/2021/10/mr-greedy-and-tale-of-minimum-tracking.html#comment-form

From the comments: https://gist.github.com/robcarver17/11ec3a6c4370aa8e10c895a483420650

a few things made a few modifications due to errors in finding "imports"
Original code:
from systems.forecasting import Rules from systems.basesystem import System from systems.forecast_combine import ForecastCombine from systems.forecast_scale_cap import ForecastScaleCap from systems.provided.dynamic_small_system_optimise.portfolio_weights import portfolioWeightsStage from systems.provided.dynamic_small_system_optimise.optimised_positions import optimisedPositions from systems.provided.dynamic_small_system_optimise.accounts import accountForOptimisedStage from systems.provided.dynamic_small_system_optimise.risk import Risk from systems.rawdata import RawData from systems.positionsizing import PositionSizing from systems.portfolio import Portfolios

My modifications:
from systems.forecasting import Rules from systems.basesystem import System from systems.forecast_combine import ForecastCombine from systems.forecast_scale_cap import ForecastScaleCap from systems.provided.dynamic_small_system_optimise.portfolio_weights_stage import portfolioWeightsStage from systems.provided.dynamic_small_system_optimise.optimised_positions_stage import optimisedPositions from systems.provided.dynamic_small_system_optimise.accounts_stage import accountForOptimisedStage from systems.provided.dynamic_small_system_optimise.risk import Risk from systems.rawdata import RawData from systems.positionsizing import PositionSizing from systems.portfolio import Portfolios from sysdata.sim.csv_futures_sim_data import csvFuturesSimData # to run with csv data

Also added the use csv data.

added this line at the end of the script.

system.accounts.optimised_portfolio()

Runs all is fine, but get no output
have added the following lines

print(system.accounts.optimised_portfolio())
print(system.accounts.optimised_portfolio().percent().stats())

to get this after a the processing 👍

1970-02-03    NaN
1970-02-04    NaN
1970-02-05    0.0
1970-02-06    0.0
1970-02-09    0.0
             ... 
2021-12-06    0.0
2021-12-07    0.0
2021-12-08    0.0
2021-12-09    0.0
2021-12-10    0.0
Freq: B, Length: 13529, dtype: float64
 Weighted account curve; use object.stats() to see methods
Traceback (most recent call last):
  File "basic_dynamic_backtest_RMedit.py", line 65, in <module>
    print(system.accounts.optimised_portfolio().percent().stats())
TypeError: 'accountCurveGroup' object is not callable

what have I missed this time?

robcarver17 · 2022-01-31T16:29:20Z

robcarver17
Jan 31, 2022
Maintainer

It's quite hard to know why you're getting a string of zeros, do you have the entire script you ran so I can just paste it in and try and replicate the error?

Having said all that, you need:
system.accounts.portfolio().percent.stats()

Note no () after the percent -it's a property, not a method of the class.

4 replies

rorymac Jan 31, 2022
Author

Rob

Here is the full script :

from sysdata.config.configdata import Config

from systems.forecasting import Rules
from systems.basesystem import System
from systems.forecast_combine import ForecastCombine
from systems.forecast_scale_cap import ForecastScaleCap
from systems.provided.dynamic_small_system_optimise.portfolio_weights_stage import portfolioWeightsStage
from systems.provided.dynamic_small_system_optimise.optimised_positions_stage import optimisedPositions
from systems.provided.dynamic_small_system_optimise.accounts_stage import accountForOptimisedStage
from systems.provided.dynamic_small_system_optimise.risk import Risk
from systems.rawdata import RawData
from systems.positionsizing import PositionSizing
from systems.portfolio import Portfolios
from sysdata.sim.csv_futures_sim_data import csvFuturesSimData # to run with csv data


def futures_system_dynamic():

    #data = dbFuturesSimData() # to run with mongo DATA
    data = csvFuturesSimData() # to run with csv data

    config = Config(
            "systems.provided.futures_chapter15.futuresconfig.yaml")


    system = System(
        [
            accountForOptimisedStage(),
            Portfolios(),
            PositionSizing(),
            RawData(),
            ForecastCombine(),
            ForecastScaleCap(),
            Rules(),
            portfolioWeightsStage(),
            optimisedPositions(),
            Risk()
        ],
        data,
        config,
    )
    system.set_logging_level("on")
    del(system.config.instrument_weights)

    system.config.instruments = ['AEX', 'AUD',  'BITCOIN', 'BOBL', 'BTP', 'BUND', 'CAC',
                                 'COPPER', 'CORN', 'CRUDE_W_mini', 'EDOLLAR',  'EUR', 'EUROSTX', 'FEEDCOW',
                                 'GBP', 'GOLD_micro', 'JPY',  'LEANHOG',
                                 'LIVECOW', 'MXP', 'NASDAQ_micro', 'NZD', 'OAT', 'PALLAD', 'PLAT', 'SHATZ', 'SMI',
                                 'SOYBEAN', 'SOYMEAL','SP500_micro', 'US10', 'US2', 'US20', 'US5',
                                 'RICE',  'SOYOIL','GASOILINE','HEATOIL',
                                 'VIX', 'WHEAT']

    system.config.notional_trading_capital = 350000
    system.config.use_instrument_div_mult_estimates = True
    system.config.use_instrument_weight_estimates = True
    system.config.instrument_div_mult_estimate['dm_max'] = 5
    system.config.buffer_method = "forecast"

    return system

system = futures_system_dynamic()
#system.accounts.optimised_portfolio()
#print(system.accounts.optimised_portfolio())
system.accounts.portfolio().percent.stats() 
#system.accounts.portfolio().stats()
#system.accounts.portfolio().curve().plot()

Running that shows it has run but shows nor output.

end of output:

Calculating forecast method buffers for WHEAT
Calculating buffered positions
Calculating pandl for instrument for WHEAT

robcarver17 Feb 1, 2022
Maintainer

That's because in most IDEs if you just paste in a load of commands it won't show you the output unless you use a print statement. So I ran your exact scriipt but with a print statement:

print(system.accounts.portfolio().percent.stats())
[[('min', '-10.74'), ('max', '8.848'), ('median', '0.07641'), ('mean', '0.102'), ('std', '1.399'), ('skew', '-0.04293'), ('ann_mean', '26.09'), ('ann_std', '22.38'), ('sharpe', '1.166'), ('sortino', '1.629'), ('avg_drawdown', '-11.23'), ('time_in_drawdown', '0.9198'), ('calmar', '0.4979'), ('avg_return_to_drawdown', '2.322'), ('avg_loss', '-0.9688'), ('avg_gain', '1.03'), ('gaintolossratio', '1.063'), ('profitfactor', '1.23'), ('hitrate', '0.5364'), ('t_stat', '8.478'), ('p_value', '2.519e-17')], ('You can also plot / print:', ['rolling_ann_std', 'drawdown', 'curve', 'percent'])]

print(system.accounts.portfolio().percent)
index
1970-02-03         NaN
1970-02-04         NaN
1970-02-05         NaN
1970-02-06         NaN
1970-02-09         NaN
                ...   
2021-12-06   -0.001475
2021-12-07    0.000000
2021-12-08   -0.001777
2021-12-09    0.180357
2021-12-10   -0.092857

rorymac Feb 2, 2022
Author

Rob
Fantastic that worked.
Thank you!
A few more questions:
The config file in this case :
config = Config( "systems.provided.futures_chapter15.futuresconfig.yaml")
are the settings in there over-ridden selectively by the script?
I assume so as forecasts weights are calculated for each instrument in the output despite them being specified in the config yaml

however when taking them out of the config this appears in the output:
WARNING: No forecast weights - using equal weights of 0.143 over all 7 trading rules in system
(I had added a few more in)
so really the question is how to specify and or optimise for various settings?

robcarver17 Feb 3, 2022
Maintainer

https://github.com/robcarver17/pysystemtrade/blob/master/docs/backtesting.md#configuration

rorymac · 2022-02-08T15:09:23Z

rorymac
Feb 8, 2022
Author

Rob

upgraded to latest 3121554
Had to to take a few days off. So lost contact with a couple of references in the backtest script, namely :

from systems.provided.dynamic_small_system_optimise.portfolio_weights_stage import portfolioWeightsStage from systems.provided.dynamic_small_system_optimise.optimised_positions_stage import optimisedPositions from systems.provided.dynamic_small_system_optimise.accounts_stage import accountForOptimisedStage from systems.provided.dynamic_small_system_optimise.risk import Risk

looking through the commits around 244a83a, there appear to be shifts from :
[systems/provided/dynamic_small_system_optimise/optimised_positions_stage.py]
to sysquant.

Is that about right ? if I dig into sysquant I should find the references ?

10 replies

robcarver17 Feb 9, 2022
Maintainer

Yes those are before buffering and the algo

After buffering (static system): system.accounts.get_buffered_position( instrument_code, roundpositions=True)
After greedy algo (dynamic system): system.optimisedPositions.get_optimised_position_df()

(Note in production I don't do this - I run the buffering and or greedy algo on the unrounded positions, taking into account my actual live current positions, rather than what the backtest thought it had on yesterday)

rorymac Feb 16, 2022
Author

It is late and perhaps I am going mad.

has anyone noticed this ?
running the backtest of the optimised using the script above, but I am trying to get the portfolio daily returns:

acc_curve=system.accounts.portfolio()
print("the daily returns")
print(acc_curve.net.daily.percent)

returns_df = acc_curve.net.daily.percent.to_frame()
returns_df.to_csv('daily_returns.csv')

The output (truncated) on the command line :

the daily returns
index
2000-01-19         NaN
2000-01-20         NaN
2000-01-21         NaN
2000-01-24         NaN
2000-01-25         NaN
                ...   
2021-12-06   -0.003421
2021-12-07    0.000000
2021-12-08   -0.001777
2021-12-09    0.178579
2021-12-10    0.000000
Freq: B, Length: 5713, dtype: float64
 Weighted account curve; use object.stats() to see methods

so that looks like the daily returns... BUT when I try to put it into a df and subsquently a csv file, I get all the instruments and the returns by instrument.

returns_df = acc_curve.net.daily.percent.to_frame()
print(returns_df)```

       AEX  AUD  BITCOIN  BOBL  BTP  BUND  CAC  COPPER  CORN  ...  SOYMEAL  SOYOIL  SP500_micro  US10  US2  US20  US5  VIX  WHEAT

index ...
2000-01-19 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2000-01-20 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2000-01-21 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2000-01-24 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2000-01-25 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2021-12-06 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2021-12-07 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2021-12-08 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2021-12-09 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN
2021-12-10 NaN NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN

[5713 rows x 40 columns]```

Ignore the NaN (at least I think you can, when I look at the file in a spreadsheet there are numbers and zeros)
I think the code is correct : https://github.com/robcarver17/pysystemtrade/blob/master/docs/backtesting.md#stage-accounting
shows it to be so.

robcarver17 Feb 16, 2022
Maintainer

Not surprisingly, (to me at least) .to_frame() turns something into a data frame. All account curve objects that are 'grouped' will do this.

What you need is .as_ts (as time series)

type(system.accounts.portfolio().as_ts)

rorymac Feb 16, 2022
Author

And Lo, it was so!

BTW that was very interesting, as I understand it the results of system.accounts.portfolio().net.daily are dataframe "like" as you point out in the doc's, pushing the results into a DF then expands the results into their constituent parts, so would that mean to sum the rows would give you same result?
Trying to do the comparison between standard and dynamic a la your chapter in trading evolved, mostly to try and understand the stats and do the exercise.
Thanks Rob

robcarver17 Feb 16, 2022
Maintainer

" pushing the results into a DF then expands the results into their constituent parts, so would that mean to sum the rows would give you same result?"

It ought to, with the caveat that the whole 'weighted' vs 'unweighted' https://github.com/robcarver17/pysystemtrade/blob/master/docs/backtesting.md#weighted_acg

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run dynamic backtest. #529

{{title}}

Replies: 2 comments 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

run dynamic backtest. #529

rorymac Jan 30, 2022

Replies: 2 comments · 14 replies

robcarver17 Jan 31, 2022 Maintainer

rorymac Jan 31, 2022 Author

robcarver17 Feb 1, 2022 Maintainer

rorymac Feb 2, 2022 Author

robcarver17 Feb 3, 2022 Maintainer

rorymac Feb 8, 2022 Author

robcarver17 Feb 9, 2022 Maintainer

rorymac Feb 16, 2022 Author

robcarver17 Feb 16, 2022 Maintainer

rorymac Feb 16, 2022 Author

robcarver17 Feb 16, 2022 Maintainer

rorymac
Jan 30, 2022

Replies: 2 comments 14 replies

robcarver17
Jan 31, 2022
Maintainer

rorymac Jan 31, 2022
Author

robcarver17 Feb 1, 2022
Maintainer

rorymac Feb 2, 2022
Author

robcarver17 Feb 3, 2022
Maintainer

rorymac
Feb 8, 2022
Author

robcarver17 Feb 9, 2022
Maintainer

rorymac Feb 16, 2022
Author

robcarver17 Feb 16, 2022
Maintainer

rorymac Feb 16, 2022
Author

robcarver17 Feb 16, 2022
Maintainer