Blank rows in rawdata.get_daily_prices after resample

[GolovinAlexx]

Hello all!

I'm learning the backtesting part of pysystemtrade. I am working with my own historical daily data.

As I understand, at Raw data stage we have "adjusted daily priced", calculated from "raw adjusted prices" using the resample("1B") function.

Business day != market day, because market may be closed on Monday for example. After resampling pandas will insert an empty row on Monday in the daily data. Now the return between Tuesday and Friday is return between Tuesday and Monday and is NaN. Return between Monday and Friday is NaN too.

Is this is correct way of code? And is it logical to damage a simple raw data by inserting blank rows at business, but not market days?

[robcarver17]

Sorry I don't follow, can you paste an example in?

[GolovinAlexx]

  from systems.provided.futures_chapter15.basesystem import futures_system
  
  system = futures_system()
  
  daily_prices = system.rawdata.get_daily_prices("EDOLLAR")
  daily_returns = system.rawdata.daily_returns("EDOLLAR")
  
  daily_prices.to_csv("edollar_daily_prices.csv")
  daily_returns.to_csv("edollar_daily_returns.csv")

For example, 2022-09-05 for prices, and 2022-09-05, 2022-09-06 for returns.

[tgibson11]

Aren't you losing a day of returns there? From 09-02 to 09-06 there is a loss that appears to unaccounted for because of the Nan. The overall return for 09-02 to 09-07 should be a small loss, but the return series only shows the gain from 09-06 to 09-07.

I would expect a forward fill of the adjusted prices at some point, either in the resampling, or when the returns are calculated.

Am I looking at this wrong?

[robcarver17]

If you forward fill the adjusted prices, your vol estimate will be biased.

[GolovinAlexx]

Well perhaps you weren't expecting it, but it's correct :-)

What do you think should happen instead?

Thanks for the answer!

Maybe I'm confusing because I trade on single stock exchange, and if the working day is a holiday, we don't have data flow like on the weekend. So, no data - no row in the database (e.g. for weekends).

But if you have several exchanges in your portfolio, then you are right - you can trade one instrument on an opened exchange and do not trade another one on a closed exchange.

But, as far as I know, in the Arab countries Sunday is the market day, and the stock exchange is open. So this Sunday's data will be lost after resampling ("1B").

Maybe theoretically we need some market calendar? And some fancy function instead of resample("1B"), where "B" is not a business, but a market day for the specific market...

[tgibson11]

Ok, you don't want the adjusted prices themselves to be forward filled, but why not do that as an intermediate step when calculating the returns?

[robcarver17]

It's a lot of work to have market calendars (believe me, we used to do this at AHL). I have absolutely no intention of maintaining them, or introducing a third party dependency on someone elses market calendar library.

And I also know about Arab countries (I was brought up in Dubai). But I don't trade any Arab markets (or Israel, which would have the same issue), so I'm not bothered.

There are 'proper' ways of getting round this, involving eg brownian bridges, but it's .not something to get your knickers or any other kind of underwear in a twist about because it makes bugger all difference.