Start building intraday database?

[js190]

I'm now collecting hourly data for ~130ish futures via IB, storing these in Parquet files for each contract. I'm only collecting instruments from the standard pysystemtrade roll schedule and config. Currently I'm not using the execution part of the codebase, only the core DO model.

This discussion is to get a lead on what would be the lowest frequency of data we could usefully use for intraday style strategies.

I'm thinking of having a core execution system which takes the pystemtrade model position as inputs and then optimises the execution as an independent alpha. We could have a libray of such execution strategies, which could open the door for pure intraday strategies which could operate as a core strategy themselves not just executuion.

• Does anyone have a prior on what the frequency data we could/should target for a system running super slow Python over IB web API?
• How much data would we need to get reliable signal, IB provide ~1 year of hourly data.
• Does it make sense that contracts closer on the curve (with more vol) would be more of interest for intraday?

[robcarver17]

That's exactly the paradigm that's in place right now, except there are only two execution strategies (market and my own simple execution algo).

For my own intraday model ideas, the plan is to have the overnight backtest generate conditional orders, which would then be conditionally executed.

For example if we think we're in a mean reversion channel 100/110, then we'd have conditional orders like: if flat, buy at 100 or below; sell at 110 or above. If long then close at 109, or stop out at 95. If short then close at 101, or stop out at 115. Position sizes would be baked into the conditional order as well.

Then basically there would be an additional method in the run_stack_handler that checks to see if conditional orders have been satisified and if so submits them as real orders.

I plan to work on this at some point in the next couple of months.

As a rule of thumb, information scales with root time. So if 20 years is adequate with daily data, then 20/root(8) = 7 years would be adequate for hourly data.

[js190]

Cool, what's your prior on appropriate frequency of intraday data for a one man band? As the frequency get's higher the model run to execution latency becomes more cruicial. 1 hour certianly seems managable - would you dare to go higher?

[ChrisAllisonMalta]

My 2 cents would be that the time period should cover a variety of different environments like highly volatile February & March 2020 as well as quieter periods like 2017 when (equity) vol was very low

[js190]

Makes sense - the more data the better then we naturally cover known and unknown events. Would you get intraday data from barchart.com?

[ChrisAllisonMalta]

I've used FirstRate Data in the recently