Get baseline running with full US retention + synthpop

[paddy-r]

Already managed full pidp retention through data generation pipeline (see #444) but Minos currently initialises but doesn't complete a year of run. Next steps:

• Running with US (i.e. non-synthpop) input pop...
  • Current issue: KeyError in mortality lookup due to missing values of sex. - > fixed in MICE integration #458 (missing-types imputed/replaced with NaN during imputation; KeyError was due to -9 values of sex in mort/fert lookups)
  • Account for missingness, either via naive imputation by hierarchical sampling from existing data (already done outside Minos, must copy over) or via MICE (already on child_poverty_May24). separated into Get MICE working with full pidp retention #457
  • Still some missingness in imputed data -> partly fixed in e7972aa for sex and ethnicity but not for region (11/29271 = 0.04% for 2020 data) -> marking as done and moving to new issue (Missingness after MICE imputation #459) as this is of more general interest for other variables with missingness; can continue here with the very small level of missingness
• Running with full synthpop (or 1% of it)...
  • Update to full GB synthpop (currently Scotland only) -> done in GB synthpop: initial try at GB synthpop in data gen #461
  • Refactor to select only necessary variables before merging US with SP, to reduce size -> done in GB synthpop: initial try at GB synthpop in data gen #461 in US_individual_upscaling, multiple commits
  • Generate synthpop from fully retained US population, this must include composite vars from Minos, which was another motivation for getting full retention through data generation. (Already done outside Minos, must copy over and test.) -> as above
  • Generate additional years of synthpop input (e.g. 2019-2021, as in IE (II)) -> moved to Full GB synthpop for fertility - final tasks #464 as want to draw a line here
  • Check correct years being selected in Minos/synthpop (i.e. go through year naming conventions used) - > done in GB synthpop: initial try at GB synthpop in data gen #461, various places, e.g. gb_scaled config files (2019, rather than 2020), DG makefiles, US_individual_upscaling, generate_repl_pop; not complete though, as (e.g.) rate tables and external data for validation are still off by a year -> moving to Full GB synthpop for fertility - final tasks #464 to complete
  • Allow for specification of synthpop as input population -> done in GB synthpop: initial try at GB synthpop in data gen #461 in various places
  • Ensure latest LSOA, ward, LA and region definitions present -> done in GB synthpop: initial try at GB synthpop in data gen #461, adapted IE code to US_individual_upscaling.add_spatial_attributes, not used anywhere yet but will be required for validation -> carried over to Full GB synthpop for fertility - final tasks #464 so as not to forget to test

[paddy-r]

Closing as all done in #458, #460, #461.