Skip to content

Optimize simulation data download not outputting unused resolutions / stratification / aggregations #720

@mknaranja

Description

@mknaranja

Motivation / Current Behaviour

When running the scripts to download and transform simulation data, a lot of output is generated:

Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_infected.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_infected_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_deaths.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_deaths_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_germany.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_germany_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_infected_state.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_infected_state_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_state.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_state_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_infected_county.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_infected_county_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_county.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_county_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_gender.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_gender_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_state_gender.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_state_gender_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_county_gender.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_county_gender_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_age.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_age_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_state_age.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_state_age_ma7.json
Information: Data has been written to /localdata1/kueh_mj/code/memilio-abm-openmp/data/pydata/Germany/cases_all_county_age.json
...

However, most of the files are not needed for a particular run as mostly only one setting (e.g. age groups or not, gender or not, and a particular geographical resolution is used).

Enhancement description

We should introduce some options of resolution for memilio-epidata to only output the files needed for the simulation.

Additional context

No response

Checklist

  • Attached labels, especially loc:: or model:: labels.
  • Linked to project

Metadata

Metadata

Assignees

Labels

class::improvementCleanup that doesn't affect functionalityloc::data handlingThis issue concerns any kind of data handling.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions