Ozone vertical profiles from limb and nadir sensors #179
Comments
|
In the old VM there used to be a shared .cdsapirc (I think it was Vincenzo's account), but there isn't anymore. Create your own config file in import os
os.environ["CDSAPI_RC"] = os.path.expanduser("~/calmanti_sandro/.cdsapirc")If you don't know how to create your .cdsapirc, you can follow these instructions: https://github.com/ecmwf/cdsapi?tab=readme-ov-file#configure However, it looks like the CDS webportal is experiencing problems at the moment. If the URL doesn't work, you might want to try again tomorrow. |
|
Ciao @malmans2 thanks for the previous fix. There's an issue with metadata of some ozone file that we need to handle. For most of the ozone files it is possible to add the time dimension by using the global attribute "time_coverage_start". This is what we did with the notebook on total ozone, and the same will probably work (not tested with all datasets though) with data files on vertical ozone profile from nadir sensor. The issue is with vertical profiles from limb sensors (see example attached). These files are missing any time related attribute and the only way to get the valid time is from the first 6 digits of the filename itself. I have checked all files from limb sensors and they use the same global attributes: the time mark is missing. Since our notebook is about the consistency of nadir and limb sensors, it would be necessary to use the same approach for adding the time dimension. From what I see in the previous ozone notebook, the most efficient whay to do so is to add a different method for deriving the time dimension, for example something like Does it exist already? Can you add this as a further method, based on a simple paring of the first digits of filenames? It is not optimal to have different rules for the metadata in the same catalogue. I will mention this in the quality assurance. However, for the notebook we probably need to handle this as I don't think it might be changed quickly in the date. |
|
Using the filename is unreliable as we have various level of caching that use custom names. However, I think we can use the attributes for this dataset as well. Here are the attributes: {
"summary": "Monthly zonal mean ozone profiles in 10-deg latitude zones and data uncertainty characterization",
"sensor": "SCIAMACHY",
"platform": "ENVISAT",
"year": "2003",
"number_of_altitude_levels": "61",
"number_of_latitude_bins": "18",
"geospatial_lat_resolution": "10 deg ",
"geospatial_lat_min": "-90",
"geospatial_lat_max": "90",
"value_for_nodata": "NaN",
"date_created": "20170828T124530",
"creator_name": "Viktoria Sofieva",
"creator_email": "[email protected]",
"address": "P.O.Box 503, 00101 Helsinki, Finland",
"naming_authority": "FMI - Finnish Meteorological Institute",
"Conventions": "CF-1.5",
"standard_name_vocabulary": "NetCDF Climate and Forecast(CF) Metadata Convention version 18",
"file_version": "fv0001",
"tracking_id": "6961ec59-ca84-4cce-8034-d48429a967a3",
"id": "C3S_OZONE-L3-LP_CONC_MZM-SCIA_ENVISAT_CCI-200302-fv0001.nc",
}It looks like we can use the def add_time_dim(ds):
if "time_coverage_start" in ds.attrs:
time_str = ds.attrs["time_coverage_start"][:6]
else:
time_str = ds.attrs["id"].split("-")[-2]
time = pd.to_datetime(time_str, format="%Y%m")
return ds.expand_dims(time=[time])A couple of comments:
|
|
Ok good. I will use this approach and adapt casa by case if necessary. |
|
Ciao @malmans2 it works but it's not sufficient and I need to think of some solution or change the objective of the analysis. Problem: some of the data files have no time attributes and no "id" attribute either. Unless you can find in the example below any other possibility to derive the time dimension there are several other cases like "cmzm", where I was not able to find any useful attribute to derive the time dimension. Unfortunately, while some of the limb files have a useful id attribute (see exemples in the attached notebook), the merged products that would be useful for the comparison/consistency analysis do not have the time information in the metadata. I'm in contact with Joao to see if we can solve this but I don't think it will be quick. It seems to me that either we can think of some (temporary?) fix which uses the file name or I need to think of some other analysis. Let me know what you think, and I'm available to discuss on the phone if necessary. |
|
Got it. I'm about to leave the office, so I'll take a look on Monday and will let you know. Worst case scenario we can probably turn off the usual caching for this analysis. Have a good weekend! |
|
Great, consider the entire "limb" dataset is about 1.2G, we're not dealing with a large dataset. See you on monday. |
What happened?
We have started to work on a new notebook for ozone .
The idea is to compare data from about 10 to 15 different number of sensors. Here we start the analysis with just one of them.
ozone_vertical-profile_v0.zip
There's an error about missing or incomplete .cdsapirc
We have already used this dataset before. Do we need to use a different/local cdsapirc or are we using any wrong import module?
Minimal Complete Verifiable Example
Relevant log output
Anything else we need to know?
No response
Environment
Note: you may need to restart the kernel to use updated packages.
The text was updated successfully, but these errors were encountered: