More metrics #54

ocefpaf · 2024-02-13T16:59:19Z

TODO (moved to #56):

parallelize update_metrics
HF Radar Stations # This is a hardcoded number in the notebook (165), is that still valid @MathewBiddle?
National Platforms # Lots of services and should be into a separate module
NGDAC Glider Days # Implemented but must change to glider-data-days

ocefpaf · 2024-02-13T19:17:04Z

@MathewBiddle this one is getting too big to review. I'll address the remaining points in another PR.

MathewBiddle · 2024-02-14T18:01:34Z

HF-Radar is hardcoded 😢 . At one point you could parse the information from http://hfrnet.ucsd.edu/sitediag/stationList.php, but that doesn't seem to be the case anymore. Let's leave it hardcoded for now and update it once we have a source.

MathewBiddle · 2024-02-14T18:03:30Z

Is this ready for review?

ocefpaf · 2024-02-14T18:59:17Z

HF-Radar is hardcoded 😢 . At one point you could parse the information from http://hfrnet.ucsd.edu/sitediag/stationList.php, but that doesn't seem to be the case anymore. Let's leave it hardcoded for now and update it once we have a source.

OK. I'll add a note to check hfrnet again in the future.

Is this ready for review?

Yep. I have some extra changes that would be nice in a fresh PR to avoid clashing with the ones here.

ocefpaf · 2024-02-14T19:07:53Z

PS: The next changes parallelize things. It takes ~7 s against +20 s from before. The more metrics we add, the more the speedup will be important (we are still missing the national platforms and that hits different data sources).

In [2]: %time update_metrics()
CPU times: user 88.1 ms, sys: 86.3 ms, total: 174 ms
Wall time: 6.59 s
Out[2]: 
     date_UTC Federal Partners Regional Associations  HF Radar Stations NGDAC Glider Days  ...  QARTOD Manuals IOOS Core Variables Metadata Records IOOS COMT Projects
0  2018-02-01               17                    11                150             52027  ...              13                  34             8600    1          <NA>
1  2022-04-22               17                    11                165             53672  ...              13                  34             7213    1             5
2  2022-07-08               17                    11                165             55448  ...              13                  34             6217    1             5
3  2022-10-05               17                    11                165             59088  ...              13                  34            24499    1             5
4  2023-01-05               17                    11                165             62042  ...              13                  34            11840    1             5
5  2024-02-14               17                    11               <NA>             76075  ...              13                  34            35249    1             5

[6 rows x 16 columns]

MathewBiddle · 2024-02-15T13:51:55Z

ioos_metrics/ioos_metrics.py

+        try:
+            num = function()
+        except Exception as err:
+            log.error(f"{err}")


ioos_metrics.ioos_metrics.update_metrics() Traceback (most recent call last): File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 424, in update_metrics num = function() ^^^^^^^^^^ File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 376, in hab_pilot_projects from pdfminer.high_level import extract_text File "C:\Users\Mathew.Biddle\programs\PyCharm Community Edition 2020.2.2\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import module = self._system_import(name, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'pdfminer.high_level' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\IPython\core\interactiveshell.py", line 3505, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-10-da9d358039b7>", line 1, in <module> df2 = ioos_metrics.ioos_metrics.update_metrics() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 426, in update_metrics log.error(f"{err}") ^^^ NameError: name 'log' is not defined

Should be fixed in the last commit.

MathewBiddle

Looks good. Just one fix of a duplicated line.

MathewBiddle · 2024-03-15T19:33:41Z

ioos_metrics/ioos_metrics.py

+        "Federal Partners": federal_partners,
+        "HAB Pilot Projects": hab_pilot_projects,
+        "IOOS Core Variables": ioos_core_variables,
+        "IOOS Core Variables": ioos_core_variables,


this is duplicated

Good catch! Fixed it.

MathewBiddle · 2024-03-15T19:34:44Z

tests/test_metrics.py

I love ❤️ this!!

MathewBiddle · 2024-04-03T13:36:12Z

Just ran this and received this error:

import ioos_metrics.ioos_metrics
df2 = ioos_metrics.ioos_metrics.update_metrics()
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\IPython\core\interactiveshell.py", line 3505, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-da9d358039b7>", line 1, in <module>
    df2 = ioos_metrics.ioos_metrics.update_metrics()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 429, in update_metrics
    message = _compare_metrics(column=column, num=num)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 65, in _compare_metrics
    elif num < old:
         ^^^^^^^^^
TypeError: '>' not supported between instances of 'int' and 'NoneType'

MathewBiddle · 2024-04-03T13:47:00Z

It looks like, since HAB Pilot Projects doesnt exist previously, this catches the if loop.

I added some print statements to help debug:

df2 = ioos_metrics.ioos_metrics.update_metrics()
column: ATN Deployments
old: 4444
num: 5298
column: COMT Projects
old: 5
num: 5
column: Federal Partners
old: 17
num: 17
column: HAB Pilot Projects
old: 9
num: None
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\IPython\core\interactiveshell.py", line 3505, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-da9d358039b7>", line 1, in <module>
    df2 = ioos_metrics.ioos_metrics.update_metrics()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 432, in update_metrics
    message = _compare_metrics(column=column, num=num)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 68, in _compare_metrics
    elif num < old:
         ^^^^^^^^^
TypeError: '>' not supported between instances of 'int' and 'NoneType'

MathewBiddle · 2024-04-03T13:49:05Z

ahh, it looks like its a problem with hab_pilot_projects()

ioos_metrics.ioos_metrics.hab_pilot_projects()
Traceback (most recent call last):
  File "C:\Users\Mathew.Biddle\programs\Miniforge\envs\ioos-metrics\Lib\site-packages\IPython\core\interactiveshell.py", line 3505, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-a4a8446718c8>", line 1, in <module>
    ioos_metrics.ioos_metrics.hab_pilot_projects()
  File "C:\Users\Mathew.Biddle\Documents\GitProjects\ioos_metrics\ioos_metrics\ioos_metrics.py", line 379, in hab_pilot_projects
    from pdfminer.high_level import extract_text
  File "C:\Users\Mathew.Biddle\programs\PyCharm Community Edition 2020.2.2\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pdfminer.high_level'

MathewBiddle · 2024-04-03T13:55:01Z

Okay, I updated pdfminer and now the HABs function works.

MathewBiddle · 2024-04-03T13:56:18Z

ioos_metrics.ioos_metrics.update_metrics() was broken too as I need to install ckanapi.

MathewBiddle · 2024-04-03T13:58:45Z

Looks like my env was all out of date. Updating my env then I'll try again.

conda env update --file environment.yml --prune

ocefpaf · 2024-04-03T14:13:08Z

I guess I could more gracefully when a dependency is missing. Let me see if I can fix those.

ocefpaf · 2024-04-03T14:27:30Z

@MathewBiddle latest commit should make the update_metrics run even when there is a missing dependency. Note that, b/c we want it to run all the way to the end, the metric will be None but the error will be in the logs like:

INFO:root:[2023-01-05] : COMT Projects equal 5 = 5.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ioos.noaa.gov:443
DEBUG:urllib3.connectionpool:https://ioos.noaa.gov:443 "GET /community/national HTTP/1.1" 301 0
DEBUG:urllib3.connectionpool:https://ioos.noaa.gov:443 "GET /community/national/ HTTP/1.1" 200 None
INFO:root:df_fed_partners[0].to_string()='0     National Oceanic and Atmospheric Administratio...\n1     National Aeronautics and Space Administration ...\n2     Bureau of Ocean Energy Manage
ment, Regulation ...\n3                        Office of Naval Research (ONR)\n4                  U.S. Army Corps of Engineers (USACE)\n5                         U.S. Geological Survey (USGS)
\n6                            Department of Energy (DOE)\n7                    Department of Transportation (DOT)\n8               U.S. Arctic Research Commission (USARC)\n9                 
    National Science Foundation (NSF)\n10                Environmental Protection Agency (EPA)\n11                       Marine Mammal Commission (MMC)\n12    Oceanographer of the Navy, repre
senting the Jo...\n13                              U.S. Coast Guard (USCG)\n14    Department of Agriculture, Cooperative State R...\n15                            Department of State (DOS)\n1
6                   Food and Drug Administration (FDA)'
INFO:root:[2023-01-05] : Federal Partners equal 17 = 17.
ERROR:root:No module named 'pdfminer'

ocefpaf · 2024-04-03T14:53:23Z

I added some print statements to help debug

Matt, I should mention that update_metrics never fails, it keeps going and logs everyting in the metric.log file. You can either inspect the logs, to figure out why some metric is None, or run the specific function by itself. Here is what happens if I run hab_pilot_projects outside of update_metrics without pdfminer.six:

from ioos_metrics.ioos_metrics import hab_pilot_projects

hab_pilot_projects()
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 1
----> 1 hab_pilot_projects()

File ~/Dropbox/pymodules/01-forks/IOOS/ioos_metrics/ioos_metrics/ioos_metrics.py:378, in hab_pilot_projects()
    368 def hab_pilot_projects():
    369     """
    370     These are the National Harmful Algal Bloom Observing Network Pilot Project awards.
    371     Currently these were calculated from the
   (...)
    376 
    377     """
--> 378     from pdfminer.high_level import extract_text
    380     url = "https://cdn.ioos.noaa.gov/media/2022/10/NHABON-Funding-Awards-FY22.pdf"
    382     data = requests.get(url)

ModuleNotFoundError: No module named 'pdfminer'

MathewBiddle · 2024-04-03T14:54:57Z

After updating the env things are looking good.

It looks like pdfminer writes a lot of stuff to the log file. (166082 lines worth) We can clean that up in a further PR.

MathewBiddle · 2024-04-03T14:59:04Z

missing

hf radar - hardcoded for now = 165
National Platforms - building into a separate module??

ocefpaf added 12 commits February 13, 2024 13:47

add docstring

27e59d2

add Regional Platforms

060eea4

remove TODO

8211545

add regional_platforms

72bcb82

add ott_projects

6182fbe

add qartod

265b950

add ioos_core_variables

97b7ee3

add ckanapi

981058d

ignore .ipynb_checkpoints

d4583f1

add metadata_records and ioos

ad61872

add pdfminer.six

51c632a

add mbon_projects and hab_pilot_projects

6d24bd8

ocefpaf marked this pull request as ready for review February 13, 2024 19:14

ocefpaf added 4 commits February 13, 2024 16:37

refactor tests

a60d0b2

add fake-useragent

4ffe3fd

use random fake agent

3e39ef7

add logging

4b76c24

ocefpaf mentioned this pull request Feb 14, 2024

Parallel #56

Merged

4 tasks

MathewBiddle reviewed Feb 15, 2024

View reviewed changes

fix typo

f2cb394

ocefpaf force-pushed the more_metrics branch from bfc4149 to f2cb394 Compare February 15, 2024 15:24

ocefpaf requested a review from MathewBiddle February 15, 2024 18:57

MathewBiddle requested changes Mar 15, 2024

View reviewed changes

remove duplicated entry

47e5346

ocefpaf requested a review from MathewBiddle March 15, 2024 23:12

update pre-commits

d768d09

ocefpaf force-pushed the more_metrics branch from 7771081 to 5d3bb95 Compare April 3, 2024 14:56

fail gracefully when a metric returns None

a08f95b

ocefpaf force-pushed the more_metrics branch from 5d3bb95 to a08f95b Compare April 3, 2024 14:58

MathewBiddle merged commit 5381080 into ioos:main Apr 3, 2024
1 check passed

ocefpaf deleted the more_metrics branch April 3, 2024 15:14

ocefpaf mentioned this pull request Apr 3, 2024

National platforms #57

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More metrics #54

More metrics #54

ocefpaf commented Feb 13, 2024 •

edited by MathewBiddle

Loading

ocefpaf commented Feb 13, 2024

MathewBiddle commented Feb 14, 2024

MathewBiddle commented Feb 14, 2024

ocefpaf commented Feb 14, 2024

ocefpaf commented Feb 14, 2024

MathewBiddle Feb 15, 2024

ocefpaf Feb 15, 2024

MathewBiddle left a comment

MathewBiddle Mar 15, 2024

ocefpaf Mar 15, 2024

MathewBiddle Mar 15, 2024

MathewBiddle commented Apr 3, 2024 •

edited

Loading

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024 •

edited

Loading

ocefpaf commented Apr 3, 2024

ocefpaf commented Apr 3, 2024

ocefpaf commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024 •

edited

Loading

More metrics #54

More metrics #54

Conversation

ocefpaf commented Feb 13, 2024 • edited by MathewBiddle Loading

ocefpaf commented Feb 13, 2024

MathewBiddle commented Feb 14, 2024

MathewBiddle commented Feb 14, 2024

ocefpaf commented Feb 14, 2024

ocefpaf commented Feb 14, 2024

MathewBiddle Feb 15, 2024

Choose a reason for hiding this comment

ocefpaf Feb 15, 2024

Choose a reason for hiding this comment

MathewBiddle left a comment

Choose a reason for hiding this comment

MathewBiddle Mar 15, 2024

Choose a reason for hiding this comment

ocefpaf Mar 15, 2024

Choose a reason for hiding this comment

MathewBiddle Mar 15, 2024

Choose a reason for hiding this comment

MathewBiddle commented Apr 3, 2024 • edited Loading

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024 • edited Loading

ocefpaf commented Apr 3, 2024

ocefpaf commented Apr 3, 2024

ocefpaf commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024

MathewBiddle commented Apr 3, 2024 • edited Loading

ocefpaf commented Feb 13, 2024 •

edited by MathewBiddle

Loading

MathewBiddle commented Apr 3, 2024 •

edited

Loading

MathewBiddle commented Apr 3, 2024 •

edited

Loading

MathewBiddle commented Apr 3, 2024 •

edited

Loading