-
Notifications
You must be signed in to change notification settings - Fork 0
Active measurements
All time ranges are [start, end) so the end time is excluded.
Script: traceroute.py (not included in this repository)
When: XT00:30 UTC
Range: 24 hours of the previous day from (X-1)T00:00 to XT00:00.
Output topic: ihr_atlas_traceroutev4_topology
Output timestamp: The timestamp field (start of measurement) from each traceroute is used to assign a message timestamp in Kafka.
What: Fetch traceroute data from the RIPE Atlas Measurement Results REST API. We fetch data for the Built-in Measurements 5051 and 5151, corresponding to traceroute measurements over UDP and ICMP.
Script: traceroute_to_bgp.py
When: XT00:30 UTC
Range: 24 hours of the previous day from (X-1)T00:00 to XT00:00.
Input topics: ihr_atlas_traceroutev4_topology, ihr_peeringdb_ix, ihr_peeringdb_netixlan, ip2asn
Output topics: ihr_bgp_traceroutev4_topology_ribs, ihr_bgp_traceroutev4_topology_updates, ihr_bgp_traceroutev4_topology_stats
Output timestamp: XT00:00
What: Convert the Atlas traceroute data into a RIB-like format with AS path, peer address, etc. All traceroutes in the input range get the same output timestamp.
Script: compute_as_visibility.py
When: XT00:30 UTC
Range: XT00:00 (corresponding to [(X-1)T00:00, XT00:00))
Input topics: ihr_bgp_traceroutev4_topology_ribs, ihr_bgp_traceroutev4_topology_updates
Output topic: ihr_bgp_traceroutev4_topology_as_visibility
Output timestamp: XT00:00
What: Iterate over the AS and IP paths and:
- count the total number AS paths in the interval (i.e., number of traceroutes in the interval)
- count in how many paths each AS occurs
- count in how many paths each AS occurs as the last hop
- build a set of IPs for each AS used as transit
- build a set of IPs for each AS used as the last hop
Script: compute_star_distribution.py
When: XT00:30 UTC
Range: XT00:00 (corresponding to [(X-1)T00:00, XT00:00))
Input topic: ihr_bgp_traceroutev4_topology_ribs
Output topic: ihr_hegemony_traceroutev4_topology_star_distribution
Output timestamp: XT00:00
What: Iterate over the IP paths. For each path, get the length (total number of hops) and the number of * hops in the path. Get the scope (last AS in the AS path) and record the (path_len, stars_in_path) tuple for each (scope, as) combination (remaining ASes) in the AS path. In the end, each (scope, as) pair has a list of (path_len, stars_in_path) pairs, indicating the number of paths and their corresponding lengths / number of * hops that were used to reach scope while going through as.
Scripts: produce_bgp_atom.py, produce_bcscore.py, produce_hege.py (all found here)
When: XT00:30 UTC
Range: XT00:00 (corresponding to [(X-1)T00:00, XT00:00))
Input topics: ihr_bgp_traceroutev4_topology_ribs, ihr_bgp_traceroutev4_topology_updates
Output topics: ihr_bgp_atom_traceroutev4_topology, ihr_bcscore_traceroutev4_topology, ihr_hegemony_traceroutev4_topology
Output timestamp: XT00:00
What: Compute AS hegemony scores.
Script: classify_dependencies.py
When: XT01:00 UTC
Range: XT00:00 (corresponding to [(X-1)T00:00, XT00:00) for traceroute and the exact timestamp for BGP)
Input topics: ihr_hegemony, ihr_hegemony_traceroutev4_topology
Output topic: ihr_hegemony_classification
Output timestamp: XT00:00
What: Classify AS dependencies by comparing the dependencies / hegemony scores calculated based on traceroute data with BGP data. The detailed process should be described somewhere.
Script: check_class_visibility.py
When: XT01:00 UTC
Range: XT00:00 (corresponding to [(X-1)T00:00, XT00:00)) for ihr_hegemony_classification and [(X-7)T00:00, XT00:00) for ihr_bgp_traceroutev4_topology_as_visibility
Input topics: ihr_hegemony_classification, ihr_bgp_traceroutev4_topology_as_visibility
Output topics: ihr_hegemony_classification_bgp_only_dependencies, ihr_hegemony_classification_equal_dependencies
Output timestamp: XT00:00
What: The script is run two times, once for bgp_only and once for equal. Read the visibility information for the last week (Note: The input ranges of the classification and visibility topics differ here, since we are interested in long-term visibilty). For each AS, built sets of transit, last hop, and unique (union of transit and last hop) IPs that were seen in traceroutes during this interval. Read the classification topic and only retrieve dependencies that are exclusively classified as bgp_only / equal for all scopes in which they occur. For each of these dependencies, write the (scope, as) pair together with the number of unique, transit, and last hop IPs to the output topic.
Script: check_ip2asn_visibility.py
When: XT01:00 UTC
Range: XT00:00 (corresponding to [(X-1)T00:00, XT00:00))
Input topics: ihr_hegemony_classification_bgp_only_dependencies, ihr_peeringdb_ix, ihr_peeringdb_netixlan, ip2asn
Output topic: , ihr_bgp_only_dependencies_ip2asn_visibility
Output timestamp: XT00:00
What: This script basically enhances the messages of the ihr_hegemony_classification_bgp_only_dependencies topic, since it includes them verbatim + additional information. Check for each AS if we have a IP-to-AS mapping either provided by ip2asn or by PeeringDB data. This script looks at bgp_only dependencies and there might exist ASes in paths from the BGP data that do not have a mapping. If a mapping is found, record the number of prefixes as well as the sum of IP addresses covered by these prefixes. (separately for ip2asn and PeeringDB data).
Script: check_transit_ips.py
When: XT01:00 UTC
Range: XT00:00 (corresponding to [(X-1)T00:00, XT00:00)) for ihr_bgp_traceroutev4_topology_ribs and [(X-7)T00:00, XT00:00) for ihr_bgp_traceroutev4_topology_as_visibility
Input topics: ihr_bgp_traceroutev4_topology_as_visibility, ihr_bgp_traceroutev4_topology_ribs, ihr_hegemony, ihr_hegemony_classification_bgp_only_dependencies
Output topic: , ihr_bgp_only_dependencies_transit_ips
Output timestamp: XT00:00
What: Check if IPs that are used for transit in traceroute are used as transit to scopes as expected by BGP. Read all ASes that are bgp_only dependencies and get the set of transit IPs, as well as the set of scopes that depend on these ASes. Go through the IP paths (*_ribs topic) and if the path includes a transit IP for one of the ASes, check if the target of that path is a scope that depends on the corresponding AS.
The script description is omitted if it is the same as above.
Script: traceroute_to_bgp.py
When: Every Monday at XT02:00 UTC
Range: The last week from (X-7)T00:00 to XT00:00.
Input topics: ihr_atlas_traceroutev4_topology, ihr_peeringdb_ix, ihr_peeringdb_netixlan, ip2asn
Output topics: ihr_bgp_traceroutev4_topology_weekly_ribs, ihr_bgp_traceroutev4_topology_weekly_updates, ihr_bgp_traceroutev4_topology_weekly_stats
Output timestamp: XT00:00
Script: compute_star_distribution.py
When: Every Monday at XT02:00 UTC
Range: XT00:00 (corresponding to [(X-7)T00:00, XT00:00))
Input topic: ihr_bgp_traceroutev4_topology_weekly_ribs
Output topic: ihr_hegemony_traceroutev4_topology_weekly_star_distribution
Output timestamp: XT00:00
Scripts: produce_bgp_atom.py, produce_bcscore.py, produce_hege.py (all found here)
When: Every Monday at XT02:00 UTC
Range: XT00:00 (corresponding to [(X-7)T00:00, XT00:00))
Input topics: ihr_bgp_traceroutev4_topology_weekly_ribs, ihr_bgp_traceroutev4_topology_weekly_updates
Output topics: ihr_bgp_atom_traceroutev4_topology_weekly, ihr_bcscore_traceroutev4_topology_weekly, ihr_hegemony_traceroutev4_topology_weekly
Output timestamp: XT00:00
Script: classify_dependencies.py
When: Every Tuesday at XT02:00 UTC
Range: (X-1)T00:00 (corresponding to [(X-8)T00:00, (X-1)T00:00) for traceroute and the exact timestamp for BGP)
Input topics: ihr_hegemony, ihr_hegemony_traceroutev4_topology_weekly
Output topic: ihr_hegemony_classification_weekly
Output timestamp: (X-1)T00:00
Script: check_class_visibility.py
When: Every Tuesday at XT02:00 UTC
Range: (X-1)T00:00 (corresponding to [(X-8)T00:00, (X-1)T00:00))
Input topics: ihr_hegemony_classification_weekly, ihr_bgp_traceroutev4_topology_as_visibility
Output topics: ihr_hegemony_classification_weekly_bgp_only_dependencies, ihr_hegemony_classification_weekly_equal_dependencies
Output timestamp: (X-1)T00:00
Script: check_ip2asn_visibility.py
When: Every Tuesday at XT02:00 UTC
Range: (X-1)T00:00 (corresponding to [(X-8)T00:00, (X-1)T00:00))
Input topics: ihr_hegemony_classification_weekly_bgp_only_dependencies, ihr_peeringdb_ix, ihr_peeringdb_netixlan, ip2asn
Output topic: , ihr_weekly_bgp_only_dependencies_ip2asn_visibility
Output timestamp: (X-1)T00:00
When: Every Tuesday at XT02:00 UTC
Range: (X-1)T00:00 (corresponding to [(X-8)T00:00, (X-1)T00:00))
Input topics: ihr_bgp_traceroutev4_topology_as_visibility, ihr_bgp_traceroutev4_topology_weekly_ribs, ihr_hegemony, ihr_hegemony_classification_weekly_bgp_only_dependencies
Output topic: , ihr_weekly_bgp_only_dependencies_transit_ips
Output timestamp: (X-1)T00:00