api_external module

The api_external module is part of the nmeta suite, but is run separately

This module runs a class and methods for an API that exposes an interface into nmeta MongoDB collections.

It leverages the Eve Python REST API Framework

class api_external.ExternalAPI(config)

Bases: baseclass.BaseClass

This class provides methods for the External API

class FlowUI

Bases: object

An object that represents a flow record to be sent in response to the WebUI. Features:

  • Flow direction normalised to direction of first packet in flow
  • Src and Dst are IP or Layer 2 to optimise screen space
  • Extra data included for hover-over tips

Note that there should not be any display-specific data (i.e. don’t send any HTML, leave this to the client code)

response()

Return a dictionary object of flow parameters for sending in response

ExternalAPI.run()

Run the External API instance

Note that API definitions are from previously imported files from api_definitions subdirectory

ExternalAPI.response_pi_rate(items)

Update the response with the packet_in rate. Hooked from on_fetched_resource_pi_rate

Returns key/values for packet-in processing time in API response: - timestamp - pi_rate

ExternalAPI.response_pi_time(items)

Update the response with the packet_time min, avg and max. Hooked from on_fetched_resource_pi_time

Returns key/values for packet-in processing time in API response: - timestamp - ryu_time_max - ryu_time_min - ryu_time_avg - ryu_time_period - ryu_time_records - pi_time_max - pi_time_min - pi_time_avg - pi_time_period - pi_time_records

If no data found within time period then returns without key/values

ExternalAPI.response_controller_summary(items)

Update the response with the packet_in rate, packet processing time stats

Hooked from on_fetched_resource_controller_summary

Rounds seconds results

ExternalAPI.response_identities_ui(items)
Populate the response with identities that are filtered:
  • Reverse sort by harvest time
  • Deduplicate by id_hash, only returning most recent per id_hash
  • Includes possibly stale records
  • Check DNS A records to see if they are from a CNAME

Hooked from on_fetched_resource_<name>

ExternalAPI.response_flows_removed_stats_count(items)

Return count of removed flows collection

ExternalAPI.response_flows_removed_src_bytes_sent(items)

Returns removed flow bytes sent by session source IP (deduplicated for flows crossing multiple switches), enriched with identity metadata.

ExternalAPI.response_flows_removed_src_bytes_received(items)

Returns removed flow bytes received by session source IP (deduplicated for flows crossing multiple switches), enriched with identity metadata.

ExternalAPI.response_flows_removed_dst_bytes_sent(items)

Returns removed flow bytes sent by session destination IP (deduplicated for flows crossing multiple switches), enriched with identity metadata.

ExternalAPI.response_flows_removed_dst_bytes_received(items)

Returns removed flow bytes received by session destination IP (dedup for flows crossing multiple switches), enriched with identity metadata.

ExternalAPI.response_flows_ui(items)
Populate the response with flow entries that are filtered:
  • Reverse sort by initial ingest time
  • Deduplicate by flow_hash, only returning most recent per flow_hash
  • Enrich with TBD

Hooked from on_fetched_resource_<name>

ExternalAPI.response_switches_count(items)

Populate the response with number of connected switches.

ExternalAPI.flow_match(flow, flows_filterlogicselector, flows_filtertypeselector, filter_string)

Passed an instance of FlowUI class, a logic selector, filter type and filter string.

Return a boolean on whether or not that theres a match.

ExternalAPI.flow_augment_record(record)

Passed a record of a single flow from the packet_ins database collection.

Create FlowUI class instance, add in known data and augment with identity data. Logic is specific to the webUI user experience.

Return the FlowUI class instance

ExternalAPI.get_flow_data_xfer(record)

Passed a record of a single flow from the packet_ins database collection.

Enrich this by looking up data transfer stats (which may not exist) in flow_rems database collection, and return dictionary of the values.

Note that the data sent (tx) and received (rx) records will have different flow hashes.

ExternalAPI.get_classification(flow_hash)

Passed flow_hash and return a dictionary of a classification object for the flow_hash (if found), otherwise a dictionary of an empty classification object.

ExternalAPI.flow_normalise_direction(record)

Passed a dictionary of an flow record and return a similar dictionary that has sources and destinations normalised to the direction of the first observed packet in the flow

ExternalAPI.get_flow_client_ip(flow_hash)

Find the IP that is the originator of a flow searching forward by flow_hash

Finds first packet seen for the flow_hash within the time limit and returns the source IP, otherwise 0,

ExternalAPI.get_id(ip_addr)

Passed an IP address. Look this up for matching identity metadata and return a string that contains either the original IP address or an identity string

ExternalAPI.get_dns_ip(service_name)

Use this to get an IP address for a DNS lookup that returned a CNAME Passed a DNS CNAME and look this up in identities collection to see if there is a DNS A record, and if so return the IP address, otherwise return an empty string.

ExternalAPI.get_host_by_ip(ip_addr)

Passed an IP address. Look this up in the identities db collection and return a host name if present, otherwise an empty string

ExternalAPI.get_location_by_mac(mac_addr)

Passed a MAC address. Look this up in the identities db collection and return a source logical location if present, otherwise an empty string

ExternalAPI.get_service_by_ip(ip_addr, alias=1)

Passed an IP address. Look this up in the identities db collection and return a service name if present, otherwise an empty string.

If alias is set, do additional lookup on success to see if service name is an alias for another name, and if so return that.

ExternalAPI.get_pi_rate(test=0)

Calculate packet-in rate by querying packet_ins database collection.

Setting test=1 returns database query execution statistics

ExternalAPI.get_pi_time()

Calculate packet processing time statistics by querying the pi_time database collection.

api_external.enumerate_eth_type(eth_type)

Passed an eth_type (in decimal) and return an enumerated version, or if not found, return the original value. Example, pass this function value 2054 and it return will be ‘ARP’

api_external.hovertext_eth_type(eth_type)

Passed an eth_type (decimal, not enumerated) and return it wrapped in extra text to convey context

api_external.enumerate_ip_proto(ip_proto)

Passed an IP protocol number (in decimal) and return an enumerated version, or if not found, return the original value. Example, pass this function value 6 and it return will be ‘TCP’

api_external.hovertext_ip_proto(ip_proto)

Passed an IP protocol number (decimal, not enumerated) and return it wrapped in extra text to convey context

api_external.hovertext_ip_addr(ip_addr)

Passed an IP address and return it wrapped in extra text to convey context