Plugins¶

What and Why?¶

One of Rumal’s major strength is its ability to augment the information generated by Thug. This enrichment is performed on the fron-end server by making use of Plugins. The additional information is often obtained by sending the current one (say domain name) to fetch some extra information (IP Address). Rumal’s GUI has a separate panel for displaying this augmented information.

As of now Rumal comes bundled with a WhoisPlugin and GeoPlugin by default. The WhoisPlugin fetches the WhoisInfo while the GeoPlugin tries to match the IP Address obtained on back-end to locations on the Earth’s map. Work is in progress to expand the set of default plugins to add more useful information by default.

Adding Removing Plugins¶

Adding and Removing plugins is a straight forward procedure. To add a plugin you just need to drop its .py file into the plugins directory of the frontend (/rumal/interface/plugins/). This section on advanced usage has details regarding how to configure the plugins.

How Plugins work?¶

Plugins come in the workflow once the scan details are retrieved from the backend. The bson object stored for the scan is then modified to add details. The bson has a key named “Plugins” under which all the generated plugin info is stored. For example a the default WhoisPlugin saves its info under. “flat_tree_node” => “some_node” =>”WhoisPlugin” key.

The plugin system works in the form of a pipeline and it is possible to add plugins such that they are dependent on each other. Rumal automatically finds out the correct order in which plugins need to be run given that dependencies are correctly specified in the .py file of the plugins.

The UI has a plugins panel which displays information by parsing the same file and automatically displays the json of the info stored in the particular plugin’s key.

Writing your own Plugins¶

Rumal Plugins are derived from the abstract class PluginBase found in rumal/interface/plug.py file. PluginBase class has been impemented using Python’s abc module. To get a better understanding of the implementation we recommend the excellent tutorial at https://pymotw.com/2/abc/

The methods with @abc.abstractproperty and @abc.abstractmethod need to be overridden while implementing the Plugin class. Below is a walk-through of how the GeoPlugin was implemented.

Relevant code from PluginBase. Implementation of some methods has been omitted.

class PluginBase(object):
    __metaclass__ = abc.ABCMeta

    @abc.abstractproperty
    def dependencies(self):
        """List of dependencies - put class names of required plugins here.
           Return blank list if none."""
        return []

    @abc.abstractproperty
    def module_dependencies(self):
        """List of non-standard lib module dependencies
           - put module names as keys and versions as
           values.Return blank list if none."""
        return {}

    def input_run(self, data):
        """Adds data to object and calls self.get_config and self.run"""

    def get_config(self):
        """Gets config file data and stores it under self.config of object instance"""

    def save_data(self):
        """Add to plugins list and return modified data."""

    def check_dependencies(self):
        """Check if all the dependencies are met."""

    @abc.abstractmethod
    def run(self):
        """Run and make changes to data"""
        # 1. Call check for dependencies
        self.check_dependencies()
        # 2. Append all changes to x.data["flat_tree"]["url_link/node"]["plugin_name"]
        # 3. Call save data
        self.save_data()

Now for implementing GeoPlugin the class must be named properly. This is important so that the enrichment daemon is able to sucessfully detect plugins. All plugin classes are required to be named as XyzPlugin. This is the name of the key under “Plugins” where the info is saved. This is also important for the plugin info to be displayed in the UI.

class GeoPlugin(PluginBase):
    @property
    def dependencies(self):
        """List of dependencies - put class names of required plugins here.
           Return blank list if none."""
        return []

    @property
    def module_dependencies(self):
        """List of non-standard lib module dependencies
           - put module names as keys and versions as
           values.Return blank list if none."""
        return
        {
            "geoip2": "2.2.0",
            "ipaddr": "2.1.11",
            "maxminddb": "1.2.0",
            "requests": "2.7.0",
        }

    def config_plugin(self):
        """ Use data in self.config_dict to configure required settings
            also create reader object as creation is expensive.
        """
        self.readers = {}
        # Now check which dbs are enabled.
        self.enabled_dbs = []
        db_path_dict = self.config_dict["db_path"]
        for name, value in self.config_dict["dbs"].iteritems():
            if value == "True":
                self.enabled_dbs.append(name)
                db_path = db_path_dict[name]
                self.readers[name] = geoip2.database.Reader(db_path)

    def pretty_response(self,response,db_type):
        # Not relevant to this example.

    def get_geo(self, ip, db_type):
        # Not relevant to this example.

    def run(self):
        """Run and make changes to data"""
        self.check_dependencies()
        self.config_plugin()
        # 2. Append all changes to x.data["flat_tree"]["url_link/node"]["plugin_name"]
        for db_type in self.enabled_dbs:  # will be set in config method
            ip_geo_map = {}  # keys are IPs and values contain respective geo data.
            # Reset ip_geo_map for each DB run.
            for node in self.data["flat_tree"]:
                node_ip = node["ip"]
                if node_ip == None:
                    node["GeoPlugin"][db_type] = {}
                    continue
                if "GeoPlugin" not in node.keys():
                    node["GeoPlugin"] = {}  # To avoid key error as further data is according to db.
                if node_ip in ip_geo_map.keys():
                    node["GeoPlugin"][db_type] = ip_geo_map[node_ip]
                else:
                    node["GeoPlugin"][db_type] = self.get_geo(node_ip, db_type)
                    ip_geo_map[node_ip] = node["GeoPlugin"][db_type]
            # 3. Call save data
        return self.save_data()

To override the methods dependencies and properties @property decorator is used. dependencies method returns a list of strings consisting of class names of other plugins that must be run before this plugin. The module_dependecies method lists the required python packages for the plugin to function. These must be manually installed by the user.

Next, the config_plugin makes use of the plugin confiugration parameters that have already been loaded in self.config_dict by the get_config method in the PluginBase class. The config_dict contains the config details in key,value form from the config file.

The most important part of the plugin is the run method. A fixed procedure must be followed for writing this method. self.check_dependencies() and self.config_plugin() must be called in the beginning and result from self.save_data() should be returned towards the end. Between these the required processing should be performed and the changes must be saved to self.data object which is passed to the next plugin. New information must always be saved to self.data[“flat_tree_node”][particular_node][“XyzPlugin”]

The configuration file should be named by the name XyzPlugin-config in the plugins directory. The example config file is shown below:

[db_path]
city = /absolute/path/to/your/db/file
anonymous_ip = None
connection_type = None
isp = None
[dbs]
city = True
anonymous_ip = False
connection_type = False
isp = False

self.config_dict stores all info from the file in the form of a nested dictonary.