Nektar::MetricExecutionTime Class Reference

Metric that finds the execution time in an output and tests it against an accepted value and tolerance. More...

#include <MetricExecutionTime.h>

Inheritance diagram for Nektar::MetricExecutionTime:

Public Member Functions
	~MetricExecutionTime () override
Public Member Functions inherited from Nektar::Metric
	Metric (TiXmlElement *metric, bool generate)
	Constructor. More...
virtual	~Metric ()=default
bool	Test (std::istream &pStdout, std::istream &pStderr)
	Calls a metric's v_Test function (or v_Generate if m_generate). More...
void	Generate (std::istream &pStdout, std::istream &pStderr)
std::string	GetType ()
	Return metric type. More...
int	GetID ()
	Return metric ID. More...
bool	SupportsAverage () const
	Return whether this metric supports averaging results from multiple runs. More...

Static Public Member Functions
static MetricSharedPtr	create (TiXmlElement *metric, bool generate)

Static Public Attributes
static std::string	type

Protected Member Functions
	MetricExecutionTime (TiXmlElement *metric, bool generate)
	Construct a new MetricExecutionTime object. More...
bool	v_Test (std::istream &pStdout, std::istream &pStderr) override
	Test output against a regular expression and its expected value. More...
void	v_Generate (std::istream &pStdout, std::istream &pStderr) override
	Generate an accepted execution time value using a test output or error stream. More...
virtual bool	v_Test (std::istream &pStdout, std::istream &pStderr)=0
	Virtual function to test the metric. Should be redefined in derived classes. More...
virtual void	v_Generate (std::istream &pStdout, std::istream &pSrderr)=0
	Virtual function to generate the metric. Should be redefined in derived classes. More...

Protected Attributes
std::regex	m_regex
	Regex used to match an execution time in a test output. More...
MetricExecutionTimeFieldValue	m_match
	Stores each execution time found in the test output. More...
bool	m_useStderr = false
	If true, use stderr for testing/generation instead of stdout. More...
Protected Attributes inherited from Nektar::Metric
int	m_id
	Stores the ID of this metric. More...
std::string	m_type
	Stores the type of this metric (uppercase). More...
bool	m_generate
	Determines whether to generate this metric or not. More...
bool	m_average = false
	Indicates whether a metric supports averaging results from multiple runs. More...
TiXmlElement *	m_metric
	Pointer to XML structure containing metric definition. More...

Detailed Description

Metric that finds the execution time in an output and tests it against an accepted value and tolerance.

Definition at line 71 of file MetricExecutionTime.h.

Constructor & Destructor Documentation

~MetricExecutionTime()

Nektar::MetricExecutionTime::~MetricExecutionTime ( )

inlineoverride

Definition at line 74 of file MetricExecutionTime.h.

    {
    }

MetricExecutionTime()

Nektar::MetricExecutionTime::MetricExecutionTime ( TiXmlElement *  metric, bool  generate  )

protected

Construct a new MetricExecutionTime object.

If the metric is not a derived class, then this constructor will parse the regular expression from the metric's element in the test file, or use a default value if this doesn't exist. It will also see if an expected execution time and tolerance have been provided for the current computer's hostname. If this is the case, these will be used by v_Test. If not, then a skip flag m_skip will be set to true and the test will automatically pass.

Parameters

metric
generate

Definition at line 65 of file MetricExecutionTime.cpp.

    : Metric(metric, generate)
{
    // If we are a derived class, do nothing
    if (m_type != "EXECUTIONTIME")
    {
        return;
    }
 
    // Parse Regex expression
    TiXmlElement *regex = metric->FirstChildElement("regex");
 
    if (regex)
    {
        ASSERTL0(regex->GetText(), "Failed to read regex element.");
        m_regex = regex->GetText();
    }
    else
    {
        // Set the regex to a default value.
        m_regex = R"(^.*Total Computation Time\s*=\s*(\d+\.?\d*).*)";
    }
 
    // Inform the Tester this metric supports averaging data from multiple runs.
    m_average = true;
    m_match   = MetricExecutionTimeFieldValue("nil");
 
    // Parse matching values if not generating.
    if (m_generate)
    {
        return;
    }
 
    TiXmlElement *value = metric->FirstChildElement("value");
    ASSERTL0(value || m_generate, "Missing value tag for metric.");
 
    bool hostnameMatch = false;
    while (value)
    {
        ASSERTL0(value->Attribute("tolerance"),
                 "Missing tolerance in execution time metric.");
        ASSERTL0(value->Attribute("hostname"),
                 "Missing hostname in execution time metric.");
        ASSERTL0(!EmptyString(value->GetText()),
                 "Missing value in execution time metric.");
 
        // Only use the value if it matches the runner's hostname.
        if (value->Attribute("hostname") == boost::asio::ip::host_name())
        {
            MetricExecutionTimeFieldValue val;
            val.m_value     = value->GetText();
            val.m_tolerance = atof(value->Attribute("tolerance"));
            m_match         = val;
 
            hostnameMatch = true;
 
            // No use finding any more values
            break;
        }
 
        value = value->NextSiblingElement("value");
    }
 
    // If no hostname match was found, pass a skip flag to the test.
    if (!hostnameMatch)
    {
        cerr << "WARNING: No execution time value provided for host "
             << boost::asio::ip::host_name() << ". Skipping metric." << endl;
        m_match.m_skip = true;
    }
}

References ASSERTL0, Nektar::EmptyString(), Nektar::Metric::m_average, Nektar::Metric::m_generate, m_match, m_regex, Nektar::MetricExecutionTimeFieldValue::m_skip, Nektar::MetricExecutionTimeFieldValue::m_tolerance, Nektar::Metric::m_type, and Nektar::MetricExecutionTimeFieldValue::m_value.

Referenced by create().

Member Function Documentation

create()

static MetricSharedPtr Nektar::MetricExecutionTime::create ( TiXmlElement *  metric, bool  generate  )

inlinestatic

Definition at line 78 of file MetricExecutionTime.h.

    {
        return MetricSharedPtr(new MetricExecutionTime(metric, generate));
    }

References MetricExecutionTime().

v_Generate()

void Nektar::MetricExecutionTime::v_Generate ( std::istream &  pStdout, std::istream &  pStderr  )

overrideprotectedvirtual

Generate an accepted execution time value using a test output or error stream.

This function generate an execution time match by finding the first execution time in the output that matches m_regex (provided in the test file).

Parameters

pStdout	Reference to test output stream.
pStderr	Reference to test error stream.

Implements Nektar::Metric.

Definition at line 266 of file MetricExecutionTime.cpp.

{
    boost::ignore_unused(pStderr);
 
    // Select istream to use
    istream &is = m_useStderr ? pStderr : pStdout;
 
    std::smatch matches;
 
    string line;
    // Vector of execution times found in the output
    vector<double> sampleTimes;
    bool matched = false;
 
    // Process output file line by line searching for regex matches
    while (getline(is, line))
    {
        // Test to see if we have a match on this line
        if (std::regex_match(line, matches, m_regex))
        {
            // If no fields in regex then throw an error
            ASSERTL0(matches.size() != 1, "No test sections in regex!");
 
            // Check each regex capture group in turn
            for (int i = 1; i < matches.size(); ++i)
            {
                // Extract the captured string
                string match(matches[i].first, matches[i].second);
                double val;
                try
                {
                    val = boost::lexical_cast<double>(match);
 
                    sampleTimes.push_back(val);
                    matched = true;
                }
                catch (boost::bad_lexical_cast &e)
                {
                    cerr << "Could not convert match " << match << " to double"
                         << endl;
                    continue;
                }
            }
        }
    }
 
    // Stores the accepted average execution time and a default tolerance.
    MetricExecutionTimeFieldValue okValue;
 
    if (matched)
    {
        // Average the found execution times and set it as the accepted value.
        double minTime = sampleTimes[0];
        for (unsigned int i = 0; i < sampleTimes.size(); ++i)
        {
            minTime = (sampleTimes[i] < minTime ? sampleTimes[i] : minTime);
        }
 
        okValue.m_value = to_string(minTime);
        m_match         = okValue;
    }
    else
    {
        cerr << "No execution times were found in the test output. Value set "
                "to 'nil'."
             << endl;
    }
 
    // If we are not a derived class then create a new structure.
    if (m_type == "EXECUTIONTIME")
    {
        // Remove values if they already exist.
        while (m_metric->FirstChildElement("value"))
        {
            ASSERTL0(
                m_metric->RemoveChild(m_metric->FirstChildElement("value")),
                "Couldn't remove value from metric!");
        }
 
        TiXmlElement *val = new TiXmlElement("value");
 
        val->SetAttribute("tolerance",
                          boost::lexical_cast<string>(m_match.m_tolerance));
        val->SetAttribute("hostname", boost::asio::ip::host_name());
        val->LinkEndChild(new TiXmlText(m_match.m_value));
        m_metric->LinkEndChild(val);
    }
}

References ASSERTL0, m_match, Nektar::Metric::m_metric, m_regex, Nektar::MetricExecutionTimeFieldValue::m_tolerance, Nektar::Metric::m_type, m_useStderr, and Nektar::MetricExecutionTimeFieldValue::m_value.

v_Test()

bool Nektar::MetricExecutionTime::v_Test ( std::istream &  pStdout, std::istream &  pStderr  )

overrideprotectedvirtual

Test output against a regular expression and its expected value.

This function will find the execution time for each test run in the output, using m_regex (provided in the test file). It will then average these times and compare them to the expected execution time for the current computer's hostname (this behaviour is defined in the constructor). If the average execution time falls outside of the tolerance then the test will fail.

If no execution time value is provided for the current computer's hostname, m_skip will be set to true and the test will automatically pass.

Parameters

pStdout	Reference to test output stream.
pStderr	Reference to test error stream.

Returns

true if the test passes, false otherwise.

Implements Nektar::Metric.

Definition at line 154 of file MetricExecutionTime.cpp.

{
    boost::ignore_unused(pStdout, pStderr);
 
    bool success = true;
 
    // Select istream to use.
    istream &is = m_useStderr ? pStderr : pStdout;
 
    std::smatch matches;
    string line;
    // Vector of execution times found in the output.
    vector<double> times;
    bool matched = false;
 
    // Process output file line by line searching for regex matches
    while (getline(is, line))
    {
        // Test to see if we have a match on this line.
        if (std::regex_match(line, matches, m_regex))
        {
            // If no matches are found then throw an error.
            if (matches.size() == 1)
            {
                cerr << "No test sections in regex!" << endl;
                return false;
            }
 
            // Check each regex capture group in turn
            for (int i = 1; i < matches.size(); ++i)
            {
                // Extract the captured string
                string match(matches[i].first, matches[i].second);
                double val;
                try
                {
                    val = boost::lexical_cast<double>(match);
 
                    // Add value to the list of found execution times.
                    times.push_back(val);
                    matched = true;
                }
                catch (boost::bad_lexical_cast &e)
                {
                    cerr << "Could not convert match " << match << " to double"
                         << endl;
                    success = false;
                    continue;
                }
            }
        }
    }
 
    if (!matched)
    {
        cerr << "No execution times were found in the test output. Consider "
                "providing a custom regex for this test metric."
             << endl;
        success = false;
    }
 
    // Find the minimum of the execution times.
    double minTime = times[0];
    for (unsigned int i = 0; i < times.size(); ++i)
    {
        minTime = (times[i] < minTime) ? times[i] : minTime;
    }
 
    // If a skip flag is set (due to no matching hostname) then return
    // successful test and output the average time.
    if (m_match.m_skip)
    {
        cerr << "Minimum execution time for host "
             << boost::asio::ip::host_name() << ": " << minTime << endl;
        return success;
    }
 
    ASSERTL0(m_match.m_value != "nil",
             "No test conditions defined for execution time.");
 
    // Check that the average time is within the tolerance.
    if (fabs(minTime - boost::lexical_cast<double>(m_match.m_value)) >
            m_match.m_tolerance &&
        !m_match.m_skip)
    {
        cerr << endl;
        cerr << "Failed tolerance match." << endl;
        cerr << "  Expected min execution time: " << m_match.m_value << " +/- "
             << m_match.m_tolerance << endl;
        cerr << "  Actual min: " << minTime << endl;
 
        for (unsigned int i = 0; i < times.size(); ++i)
        {
            cerr << "       Run " << i << ": " << times[i] << endl;
        }
 
        success = false;
    }
 
    return success;
}

References ASSERTL0, m_match, m_regex, Nektar::MetricExecutionTimeFieldValue::m_skip, Nektar::MetricExecutionTimeFieldValue::m_tolerance, m_useStderr, and Nektar::MetricExecutionTimeFieldValue::m_value.

Member Data Documentation

m_match

MetricExecutionTimeFieldValue Nektar::MetricExecutionTime::m_match

protected

Stores each execution time found in the test output.

Definition at line 89 of file MetricExecutionTime.h.

Referenced by MetricExecutionTime(), v_Generate(), and v_Test().

m_regex

std::regex Nektar::MetricExecutionTime::m_regex

protected

Regex used to match an execution time in a test output.

Definition at line 87 of file MetricExecutionTime.h.

Referenced by MetricExecutionTime(), v_Generate(), and v_Test().

m_useStderr

bool Nektar::MetricExecutionTime::m_useStderr = false

protected

If true, use stderr for testing/generation instead of stdout.

Definition at line 91 of file MetricExecutionTime.h.

Referenced by v_Generate(), and v_Test().

type

std::string Nektar::MetricExecutionTime::type

static

Initial value:

=
    GetMetricFactory().RegisterCreatorFunction("EXECUTIONTIME",
                                               MetricExecutionTime::create)

Definition at line 83 of file MetricExecutionTime.h.