Skip to main content

Metrics

Overview​

The Agreement conformed from several Guarantees contain all the information needed to compute them. Each guarantee contains an objective, which can involve one or more metrics. The metric is the instrument used to fetch the required information and extract a value for the system to verify the fulfilment of the objective.

Collectors​

The collectors are the component which receiving a metric as input, are able to connect to the different data sources to extract the information needed for it, and compute the final result.

Actually, there are 3 different collectors available. The metrics are modeled as a JSON object following the iAgree specification. Each collector has his own syntax and they are similar in order to be consistent but there are some differences in order to extend the features.

Event Collector​

The Event Collector is capable of using different data sources as input and join the information in order to correlate different payloads composing more complex metrics. Actually it can fetch information from Github, Pivotal Tracker, Travis CI, CodeClimate and Heroku.

Patterns​

The metrics available to use in the Event Collector follow 4 different patterns:

  1. Number of [Event] in [Tool] every [Period] by [TEAM|MEMBER]
  2. [MAX|MIN|AVG|STD|NEWEST|LATEST] [Property] value of [Event] in [Tool] every [Period] by team.
  3. [Frequency] distribution of [Event] in [Tool] every [Period] by team.
  4. Percentage of [Event1] in [Tool1] correlated with [Event2] in [Tool2] within [window] every [Period] by team.

Right now these are the events available for each tool:

ToolEvents
GitHubNew branches, Open PRs, Merged PRs, Closed PRs, PR additions/deletions
Pivotal TrackerStarted stories, Finished stories, Delivered stories, Accepted stories
HerokuReleases, Builds
TravisBuilds
CodeClimateCoverage reports

Here we are showing 4 different examples with their corresponding JSON format using the patterns above. Bear in mind that this is the DSL of the metric and the period is not displayed because it belongs to the guarantee:

1. NUMBER_PT_STARTEDSTORIES: Number of STARTED_STORIES in PIVOTAL every DAY by MEMBER

{
    "metric": {
      "computing": "actual",
      "element": "number",
      "event": {
        "pivotal": {
          "activity": {
            "highlight": "finished"
          }
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }
}

2. VALUE_CC_COVERAGE_AVG: AVG COVERAGE value of COVERAGE_REPORT in CODECLIMATE every WEEK by team.


{
    "metric": {
      "computing": "actual",
      "element": {
        "value": {
          "parameter": "attributes.covered_percent",
          "return": "avg",
          "traceback": true
        }
      },
      "event": {
        "codeclimate": {
          "coverage": {}
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }
}

3. STDEV_TR_SUCCESSFULBUILDS_DAILY: DAILY distribution of SUCCESSFUL_BUILDS in TRAVIS every WEEK by team

{
    "metric": {
      "computing": "string",
      "element": {
        "stdev": {
          "period": "daily"
        }
      },
      "event": {
        "travis": {
          "builds_public": {
            "@type": "build",
            "state": "passed"          
          }
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }
}

4. PERCENTAGE_GH_NEWBRANCH_PT_STARTEDSTORIES: Percentage of NEW_BRANCH in GITHUB correlated with START_STORY in PIVOTAL_TRACKER within 1_HOUR every WEEK by team.

{
    "metric": {
      "computing": "actual",
      "element": {
        "percentage": {
          "related": {
            "github": {
              "events": {
                "type": "CreateEvent",
                "payload": {
                  "ref_type": "branch"
                }
              }
            },
            "window": 3600
          }
        }
      },
      "event": {
        "pivotal": {
          "activity": {
            "highlight": "started"
          }
        }
      },
      "scope": {
        "$ref": "#/context/definitions/scopes/development"
      }
    }
}

For a full list of metric pattern examples, please follow this link where you can found the following examples:

Non correlation metrics:

Metric TypeToolEventDSL Definition
NumberGitHubNew BranchesShow definition
NumberGitHubOpen PRShow definition
NumberGitHubMerged PRShow definition
NumberPivotal TrackerStarted StoriesShow definition
NumberPivotal TrackerFinished StoriesShow definition
NumberPivotal TrackerDelivered StoriesShow definition
NumberPivotal TrackerAccepted StoriesShow definition
NumberHerokuReleasesShow definition
NumberTravisBuildsShow definition
NumberTravisFailed BuildsShow definition
NumberCodeClimateCoverage ReportsShow definition
NumberCodeClimateCoverage Reports over 80Show definition
ValueCodeClimateCoverageShow definition
ValueCodeClimateCoverage OffsettedShow definition
STDevGitHubDaily Merged PRShow definition

Correlation metrics:

Metric TypeTool 1Event 1Tool 2Event 2Correlation TypeDSL Definition
PercentageGitHubNew BranchPivotal TrackerStarted StoriesTime WindowShow definition
PercentageGitHubNew BranchPivotal TrackerStarted StoriesBindShow definition
PercentageGitHubOpen PRPivotal TrackerFinished StoriesTime WindowShow definition
PercentageGitHubOpen PRPivotal TrackerFinished StoriesBindShow definition
PercentageGitHubMerge PRPivotal TrackerDelivered StoriesTime WindowShow definition
PercentageGitHubMerge PRPivotal TrackerDelivered StoriesBindShow definition
PercentageGitHub WrapperNew BranchPivotal TrackerStarted StoriesTime WindowShow definition
PercentageHerokuReleasesPivotal TrackerDelivered StoriesTime WindowShow definition
PercentageHerokuReleasesPivotal TrackerDelivered StoriesBindShow definition
PercentageTravisSuccessful BuildsTravisBuildsTime WindowShow definition
PercentageCodeClimateCoverage reports over 80CodeClimateCoverage reportsTime WindowShow definition

Pivotal Tracker Collector​

The PT Collector connects with Pivotal Tracker and can produce more specific metrics using the different payloads.

Examples​

There is a webpage with available examples for public usage.

GitHub Collector​

Using GitHub API v4, it can access to the information from GitHub Projects Kanbans in order to keep track of the different cards and the changes.

Examples​

There is a webpage with available examples for public usage.