madmom.evaluation¶
Evaluation package.
-
madmom.evaluation.find_closest_matches(detections, annotations)[source]¶ Find the closest annotation for each detection.
Parameters: detections : list or numpy array
Detected events.
annotations : list or numpy array
Annotated events.
Returns: indices : numpy array
Indices of the closest matches.
Notes
The sequences must be ordered.
-
madmom.evaluation.calc_errors(detections, annotations, matches=None)[source]¶ Errors of the detections to the closest annotations.
Parameters: detections : list or numpy array
Detected events.
annotations : list or numpy array
Annotated events.
matches : list or numpy array
Indices of the closest events.
Returns: errors : numpy array
Errors.
Notes
The sequences must be ordered. To speed up the calculation, a list of pre-computed indices of the closest matches can be used.
-
madmom.evaluation.calc_absolute_errors(detections, annotations, matches=None)[source]¶ Absolute errors of the detections to the closest annotations.
Parameters: detections : list or numpy array
Detected events.
annotations : list or numpy array
Annotated events.
matches : list or numpy array
Indices of the closest events.
Returns: errors : numpy array
Absolute errors.
Notes
The sequences must be ordered. To speed up the calculation, a list of pre-computed indices of the closest matches can be used.
-
madmom.evaluation.calc_relative_errors(detections, annotations, matches=None)[source]¶ Relative errors of the detections to the closest annotations.
Parameters: detections : list or numpy array
Detected events.
annotations : list or numpy array
Annotated events.
matches : list or numpy array
Indices of the closest events.
Returns: errors : numpy array
Relative errors.
Notes
The sequences must be ordered. To speed up the calculation, a list of pre-computed indices of the closest matches can be used.
-
class
madmom.evaluation.EvaluationMixin[source]¶ Evaluation mixin class.
This class has a name attribute which is used for display purposes and defaults to ‘None’.
METRIC_NAMES is a list of tuples, containing the attribute’s name and the corresponding label, e.g.:
The attributes defined in METRIC_NAMES will be provided as an ordered dictionary as the metrics property unless the subclass overwrites the property.
FLOAT_FORMAT is used to format floats.
-
metrics¶ Metrics as a dictionary.
-
tostring(**kwargs)[source]¶ Format the evaluation metrics as a human readable string.
Returns: str
Evaluation metrics formatted as a human readable string.
Notes
This is a fallback method formatting the metrics dictionary in a human readable way. Classes inheriting from this mixin class should provide a method better suitable.
-
-
class
madmom.evaluation.SimpleEvaluation(num_tp=0, num_fp=0, num_tn=0, num_fn=0, name=None, **kwargs)[source]¶ Simple Precision, Recall, F-measure and Accuracy evaluation based on the numbers of true/false positive/negative detections.
Parameters: num_tp : int
Number of true positive detections.
num_fp : int
Number of false positive detections.
num_tn : int
Number of true negative detections.
num_fn : int
Number of false negative detections.
name : str
Name to be displayed.
Notes
This class is only suitable for a 1-class evaluation problem.
-
num_tp¶ Number of true positive detections.
-
num_fp¶ Number of false positive detections.
-
num_tn¶ Number of true negative detections.
-
num_fn¶ Number of false negative detections.
-
num_annotations¶ Number of annotations.
-
precision¶ Precision.
-
recall¶ Recall.
-
fmeasure¶ F-measure.
-
accuracy¶ Accuracy.
-
-
class
madmom.evaluation.Evaluation(tp=None, fp=None, tn=None, fn=None, **kwargs)[source]¶ Evaluation class for measuring Precision, Recall and F-measure based on numpy arrays or lists with true/false positive/negative detections.
Parameters: tp : list or numpy array
True positive detections.
fp : list or numpy array
False positive detections.
tn : list or numpy array
True negative detections.
fn : list or numpy array
False negative detections.
name : str
Name to be displayed.
-
num_tp¶ Number of true positive detections.
-
num_fp¶ Number of false positive detections.
-
num_tn¶ Number of true negative detections.
-
num_fn¶ Number of false negative detections.
-
-
class
madmom.evaluation.MultiClassEvaluation(tp=None, fp=None, tn=None, fn=None, **kwargs)[source]¶ Evaluation class for measuring Precision, Recall and F-measure based on 2D numpy arrays with true/false positive/negative detections.
Parameters: tp : list of tuples or numpy array, shape (num_tp, 2)
True positive detections.
fp : list of tuples or numpy array, shape (num_fp, 2)
False positive detections.
tn : list of tuples or numpy array, shape (num_tn, 2)
True negative detections.
fn : list of tuples or numpy array, shape (num_fn, 2)
False negative detections.
name : str
Name to be displayed.
Notes
The second item of the tuples or the second column of the arrays denote the class the detection belongs to.
-
class
madmom.evaluation.SumEvaluation(eval_objects, name=None)[source]¶ Simple class for summing evaluations.
Parameters: eval_objects : list
Evaluation objects.
name : str
Name to be displayed.
-
num_tp¶ Number of true positive detections.
-
num_fp¶ Number of false positive detections.
-
num_tn¶ Number of true negative detections.
-
num_fn¶ Number of false negative detections.
-
num_annotations¶ Number of annotations.
-
-
class
madmom.evaluation.MeanEvaluation(eval_objects, name=None, **kwargs)[source]¶ Simple class for averaging evaluation.
Parameters: eval_objects : list
Evaluation objects.
name : str
Name to be displayed.
-
num_tp¶ Number of true positive detections.
-
num_fp¶ Number of false positive detections.
-
num_tn¶ Number of true negative detections.
-
num_fn¶ Number of false negative detections.
-
num_annotations¶ Number of annotations.
-
precision¶ Precision.
-
recall¶ Recall.
-
fmeasure¶ F-measure.
-
accuracy¶ Accuracy.
-
-
madmom.evaluation.tostring(eval_objects, **kwargs)[source]¶ Format the given evaluation objects as human readable strings.
Parameters: eval_objects : list
Evaluation objects.
Returns: str
Evaluation metrics formatted as a human readable string.
-
madmom.evaluation.tocsv(eval_objects, metric_names=None, float_format='{:.3f}', **kwargs)[source]¶ Format the given evaluation objects as a CSV table.
Parameters: eval_objects : list
Evaluation objects.
metric_names : list of tuples, optional
List of tuples defining the name of the property corresponding to the metric, and the metric label e.g. (‘fp’, ‘False Positives’).
float_format : str, optional
How to format the metrics.
Returns: str
CSV table representation of the evaluation objects.
Notes
If no metric_names are given, they will be extracted from the first evaluation object.
-
madmom.evaluation.totex(eval_objects, metric_names=None, float_format='{:.3f}', **kwargs)[source]¶ Format the given evaluation objects as a LaTeX table.
Parameters: eval_objects : list
Evaluation objects.
metric_names : list of tuples, optional
List of tuples defining the name of the property corresponding to the metric, and the metric label e.g. (‘fp’, ‘False Positives’).
float_format : str, optional
How to format the metrics.
Returns: str
LaTeX table representation of the evaluation objects.
Notes
If no metric_names are given, they will be extracted from the first evaluation object.
-
madmom.evaluation.evaluation_io(parser, ann_suffix, det_suffix, ann_dir=None, det_dir=None)[source]¶ Add evaluation input/output and formatting related arguments to an existing parser object.
Parameters: parser : argparse parser instance
Existing argparse parser object.
ann_suffix : str
Suffix of the annotation files.
det_suffix : str
Suffix of the detection files.
ann_dir : str, optional
Use only annotations from this folder (and sub-folders).
det_dir : str, optional
Use only detections from this folder (and sub-folders).
Returns: io_group : argparse argument group
Evaluation input / output argument group.
formatter_group : argparse argument group
Evaluation formatter argument group.