Skip to content

Commit 4b63fea

Browse files
diosmosissgiehlmichalkleiner
authored
use RecordBuilders in CustomDimensions plugin (matomo-org#20720)
* introduce RecordBuilder concept and re-organize Goals archiving code via RecordBuilders * fix loop iteration bug * split ecommerce records recordbuilder into 3 separate records * make sure Goals::getRecordMetadata() behaves like old archiver code * make sure recordbuilder archive processor is restored after being used since archiving is a recursive process * just make ArchiveProcessor a parameter * check for plugin before calling buildMultiplePeriod() * do not invoke record builders if archiver has no plugin (happens during tests) * insert empty DataTables (as this appears to be the existing behavior before this change) * add RecordBuilder class name to aggregation query hint * clear up in-source todo * attempt only archiving requested report if range archive and the record needed is created by a RecordBuilder * refactor ArchiveSelector::getArchiveIds() to provide result with string keys * when all found archives are partial archives, check that requested data is present within them. if some are not present, only archive those in a new partial archive. * return correct value in Model::getRecordsContainedInArchives() * fix if formatting * existingArchives can be falsy * existing archives can be null if the check is not relevant to the current archive request * do not archive dependent segments if only processing the specific requested report * fix more tests * fix LoaderTest * make sure if archiving specific reports for a single plugin that archiver class instances will not be created * add filterRecordBuilders event * if it looks like the requested records are numeric, prioritize the numeric archive table, otherwise blob archive table * fix copy-paste error * add dummy test for numeric values * add test for partial archiving of numeric records for ranges and fix typo causing this to fail * lessen code redundancy in Archive.php, use Piwik\\Request and do not yet mark RecordBuilder as api * fix type hint * fix php-cs errors * fix failing tests * fix failing tests (really) * Add Archiver::getDependentSegmentsToArchive() so plugins do not have to implement aggregation methods in Archiver to process dependent archives. * move sitesearch category archiving to record builder * unfinished * move rest of actions archiving to ActionReports + add option for blob Records to set custom column aggregation ops * simplify, add typehints and get simple tests to pass * only do recursive row count in RecordBuilder if there is a numeric record that depends on it * fix isEnabled calls * only add idarchive to Archive.php idarchive cache if it is not already there (makes debugging a little less confusing) * remove unneeded TODO * when forcing new archive because timestamp is too old, do not report any existing archives * report no existing archives if done flag is different + add tests * update uses of ARCHIVE_DEPENDENT static variable * array intersect instead of array dif * ArchivingHelper::updateActionsTableWithRowQuery() needs access to every report being built in case the action type is NULL * should never insert datatable maps * change ordering of row deletion * use correct re-introduced variable * fix ranking query override * fix OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest * one more test failure fix * fix one more test * move content archiving code to RecordBuilder * start on using record builders for custom dimensions * remove comment * remove unused imports * remove unused imports * make sure when numeric records that depend on blob records are archived, the blob records archived first * remove unneeded unset * handle ranking query summary row * ranking query can be null * ignore interactions that do not have an impression * fix analyze archive table test * handle rankingquery summary row + convert values to float since they can sometimes be strings like "0.000" * accidentally fixing BlobReportLimitingTest. maybe. * remove no longer needed DataArray subclass * fix phpcs * remove unneeded newline * add clarifying comment * move datatable modification methods to DataTable & Row since it will probably come up in review * use new methods in DataTable, Row * fix unrefactored function calls * fix copy paste errors * allow using custom column aggregation ops when calling new DataTable methods * add newline for ContentRecords * newline * use siteAware cache for RecordBuilder array * better typehints in RecordBuilder * allow dependent segments to specific different plugin than the one containing the archiver * add typehint * return summed row in Row::sumRowWithLabelToSubtable() * allow descendents of RecordBuilder to manually insert records if needed * ignore any records that are not declared in the record metadata (which can happen, for instance, when a goal has been deleted but is still referred to in log data) * apply review feedback * remove stray debugging change * Update variable name for consistency * Remove unnecessary array_filter since a valid class name never has an empty segment * Add TODOs * add comment on why we look for data within partial archives prior to reporting whether archives were found or not * typehint fixes + make insertBlobRecord (formerly insertRecord) protected for use in RecordBuilders that need to manually insert data * more typehints * in aggregateNumericMetrics() allow operationsToApply to be array mapping column name to op * apply review feedback --------- Co-authored-by: Stefan Giehl <[email protected]> Co-authored-by: Michal Kleiner <[email protected]>
1 parent e13e124 commit 4b63fea

7 files changed

Lines changed: 439 additions & 366 deletions

File tree

‎core/DataArray.php‎

Lines changed: 2 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
namespace Piwik;
1010

1111
use Exception;
12+
use Piwik\DataTable\Filter\EnrichRecordWithGoalMetricSums;
1213
use Piwik\Tracker\GoalManager;
1314

1415
/**
@@ -378,31 +379,7 @@ public function enrichMetricsWithConversions()
378379
protected function enrichWithConversions(&$data)
379380
{
380381
foreach ($data as &$values) {
381-
if (!isset($values[Metrics::INDEX_GOALS])) {
382-
continue;
383-
}
384-
385-
$revenue = $conversions = 0;
386-
foreach ($values[Metrics::INDEX_GOALS] as $idgoal => $goalValues) {
387-
// Do not sum Cart revenue since it is a lost revenue
388-
if ($idgoal >= GoalManager::IDGOAL_ORDER) {
389-
$revenue += $goalValues[Metrics::INDEX_GOAL_REVENUE];
390-
$conversions += $goalValues[Metrics::INDEX_GOAL_NB_CONVERSIONS];
391-
}
392-
}
393-
$values[Metrics::INDEX_NB_CONVERSIONS] = $conversions;
394-
395-
// 25.00 recorded as 25
396-
if (round($revenue) == $revenue) {
397-
$revenue = round($revenue);
398-
}
399-
$values[Metrics::INDEX_REVENUE] = $revenue;
400-
401-
// if there are no "visit" column, we force one to prevent future complications
402-
// eg. This helps the setDefaultColumnsToDisplay() call
403-
if (!isset($values[Metrics::INDEX_NB_VISITS])) {
404-
$values[Metrics::INDEX_NB_VISITS] = 0;
405-
}
382+
EnrichRecordWithGoalMetricSums::enrichWithConversions($values);
406383
}
407384
}
408385

‎core/DataTable/Filter/EnrichRecordWithGoalMetricSums.php‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
<?php
2+
/**
3+
* Matomo - free/libre analytics platform
4+
*
5+
* @link https://matomo.org
6+
* @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
7+
*
8+
*/
9+
10+
namespace Piwik\DataTable\Filter;
11+
12+
use Piwik\DataTable\BaseFilter;
13+
use Piwik\Metrics;
14+
use Piwik\Tracker\GoalManager;
15+
16+
/**
17+
* This filter will check for goal metrics in every row in a DataTable (that is, the Metrics::INDEX_GOALS
18+
* column), and if found, adds the sum of all goal conversions and sum of revenue as two new fields.
19+
*
20+
* This filter is used by RecordBuilders during archiving.
21+
*/
22+
class EnrichRecordWithGoalMetricSums extends BaseFilter
23+
{
24+
public function filter($table)
25+
{
26+
foreach ($table->getRows() as $row) {
27+
$columns = $row->getColumns();
28+
self::enrichWithConversions($columns);
29+
$row->setColumns($columns);
30+
31+
$subtable = $row->getSubtable();
32+
if ($subtable) {
33+
$this->filter($subtable);
34+
}
35+
}
36+
}
37+
38+
public static function enrichWithConversions(&$values): void
39+
{
40+
if (!isset($values[Metrics::INDEX_GOALS])) {
41+
return;
42+
}
43+
44+
$revenue = $conversions = 0;
45+
foreach ($values[Metrics::INDEX_GOALS] as $idgoal => $goalValues) {
46+
// Do not sum Cart revenue since it is a lost revenue
47+
if ($idgoal >= GoalManager::IDGOAL_ORDER) {
48+
$revenue += $goalValues[Metrics::INDEX_GOAL_REVENUE];
49+
$conversions += $goalValues[Metrics::INDEX_GOAL_NB_CONVERSIONS];
50+
}
51+
}
52+
$values[Metrics::INDEX_NB_CONVERSIONS] = $conversions;
53+
54+
// 25.00 recorded as 25
55+
if (round($revenue) == $revenue) {
56+
$revenue = round($revenue);
57+
}
58+
$values[Metrics::INDEX_REVENUE] = $revenue;
59+
60+
// if there are no "visit" column, we force one to prevent future complications
61+
// eg. This helps the setDefaultColumnsToDisplay() call
62+
if (!isset($values[Metrics::INDEX_NB_VISITS])) {
63+
$values[Metrics::INDEX_NB_VISITS] = 0;
64+
}
65+
}
66+
}

‎core/Metrics.php‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010

1111
use Piwik\Cache as PiwikCache;
1212
use Piwik\Columns\Dimension;
13+
use Piwik\Tracker\GoalManager;
1314

1415
require_once PIWIK_INCLUDE_PATH . "/core/Piwik.php";
1516

@@ -581,4 +582,66 @@ public static function getPercentVisitColumn()
581582
$percentVisitsLabel = str_replace(' ', '&nbsp;', Piwik::translate('General_ColumnPercentageVisits'));
582583
return $percentVisitsLabel;
583584
}
585+
586+
/**
587+
* This is a utility method used when building records through log aggregation.
588+
*
589+
* In records with per-goal conversion metrics the metrics are stored within DataTable Rows
590+
* as a column with an array a value. The array is indexed by the goal ID and the column name
591+
* is set to `Metrics::INDEX_GOALS`, for example:
592+
*
593+
* ```
594+
* $columns = [
595+
* Metrics::INDEX_GOALS = [
596+
* $idGoal => [
597+
* // ... conversion metrics ...
598+
* ],
599+
* ],
600+
* ];
601+
* $row = new Row([DataTable::COLUMNS => $columns]);
602+
* ```
603+
*
604+
* This methods returns an array like `$columns` above based on a goal ID and a row of
605+
* metric values for the goal. The result can be added directly to a DataTable record via `sumRowWithLabel()`.
606+
*
607+
* The goal metrics returned will differ based on whether the goal is user defined or an ecommerce goal.
608+
*
609+
* @param int $idGoal
610+
* @param array $goalsMetrics
611+
* @return array
612+
*/
613+
public static function makeGoalColumnsRow(int $idGoal, array $goalsMetrics): array
614+
{
615+
if ($idGoal > GoalManager::IDGOAL_ORDER) { // user defined goal
616+
$columns = [
617+
Metrics::INDEX_GOAL_NB_CONVERSIONS,
618+
Metrics::INDEX_GOAL_NB_VISITS_CONVERTED,
619+
Metrics::INDEX_GOAL_REVENUE,
620+
];
621+
} else if ($idGoal == GoalManager::IDGOAL_ORDER) { // ecommerce order
622+
$columns = [
623+
Metrics::INDEX_GOAL_NB_CONVERSIONS,
624+
Metrics::INDEX_GOAL_NB_VISITS_CONVERTED,
625+
Metrics::INDEX_GOAL_REVENUE,
626+
Metrics::INDEX_GOAL_ECOMMERCE_REVENUE_SUBTOTAL,
627+
Metrics::INDEX_GOAL_ECOMMERCE_REVENUE_TAX,
628+
Metrics::INDEX_GOAL_ECOMMERCE_REVENUE_SHIPPING,
629+
Metrics::INDEX_GOAL_ECOMMERCE_REVENUE_DISCOUNT,
630+
Metrics::INDEX_GOAL_ECOMMERCE_ITEMS,
631+
];
632+
} else { // idGoal == GoalManager::IDGOAL_CART (abandoned cart)
633+
$columns = [
634+
Metrics::INDEX_GOAL_NB_CONVERSIONS,
635+
Metrics::INDEX_GOAL_NB_VISITS_CONVERTED,
636+
Metrics::INDEX_GOAL_REVENUE,
637+
Metrics::INDEX_GOAL_ECOMMERCE_ITEMS,
638+
];
639+
}
640+
641+
$values = [];
642+
foreach ($columns as $column) {
643+
$values[$column] = $goalsMetrics[$column] ?? 0;
644+
}
645+
return $values;
646+
}
584647
}

0 commit comments

Comments
 (0)