Skip to content

Commit

Permalink
*: Rename ReportGenerationQuery to ReportQuery
Browse files Browse the repository at this point in the history
  • Loading branch information
Chance Zibolski committed May 7, 2019
1 parent 0cbcb85 commit 7af7c16
Show file tree
Hide file tree
Showing 100 changed files with 1,291 additions and 1,291 deletions.
2 changes: 1 addition & 1 deletion Documentation/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Custom Resources:

- [Reports](report.md)
- [Roll-up Reports](rollup-reports.md)
- [ReportGenerationQueries](reportgenerationqueries.md)
- [ReportQueries](reportqueries.md)
- [ReportDataSources](reportdatasources.md)
- [StorageLocations](storagelocations.md)

26 changes: 13 additions & 13 deletions Documentation/metering-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ There are 6 custom resources that Operator Metering defines that you need to be
- `PrestoTables`: Defines a table in Presto. A PrestoTable can be "unmanaged" to expose a table that already exists, or "managed" to instruct metering to create a table as a result of the resource being created.
- `HiveTables`: Defines a table in Hive. When created, it instructs metering to create the table in Hive which causes the table to be available to Presto.
- `ReportDataSources`: Controls what data is available (Prometheus data, AWS billing data, Presto tables, views into other tables).
- `ReportGenerationQueries`: Controls how we query the data available within ReportDataSources. If referenced by a `Report` it will manage what it will be reporting on when the report is run. If it's referenced by a `ReportDataSource` it will instruct metering to create a view within Presto based on the rendered query.
- `Reports`: Causes reports to be generated using the configured `ReportGenerationQuery` resource. This is the primary resource an end-user of Operator Metering would interact with. Can be configured to run on a schedule.
- `ReportQueries`: Controls how we query the data available within ReportDataSources. If referenced by a `Report` it will manage what it will be reporting on when the report is run. If it's referenced by a `ReportDataSource` it will instruct metering to create a view within Presto based on the rendered query.
- `Reports`: Causes reports to be generated using the configured `ReportQuery` resource. This is the primary resource an end-user of Operator Metering would interact with. Can be configured to run on a schedule.

In the sections below, we will cover the resources described above in more detail.

Expand All @@ -55,7 +55,7 @@ There are many types of ReportDataSources, with the most common being a "Prometh

A PrometheusMetricsImporter ReportDataSource instructs the reporting-operator to create a database table for storing Prometheus metric data and to being the process of importing Prometheus metrics.
You can also define a AWSBilling ReportDataSource to create table pointing at an existing S3 bucket containing AWS Cost and Usage reports.
Additionally, there are GenerationQueryView ReportDataSource's which create views in Presto, and PrestoTable ReportDataSource's which just expose an existing PrestoTable as a ReportDataSource.
Additionally, there are ReportQueryView ReportDataSource's which create views in Presto, and PrestoTable ReportDataSource's which just expose an existing PrestoTable as a ReportDataSource.


#### PrometheusMetricsImporter ReportDataSources
Expand Down Expand Up @@ -99,14 +99,14 @@ By default, Operator Metering has an section in the `Metering` resource for conf
For more details on configuring this read the [AWS billing correlation section in the Metering Configuration doc][metering-aws-billing-conf].


#### ReportGenerationQueryView View ReportDataSources
#### ReportReportQueryView View ReportDataSources

A `ReportGenerationQuery` ReportDataSource configures the reporting-operator to create a Presto view based on the query specified.
A `ReportQuery` ReportDataSource configures the reporting-operator to create a Presto view based on the query specified.

When the ReportDataSource is created, the reporting-operator:

- Checks if the `ReportDataSource` has a view created for it yet by checking the `status.tableRef` field.
- If it field is empty, it creates a view by creating a `PrestoTable` resource with `spec.view` set to true, and `spec.query` set the rendered value of the ReportGenerationQuery's spec.query field.
- If it field is empty, it creates a view by creating a `PrestoTable` resource with `spec.view` set to true, and `spec.query` set the rendered value of the ReportQuery's spec.query field.
- It then waits for the PrestoTable's `spec.tableName` to be set, and updates the ReportDataSource's `spec.tableRef` to indicate the view was created successfully.

#### PrestoTable ReportDataSources
Expand All @@ -118,22 +118,22 @@ When the ReportDataSource is created, the reporting-operator:
- Lookups the `PrestoTable` resource and verifies it's `status.tableName` is set.
- If the `status.tableName` is set then it will update the ReportDataSource's `spec.tableRef` to the tableName.

### ReportGenerationQuery
### ReportQuery

For user-docs containing a description of the fields, and examples, see [ReportGenerationQueries][reportgenerationqueries].
For user-docs containing a description of the fields, and examples, see [ReportQueries][reportqueries].

When the metering operator sees a new `ReportGenerationQuery` in its namespace, it will reprocess anything that depends on it (ReportDataSources or Reports).
When the metering operator sees a new `ReportQuery` in its namespace, it will reprocess anything that depends on it (ReportDataSources or Reports).

### Report

For user-docs containing a description of the fields, and examples, see [Reports][reports].

When a `Report` is created, and it sees the creation event, it does the following to generate the results:

- Retrieve the `ReportGenerationQuery` for the Report.
- For each `ReportGenerationQuery`, `ReportDataSource` and `Report` input in the `spec.inputs`, retrieve and validate them, and any of their dependencies.
- Retrieve the `ReportQuery` for the Report.
- For each `ReportQuery`, `ReportDataSource` and `Report` input in the `spec.inputs`, retrieve and validate them, and any of their dependencies.
- Update the Report status to indicate we're beginning to generate the report.
- Evaluate the `ReportGenerationQuery` template, passing the StartPeriod & EndPeriod into the template context.
- Evaluate the `ReportQuery` template, passing the StartPeriod & EndPeriod into the template context.
- Create a database table using Hive named after the Report. This table is either configured to use HDFS or S3 based on the Report's StorageLocation.
- Execute the query using Presto.
- Update the Report status that everything succeeded.
Expand All @@ -145,5 +145,5 @@ When a `Report` is created, and it sees the creation event, it does the followin
[metering-aws-billing-conf]: metering-config.md#aws-billing-correlation
[storagelocations]: storagelocations.md
[reportdatasources]: reportdatasources.md
[reportgenerationqueries]: reportgenerationqueries.md
[reportqueries]: reportqueries.md
[reports]: report.md
36 changes: 18 additions & 18 deletions Documentation/report.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ kind: Report
metadata:
name: pod-cpu-request-hourly
spec:
generationQuery: "pod-cpu-request"
query: "pod-cpu-request"
reportingStart: "2018-07-01T00:00:00Z"
schedule:
period: "hourly"
Expand All @@ -41,22 +41,22 @@ kind: Report
metadata:
name: pod-cpu-request-hourly
spec:
generationQuery: "pod-cpu-request"
query: "pod-cpu-request"
reportingStart: "2018-07-01T00:00:00Z"
reportingEnd: "2018-07-31T00:00:00Z"
```

### generationQuery
### query

Names the `ReportGenerationQuery` used to generate the report.
The generation query controls the schema of the report as well how the results are processed.
Names the `ReportQuery` used to generate the report.
The report query controls the schema of the report as well how the results are processed.

*`generationQuery` is a required field.*
*`query` is a required field.*

Use `kubectl` to obtain a list of available `ReportGenerationQuery` objects:
Use `kubectl` to obtain a list of available `ReportQuery` objects:

```
kubectl -n $METERING_NAMESPACE get reportgenerationqueries
kubectl -n $METERING_NAMESPACE get reportqueries
NAME AGE
cluster-cpu-capacity 23m
cluster-cpu-capacity-raw 23m
Expand Down Expand Up @@ -105,7 +105,7 @@ pod-memory-usage 23m
pod-memory-usage-raw 23m
```

ReportGenerationQueries with the `-raw` suffix are used by other ReportGenerationQueries to build more complex queries, and should not be used directly for reports.
ReportQueries with the `-raw` suffix are used by other ReportQueries to build more complex queries, and should not be used directly for reports.

`namespace-` prefixed queries aggregate Pod CPU/memory requests by namespace, providing a list of namespaces and their overall usage based on resource requests.

Expand All @@ -120,11 +120,11 @@ The `aws-ec2-billing-data` report is used by other queries, and should not be us
For a complete list of fields each report query produces, use `kubectl` to get the object as JSON, and check the `columns` field:

```
kubectl -n $METERING_NAMESPACE get reportgenerationqueries namespace-memory-request -o json
kubectl -n $METERING_NAMESPACE get reportqueries namespace-memory-request -o json
{
"apiVersion": "metering.openshift.io/v1alpha1",
"kind": "ReportGenerationQuery",
"kind": "ReportQuery",
"metadata": {
"name": "namespace-memory-request",
"namespace": "metering"
Expand Down Expand Up @@ -220,7 +220,7 @@ kind: Report
metadata:
name: pod-cpu-request-hourly
spec:
generationQuery: "pod-cpu-request"
query: "pod-cpu-request"
schedule:
period: "hourly"
reportingStart: "2018-01-01T00:00:00Z"
Expand All @@ -242,7 +242,7 @@ kind: Report
metadata:
name: pod-cpu-request-hourly
spec:
generationQuery: "pod-cpu-request"
query: "pod-cpu-request"
schedule:
period: "weekly"
reportingStart: "2018-07-01T00:00:00Z"
Expand All @@ -256,7 +256,7 @@ For reports with a schedule set, it will not wait for each period's reportingEnd

### Inputs

The `spec.inputs` field of a Report can be used to override or set values defined in a [ReportGenerationQuery's spec.input field][query-inputs].
The `spec.inputs` field of a Report can be used to override or set values defined in a [ReportQuery's spec.input field][query-inputs].

It is a list of name-value pairs:

Expand All @@ -273,8 +273,8 @@ For an example of how this can be used, see it in action [in a roll-up report](r

Report data is stored in the database much like metrics themselves, and can thus be used in aggregated or roll-up reports. A simple use case for a roll-up report is to spread the time required to produce a report over a longer period of time: instead of requiring a monthly report to query and add all data over an entire month, the task can be split into daily reports that each run over a thirtieth of the data.

A custom roll-up report requires a custom generation query.
The ReportGenerationQuery template processor provides a function: `reportTableName` that can get the necessary table name [from a report name](rollup-reports.md#2-create-the-aggregation-query).
A custom roll-up report requires a custom report query.
The ReportQuery template processor provides a function: `reportTableName` that can get the necessary table name [from a report name](rollup-reports.md#2-create-the-aggregation-query).

Below is an snippet taken from a built-in query:

Expand Down Expand Up @@ -305,7 +305,7 @@ spec:
```
# aggregated-report.yaml
spec:
generationQuery: "namespace-cpu-usage"
query: "namespace-cpu-usage"
inputs:
- name: "NamespaceCPUUsageReportName"
value: "namespace-cpu-usage-hourly"
Expand All @@ -323,4 +323,4 @@ The `status` field of a `Report` currently has two fields:
- `lastReportTime`: Indicates the time Metering has collected data up to.

[rfc3339]: https://tools.ietf.org/html/rfc3339#section-5.8
[query-inputs]: reportgenerationqueries.md#query-inputs
[query-inputs]: reportqueries.md#query-inputs
24 changes: 12 additions & 12 deletions Documentation/reportdatasources.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

A `ReportDataSource` is a custom resource that represents how to store data, such as where it should be stored, and in some cases, how the data is to be collected.

There are currently four types of ReportDataSource's: `prometheusMetricsImporter`, `awsBilling`, `generationQueryView` and `prestoTable`.
There are currently four types of ReportDataSource's: `prometheusMetricsImporter`, `awsBilling`, `reportQueryView` and `prestoTable`.
Each has a corresponding configuration section within the `spec` of a `ReportDataSource`.
The main effect that creating a ReportDataSource has is that it causes the metering operator to create a table in Presto or Hive.
Depending on the type of ReportDataSource it then may do other additional tasks.
Expand All @@ -23,9 +23,9 @@ To read more details on how the different ReportDataSources work, read the [mete
- `bucket`: Bucket name to store data into.
- `prefix`: Path within the bucket where to store data.
- `region`: The region where bucket is located.
- `generationQueryView`: If this section is present, then the `ReportDataSource` will be configured to create a View in Presto using the rendered `spec.query` as the query for the view.
- `queryName`: The name of a [ReportGenerationQuery][reportgenerationquery] to create a view from.
- `inputs`: A list of inputs this report query accepts to control its behavior. For more in depth details, see the [ReportGenerationQuery query inputs][query-inputs] documentation.
- `reportQueryView`: If this section is present, then the `ReportDataSource` will be configured to create a View in Presto using the rendered `spec.query` as the query for the view.
- `queryName`: The name of a [ReportQuery][reportquery] to create a view from.
- `inputs`: A list of inputs this report query accepts to control its behavior. For more in depth details, see the [ReportQuery query inputs][query-inputs] documentation.
- `storage`: This section controls the `StorageLocation` options, allowing you to control on a per ReportDataSource level, where data is stored.
- `storageLocationName`: The name of the `StorageLocation` resource to use.
- `prestoTable`: If present, then the `ReportDataSource` will simply make it possible to reference a database table within Presto as a ReportDataSource.
Expand Down Expand Up @@ -75,16 +75,16 @@ spec:
url: http://custom-prometheus-instance:9090
```

## ReportGenerationQuery View Datasource
## ReportQuery View Datasource

For ReportDataSources with a `spec.generationQueryView` present, a Presto view will be created using the rendered output of a specified [ReportGenerationQuery][reportgenerationquery]'s `spec.query` field.
For ReportDataSources with a `spec.reportQueryView` present, a Presto view will be created using the rendered output of a specified [ReportQuery][reportquery]'s `spec.query` field.
This enables abstracting away the details of more complex queries by exposing them as a database table whose content is based on the result of of the query the view is based on.
It also enables re-use by allowing you to create a view containing the complexities of a query allowing other queries to simply query it as a regular table.

### Example ReportGenerationQuery View Datasource
### Example ReportQuery View Datasource

This example exposes the `pod-memory-request-raw` ReportGenerationQuery as a view.
The schema is based on the `spec.columns` of the ReportGenerationQuery.
This example exposes the `pod-memory-request-raw` ReportQuery as a view.
The schema is based on the `spec.columns` of the ReportQuery.

```
apiVersion: metering.openshift.io/v1alpha1
Expand All @@ -94,7 +94,7 @@ metadata:
labels:
operator-metering: "true"
spec:
generationQueryView:
reportQueryView:
queryName: pod-memory-request-raw
```

Expand Down Expand Up @@ -145,7 +145,7 @@ spec:
[default-storage-location]: storagelocations.md#default-storagelocation
[architecture]: metering-architecture.md
[presto-types]: https://prestodb.io/docs/current/language/types.html
[reportgenerationqueries]: reportgenerationqueries.md
[reportgenerationquery]: reportgenerationqueries.md
[query-inputs]: reportgenerationqueries.md#query-inputs
[reportquery]: reportqueries.md
[query-inputs]: reportqueries.md#query-inputs
[prestotables]: prestotables.md
Loading

0 comments on commit 7af7c16

Please sign in to comment.