Kubernetes Measure Definitions

Prerequisites/troubleshooting lack of K8s metrics in Ternary:

It is important to note that we use Google's native Cloud Monitoring to gather Kubernetes metrics. It is enabled by default on GKE clusters. If you do not have it enabled, you will need to enable it for Ternary to ingest your Kubernetes cluster, namespace, workload and pod metrics.

Another issue we have seen, if Cloud Monitoring is enabled and we are still not seeing metrics, is permissions related issues in publishing the metrics to the GCP console in Cloud Monitoring. For example, if you use a custom service account and not the default Compute Engine service account the absence of metrics could be explained by missing roles such as roles/logging.logWriter, roles/monitoring.metricWriter, roles/stackdriver.resourceMetdata.writer. You can check if Kubernetes metrics are being published by navigating to the Metrics Explorer in the GCP console and replicating this query:

Metric: Kubernetes Container - CPU Usage time
Filters: project_id=your_project_id
groupingFunction: Sum
groupByLabels:
- cluster_name
- node_name
- top_level_controller_name
- top_level_controller_type
- container_name
- location
- namespace_name
- pod_name
perSeriesAligner: Rate
alignmentPeriod: 1h

If this query returns no results the issue may be in Cloud Monitoring metrics for Kubernetes being published to the GCP console.

Kubernetes Container Measure Definition

Metric	Definition
limitCPUCores	Core-Days
limitMemoryBytes	Byte-Days
requestCPUCores	k8s_container.request_cpu_cores / 24.0 = requestCPUCores in Core-Days
requestCPUCost	(requestCPUCores) * (CPU USD Price) = Dollars
requestMemoryBytes	k8s_container.request_memory_bytes / 24.0 = requestMemoryBytes in Byte-Days
requestMemoryCost	k8s_container.request_memory_bytes * k8s_node_detail.ram_usd_price / 1073741824.0 = requestMemoryCost in USD (There are 1073741824.0 bytes per GiB)
totalRequestCost	requestMemoryCost + requestCPUCost = totalRequestCost in USD
totalWasteCost	requestMemoryCost + requestCPUCost - usedCPUCost - usedMemoryCost = totalWasteCost in USD
usedCPUCores	k8s_container.used_cpu_cores / 24.0 = usedCPUCores in Core-Days
usedCPUCost	k8s_container.used_cpu_cores * k8s_node_detail.cpu_usd_price = usedCPUCost in USD
usedMemoryBytes	k8s_container.used_memory_bytes / 24.0 = usedMemoryBytes in Byte-Days
usedMemoryCost	k8s_container.used_memory_bytes * k8s_node_detail.ram_usd_price / 1073741824.0 = usedMemoryCost in USD
wasteCPUs	requestCPUCores - usedCPUCores = wasteCPUs in Core-Days
wasteMemoryBytes	requestMemoryBytes - usedMemoryBytes = wasteMemoryBytes in Byte-Days
wasteCPUCost	requestCPUCost - usedCPUCost = wasteCPUCost in USD
wasteMemoryCost	requestMemoryCost - usedMemoryCost = wasteMemoryCost in USD

Kubernetes Node Measure Definition

Metric	Definition
allocatableCPUUtilization	allocatable_cpu_utilization / 24.0 = allocatableCPUUtilization
allocatableMemoryUtilization	allocatable_memory_utilization / 24.0 = allocatableMemoryUtilization
CPUCoreUsageTime	cpu_core_usage_time / 24.0 = CPUCoreUsageTime in Core-Days
totalCost	totalMemoryCost + totalCPUCost = totalCost in USD
totalCPUCores	k8s_node.total_cpu_cores / 24.0 = totalCPUCores in Core-Days
totalCPUCost	k8s_node.total_cpu_cores * k8s_node_detail.cpu_usd_price = totalCPUCost in USD
totalMemoryCost	k8s_node.total_memory_bytes * k8s_node_detail.ram_usd_price / 1073741824.0 = totalMemoryCost in USD
totalMemoryBytes	k8s_node.total_memory_bytes / 24.0 = totalMemoryBytes in Byte-Days
usedMemoryBytes	used_memory_bytes / 24.0 = usedMemoryBytes in Byte-Days
wasteCPUs	totalCPUCores - CPUCoreUsagetime = wasteCPUs in Core-Days
wasteMemoryBytes	totalMemorybytes - usedMemoryBytes = wasteMemoryBytes in Bytes