Kubernetes Measure Definitions
Prerequisites/troubleshooting lack of K8s metrics in Ternary:
It is important to note that we use Google's native Cloud Monitoring to gather Kubernetes metrics. It is enabled by default on GKE clusters. If you do not have it enabled, you will need to enable it for Ternary to ingest your Kubernetes cluster, namespace, workload and pod metrics.
Another issue we have seen, if Cloud Monitoring is enabled and we are still not seeing metrics, is permissions related issues in publishing the metrics to the GCP console in Cloud Monitoring. For example, if you use a custom service account and not the default Compute Engine service account the absence of metrics could be explained by missing roles such as roles/logging.logWriter, roles/monitoring.metricWriter, roles/stackdriver.resourceMetdata.writer. You can check if Kubernetes metrics are being published by navigating to the Metrics Explorer in the GCP console and replicating this query:
- Metric: Kubernetes Container - CPU Usage time
- Filters: project_id=your_project_id
- groupingFunction: Sum
- groupByLabels:
- cluster_name
- node_name
- top_level_controller_name
- top_level_controller_type
- container_name
- location
- namespace_name
- pod_name
- perSeriesAligner: Rate
- alignmentPeriod: 1h
If this query returns no results the issue may be in Cloud Monitoring metrics for Kubernetes being published to the GCP console.
Kubernetes Container Measure Definition
| Metric | Definition |
|---|---|
| limitCPUCores | Core-Days |
| limitMemoryBytes | Byte-Days |
| requestCPUCores | k8s_container.request_cpu_cores / 24.0 = requestCPUCores in Core-Days |
| requestCPUCost | (requestCPUCores) * (CPU USD Price) = Dollars |
| requestMemoryBytes | k8s_container.request_memory_bytes / 24.0 = requestMemoryBytes in Byte-Days |
| requestMemoryCost | k8s_container.request_memory_bytes * k8s_node_detail.ram_usd_price / 1073741824.0 = requestMemoryCost in USD (There are 1073741824.0 bytes per GiB) |
| totalRequestCost | requestMemoryCost + requestCPUCost = totalRequestCost in USD |
| totalWasteCost | requestMemoryCost + requestCPUCost - usedCPUCost - usedMemoryCost = totalWasteCost in USD |
| usedCPUCores | k8s_container.used_cpu_cores / 24.0 = usedCPUCores in Core-Days |
| usedCPUCost | k8s_container.used_cpu_cores * k8s_node_detail.cpu_usd_price = usedCPUCost in USD |
| usedMemoryBytes | k8s_container.used_memory_bytes / 24.0 = usedMemoryBytes in Byte-Days |
| usedMemoryCost | k8s_container.used_memory_bytes * k8s_node_detail.ram_usd_price / 1073741824.0 = usedMemoryCost in USD |
| wasteCPUs | requestCPUCores - usedCPUCores = wasteCPUs in Core-Days |
| wasteMemoryBytes | requestMemoryBytes - usedMemoryBytes = wasteMemoryBytes in Byte-Days |
| wasteCPUCost | requestCPUCost - usedCPUCost = wasteCPUCost in USD |
| wasteMemoryCost | requestMemoryCost - usedMemoryCost = wasteMemoryCost in USD |
Kubernetes Node Measure Definition
| Metric | Definition |
|---|---|
| allocatableCPUUtilization | allocatable_cpu_utilization / 24.0 = allocatableCPUUtilization |
| allocatableMemoryUtilization | allocatable_memory_utilization / 24.0 = allocatableMemoryUtilization |
| CPUCoreUsageTime | cpu_core_usage_time / 24.0 = CPUCoreUsageTime in Core-Days |
| totalCost | totalMemoryCost + totalCPUCost = totalCost in USD |
| totalCPUCores | k8s_node.total_cpu_cores / 24.0 = totalCPUCores in Core-Days |
| totalCPUCost | k8s_node.total_cpu_cores * k8s_node_detail.cpu_usd_price = totalCPUCost in USD |
| totalMemoryCost | k8s_node.total_memory_bytes * k8s_node_detail.ram_usd_price / 1073741824.0 = totalMemoryCost in USD |
| totalMemoryBytes | k8s_node.total_memory_bytes / 24.0 = totalMemoryBytes in Byte-Days |
| usedMemoryBytes | used_memory_bytes / 24.0 = usedMemoryBytes in Byte-Days |
| wasteCPUs | totalCPUCores - CPUCoreUsagetime = wasteCPUs in Core-Days |
| wasteMemoryBytes | totalMemorybytes - usedMemoryBytes = wasteMemoryBytes in Bytes |
Updated 5 months ago