Ternary's operations naturally entail assuming privileges of some kind in a customer’s cloud presence. As a result, it is expected that enterprise customers with built-out security organizations may be interested in understanding more about how Ternary protects access to their cloud environment.

Our methodology is intended to please even the most exacting security teams with practices that are beyond the industry average and even sometimes not broadly pushed for adoption by the major cloud providers yet.

Internal IT controls

• All customer data that has already been copied into Ternary’s cloud environment, is stored in datasets and storage buckets with public access disabled and IAM policies installed to only allow access to authorized team members.
• Only a subset of Ternary employees with need to access production, have access to the above production and customer data. This employee list is automatically controlled by an access group rather than giving direct access to individuals. In addition, the IAM policies which grant access to these groups are managed in Terraform.
• Different subsets of Ternary employees with different needs are assigned to groups that grant the minimal privilege to do what they need. For example, developers have greater permissions (read-write) than customer success (read only.)
• Audit logs are retained for any authorized operation; both within Google Cloud itself and within our product. For example, if a CS or Engineer requires access to your customer data, an audit log is written whenever they gain access to your environment.
• All Ternary employee machines are registered with a mobile device management solution which enforces password policies, screen locks, and allows for location detection and remote wipe in the case of loss.

Production controls

We have two projects in our organization for production usage.

• ternary-prod: Where our web application, task runner, and core infrastructure run.
• ternary-prod-cacc: A shell project that exists only to house service accounts that are granted diverse access to all customer’s clouds (cacc stands for customer access.). No Ternary employee has direct access to this project by default, but Organization Administrators would have access to give themselves permissions on the project if they had a business reason to.

In terms of regions and datacenters in use, data processing, compute and storage occur in the Google Cloud US multiregion, for services where multiregions are available; and the us-central1 region of Google Cloud where one has to choose a specific region. If source data (e.g. billing exports) are located in a specific zone or location outside these two areas, the data is copied in by way of a Cloud Storage bucket.

Our web application runs as a specific service account user: [email protected]. The API account also has complete access to the internal Ternary data model in Firestore. This is essential for providing the user interface of My Ternary to you.

To keep the data in Ternary fresh, we use a separate service account: [email protected] . This service account is used by Cloud Scheduler and Cloud Tasks multiple times every day to launch tasks to keep customer data up-to-date.

Both taskrunner@ and apiserver@ service accounts can impersonate the service accounts which you have granted access to your clouds, but do not bear any default privileges over your data until they impersonate the tenant based service accounts, and then only hold those privileges until the impersonation session ends (which could be within the scope of a single request.)

Finally, we use a separate service to manage web based access to our BigQuery dataset, called Cube.js. It runs as a service account: [email protected] . It has access to our full BigQuery data environment, but when accessing this service from the Web, it is required to provide a token that scopes the access to a particular tenant. It's not possible to generate a token to this service that has full access to query data across multiple customers. cubejs@ service account may not impersonate your customer service account, and only has access to data that has already been placed within our environment.

Customer data model

When a customer is created in Ternary, a new BigQuery dataset is created for that customer inside ternary-prod. The customer specific data includes:

• Copies of your billing exports from any cloud provider you've given us access to
• Copies of developer metrics from any cloud provider (e.g. Stackdriver, Cloudwatch)
• Copies of cloud resource manifests (if applicable)
• Processed information from your BigQuery INFORMATION_SCHEMA (GCP only)

Your organization's data lives exclusively in this. No sensitive data is stored outside of that private dataset. Data around users, permissions, and non-sensitive metadata regarding your organization is stored in Google Cloud Firestore, and this data is namespaced to your tenant but not completely airgapped/isolated from the same data of other tenants.

Data in BigQuery is by default retained for 1 year, but is adjustable on a per-tenant basis, upon request.

Service account

For every distinct customer of Ternary, a specific service account is created to manage access to that customer's cloud resources. This means that no one service account has direct access to multiple customers.

Of course, we need to use this service account when accessing your data; therefore some other accounts, and our employees, have access to impersonate this service account on-demand. Doing so generates an audit log which you can request.

This service account looks like this: [email protected] . The nonce ('xxx') is randomly generated, and is not associated with any other identifier in our platform and cannot be reverse engineered back to your company name.

Customer access controls

Here are our overarching principles regarding customer data access.

• The core user facing product is never directly accessing the user’s cloud, but instead is acting on a copy of it that has been fetched into our data store by customer access mechanism described below.
• When the customer’s cloud is accessed, it is using a specific service account that is not granted access to any other customer’s clouds but their own.
• When the customer’s cloud is accessed, the access is initiated from a completely different runtime path and security context than the web application. In other words, no vulnerability in the web application itself (other than by updating the data model and changing the terms or pattern of the data access using the service account) could result in direct API access through to the customer’s cloud. Only an asynchronous task runner that runs under a different deployment and service account would be able to assume the identity of the customer access service account and get through to the customer cloud instance.
• No service account keys, access keys, secret keys are to be generated / shared with customers. That means that we do not use a service account key to authenticate ourselves as the user that can access a customer’s cloud. Conversely, we would uniformly reject any proposal for Ternary platform integration with a customer that included the exchange of pre-shared secret information and instead we would work hard to get them implemented with the proper, secure technique.

In this following subsections, we go into the specifics about how we access data within each cloud provider, what permissions are requested, and what we use them for. If you don't use a particular cloud provider, feel free to skip around to the next sections.

Google Cloud Platform

Methodology

The customer grants IAM access to the aforementioned service account. Since we live on GCP and you are using GCP, there is no additional indirection needed. The IAM access is granted on any eligible GCP resource: project, folder, or organization.

Permissions

Here are all the details of roles that we request to be assigned to our service account on your organization, and the reasons why we need it.

Amazon Web Services

Methodology

There are two authentication approaches for AWS customer data. For more information on this approach, please read our AWS onboarding instructions.

In the most modern one, the same constructs as above are used to protect access to customer clouds. AWS recognizes Google as a federated identity provider. The tokens from the Google service accounts are valid for use on AWS using a feature called AssumeRoleWithWebIdentity. Customers grant AWS access to the web identity matching our Google service account, and no further static credentials or keys need to be generated, transmitted or stored.

In our last-generation authentication mechanism, we use a combination of short lived access keys to our AWS account, and having customers grant IAM access to our AWS account number + a randomly generated ExternalID. This is the method promoted by AWS: https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html . The short lived access keys allow access into our primary AWS account from GCP, and are rotated by a lambda every hour and made available in GCP Secret Manager. When Ternary needs to access an AWS account under this mechanism, we use this key to jump into an IAM identity inside the AWS account, which then allows us to assume the customer role outside our AWS account.

Our goal is to migrate every Ternary AWS customer to the modern approach over a period of 1 year.

Permissions

The most current list of permissions can be found here.

• s3:ListAllMyBuckets, s3:GetLifecycleConfiguration: Collect usage and optimizationinformation for Amazon S3.
• cloudwatch:GetMetricData, cloudwatch:ListMetricStreams, cloudwatch:ListTagsForResource, cloudwatch:GetMetricStatistics, cloudwatch:ListMetrics: Collect generic developer metric information for multiple AWS services to power all of our distinct AWS features.
• compute-optimizer:GetEC2InstanceRecommendations, compute-optimizer:GetAutoScalingGroupRecommendations, compute-optimizer:GetEBSVolumeRecommendations, compute-optimizer:GetLambdaFunctionRecommendations: Retrieve AWS's own optimization information for EC2.
• rds:DescribeReservedDBInstancesOfferings: Retrieve AWS's own optimization information for RDS.
• ce:GetReservationPurchaseRecommendation, ce:GetSavingsPlansPurchaseRecommendation,
• ce:GetRightsizingRecommendation: Retrieve AWS Cost Explorer optimization information.
• EC2:DescribeVolumes, EC2:DescribeInstances, lambda:ListFunctions, lambda:ListProvisionedConcurrencyConfigs: Retrieve information for our AWS Compute feature page.

Microsoft Azure

Methodology

To access Azure customer data, we use a randomly generated client certificate that is unique to your tenant (your Ternary customer space.) This client certificate is signed by a well-known Ternary root authority. That root authority is self signed, but it is available for perusal on our official knowledge base, and can be distributed to you by our customer success team. The customer then grants access to the client certificate to Azure resources.

The private key of the certificate is not available in plain text but is instead stored within GCP secret manager and only retrieved when needed by our background task runner. However, the public key is visible to the customer via web browser, so that they can grant access to it.

For more information on this approach, and information on validating the public part of a client certificate that may be associated to your Tenant, please read our Azure onboarding instructions.

Permissions

Today, Ternary is not accessing more than your Azure bill. You provision read access to the storage container containing your bill information to the user identified by the client certificate, and we read from this storage container.