Metrics

What are metrics?

A custom extension that helps us to monitor the deployment is automatically added to your deployment. This extension can not be removed.

It collects various metrics about the cluster health and some anonymous metrics about the usage of the cluster.

Cloud-IAM REST API /deployments/{deployment_id}/metrics endpoint yield functional metrics about your Keycloak in OpenMetrics format textual representation that came from Prometheus textual representation.

A response from the endpoint will return lines (called a MetricPoint) like this one:

keycloak_login_attempts{application="keycloak",client_id="account-console",realm="master"} 2 1645446218620

For each MetricPoint:

• A MetricPoint in a Metric with the type Counter MUST have one value called Total (2 in the line example). A Total is a non-NaN and MUST be monotonically non-decreasing over time, starting from 0.
• A MetricPoint in a Metric with the type Counter SHOULD have a Timestamp (1645446218620 from the line example) value called Created. This can help ingestors discern between new metrics and long-running ones it did not see before.
• A MetricPoint in a Metric Counter's Total MAY reset to 0. If present, the corresponding Created time MUST also be set to the timestamp of the reset.

Prometheus configuration

Here is an example of Prometheus configuration to scrape metrics from Cloud-IAM deployment. You might need to request for a service account to the Cloud-IAM support team.

scrape_configs:
- job_name: cloud-iam
  scrape_interval: 30s
  scheme: https
  metrics_path: /deployments/{deployment-id}/metrics
  static_configs:
  - targets:
    - api.cloud-iam.com
    labels:
      host: '{my-deployment}'
  oauth2:
    client_id: '{service-account-client-id}'
    client_secret: '{service-account-client-secret}'
    token_url: 'https://iam.cloud-iam.com/auth/realms/cloud-iam/protocol/openid-connect/token'
    endpoint_params:
      grant_type: 'client_credentials'

The metrics can be viewed for example through a Grafana dashboard. Cloud-IAM maintain a public dashboard

Deployment status metrics

Alongside the Keycloak metrics above, Cloud-IAM also exposes the deployment's own status as Prometheus metrics. This plugs deployment health directly into your monitoring stack. No custom script, no push gateway, just a scrape target.

GET /deployments/{deploymentId}/monitoring/metrics returns the deployment status in OpenMetrics text format.

# HELP cloud_iam_deployment_up Whether the deployment is serving: 1 when its lifecycle status is RUNNING, 0 otherwise.
# TYPE cloud_iam_deployment_up gauge
cloud_iam_deployment_up{deployment_id="123e4567-e89b-12d3-a456-426614174000"} 1
# HELP cloud_iam_deployment_status Deployment lifecycle status: 1 on the status the deployment is currently in, 0 on the others.
# TYPE cloud_iam_deployment_status gauge
cloud_iam_deployment_status{deployment_id="123e4567-e89b-12d3-a456-426614174000",status="RUNNING"} 1
cloud_iam_deployment_status{deployment_id="123e4567-e89b-12d3-a456-426614174000",status="CREATING"} 0
cloud_iam_deployment_status{deployment_id="123e4567-e89b-12d3-a456-426614174000",status="STARTING"} 0
cloud_iam_deployment_status{deployment_id="123e4567-e89b-12d3-a456-426614174000",status="STOPPED"} 0
cloud_iam_deployment_status{deployment_id="123e4567-e89b-12d3-a456-426614174000",status="ERROR"} 0
cloud_iam_deployment_status{deployment_id="123e4567-e89b-12d3-a456-426614174000",status="UNKNOWN"} 0
cloud_iam_deployment_status{deployment_id="123e4567-e89b-12d3-a456-426614174000",status="DELETING"} 0

Two metrics come out of this endpoint.

cloud_iam_deployment_up answers one question: is the deployment serving traffic right now. It reads 1 when the lifecycle status is RUNNING, 0 for everything else. This is usually the first alert you want to set up.

cloud_iam_deployment_status gives you the full lifecycle instead of a single value. Every scrape emits every possible status, the current one at 1 and all the others at 0. So if your deployment is running, you'll see RUNNING at 1, and CREATING, STARTING, STOPPED, ERROR, UNKNOWN and DELETING all at 0.

Emitting every status on every scrape matters for alerting. A rule like cloud_iam_deployment_status{status="ERROR"} == 1 only fires if the ERROR series already exists in Prometheus. If only the current status were emitted, that series would never exist until the deployment is already broken, and the alert would never fire when you actually need it. Sending every status, zeros included, keeps the series there from the start, ready to alert on the moment it flips to 1.

Scraping deployment status

The endpoint expects the same authentication as the rest of the Cloud-IAM API, with the Monitoring role on the service account.

scrape_configs:
- job_name: cloud-iam-deployment-status
  scrape_interval: 30s
  scheme: https
  metrics_path: /deployments/{deployment-id}/monitoring/metrics
  static_configs:
  - targets:
    - api.cloud-iam.com
    labels:
      host: '{my-deployment}'
  oauth2:
    client_id: '{service-account-client-id}'
    client_secret: '{service-account-client-secret}'
    token_url: 'https://iam.cloud-iam.com/auth/realms/cloud-iam/protocol/openid-connect/token'
    endpoint_params:
      grant_type: 'client_credentials'

This endpoint is available on every plan, freemium included.

Works through every lifecycle state

Unlike the Keycloak metrics endpoint above, which has nothing to scrape while a deployment isn't running, this one responds through the whole lifecycle, including while it's starting up or being deleted. That's the point: you can watch a deployment go from CREATING to RUNNING, not just poll it once it's already up.

Keycloak Metrics

Metrics for Keycloak versions from 23.0.0

Since 23.0.0, the http metrics are not enabled by default. In order to gather this information, the http metrics collection must be enabled.

In the configuration panel of the deployment in the Cloud-IAM Console, set:

• METRICS_HTTP_MONITOR_ENABLED to true
• METRICS_HTTP_MONITOR_RESOURCE_FILTER to select which paths will be collected (ex: openid-connect to gather statistics about Open ID endpoints). The value support multiple comma separated value to collect multiple paths. The filter must not be empty.

If instead of percentiles you prefer to get elapsed time buckets, please set METRICS_HTTP_MONITOR_USE_PERCENTILES to false.

Metrics for Keycloak versions from 21.0.0:

keycloak_informationcounter: Deployment information

keycloak_information_totalcounter: Deployment information

keycloak_user_event_totalcounter: Number of user events per type, realm

keycloak_admin_event_totalcounter: Number of admin events per type, realm

keycloak_logins_totalcounter: Number of login success per realm

keycloak_failed_login_attempts_totalcounter: Number of failed login attempts per realm

keycloak_registrations_totalcounter: Number of user registration success per realm

keycloak_registrations_errors_totalcounter: Number of failed user registration per realm

keycloak_client_logins_totalcounter: Number of client login success per realm

keycloak_failed_client_login_attempts_totalcounter: Number of failed client login attempts per realm

keycloak_refresh_tokens_totalcounter: Number of refresh token usage success per realm

keycloak_refresh_tokens_errors_totalcounter: Number of failed refresh token per realm

keycloak_code_to_tokens_totalcounter: Number of code to token success per realm

keycloak_code_to_tokens_errors_totalcounter: Number of failed code to token per realm

keycloak_response_totalcounter: Total number of success responses per code, method, resource

keycloak_response_errors_totalcounter: Total number of failed responses per code, method, resource

keycloak_monthly_active_user_count_totalcounter: Monthly active user per realm

keycloak_request_duration_secondsgauge: Requests duration

keycloak_request_duration_seconds_bucketgauge: Requests duration

keycloak_request_duration_seconds_countcounter: Request count

keycloak_request_duration_seconds_maxcounter: Maximum request duration

keycloak_request_duration_seconds_sumcounter: Total request duration

Metrics for Keycloak versions prior to 21.0.0:

keycloak_client_loginscounter Total successful client logins

keycloak_code_to_tokenscounter Total number of successful code to token

keycloak_login_attemptscounter Total number of login attempts

keycloak_loginscounter Total successful logins

keycloak_refresh_tokenscounter Total number of successful token refreshes

keycloak_refresh_tokens_errorscounter Total number of failed token refreshes

keycloak_request_duration_buckethistogram Request duration

keycloak_request_duration_counthistogram Request duration

keycloak_request_duration_sumhistogram Request duration

keycloak_response_errorscounter Total number of error responses

keycloak_user_event_LOGINcounter Generic Keycloak User event

keycloak_user_event_LOGIN_ERRORcounter Generic Keycloak User event

keycloak_user_event_REGISTERcounter Generic Keycloak User event

keycloak_user_event_REGISTER_ERRORcounter Generic KeyCloak User event

keycloak_user_event_LOGOUTcounter Generic KeyCloak User event

keycloak_user_event_LOGOUT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CODE_TO_TOKENcounter Generic KeyCloak User event

keycloak_user_event_CODE_TO_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_LOGINcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REFRESH_TOKENcounter Generic KeyCloak User event

keycloak_user_event_REFRESH_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_VALIDATE_ACCESS_TOKENcounter Generic KeyCloak User event

keycloak_user_event_VALIDATE_ACCESS_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_INTROSPECT_TOKENcounter Generic KeyCloak User event

keycloak_user_event_INTROSPECT_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_FEDERATED_IDENTITY_LINKcounter Generic KeyCloak User event

keycloak_user_event_FEDERATED_IDENTITY_LINK_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_FEDERATED_IDENTITYcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_FEDERATED_IDENTITY_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_EMAILcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_EMAIL_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PROFILEcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PROFILE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PASSWORDcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PASSWORD_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_TOTPcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_TOTP_ERRORcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_EMAILcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_EMAIL_ERRORcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_PROFILEcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_PROFILE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_TOTPcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_TOTP_ERRORcounter Generic KeyCloak User event

keycloak_user_event_GRANT_CONSENTcounter Generic KeyCloak User event

keycloak_user_event_GRANT_CONSENT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_CONSENTcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_CONSENT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REVOKE_GRANTcounter Generic KeyCloak User event

keycloak_user_event_REVOKE_GRANT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_SEND_VERIFY_EMAILcounter Generic KeyCloak User event

keycloak_user_event_SEND_VERIFY_EMAIL_ERRORcounter Generic KeyCloak User event

keycloak_user_event_SEND_RESET_PASSWORDcounter Generic KeyCloak User event

keycloak_user_event_SEND_RESET_PASSWORD_ERRORcounter Generic KeyCloak User event

keycloak_user_event_SEND_IDENTITY_PROVIDER_LINKcounter Generic KeyCloak User event

keycloak_user_event_SEND_IDENTITY_PROVIDER_LINK_ERRORcounter Generic KeyCloak User event

keycloak_user_event_RESET_PASSWORDcounter Generic KeyCloak User event

keycloak_user_event_RESET_PASSWORD_ERRORcounter Generic KeyCloak User event

keycloak_user_event_RESTART_AUTHENTICATIONcounter Generic KeyCloak User event

keycloak_user_event_RESTART_AUTHENTICATION_ERRORcounter Generic KeyCloak User event

keycloak_user_event_INVALID_SIGNATUREcounter Generic KeyCloak User event

keycloak_user_event_INVALID_SIGNATURE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REGISTER_NODEcounter Generic KeyCloak User event

keycloak_user_event_REGISTER_NODE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UNREGISTER_NODEcounter Generic KeyCloak User event

keycloak_user_event_UNREGISTER_NODE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_USER_INFO_REQUESTcounter Generic KeyCloak User event

keycloak_user_event_USER_INFO_REQUEST_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LINK_ACCOUNTcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LINK_ACCOUNT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LOGINcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_FIRST_LOGINcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_FIRST_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_POST_LOGINcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_POST_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RESPONSEcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RESPONSE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RETRIEVE_TOKENcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RETRIEVE_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IMPERSONATEcounter Generic KeyCloak User event

keycloak_user_event_IMPERSONATE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CUSTOM_REQUIRED_ACTIONcounter Generic KeyCloak User event

keycloak_user_event_CUSTOM_REQUIRED_ACTION_ERRORcounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTIONScounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTIONS_ERRORcounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTION_TOKENcounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTION_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INFOcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INFO_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_REGISTERcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_REGISTER_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_UPDATEcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_UPDATE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_DELETEcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_DELETE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INITIATED_ACCOUNT_LINKINGcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INITIATED_ACCOUNT_LINKING_ERRORcounter Generic KeyCloak User event

keycloak_user_event_TOKEN_EXCHANGEcounter Generic KeyCloak User event

keycloak_user_event_TOKEN_EXCHANGE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_AUTHcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_AUTH_ERRORcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_VERIFY_USER_CODEcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_VERIFY_USER_CODE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_CODE_TO_TOKENcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_CODE_TO_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_AUTHREQID_TO_TOKENcounter Generic KeyCloak User event

keycloak_user_event_AUTHREQID_TO_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_PERMISSION_TOKENcounter Generic KeyCloak User event

keycloak_user_event_PERMISSION_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_DELETE_ACCOUNTcounter Generic KeyCloak User event

keycloak_user_event_DELETE_ACCOUNT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_PUSHED_AUTHORIZATION_REQUESTcounter Generic KeyCloak User event

keycloak_user_event_PUSHED_AUTHORIZATION_REQUEST_ERRORcounter Generic KeyCloak User event

keycloak_admin_event_ACTIONcounter Generic KeyCloak Admin event

keycloak_admin_event_CREATEcounter Generic KeyCloak Admin event

keycloak_admin_event_UPDATEcounter Generic KeyCloak Admin event

keycloak_admin_event_DELETEcounter Generic KeyCloak Admin event

Database metrics

These metrics describe the PostgreSQL database backing your deployment, not Keycloak itself. They are available on Keycloak 21.0.0 and later.

Unlike the Keycloak metrics above, they are gauges, not counters: each value is a snapshot of what the database was doing at the moment Cloud-IAM read it, once a minute. A query that starts and finishes between two readings is never seen, and applying rate() or increase() to them is meaningless.

They all carry host="database" rather than a Keycloak node name, since they come from the database rather than from a cluster member.

pg_active_queries_countgauge: Number of non-idle client queries in flight

pg_active_queries_max_age_secondsgauge: Age of the oldest non-idle client query, in seconds

pg_active_queries_bucket_countgauge: Number of non-idle client queries running for at least ge seconds

Labels

• datname — the database the session is connected to.
• usename — the database role the session authenticated as.
• application_name — the application name the client declared, when it declares one.
• ge — on pg_active_queries_bucket_count only: the age threshold in seconds. One series per threshold: 1, 10, 60, 300, 900, 3600.

ge is cumulative. ge="60" counts every query running for at least 60 seconds, so it already includes the queries counted by ge="300" and ge="3600". Summing or stacking these series counts the same queries several times.

A MetricPoint looks like this:

pg_active_queries_bucket_count{application_name="psql",deployment="my-deployment",datname="cloudiam",ge="60",host="database",usename="cloud_iam"} 1 1789047401333

One query has been running for at least 60 seconds. Reading the same point across thresholds tells you whether a single query is stuck or whether queries are piling up.

What to watch

pg_active_queries_max_age_seconds is the one to alert on. A query that runs for a long time holds its locks for the whole duration and prevents VACUUM from reclaiming dead rows behind it, so the longer it runs the more the table bloats. Cloud-IAM monitors this on your behalf, but exposing it lets you correlate it with your own workload.