Observability

To ensure the deployment are running properly, Cloud-IAM has set a lot of monitoring probes and log collection at various levels. Some of those are re exposed through our Cloud-IAM API.

All the data are kept for 30 days on Cloud-IAM infrastructure.

Logs

What would be a managed service without real-time logs? Cloud-IAM dashboard displays real-time logs for every dedicated deployments (starting from Roaring Rabbit plan).

Logs access are the best way to understand what is going on with your deployment and how your custom extensions are doing.

Keycloak

Load balancer

Logs ingestion

Keycloak

Keycloak cluster logs can be integrated into customer's own logging infrastructure when polling Cloud-IAM REST API /deployments/{deploymentId}/logs endpoint with the source field to keycloak.

$ curl -H "Authorization: Bearer $TOKEN" 'https://api.cloud-iam.com/deployments/00000000-0000-0000-0000-000000000000/logs?source=keycloak&since=2023-02-26T23:17:06Z' | jq .
[
  "2023-03-07T10:00:05.85104771Z 2023-03-07 10:00:05,850 WARN  [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-727) REQUIRED and ALTERNATIVE elements at same level! Those alternative executions will be ignored: [auth-cookie, auth-cookie]\n", 
  ...
]

$ curl -H "Authorization: Bearer $TOKEN" 'https://api.cloud-iam.com/deployments/00000000-0000-0000-0000-000000000000/logs?source=keycloak&since=2023-02-26T23:17:06Z' | jq .
[
  "2023-03-07T10:00:05.85104771Z 2023-03-07 10:00:05,850 WARN  [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-727) REQUIRED and ALTERNATIVE elements at same level! Those alternative executions will be ignored: [auth-cookie, auth-cookie]\n", 
  ...
]

Load balancer

Load balancer cluster logs can be integrated into customer's own logging infrastructure when polling Cloud-IAM REST API /deployments/{deploymentId}/logs endpoint with the source field to load-balancer.

$ curl -H "Authorization: Bearer $TOKEN" 'https://api.cloud-iam.com/deployments/00000000-0000-0000-0000-000000000000/logs?source=load-balancer&since=2023-02-26T23:17:06Z' | jq .
[
  "2023-03-07T10:00:05.85104771Z 1.2.3.4 - - [07/Mar/2023:10:00:05 +0000] \"GET / HTTP/1.0\" 403 0.000 548 \"Mozilla/5.0 (Linux; Android 9; Nokia 7.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36\" \"-\" \"-\" \"-\" \"-\" \"h:-\"",
  ...
]

$ curl -H "Authorization: Bearer $TOKEN" 'https://api.cloud-iam.com/deployments/00000000-0000-0000-0000-000000000000/logs?source=load-balancer&since=2023-02-26T23:17:06Z' | jq .
[
  "2023-03-07T10:00:05.85104771Z 1.2.3.4 - - [07/Mar/2023:10:00:05 +0000] \"GET / HTTP/1.0\" 403 0.000 548 \"Mozilla/5.0 (Linux; Android 9; Nokia 7.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36\" \"-\" \"-\" \"-\" \"-\" \"h:-\"",
  ...
]

Metrics

A custom extension that helps us to monitor the deployment is automatically added to your deployment. This can not be removed.

It collects various metrics about the cluster health and some anonymous metrics about the usage of the cluster.

Cloud-IAM REST API /deployments/{deployment_id}/metrics endpoint yield functional metrics about your Keycloak in OpenMetrics format textual representation that came from Prometheus textual representation.

A response from the endpoint will return lines (called a MetricPoint) like this one:

keycloak_login_attempts{application="keycloak",client_id="account-console",realm="master"} 2 1645446218620

keycloak_login_attempts{application="keycloak",client_id="account-console",realm="master"} 2 1645446218620

For each MetricPoint:

A MetricPoint in a Metric with the type Counter MUST have one value called Total (2 in the line example). A Total is a non-NaN and MUST be monotonically non-decreasing over time, starting from 0.
A MetricPoint in a Metric with the type Counter SHOULD have a Timestamp (1645446218620 from the line example) value called Created. This can help ingestors discern between new metrics and long-running ones it did not see before.
A MetricPoint in a Metric Counter's Total MAY reset to 0. If present, the corresponding Created time MUST also be set to the timestamp of the reset.

Prometheus configuration

Here is an example of Prometheus configuration to scrape metrics from Cloud-IAM deployment. You might need to request for a service account to the Cloud-IAM support team.

scrape_configs:
- job_name: cloud-iam
  scrape_interval: 30s
  scheme: https
  metrics_path: /deployments/{deployment-id}/metrics
  static_configs:
  - targets:
    - api.cloud-iam.com
    labels:
      host: '{my-deployment}'
  oauth2:
    client_id: '{service-account-client-id}'
    client_secret: '{service-account-client-secret}'
    token_url: 'https://iam.cloud-iam.com/auth/realms/cloud-iam/protocol/openid-connect/token'
    endpoint_params:
      grant_type: 'client_credentials'

scrape_configs:
- job_name: cloud-iam
  scrape_interval: 30s
  scheme: https
  metrics_path: /deployments/{deployment-id}/metrics
  static_configs:
  - targets:
    - api.cloud-iam.com
    labels:
      host: '{my-deployment}'
  oauth2:
    client_id: '{service-account-client-id}'
    client_secret: '{service-account-client-secret}'
    token_url: 'https://iam.cloud-iam.com/auth/realms/cloud-iam/protocol/openid-connect/token'
    endpoint_params:
      grant_type: 'client_credentials'

Metrics for Keycloak versions from 23.0.0

Since 23.0.0, the http metrics are not enabled by default. In order to gather this information, the http metrics collection must be enabled.

In the configuration panel of the deployment in the Cloud-IAM Console, set:

METRICS_HTTP_MONITOR_ENABLED to true
METRICS_HTTP_MONITOR_RESOURCE_FILTER to select which paths will be collected (ex: openid-connect to gather statistics about Open ID endpoints). The value support multiple comma separated value to collect multiple paths. The filter must not be empty.

If instead of percentiles you prefer to get elapsed time buckets, please set METRICS_HTTP_MONITOR_USE_PERCENTILES to false.

Metrics for Keycloak versions from 21.0.0:

keycloak_user_event_totalcounter: Number of user events per type, realm

keycloak_admin_event_totalcounter: Number of admin events per type, realm

keycloak_logins_totalcounter: Number of login success per realm

keycloak_failed_login_attempts_totalcounter: Number of failed login attempts per realm

keycloak_registrations_totalcounter: Number of user registration success per realm

keycloak_registrations_errors_totalcounter: Number of failed user registration per realm

keycloak_client_logins_totalcounter: Number of client login success per realm

keycloak_failed_client_login_attempts_totalcounter: Number of failed client login attempts per realm

keycloak_refresh_tokens_totalcounter: Number of refresh token usage success per realm

keycloak_refresh_tokens_errors_totalcounter: Number of failed refresh token per realm

keycloak_code_to_tokens_totalcounter: Number of code to token success per realm

keycloak_code_to_tokens_errors_totalcounter: Number of failed code to token per realm

keycloak_response_totalcounter: Total number of success responses per code, method, resource

keycloak_response_errors_totalcounter: Total number of failed responses per code, method, resource

keycloak_monthly_active_user_count_totalcounter: Monthly active user per realm

keycloak_request_duration_secondsgauge: Requests duration

keycloak_request_duration_seconds_countcounter: Request count

keycloak_request_duration_seconds_maxcounter: Maximum request duration

keycloak_request_duration_seconds_sumcounter: Total request duration

Metrics for Keycloak versions prior to 21.0.0:

keycloak_client_loginscounter Total successful client logins

keycloak_code_to_tokenscounter Total number of successful code to token

keycloak_login_attemptscounter Total number of login attempts

keycloak_loginscounter Total successful logins

keycloak_refresh_tokenscounter Total number of successful token refreshes

keycloak_refresh_tokens_errorscounter Total number of failed token refreshes

keycloak_request_duration_buckethistogram Request duration

keycloak_request_duration_counthistogram Request duration

keycloak_request_duration_sumhistogram Request duration

keycloak_response_errorscounter Total number of error responses

keycloak_user_event_LOGINcounter Generic Keycloak User event

keycloak_user_event_LOGIN_ERRORcounter Generic Keycloak User event

keycloak_user_event_REGISTERcounter Generic Keycloak User event

keycloak_user_event_REGISTER_ERRORcounter Generic KeyCloak User event

keycloak_user_event_LOGOUTcounter Generic KeyCloak User event

keycloak_user_event_LOGOUT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CODE_TO_TOKENcounter Generic KeyCloak User event

keycloak_user_event_CODE_TO_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_LOGINcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REFRESH_TOKENcounter Generic KeyCloak User event

keycloak_user_event_REFRESH_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_VALIDATE_ACCESS_TOKENcounter Generic KeyCloak User event

keycloak_user_event_VALIDATE_ACCESS_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_INTROSPECT_TOKENcounter Generic KeyCloak User event

keycloak_user_event_INTROSPECT_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_FEDERATED_IDENTITY_LINKcounter Generic KeyCloak User event

keycloak_user_event_FEDERATED_IDENTITY_LINK_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_FEDERATED_IDENTITYcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_FEDERATED_IDENTITY_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_EMAILcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_EMAIL_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PROFILEcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PROFILE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PASSWORDcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_PASSWORD_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_TOTPcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_TOTP_ERRORcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_EMAILcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_EMAIL_ERRORcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_PROFILEcounter Generic KeyCloak User event

keycloak_user_event_VERIFY_PROFILE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_TOTPcounter Generic KeyCloak User event

keycloak_user_event_REMOVE_TOTP_ERRORcounter Generic KeyCloak User event

keycloak_user_event_GRANT_CONSENTcounter Generic KeyCloak User event

keycloak_user_event_GRANT_CONSENT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_CONSENTcounter Generic KeyCloak User event

keycloak_user_event_UPDATE_CONSENT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REVOKE_GRANTcounter Generic KeyCloak User event

keycloak_user_event_REVOKE_GRANT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_SEND_VERIFY_EMAILcounter Generic KeyCloak User event

keycloak_user_event_SEND_VERIFY_EMAIL_ERRORcounter Generic KeyCloak User event

keycloak_user_event_SEND_RESET_PASSWORDcounter Generic KeyCloak User event

keycloak_user_event_SEND_RESET_PASSWORD_ERRORcounter Generic KeyCloak User event

keycloak_user_event_SEND_IDENTITY_PROVIDER_LINKcounter Generic KeyCloak User event

keycloak_user_event_SEND_IDENTITY_PROVIDER_LINK_ERRORcounter Generic KeyCloak User event

keycloak_user_event_RESET_PASSWORDcounter Generic KeyCloak User event

keycloak_user_event_RESET_PASSWORD_ERRORcounter Generic KeyCloak User event

keycloak_user_event_RESTART_AUTHENTICATIONcounter Generic KeyCloak User event

keycloak_user_event_RESTART_AUTHENTICATION_ERRORcounter Generic KeyCloak User event

keycloak_user_event_INVALID_SIGNATUREcounter Generic KeyCloak User event

keycloak_user_event_INVALID_SIGNATURE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_REGISTER_NODEcounter Generic KeyCloak User event

keycloak_user_event_REGISTER_NODE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_UNREGISTER_NODEcounter Generic KeyCloak User event

keycloak_user_event_UNREGISTER_NODE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_USER_INFO_REQUESTcounter Generic KeyCloak User event

keycloak_user_event_USER_INFO_REQUEST_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LINK_ACCOUNTcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LINK_ACCOUNT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LOGINcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_FIRST_LOGINcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_FIRST_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_POST_LOGINcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_POST_LOGIN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RESPONSEcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RESPONSE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RETRIEVE_TOKENcounter Generic KeyCloak User event

keycloak_user_event_IDENTITY_PROVIDER_RETRIEVE_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_IMPERSONATEcounter Generic KeyCloak User event

keycloak_user_event_IMPERSONATE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CUSTOM_REQUIRED_ACTIONcounter Generic KeyCloak User event

keycloak_user_event_CUSTOM_REQUIRED_ACTION_ERRORcounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTIONScounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTIONS_ERRORcounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTION_TOKENcounter Generic KeyCloak User event

keycloak_user_event_EXECUTE_ACTION_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INFOcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INFO_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_REGISTERcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_REGISTER_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_UPDATEcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_UPDATE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_DELETEcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_DELETE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INITIATED_ACCOUNT_LINKINGcounter Generic KeyCloak User event

keycloak_user_event_CLIENT_INITIATED_ACCOUNT_LINKING_ERRORcounter Generic KeyCloak User event

keycloak_user_event_TOKEN_EXCHANGEcounter Generic KeyCloak User event

keycloak_user_event_TOKEN_EXCHANGE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_AUTHcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_AUTH_ERRORcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_VERIFY_USER_CODEcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_VERIFY_USER_CODE_ERRORcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_CODE_TO_TOKENcounter Generic KeyCloak User event

keycloak_user_event_OAUTH2_DEVICE_CODE_TO_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_AUTHREQID_TO_TOKENcounter Generic KeyCloak User event

keycloak_user_event_AUTHREQID_TO_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_PERMISSION_TOKENcounter Generic KeyCloak User event

keycloak_user_event_PERMISSION_TOKEN_ERRORcounter Generic KeyCloak User event

keycloak_user_event_DELETE_ACCOUNTcounter Generic KeyCloak User event

keycloak_user_event_DELETE_ACCOUNT_ERRORcounter Generic KeyCloak User event

keycloak_user_event_PUSHED_AUTHORIZATION_REQUESTcounter Generic KeyCloak User event

keycloak_user_event_PUSHED_AUTHORIZATION_REQUEST_ERRORcounter Generic KeyCloak User event

keycloak_admin_event_ACTIONcounter Generic KeyCloak Admin event

keycloak_admin_event_CREATEcounter Generic KeyCloak Admin event

keycloak_admin_event_UPDATEcounter Generic KeyCloak Admin event

keycloak_admin_event_DELETEcounter Generic KeyCloak Admin event

Observability ​

Logs ​

Keycloak ​

Load balancer ​

Logs ingestion ​

Keycloak ​

Load balancer ​

Metrics ​

Prometheus configuration ​

Metrics for Keycloak versions from 23.0.0 ​

Metrics for Keycloak versions from 21.0.0: ​

Metrics for Keycloak versions prior to 21.0.0: ​

Observability

Logs

Keycloak

Load balancer

Logs ingestion

Keycloak

Load balancer

Metrics

Prometheus configuration

Metrics for Keycloak versions from 23.0.0

Metrics for Keycloak versions from 21.0.0:

Metrics for Keycloak versions prior to 21.0.0: