Keycloak Logs

Cloud-IAM Console - Keycloak logs Feature

How Cloud-IAM ensures monitoring

To ensure your Keycloak deployment runs properly, Cloud-IAM has implemented:

• Continuous monitoring
• Log collection at different system levels

This setup gives you complete visibility into your deployment's health and activities. All logs are securely stored on Cloud-IAM infrastructure for 30 days.

What are Keycloak Logs?

Cloud-IAM has set a lot of monitoring probes and log collection at various levels. Keycloak Logs show all system events happening in real time on your Keycloak deployment and can be filtered directly from the console by log levels.

Keycloak Logs include:

• Trace — Highly detailed, low-level information. Mostly used for deep debugging.
• Debug — Useful information for development and diagnosing specific problems.
• Info — General information about normal operations of the system.
• Warn — Warnings that indicate potential problems. No immediate action is always required but should be reviewed.
• Error — Errors that affect the normal operation of the system. These require attention.

How to access Keycloak Logs

Follow these easy steps to find your Keycloak Logs:

1. Open Cloud-IAM console
2. Select the Keycloak deployment you want to audit.
3. Click on Observabiliy
4. Then select Keycloak Logs

You will now see all the keycloak logs related to your deployment.

Cloud-IAM Console - Keycloak Logs access

Keycloak Logs ingestion

API description

The API for fetching the logs needs to be polled. The API will return in the response body, an array of lines composed of a timestamp and its message:

[
  "2023-03-07T10:00:05.85104771Z 2023-03-07 10:00:05,850 WARN  [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-727) REQUIRED and ALTERNATIVE elements at same level! Those alternative executions will be ignored: [auth-cookie, auth-cookie]\n", 
  ...
]

The response come with HTTP headers that will drive the next polling url and delay.

• x-stream-polling-url: the url to call to fetch the next batch of logs. It will contain an additional since parameter to continue just after the last log line that was returned.
• x-stream-polling-delay: the delay in seconds to wait before polling the next batch of logs.
• x-stream-has-more: an optional header that indicates that additional log lines are already available and can be fetched without waiting. If set, x-stream-polling-delay value will be 0.

Delay on logs availability

For various performance reasons, the logs are buffered at several layers. It can be in Keycloak or in the load balancer memory, on disk, on our log ingestion pipeline or in the replication process across our nodes.

For this reason, we usually see a delay of a few minutes between the logs generation and its availability through the Logging API.

Keycloak logs description

Keycloak cluster logs can be integrated into customer's own logging infrastructure when polling Cloud-IAM REST API /deployments/{deploymentId}/logs endpoint with the source field to keycloak.

$ curl -H "Authorization: Bearer $TOKEN" 'https://api.cloud-iam.com/deployments/00000000-0000-0000-0000-000000000000/logs?source=keycloak&since=2023-02-26T23:17:06Z' | jq .
[
  "2023-03-07T10:00:05.85104771Z 2023-03-07 10:00:05,850 WARN  [org.keycloak.authentication.DefaultAuthenticationFlow] (executor-thread-727) REQUIRED and ALTERNATIVE elements at same level! Those alternative executions will be ignored: [auth-cookie, auth-cookie]\n", 
  ...
]

Fetch with filebeat for Elasticsearch

Filebeat is a lightweight shipper for forwarding and centralizing log data maintained by the Elasticsearch community. It can be used in combination of Logstash to collect / transform / parse and inject logs in your log infrastructure.

Here is a snippet of a filebeat input configuration to fetch the logs from the Cloud-IAM API. You will need to adjust the output section accordingly to your log infrastructure settings.

Make sure to replace the values for {service-account}, {service-account-password} and {deployment-id}. Adjust the source parameter line 8.

filebeat.inputs:
- type: cel
  enabled: true
  auth.oauth2.client.id: {service-account}
  auth.oauth2.client.secret: {service-account-password}
  auth.oauth2.token_url: https://iam.cloud-iam.com/auth/realms/cloud-iam/protocol/openid-connect/token
  interval: 60s
  resource.url: https://api.cloud-iam.com/deployments/{deployment-id}/logs?source=keycloak
  program: |
    get(state.url)
    .as(request, 
      {
        "events": request.Body.decode_json().map(e, 
            {
              "@timestamp": string(e).split_n(" ", 2)[0], 
              "message": string(e).split_n(" ", 2)[1]
            }), 
        "want_more": request.Header.exists(e, e == 'X-Stream-Has-More'),  
        "url": request.Header['X-Stream-Polling-Url'][0]
      }
    )

output.file:
  enabled: false
  path: "/tmp/"
  filename: cloud-iam-keycloak.log
  permissions: 0640
  codec.format:
    string: '%{[message]}'

output.elasticsearch:
  enabled: false
  hosts: ["http://localhost:9200"]
  api_key: "xxxxxxx:yyyyyyy"
  index: "cloud-iam-keycloak-%{+yyyy.MM.dd}"

For testing purpose, you can use this configuration file in a docker container:

$ docker run --rm -it -v $PWD/filebeat.yml:/usr/share/filebeat/filebeat.yml \
                      -v $PWD/output:/tmp \ 
                      docker.elastic.co/beats/filebeat:8.17.4