Cams Administrator's Guide

Policy Server Clustering Overview

This document provides an overview of Cams policy server clustering: a way to run multiple Cams policy servers to implement fault tolerance and increased scalability. Understanding the information in this document is important for effective deployment and administration of multiple Cams policy servers in a clustered configuration.

Other related documents on Cams policy server clustering include:

Policy Server Clustering Quick Start - instructions for deploying a Cams cluster containing two servers
Policy Server Clustering - additional details on other configuration options, debugging, etc

Clustering Benefits

The primary benefits of running multiple Cams policy servers in a cluster include:

Fault Tolerance - the ability of the distributed environment in which Cams is running to gracefully handle the failure of a Cams policy server (or the host on which it is running) and its possible recovery.
Increased Scalability - the ability to handle an increasing number of security transactions (authentication, access control checks, etc.) as the number of users, Cams web agents, and protected resources increases. You may add as many Cams policy servers as needed to a Cams cluster.

The primary mechanism by which Cams provides fault tolerance is failover. Cams web agents configured to communicate with multiple Cams policy servers in a cluster will automatically detect a failed Cams policy server and avoid further communication with it until it has recovered. Instead, security requests are sent to an accessible Cams policy server.

Scalability is provided in the Cams environment by use of load balancing, which refers to the ability of Cams web agents to delegate security requests to different Cams policy servers based on an algorithm and/or heuristic. For example, a round-robin load balancing strategy simply loops over a finite list of Cams policy servers.

Clustering Requirements

Cams clustering has various configuration, networking, and licensing requirements.

System Requirements

Cams clustering has the following system requirements:

Every Cams policy server must run on a separate computer system (host)
Each computer system should have a local file system large enough to store Cams policy server configuration files for the entire cluster and per-server log files
A shared file system (not hosted on any of the Cams policy server computer systems) is recommended for storage of master copies of Cams policy server cluster files
For ease of configuration, each Cams policy server computer system should have access to the shared file system from which Cams policy server cluster files can be copied. This must be done in a way such that failure or inaccessiblility of the shared file system will not impact operation of the Cams policy server.

Configuration Requirements

One of the most significant factors when running multiple Cams policy servers in clustered environment is maintaining consistent and compatible configuration settings. Consequently, Cams policy server clustering has the following configuration requirements:

Every Cams policy server in a cluster must run exactly the same software version. For example, you cannot run version 2.0 on one Cams policy server and version 2.01 on another Cams policy server in the same cluster.
Every Cams policy server in a cluster must deploy the same security domains. For example, if MyCamsServer1 hosts security domains system and mydomain, then MyCamsServer1 in the same cluster must host the same security domains. NOTE: This will normally occur automatically based on access to shared configuration settings.
All security domain configuration files (e.g., security-domain-registry.xml, security-domain.xml, access-control-policy.xml, and login-config.xml must contain the same contents for every Cams policy server in a given cluster.
Every Cams policy server in a cluster must authenticate a given Cams web agent using the same credentials (e.g. username and password)

Network Requirements

Cams policy server clustering has various network-related requirements and/or recommendations including:

The machines you will be using as Cams policy server hosts for the cluster must have permanently assigned, static IP addresses. You cannot use dynamically-assigned IP addresses (e.g. DHCP) in a clustering environment.
The machines you will be using as Cams policy server hosts for the cluster should not be multi-homed (support multiple static IP addresses). When a Cams policy server is started on a given host, its IP address is queried and a Cams policy server-specific registration file is loaded. If multiple IP addresses are assigned to a Cams policy server host, the wrong IP address may be used.
The Cams policy servers in a cluster should be located on the same local area network (LAN). Although Cams 2.0 does not currently require this constraint, it is anticipated that use of IP multicast will be used in subsequent Cams policy server releases.

Licensing Requirements

Production Cams licenses are issued by host IP address and maximum number of concurrent sessions. Licensing requirements include:

Every Cams policy server must be licensed by issuance of its own cams-license-keys.xml file that declares the valid Cams policy server IP addresses.
Every Cams policy server must be licensed for the same number of concurrent sessions. If necessary, you must upgrade a Cams policy server license to make it consistent with other Cams policy server licenses in a cluster. The Cam cluster as a whole is licensed for the specified maximum number of concurrent sessions (i.e. the maximum number of concurrent sessions is not addititive).

NOTE: You do not need a special Cams policy server license to run it in a cluster.

Please visit the Cams support page for more details.

Recommended Topology

Figure 1 shows a recommended Cams deployment with two Cams web agents, and a Cams cluster containing two Cams policy servers. Key aspects of this topology include:

Each Cams web agent is configured to use a named Cams cluster and a list of the Cams policy servers within the cluster
Each Cams policy server is installed on a separate computer system with its own local file system
Cams policy servers should read policy and configuration files only from a local disk, not from a shared file system.
Cams administrators should edit a master copy of Cams policy server files, including: configuration files, access control policies, login configurations, Java classes, executables, scripts, license keys, etc on a file system separate from Cams policy server host files sytems. All files should then be copied to each Cams policy server local filesystem for use by that host.

NOTE: Using a shared file system across Cams policy server hosts is not recommended because it is a single point of failure and therefore decreases fault tolerance.

If you prefer to use a shared file system to simplify Cams policy server administration, we recommend a fault-tolerant RAID system. The recommended strategy for centralizing Cams policy server configuration is to edit files in a single location, then copy them to a server-specific local file system.

Figure 1 - The recommended Cams policy server cluster deployment topology

How Cams Policy Server Clustering Works

This section provides an overview of Cams policy server cluster configuration settings, failover, and load balancing mechanisms. More detail is provided in Policy Server Clustering.

Cams Policy Server Configuration

When each Cams policy server starts on its own computer system, it is assigned a cluster name and a server name. This is done using a Cams policy server registration file, which is either CAMS_HOME/conf/cams-reg-default.conf or CAMS_HOME/conf/cams-reg-IP_ADDRESS.conf, where IP_ADDRESS is the static Internet Protocol address assigned to the Cams policy server host. For example, for a system with IP address 192.168.1.100:

CAMS_HOME/conf/cams-reg-192.168.1.100.conf

Each Cams policy server must load a different registration file to be assigned a unique server name, so use of the IP address-specific file is recommended. The Cams policy server registration file contains the following properties:

cams.server.name=MyCamsServer
cams.cluster.name=MyCamsCluster

The value of cams.server.name and cams.cluster.name may contain only alpha-numeric characters: A-Z, a-z,and 0-9. Spaces, tabs, periods, underscores, dashes, and all punctuation characters are prohibited.

Configuration files that are loaded from a local file system are:

cams.conf - Cams policy server configuration file
cams-license-keys.xml - Cams policy server license file
security-domain-registry.xml - Cams security domain registry
security-domain.xml, access-control-policy.xml, and login-config.xml - security domain configuration files

Except for server-specific properties, like cams.server.name, each Cams policy server in a cluster generally loads the same configuration settings as all other servers in the cluster. This ensures that all security services are available to Cams web agents from all Cams policy servers in a cluster.

Cams Web Agent Configuration

Every Cams web agent must specify the Cams cluster that it will use by setting the following web agent configuration property (in cams-webagent.conf):

cams.cluster.name=MyCamsCluster

This value must exactly match the cluster name configured for each Cams policy server in the cluster.

A Cams web agent may be configured to use one or more Cams policy servers within a cluster. In general, Cams web agents should be configured to use all Cams policy servers in the cluster. The URL for each Cams policy server is specified using properties of the following form within cams-webagent.conf:

cams.server.url.<Cams policy server name>=cams://<Cams policy server host>:<Cams policy server port>

For example, the following properties set two Cams policy servers for use by a Cams web agent:

cams.server.url.MyCamsServer1=cams://host1.mydomain.com:9191
cams.server.url.MyCamsServer2=cams://host2.mydomain.com:9191

The Cams policy server name must exactly match the value configured by the Cams policy server at the associated URL. Each Cams policy server name must be unique within the cluster. All other connection parameters within cams-webagent.conf, like agent authentication credentials, object pool parameters, etc. are shared for each configured Cams policy server connection.

Cams Policy Server Failover

Cams web agents are responsible for detecting unavailable Cams policy servers in a cluster and failing over to an accessible Cams policy server. Figure 2 shows a Cams cluster in which policy host 2 has failed. Cams web agents automatically detect the failed Cams policy servers and failover to live Cams policy servers.

A Cams policy server is considered to have failed under any of the following circumstances:

The Cams agent is unable to create a connection to the Cams policy server because the server host is down or otherwise inaccessible via the network, the Cams policy server is not running, the policy server is not listening on the requested TCP/IP port, Cams web agent authentication with the Cams policy server has failed, or the agent has been denied access to use the Cams policy server.
The Cams policy server host fails after the Cams web agent has successfully connected.

Figure 2 - Failure of a Cams policy server host

Cams Policy Server Recovery

Cams web agents are also responsible for detecting recovered or newly accessible Cams policy servers. They do so by attempting to connect with the failed policy server for new authentication requests only. Once a failed server comes back on line, it begins to receive authentication requests and the associated access checks after users successfully authenticate.

Cams Policy Server Load Balancing

Cams supports round-robin load balancing. Each Cams web agent maintains a list of available Cams policy servers and delegates requests in order. Because Cams does not support replication of user sessions across Cams policy servers, not every request can be load balanced. Once a user has authenticated with a given Cams policy server, all subsequent requests for that user (including additional authentications with other security domains) will be sent to the same policy server. This sticky server strategy ensures that access control checks will have access to the user's session.

Managing Cams Policy Server Files in Cluster

Every Cams policy server is a member of a Cams cluster, even if the cluster contains only one server. Managing multiple Cams policy servers within a cluster adds a little extra setup and discipline to ensure that all servers in a cluster are running the same configuration. Other than that, Cams clustering is largely transparent.

NOTE: The most important single consideration when managing a cluster of Cams policy servers is to ensure that all servers use the same access control policy and login configuration information. Consequently, it is very important to use consistent and reliable administration practices to ensure that every Cams policy server uses the same configuration files, Java classes, etc. The basic procedure is:

Edit Cams policy server master files on the shared file system
Copy Cams policy server files to each Cams policy server local file system
Restart each Cams policy server to load the new configuration

Each Cams policy server in a cluster largely uses the same configuration files as all other policy servers, but server-specific configuration settings can also be specified. Regardless, the most reliable strategy is to copy all files (even those that are for a different Cams policy server or another cluster) to every Cams policy server host.

More details are provided in Policy Server Clustering.

Clustering Limitations

Cams clustering has the following limitations:

Each Cams policy server in a cluster must run on a separate host machine, each with a single IP address. You cannot run multiple Cams policy servers on the same host by listening on different TCP/IP ports. Deploying all Cams policy servers within a Cams cluster on the same host does not provide fault tolerance, so use of multiple hosts is required.
Cams policy servers running in the same Cams cluster do not replicate authenticated user sessions among themselves. If the Cams policy server that authenticated a user is shutdown or the host on which a Cams policy server is running fails, that user's Cams session is lost and the user will be prompted to authenticate by another Cams policy server when necessary.
Cams policy server transaction logs (authentication, access control, etc) are written on an individual Cams policy server basis. Subsequently, security auditing and log file analysis is somewhat less centralized in a Cams cluster.

Back | Next | Contents