Installing a site of several servers

A single server is the ordinary Xema deployment and needs none of this — see the README. This is for a site: two or more servers on one LAN acting as one contact centre, sharing a database, a broker and a set of agents.

Read the two tiers first, because confusing them is the main way to get this wrong:

• A site is servers in one place working as one contact centre. They share a database and a broker. Everything below builds one.
• A cluster is one or more sites, which may be a thousand kilometres apart and each of which keeps working alone. A site of several servers is still one cluster.

Throughout: declaring is separate from doing. A command that records a decision does not act on a machine; xema node apply is the one thing that starts and stops services.

What a second server does and does not give you

A single server holds every role — the database, the broker, Redis, Asterisk and every Xema service — and has no redundancy. That is the ordinary deployment and it is not a compromise.

Adding servers gives you:

• Capacity. More concurrent calls and more agents than one machine carries.
• One agent pool. A call arriving at any server can be answered by an agent signed in at any other, so agents are not divided up by which machine they happened to connect to.
• Somewhere to put the load. Queues, dialers and recording spread across machines.

Adding servers does not, by itself, give you redundancy:

A site still has one database, one broker and one Redis. If they are all on the server that fails, the site stops — with three servers as surely as with one. What you gain is that a server which holds none of them can fail without taking the site with it.

Real redundancy of those three is a separate exercise — Galera or InnoDB Cluster for MariaDB, Redis Sentinel, quorum queues for RabbitMQ — and it belongs within a site, never across sites, because every one of them is designed to stop when it loses quorum. None of it is set up by anything below. Plan for it deliberately, or accept that the server holding the shared parts is the one that must not fail.

Before you start

Decide three things, because changing them later is work:

The first server you set up becomes the site. If one of your servers is already a working call centre, make that one the site — it keeps everything it has. The others join it empty.

1. The first server

Install the software

curl -fsSL https://www.xema.in/install-xema.sh | sudo bash

This installs binaries and nothing else. Only Manager is running when it finishes.

Connect it to a database

sudo xema connect-database --server localhost --user xema --provision

--provision creates the MariaDB user with a generated password, using your own login over the unix socket — so sudo is the whole credential and no password is typed or stored by hand. It also creates the databases and applies the schema.

Make it a site

xema cluster init --site alpha

init keeps everything this server already has, records what it is currently running as its roles, and records that every trunk it has is connected to it.

Tell the site its shared passwords

The other servers need to reach this one's Redis and broker, and they need a password to authenticate to each other with:

xema site secret set redis --generate
xema site secret set broker --username xema
xema site secret set node --generate

node is not optional on a site of several. Without it a call cannot be carried from the server a caller landed on to the server their agent is on, and servers cannot hand recordings to one another.

These commands record a password; they do not set it on the thing itself. You must still configure MariaDB, Redis and RabbitMQ to accept connections from the other servers, with the passwords you just recorded. Rotation is two deliberate acts in order: change the infrastructure, then record it here, then let the servers pick it up. The other way round tells every server a password that does not work yet.

Specifically, on this server:

• Redis — bind beyond 127.0.0.1, and requirepass set to what you recorded. An unauthenticated Redis reachable on a network is worth more to an attacker than most databases.
• RabbitMQ — a real user with the recorded password. guest cannot be used: RabbitMQ restricts it to localhost, so every other server will fail to authenticate.
• MariaDB — listening beyond localhost, and the Xema user permitted from the other servers.

Check it

xema site

It names anything it cannot work out, and anything that only a cluster could know is wrong — a part listening on localhost that the other servers will never reach, a password nobody recorded.

2. Every other server

Install the software

Same command as above.

Join the site

On the first server, mint a pass:

xema cluster token

Then on the new one:

xema cluster join --token <pass>

Two things to know:

• A server joins empty. Joining points it at the site's database, so anything in its own stops being read — not merged, not moved. Two working call centres are never made into one site.
• It arrives with no roles and runs nothing. Whatever it happened to be running before is not the site's business. This is deliberate, and it is the one surprise in the flow.

It needs no connect-database: the token carries the site's database, and joining writes it to /etc/xema/common.yaml.

Say what it is for

xema role add bff
xema role add queue
xema node apply

Roles are: bff queue dialer simplecdr metrics astermq fastagi tracer sipper ava. Manager is not a role — it always runs.

Do not give it database, broker or redis unless it really is the site's. A site with two servers declaring broker has none, because nothing can say which is meant.

Point it at the site's shared parts

xema node wire
sudo xema restart all

node wire writes /etc/xema/cluster.generated.yaml from the roster — the addresses and passwords of the site's database, broker and Redis, worked out rather than typed. Services read settings at startup, so they need restarting once for it to take effect. Manager reconverges this by itself afterwards.

3. Once the servers are up

Say where each trunk is connected

In the console, on each trunk's page, set Connected to node. Only that server configures the trunk; every other server can still place calls through it, by handing them to that one.

An unassigned trunk on a site of several is carried by nobody — calls on it do not arrive. xema site reports it, and so does the Servers page.

Check the whole thing

xema site          # servers, shared parts, what the site holds, what each server is running
xema nodes         # the same in detail, subject by subject
xema cluster status

xema site has two halves that do not stand in for one another. Configured comes from the database and is shown even with every server switched off. Running is asked of the servers themselves — a server that does not answer is shown as not answering, never filled in from a remembered copy.

Decide what a server may do to itself

xema site policy            # show it
xema site policy graceful   # the default

When a server notices it is not running what the site holds, this says how much it may disturb to put that right — silent for what nothing notices, graceful to rewrite Asterisk configuration and reload. Restarting a service is never done unasked at any level.

Upgrading a site

Re-run the installer on each server. It never switches anything off, so a server keeps running what it was.

Schema is not applied automatically — several servers sharing one database must not race each other to migrate it. Manager says at startup when the database is behind the build:

xema database migrate     # once, on any server in the site

When something is wrong

xema site and xema nodes are the first two things to run, in that order. Both work on a server whose Manager will not start, because they read the database directly.