Troubleshooting

If your Grape instance does seems to not have started properly after an upgrade or reboot please follow this procedure.

Note: In this guide we assume that you have already checked for common problems like insufficient diskspace or memory as well as hardware and software requirements.

Run “grape status”

The command “grape status” performs some basic checks on the stack, such as container and service status.

A successful run looks like this:

grape@grapetest:~$ grape status
                                             __
   ____ __________ _____  ___     ________  / /___  ______
  / __ `/ ___/ __ `/ __ \/ _ \   / ___/ _ \/ __/ / / / __ \
 / /_/ / /  / /_/ / /_/ /  __/  (__  )  __/ /_/ /_/ / /_/ /
 \__, /_/   \__,_/ .___/\___/  /____/\___/\__/\__,_/ .___/
/____/          /_/                               /_/

Grape SETUP Version: 3.5.5


Options for selected command 'status':
TODO

General options:
 * verbose              False
 * debug                False

Running Grape status check
 * Proxy:
    Container: running
 * Sentry:
    Container: running
 * Grape:
    Image:     onpremises-3.5.5
    Container: running
    Processes:
      chatgrape-celery-beat: running
      grape-check: stopped
      chatgrape-ws: running
      chatgrape-rr: running
      chatgrape-celery-worker-notifications: running
      chatgrape-celery-worker: running

Command 'status' returned no errors.

If a container shows “migrating” the necessary database migrations are still being run, in this case please be patient and try again after some minutes.

Look at the Service status

If the service is not running it is necessary to look at the status of the corresponding service.

Please run docker-compose ps in the configuration directory for:

  • Proxy: /data/grape/config/proxy
  • Sentry: /data/grape/config/sentry
  • Grape: /data/grape/config/application

If the status shows that the service has failed please try restarting it with docker-compose restart and wait until it either succeeds or fails again.

Note: Certain combinations of docker and linux distributions sometimes lead to errors like “Cannot create/destroy containers” or a general hangup while trying to restart containers. Should this happen you will have to fully restart the docker daemon (e.g. systemctl restart docker.service) and/or reboot the VM/Server.

Check the output of the docker containers

This only affects Sentry and Grape.

Grape

The following containers should be checked for Grape:

  • application_grape-runtime_1
  • application_postgresql_1
  • application_elasticsearch_1
  • application_redis_1
  • application_pgbouncer_1

The command to run is docker logs CONTAINERNAME | tail -n 30. This gives you the last 30 lines of output only, for they should already contain the information to track down the problem.

grape-runtime

If the grape-runtime container failed with

ERROR: The finalization script failed!
ERROR: Please try redeploying grape, if the problem persists please contact the Grape support team!
ERROR: Exiting

The most common cause is a failed migration due to the database containers failing/starting up too slow.

Restart the runtime container with docker restart application_grape-runtime_1.

If this didn’t help check the PostgreSQL, Redis and Elasticsearch containers next.

If they are not running try restarting them with docker restart CONTAINERNAME.

Once the databases are up and running please restart the grape-runtime container next by running docker restart application_grape-runtime_1.

postgresql

If PostgreSQL failed with the error FATAL: the database system is starting up the reason is most likely an unclean shutdown of the database caused by a container or VM/Server crash.

To fix this run the fix_postgresql_grape.sh helper script located in /data/grape/data/setup/current/extra.