If your Grape instance seems to not have started properly after an upgrade or reboot please follow this procedure.

Note: Make sure to check common server problems like:

  • insufficient disk space

  • insufficient memory

  • network not available

  • high CPU load (top, uptime)

Troubleshooting Flowchart:


Grape is offline / HTTP 502

grape status or docker ps

The command grape status performs some basic checks on the stack, such as container and service status.

A successful run looks like this:

grape@grapetest:~$ grape status
   ____ __________ _____  ___     ________  / /___  ______
  / __ `/ ___/ __ `/ __ \/ _ \   / ___/ _ \/ __/ / / / __ \
 / /_/ / /  / /_/ / /_/ /  __/  (__  )  __/ /_/ /_/ / /_/ /
 \__, /_/   \__,_/ .___/\___/  /____/\___/\__/\__,_/ .___/
/____/          /_/                               /_/

Grape SETUP Version: 4.2.0

General options:
 * verbose              False
 * debug                False

Running Grape status check
 * Proxy:
    Service:   running
    Container: running
 * Sentry:
    Service:   running
    Container: running
 * Grape:
      migrate:              running
      rr:                   running
      ws:                   running
      beat:                 running
      worker-emails:        running
      worker-notifications: running
      worker-default:       running

Command 'status' returned no errors.

If a container shows “migrating” the necessary database migrations are still being run, in this case please be patient and try again after some minutes.

You can also run docker ps which lists all running containers and their statuses.

Look at the Service status

If the service is not running it is necessary to look at the status of the corresponding service.

Please run docker-compose ps in the configuration directory for:

  • Proxy: /data/grape/config/proxy

  • Sentry: /data/grape/config/sentry

  • Grape: /data/grape/config/application

If the status shows that the service has failed please try restarting it with docker-compose restart and wait until it either succeeds or fails again.

Note: Certain combinations of docker and linux distributions sometimes lead to errors like “Cannot create/destroy containers” or a general hangup while trying to restart containers. Should this happen you will have to fully restart the docker daemon (e.g. systemctl restart docker.service) and/or reboot the VM/Server.

Try to restart the proxy

docker restart grape-proxy

Before you dig deeper into the cause, you can try to just restart the proxy and see if this recovers the service.

Check the output of the docker containers

docker-compose logs -f

This only affects Sentry and Grape.

Check individual container logs

The following containers should be checked for Grape:

  • application_rr_1

  • application_ws_1

  • application_postgresql_1

  • application_elasticsearch_1

  • application_redis_1

  • application_pgbouncer_1

The command to run is docker logs -f CONTAINERNAME --tail 100. This shows the last 100 lines of output and follows the log.


If the migrations step in grape install failed with

ERROR: The finalization script failed!
ERROR: Please try redeploying grape, if the problem persists please contact the Grape support team!
ERROR: Exiting

The most common cause is a failed migration due to the database containers failing/starting up too slow.

You can confirm missing migrations by entering the migration container and checking manually. Use the following commands:

  • docker exec -it application_migrate_1

  • docker exec -it application_migrate_1 python showmigrations

If any migrations have not been applied, they won’t appear checked, as in this example:

 [X] 0001_initial
 [X] 0002_auto_20170627_0948
 [X] 0003_pre_populate_default_teams
 [X] 0004_pre_populate_default_teams_fixed
 [ ] 0005_ad_imports_blank  # <- not run
 [ ] 0006_team_crowd_imports  # <- not run
 [X] 0001_initial
 [X] 0001_initial

If this is the case, you need to report that problem to Grape immediately.

Restart the runtime containers with grape restart.

If this didn’t help check the PostgreSQL, Redis and Elasticsearch containers next.

If they are not running try restarting them with docker restart CONTAINERNAME.

Once the databases are up and running please restart the runtime containers next by running grape restart.


If PostgreSQL failed with the error FATAL:  the database system is starting up the reason is most likely an unclean shutdown of the database caused by a container or VM/Server crash.

To fix this run the helper script located in /data/grape/data/setup/current/extra.

Grape is online

Two-Factor Authentication keeps failing

This can be caused by the host’s local time being out of sync, please make sure the host’s clock is up to date by using e.g. ntpd.

Problems using Grape (Error 500) - check Sentry

Check if you can see an error in Sentry at the time the problem happened. It helps the Grape team if you have an error in Sentry that correlates to your problem.

More details on Sentry

Self-test page

You can open the self-test page in the browser for a quick overview:


Note: only Grape admin users can access this page

Specific Problems

Disk is full


  • Sentry and error logs contain:

    • “No space left on device”

    • “Cannot create temporary file”

    • “AuthorizationException(403, ‘cluster_block_exception’, ‘blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];’)”

  • df -h shows that there is no space left

  • Elasticsearch went into read-only mode


  1. Quickly free some space: find big files

    • cd /data/grape

    • du -sh *

    • ls -ld *

    • go into subdirectories and repeat

  2. Delete it:

    • rm <filename>

    • df -h (to check if we have some space again)

  3. Recover elasticsearch:

    • docker exec -it application_elasticsearch_1 bash

    • curl -XPUT http://localhost:9200/_settings -H 'Content-Type: application/json' -d '{"index": {"blocks": {"read_only_allow_delete": "false"}}}'

Problems after upgrading Grape

My Grape instance just show the “We are busy updating our product” page (HTTP 502)

Please check the output of the runtime container:

docker logs application_rr_1 | tail -n 30

Grape (gunicorn process) uses ~100%

First try restarting the grape-application by running

cd /data/grape/config/application && docker-compose restart

and wait until it comes back up.

If this does not help:

Inspect what the process is doing:

  1. Find the process id of gunicorn E.g. with ps (as root on the host machine)

ps aux | grep gunicorn
  1. Inspect it using strace


A bug appeared after upgrading to a new version

  1. Use your VM snapshot to recover the last working version.

  2. If you don’t have a VM snapshot:

    1. Take a VM snapshot!

    2. Look up the previous release in the changelog or your inbox.

    3. Downgrade to that version, e.g. onpremises-4.28.1

    grape install --custom-version onpremises-4.28.1

    YMMV with this approach. Always make a backup before an upgrade!

I want see what grape commands have been called in the past

The grape commands logs all it’s output to the file /data/grape/logs/grape_setup.log, you’ll find all the information about previous runs in there.