Multi-agent setup using a separate Docker host

This guide explains how to add Netpicker capacity on a different Docker host. The main Netpicker system keeps running the core stack. The second Docker host runs an extra agent, an extra kibbitzer, and its own local redis container.

This is useful when a customer wants to place workers closer to the network devices, or when the main Docker host should not carry all backup and device-connection load.

Architecture

• Main Docker host: runs the normal Netpicker stack, including api, frontend, db, redis, celery, and the default local workers.
• Second Docker host: runs an additional agent, kibbitzer, and redis.
• API connection: the second host agent and kibbitzer point to the exposed API port on the main host.
• Redis connection: the second host agent and kibbitzer use the Redis container on the second host.

Example addresses used in this guide:

Main Netpicker Docker host: 10.10.10.20
Main API endpoint:          http://10.10.10.20:8000
Main agent WebSocket:      ws://10.10.10.20:8000/api/v1/agents/ws
Second Docker host:        10.10.10.30
Second host Redis:         redis://redis:6379/4

Replace these values with the customer environment values.

Ports required between the hosts

• 8000/tcp from the second Docker host to the main Docker host for the Netpicker API.

The second Redis instance is local to the second Docker host and does not need to be exposed to the main host or to the network. Keep it internal to the Docker Compose network on the second host.

If the main Netpicker UI is exposed on port 80, that is for users and the frontend. The remote containers need the API port, normally 8000.

Step 1: expose the API on the main Docker host

The default Netpicker Compose file already exposes the API as 8000:8000. If it is not exposed in your deployment, add an override on the main Docker host.

Create a file on the main Docker host named docker-compose.remote-workers-main.yml:

services:
  api:
    ports:
      - "8000:8000"

Start or update the main stack:

docker compose \
  -f docker-compose.yml \
  -f docker-compose.remote-workers-main.yml \
  up -d

Verify from the second Docker host that the API port is reachable:

nc -vz 10.10.10.20 8000

If nc is not installed, use curl:

curl http://10.10.10.20:8000/api/v1/status

Step 2: create an extra Netpicker agent record

Every agent container identifies itself with AGENT_ID. The value must match an agent record in Netpicker.

The default installation creates one default agent. For a remote second host, create an additional agent in Netpicker first, then copy that generated agent ID into the remote Compose file.

You can create the extra agent through the Netpicker UI if your build exposes agent management. You can also use the API:

curl -X POST "http://10.10.10.20:8000/api/v1/agents/default" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"remote-agent-host-2"}'

The response contains the agent ID to use as AGENT_ID. Use a different ID for every remote agent container.

Step 3: create the remote Compose file on the second Docker host

On the second Docker host, create a directory for the remote Netpicker workers:

mkdir -p /opt/netpicker-remote-workers/textfsm
cd /opt/netpicker-remote-workers

Create docker-compose.yml on the second Docker host:

services:
  redis:
    image: redis:7-alpine3.21
    labels:
      netpicker.io: service
    volumes:
      - redis:/data
    command: "--save 60 1 --loglevel warning"
    restart: always
    healthcheck:
      test: ["CMD", "redis-cli", "--raw", "incr", "ping"]

  agent:
    hostname: remote-agent-host-2
    image: "netpicker/agent:latest"
    labels:
      netpicker.io: service
      service.netpicker.io: agent
    environment:
      AGENT_ID: "REPLACE_WITH_REMOTE_AGENT_ID"
      AGENT_HOST: "remote-agent-host-2"
      WS_ENDPOINT: "ws://10.10.10.20:8000/api/v1/agents/ws"
      SHENV_API_URL: "http://10.10.10.20:8000"
      REDIS_URL: "redis://redis:6379/4"
      BACKUP_SVC_ADDR: "kibbitzer"
      BACKUP_SVC_PORT: 9696
      CLI_PROXY_ADDR: "0.0.0.0"
      SHARED_SSH_TTL: 180
      NO_PROXY: "10.10.10.20,localhost,127.0.0.1"
    volumes:
      - secret:/run/secrets
      - dc-vol:/dc-vol
    restart: always
    depends_on:
      redis:
        condition: service_healthy
      kibbitzer:
        condition: service_started
    healthcheck:
      test: "echo LST | nc -v 127.0.0.1 8765"
      start_period: 12s
      interval: 10s

  kibbitzer:
    image: "netpicker/kibbitzer:latest"
    labels:
      netpicker.io: service
      service.netpicker.io: kibbitzer
    environment:
      LOG_LEVEL: INFO
      SHENV_API_URL: "http://10.10.10.20:8000"
      SHENV_API_TIMEOUT: 30
      REDIS_URL: "redis://redis:6379/4"
      CELERY_BROKER_URL: "redis://redis:6379/4"
      CELERY_RESULT_BACKEND: "redis://redis:6379/4"
      SHENV_PRIVILEGED_PLATFORMS: "gigamon_gigavue arista_eos"
      SHENV_SHOW_RUN_aruba_os: "show configuration"
      SHENV_SHOW_RUN_corvil: "show config"
      SHENV_SHOW_RUN_gigamon: "show running"
      SHENV_SHOW_RUN_meinberg: "generate_config_backup && cat /var/tmp/lantime_config.backup"
      SHENV_SHOW_RUN_mikrotik_routeros: "export"
      SHENV_SHOW_RUN_nokia_srl: "info"
      SHENV_SHOW_RUN_paloalto_panos: "show config running"
      SHENV_TAG_SHOW_RUN_mikrotik_routeros_v7: "export show-sensitive"
    volumes:
      - secret:/run/secrets
      - transferium:/transferium
      - kibbitzer:/celery-worker
      - ./textfsm:/textfsm
    restart: always
    depends_on:
      redis:
        condition: service_healthy
    healthcheck:
      test: echo "ping Mac\n" | nc -v 127.0.0.1 9696
      start_period: 15s
      interval: 30s

volumes:
  dc-vol:
  kibbitzer:
  redis:
  secret:
  transferium:

Replace:

• 10.10.10.20 with the main Netpicker Docker host IP or DNS name.
• REPLACE_WITH_REMOTE_AGENT_ID with the agent ID created in Netpicker.

The service names are intentional:

• redis lets the remote agent and kibbitzer use redis://redis:6379/4 on the second host Docker network.
• kibbitzer matches the agent default backup service name. The example also sets BACKUP_SVC_ADDR: "kibbitzer" explicitly.

Step 4: use HTTPS/WSS when the API is behind TLS

If the main Netpicker API is exposed through HTTPS, adjust the remote agent and kibbitzer environment variables like this:

WS_ENDPOINT: "wss://netpicker.example.com/api/v1/agents/ws"
SHENV_API_URL: "https://netpicker.example.com"

The Redis URLs stay local to the second Docker host:

REDIS_URL: "redis://redis:6379/4"
CELERY_BROKER_URL: "redis://redis:6379/4"
CELERY_RESULT_BACKEND: "redis://redis:6379/4"

Step 5: start the remote workers

Run this on the second Docker host:

docker compose pull
docker compose up -d

Check the remote containers:

docker compose ps
docker compose logs -f redis agent kibbitzer

Step 6: verify from Netpicker

From any system that can reach the main API, list the configured agents:

curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://10.10.10.20:8000/api/v1/agents/default"

List live connected agents:

curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://10.10.10.20:8000/api/v1/agents/live_agents"

The remote agent should appear as connected. Its host value should match remote-agent-host-2.

How Netpicker distributes work across agents

Netpicker tracks which agent owns a device with the device agent_id field.

• If a device already has an agent_id, readout work is sent to that live agent.
• If a device has no agent_id yet, Netpicker splits that work across the live agents.
• When an agent successfully connects to devices, Netpicker can update those devices with the agent that reached them.

This means remote agents are useful when different Docker hosts can reach different network zones. Put the remote agent on the host with the right routing, firewall, VPN, or jump-host access.

Why the second host has its own Redis

The remote agent and kibbitzer use Redis for local worker state and queue/backend data. Running Redis on the same second Docker host keeps this traffic local and avoids exposing the main Redis instance over the network.

The remote agent still connects to the main Netpicker API over the exposed API port. The remote kibbitzer also uses the main Netpicker API URL for API calls, but its Redis settings point to the local second-host Redis container.

Scaling kibbitzer on the remote host

Kibbitzer is a Celery worker. To add more kibbitzer capacity on the second host, add more kibbitzer services that use the same local Redis container:

REDIS_URL: "redis://redis:6379/4"
CELERY_BROKER_URL: "redis://redis:6379/4"
CELERY_RESULT_BACKEND: "redis://redis:6379/4"

You can also tune worker concurrency per container:

environment:
  CELERY_CONCURRENCY: 4

Increase capacity gradually and watch CPU, memory, Redis latency, API load, and device-side rate limits.

Optional: Redis password on the second host

If you protect the second Redis instance with a password, include it in the Redis URLs used by the remote containers:

REDIS_URL: "redis://:YOUR_REDIS_PASSWORD@redis:6379/4"
CELERY_BROKER_URL: "redis://:YOUR_REDIS_PASSWORD@redis:6379/4"
CELERY_RESULT_BACKEND: "redis://:YOUR_REDIS_PASSWORD@redis:6379/4"

Store real passwords in an environment file or secret manager rather than committing them to a Compose file.

Troubleshooting

Remote agent starts but does not connect

• Check that AGENT_ID exists in Netpicker.
• Check that WS_ENDPOINT points to the main API WebSocket endpoint.
• Use ws:// for plain HTTP API access and wss:// for HTTPS API access.
• Check that the second Docker host can reach the main host API port.

docker compose logs -f agent
nc -vz 10.10.10.20 8000

Remote agent cannot reach kibbitzer

• Keep the kibbitzer service name as kibbitzer, or set BACKUP_SVC_ADDR to the actual service name.
• Check that the agent and kibbitzer are in the same Compose project/network.
• Check the kibbitzer healthcheck and logs.

docker compose logs -f agent kibbitzer

Remote kibbitzer does not pick up jobs

• Check that SHENV_API_URL points to the main API exposed port.
• Check that REDIS_URL, CELERY_BROKER_URL, and CELERY_RESULT_BACKEND point to the local Redis service on the second host.
• Check the local Redis health on the second host.

docker compose exec redis redis-cli ping
docker compose logs -f kibbitzer

Redis is healthy but workers still fail

• Confirm the Redis database number is /4 for the remote agent and kibbitzer settings.
• Confirm Redis authentication, if enabled.
• Confirm no firewall or proxy is interrupting API connections from the second host to the main host.

Work only runs on the original agent

Check whether devices are already pinned to the original agent through their agent_id. Devices pinned to a specific agent will keep using that agent while it is live. Unassigned devices can be distributed across live agents during readout.

Summary

• The second Docker host does not need the full Netpicker stack.
• Run agent, kibbitzer, and a second local redis container on the second host.
• Set the remote agent WS_ENDPOINT to the main host exposed API WebSocket URL.
• Set the remote agent and kibbitzer SHENV_API_URL to the main host exposed API URL.
• Point the remote agent, kibbitzer, and Celery Redis URLs to the local second-host Redis service.
• Only the main API port needs to be reachable from the second host.