Skip to main content

ZeroMQ - MISP Publish/Subscribe

Use MISP’s publish/subscribe interface when a trusted integration needs activity notifications with low latency. ZeroMQ emits a live stream; a consumer can react to event changes, sightings or other configured activity and fetch current state through the REST API when needed.

Note

This chapter follows MISP 3.0.0, 3.0_wip commit 99ebee729. The stream operates at instance level: normal per-user REST distribution filtering is not applied to each subscriber. Treat access to the stream as privileged access to the enabled notifications, including activity on unpublished or restricted data.

Choose notifications or polling

Use REST polling for bounded, account-scoped queries. Use notifications to react promptly, while retaining a recovery/reconciliation path. ZeroMQ PUB/SUB does not provide durable replay or consumer acknowledgements: disconnected, slow or restarting consumers can miss messages.

Activity notifications trigger processing, while authorised REST queries and reconciliation recover current state. The stream has a broader trust boundary than a user’s REST queries.

Activity notifications trigger processing, while authorised REST queries and reconciliation recover current state. The stream has a broader trust boundary than a user’s REST queries.

Do not expose a broad instance stream directly to a partner who should see only selected events. Use native synchronization, an appropriately scoped API account or a filtered integration boundary.

Configure ZeroMQ

As a site administrator, open Administration → Server Settings & Maintenance → Integrations → ZeroMQ. In MISP 3.0, integrations are grouped pages rather than the old Plugin tab. The settings retain their Plugin.ZeroMQ_* names.

The MISP 3.0 ZeroMQ settings page, showing the actual integration controls.

The MISP 3.0 ZeroMQ settings page, showing the actual integration controls.
Setting Purpose
Plugin.ZeroMQ_enable Enable MISP’s ZeroMQ publishing path. The publisher service and dependencies must also work.
Plugin.ZeroMQ_host, ZeroMQ_port Publisher bind address and port; the defaults are 127.0.0.1 and 50000. A subscriber must be able to reach that address.
Plugin.ZeroMQ_<type>_notifications_enable Select individual supported activity types. Enable only the required channels.
Plugin.ZeroMQ_include_attachments Whether attachment content is included in supported payloads. Consider volume and data exposure.
Plugin.ZeroMQ_username, ZeroMQ_password Optional ZeroMQ PLAIN authentication. PLAIN credentials do not encrypt the transport.
Plugin.ZeroMQ_redis_* Redis connection/database/namespace for the publisher queue. This is separate from the subscriber’s TCP endpoint.

Keep the publisher on a trusted local/private path. Use an authenticated, encrypted transport boundary for remote access; setting a username/password alone is not TLS. A queue in Redis does not make ZeroMQ subscribers durable.

The publisher is app/files/scripts/mispzmq/mispzmq.py, using MISP’s Python environment. Follow the installation or container’s service-management instructions. When a supervisor owns the service, use that supervisor; Plugin.ZeroMQ_supervisor_managed is a CLI-only setting. Do not start a second publisher competing for the same bind address or PID file.

Check the service

Use System health → Services in Server Settings & Maintenance to inspect service checks. Check the selected Redis connection, Python pyzmq/Redis dependencies, publisher process and bind address. The publisher error log is normally app/tmp/logs/mispzmq.error.log. With a container deployment, inspect its service logs and configuration as well.

The MISP 3.0 Services health page. Diagnostic results depend on enabled services and the deployment.

The MISP 3.0 Services health page. Diagnostic results depend on enabled services and the deployment.

A process check alone does not prove delivery to a subscriber. Test connectivity and decoding with synthetic data in an isolated setup, and monitor both publisher and consumer health.

Message framing and topics

The publisher sends one ZeroMQ frame containing the topic, one space and a JSON document:

misp_json_attribute {"Attribute":{"uuid":"...","value":"example.org"},"action":"add"}

Split at the first space, then decode the remainder as JSON. Do not assume a multipart [topic, body] message. Attribute/event notifications, audit entries and keep-alives have different payload shapes; the examples below are abbreviated illustrations, not full schemas.

Topic Contents and interpretation
misp_json A full event serialization when sent through the full-event publishing path. This is distinct from every event metadata change.
misp_json_event Event change notifications, with an action indicating the operation/path.
misp_json_attribute Attribute change payloads; inspect action and deletion state.
misp_json_object, misp_json_object_reference Object and reference changes.
misp_json_tag Tag-related notifications.
misp_json_sighting Sighting notifications.
misp_json_warninglist Warninglist changes.
misp_json_user, misp_json_organisation Account/organisation activity. These are privileged operational data, not indicator feeds.
misp_json_audit Audit-log notifications.
misp_json_workflow Notifications emitted by workflow publishing actions.
misp_json_self Publisher idle keep-alive/status messages, not event data.

The 3.0 publisher no longer lists the old conversation topic. Do not build new discussion integrations around it. Notes and other research data should be retrieved through their current APIs; do not invent a topic for every model.

Most change paths include an action, but not every message does. Full-event payloads and keep-alives are examples where a consumer must not require that field. In the pinned publisher, an idle Redis poll produces a keep-alive after about 10 seconds, not a guaranteed once-per-minute schedule; sustained traffic changes the idle behaviour.

Build a subscriber

Install pyzmq in your integration’s virtual environment. This example subscribes to attribute changes and idle status, decodes the actual wire format and handles an unexpected payload:

import json
import logging
import os
import zmq

def decode_notification(frame):
    topic, separator, body = frame.partition(b" ")
    if not separator:
        raise ValueError("Notification has no topic/body separator")
    payload = json.loads(body.decode("utf-8"))
    if not isinstance(payload, dict):
        raise ValueError("Expected a JSON object")
    return topic.decode("utf-8"), payload

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.LINGER, 0)
socket.setsockopt(zmq.RCVHWM, 1000)
socket.setsockopt(zmq.RCVTIMEO, 30000)
for topic in (b"misp_json_attribute", b"misp_json_self"):
    socket.setsockopt(zmq.SUBSCRIBE, topic)
socket.connect(os.environ.get("MISP_ZMQ_URL", "tcp://127.0.0.1:50000"))

try:
    while True:
        try:
            topic, payload = decode_notification(socket.recv())
        except zmq.Again:
            logging.warning("No notification received; check stream health")
            continue
        except (UnicodeDecodeError, ValueError):
            logging.exception("Invalid notification")
            continue
        if topic == "misp_json_self":
            continue
        if topic != "misp_json_attribute":
            continue
        attribute = payload.get("Attribute")
        if isinstance(attribute, dict) and attribute.get("uuid"):
            print(payload.get("action", "unknown"), attribute["uuid"])
            # Queue idempotent processing; do not dump the whole payload to logs.
finally:
    socket.close()
    context.term()

Subscriptions are prefix filters, not access-control rules. Check the decoded topic before dispatching it. A receive timeout is an operational signal, not proof that a particular event was lost. The example does not configure PLAIN credentials; add them before connect if the deployment requires that mechanism, through a protected transport boundary.

MISP also ships tools/misp-zmq/sub.py. Use the tool from the matching checkout and inspect --help; for example, --only misp_json_attribute filters its output. It prints payloads, so use it with synthetic data or an appropriately controlled terminal/log destination.

Process changes and recover

  • Retain UUIDs and modification times. An update keeps identity; receiving another message is not evidence that the target should create another copy.
  • Handle deletes, IDS-flag changes, expiry and filter changes as well as additions. Retire target records when they no longer meet the integration’s approved selection.
  • Treat the notification as a trigger to queue work. Fetch the current authorised object when needed, and reconcile after downtime. A REST 404 can also mean restricted access.
  • Keep downstream failures and retries out of the socket receive loop where practical. A slow sink can exhaust queues/high-water marks and lose ZeroMQ messages.
  • Do not use sightings as a substitute for an event update, or a keep-alive as proof that the source’s API and the entire processing pipeline are healthy.

Kafka is a separate option

MISP also supports Kafka publishing under Integrations → Kafka, with Plugin.Kafka_enable, broker/client configuration, notification toggles and per-type topics. It requires the relevant PHP Kafka dependency and broker setup. Its topic names and payloads should be checked against the installed configuration; do not reuse the ZeroMQ frame parser on Kafka records.

Kafka can provide broker retention and consumer offsets when configured appropriately. Those properties still require a consumer’s own acknowledgement, retry, idempotency and recovery design. Kafka publishing also needs an explicit data-access boundary; it is not native MISP sharing-group synchronization. See External Connectors.