Skip to content

IPC interface  #152

Open
Open
IPC interface #152
Labels
rfcRequest For Comments (ongoing discussion / research needed).

Description

@foxcpp
foxcpp
opened on Oct 20, 2019

Real use-cases:

  • IMAP updates generated by maddyctl should be pushed to connected clients.

Potential use-cases:

  • Introspection capabilities (such as listing enqueued messages, connected users, etc)
  • It is easier to let server synchronize certain operations (take "remove message from queue" as an example) instead of breaking invariants and hoping server code will recover gracefully.

Non-use-cases:

  • Moving to microservice-oriented architecture

Considerations:

  • We don't intend to make IPC interface stable, thus it should be possible to detect version incompatibility between client and server to prevent problems (user accidentally having different maddyctl and maddy versions). Alternative solution is to merge maddy and maddyctl executable (having maddy server to run server and maddy ctl for various management).

  • It should be possible to pass arbitrary structures (e.g. IMAP updates) and not just simple arguments.

  • It should be possible to detect config mismatch between running server and filesystem, maddyctl should prefer requesting configuration from server over reading it on its own. This is important to prevent actions affecting wrong modules or not working at all.

  • Access to IPC socket equals read-write access to state directory, socket should be inaccessible for untrusted programs and hence security is not considered.

Activity

  1. foxcpp
    added
    rfcRequest For Comments (ongoing discussion / research needed).
    on Oct 20, 2019

  2. foxcpp commented on Oct 22, 2019

    @foxcpp
    foxcpp
    on Oct 22, 2019
    OwnerAuthor

    Documenting some ideas I have.

    IPC endpoint

    IPC endpoint is a regular module, just like others. It binds on Unix socket, usually /run/maddy/ipc.sock. If somebody wants to run maddy on Windows (apparently, some people do) - there is a possibility to make it use TCP.
    Configuration:

    ipc unix://${MADDYRUNTIME}/ipc.sock

    Wire

    Wire protocol is very simple. It is request-response with request being a "method call", so it is RPC. Both requests and responses are represented LF-terminated strings, structured as follows:
    Request: MODULE_INSTANCE METHOD PAYLOAD
    Success response: ok PAYLOAD.
    Failure response: no ERROR_TEXT
    Where PAYLOAD is arbitrary string without LFs inside, depending on method it can be simple string (e.g. queue ID) or JSON-object with complex structure (e.g IMAP update). Payload can be empty.
    Example:

    local_mailboxes PushUpdate {"a": 1}

    Invocation

    Module instance that can handle IPC calls implements the following interface:

    type IPCTarget interface {
  IPCCall(method string, payload []byte) ([]byte, error)
}

    ipc module looks up module instance by its name, checks for IPCTarget interface and calls IPCCall with corresponding arguments.

    Versioning

    As a convention, Version method is defined by modules and returns "interface version" as an integer. This value is incremented each time interface changes. Clients are expected to check the version and require exact match before doing anything.

    Configuration access

    ipc module itself provides the following methods:
    ListModules, response is a space-separated list of registered modules.
    ListInstances, response is a space-separated list of registered module instance.
    StartTime, response is a Unix timestamp indicating server start-up time (more precisely, ipc module initialization time)
    DumpConfig, response is a JSON-encoded parsed configuration tree.

    When server is running, maddyctl (or any other IPC client) should not read config files from disk and instead should request them from the server to make sure it is using the same configuration.
    StartTime method is used to detect whether configuration server using is outdated (file modification date > start time) to print a warning for user. DumpConfig is used to request the actual configuration.

  3. foxcpp commented on Dec 7, 2019

    @foxcpp
    foxcpp
    on Dec 7, 2019
    OwnerAuthor

    Decision: Not needed now, statistics should be collected through analysis of machine-readable logs, IMAP updates passing should be implemented using DB-specific methods (for example, using NOTIFY from PostgreSQL, this will also work well with replication).

    Closing.

  4. foxcpp
    closed this as completedon Dec 7, 2019
  5. foxcpp
    removed
    rfcRequest For Comments (ongoing discussion / research needed).
    on Dec 7, 2019
  6. foxcpp
    reopened this on Nov 27, 2020

  7. foxcpp commented on Nov 27, 2020

    @foxcpp
    foxcpp
    on Nov 27, 2020
    OwnerAuthor

    It might be a good idea to add some sort of a management API, e.g. for use with some control dashboard or whatever.

  8. theduke commented on Nov 27, 2020

    @theduke
    theduke
    on Nov 27, 2020
    Contributor

    I'm happy to work on an implementation.

    The endpoints that seem useful:

    POST     /imap/accounts/BACKEND/USERNAME   # create account
DELETE /imap/accounts/BACKEND/USERNAME    # delete account
GET       /imap/accounts/BACKEND?limit=xx&offset=xx  # list accounts
GET       /imap/accounts/BACKEND/USERNAME/mailboxes?limit=xx&offset=xx # list mailboxes
POST     /imap/accounts/BACKEND/USERNAME/mailboxes/MBOX # create mailbox
DELETE /imap/accounts/BACKEND/USERNAME/mailboxes/MBOX  # delete mailbox
PUT       /imap/accounts/BACKEND/USERNAME/mailboxes/MBOX?rename-to=NEWNAME # update mailbox

  9. foxcpp commented on Nov 27, 2020

    @foxcpp
    foxcpp
    on Nov 27, 2020
    OwnerAuthor

    Wondering, how a REST-style HTTP API would look like, taking maddy's "modules framework" into account.

    Probably /MODULE_INSTANCE/whatever_module_defines e.g. /local_mailboxes/accounts. That is, pass API requests to a particular module instance based on first part of URL.

    Probably something like this as an optional interface to be implemented by any module that wishes to offer an external API:

    type APIModule interface {
  ServeAPI(w http.ResponseWriter, r *http.Request)
}

  10. theduke commented on Nov 27, 2020

    @theduke
    theduke
    on Nov 27, 2020
    Contributor

    That seems complicated from the consumers perspective and would make something like a dashboard or useful integration that is module agnostic quite cumbersome.

    It would be like maddyctl not having maddyctl imap-acct create but maddyctl local_mailboxes create.

    There could be an interface for operations that each kind of module (storage/auth/etc) is expected to support.
    The modules can add additional endpoints where appropriate, and still respond with a 501 Not Implemented status code if necessary.

  11. theduke commented on Nov 27, 2020

    @theduke
    theduke
    on Nov 27, 2020
    Contributor

    Like:

    type StorageApiModule interface {
  ListAccounts(options ListAccountsOptions) ([]StorageAccount, error)
  // ...
  Serve(w http.ResponseWriter, r *http.Request)
}

  12. foxcpp commented on Nov 29, 2020

    @foxcpp
    foxcpp
    on Nov 29, 2020
    OwnerAuthor

    It would be like maddyctl not having maddyctl imap-acct create but maddyctl local_mailboxes create.

    There is --cfg-block argument with default value of local_mailboxes. The problem is that maddy permits multiple independent storage backend "instances" to be used in the same config, same goes for queues, authentication dbs and so on.

    My current idea is to have "canonical" meaning for some of instance names, e.g. local_mailboxes referring to local IMAP storage if it is used, local_authdb being an authentication data provider used for local accounts, etc.

    While it is possible to "alias" to canonical names like maddyctl does it, I believe it is acceptable for an API to skip this abstraction layer.

  13. foxcpp
    added this to Maddy Mail Server Task Trackeron Oct 28, 2026
  14. github-project-automation
    moved this to Triage in Maddy Mail Server Task Trackeron Oct 28, 2026
  15. foxcpp
    moved this from Triage to TODO in Maddy Mail Server Task Trackeron Oct 28, 2026
  16. foxcpp
    added
    rfcRequest For Comments (ongoing discussion / research needed).
    on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    rfcRequest For Comments (ongoing discussion / research needed).

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions