WSC 3.0 — Emails and Background Queue

  • WoltLab Suite Core 3.0 ships with a brand new email API. The old API pretty much was the same since Community Framework 1.x and was neither flexible nor fail-safe. Without doing manual extra work you were restricted to either text/plain or text/html only emails (no combination of both), support for own headers was rudimentary and if an email delivery failed everything exploded. While one could catch the exception, one would have to decide whether to drop the email or queue it manually.
    The new API solves all of these problems: You can freely mix plain text and HTML “Mime Parts”, you have proper support for message headers (especially the Message-IDs) and we even put in an event that allows you to modify emails just before they are queued for delivery. And don’t worry if an email fails to send: sending will automatically be retried several times without requiring assistance from you.


    Sending an email


    Let’s go ahead and send our first email. We start off by creating a new \wcf\system\email\SimpleEmail object and filling in the subject:

    PHP
    $message = new SimpleEmail();
    $message->setSubject('Checking out the fancy new emails!');

    The recipient is important as well. Instead of extracting the username and email address from a user, you simply pass the \wcf\data\user\User object as the recipient:

    PHP
    // set current user as a recipient
    $message->setRecipient(WCF::getUser());

    Next is the message text (body) of the email. When using the SimpleEmail class you get the overall layout of the email for free:

    PHP
    $message->setMessage('Boring non-bold text');
    $message->setHtmlMessage('<b>Fancy bold text</b>');

    Looks like our email is ready for delivery. As you might have noticed: You don’t have to care about the inner workings of an email. Simply specify what you need and WoltLab Suite Core will make a pretty email out of it. So, let’s put it into the email queue:

    PHP
    $message->send();

    send() will prepare the email for delivery and checks whether you provided all the needed data. It will automatically queue the email for delivery. Once the email is safe in the background queue, your work is done. You don’t need to care about it any longer. It will eventually get delivered to the recipient.


    If you need more fine grained control about the email and SimpleEmail does not cut it, there is \wcf\system\email\Email which is used internally by SimpleEmail. You can get the underlying Email object by calling SimpleEmail::getEmail(). It allows you to specify multiple recipients, custom templates, attachments and much more. For further details we recommend taking a look inside the classes in \wcf\system\email\*. They should be pretty self explanatory.


    One final note when working directly with Email: Check whether it makes sense to generate the Message-Id deterministically on your own and provide a proper references header. This will create a nice email thread in the recipient’s email program. Consider you are running a shop and a user orders some products. You would then create a Message-Id like com.example.shop-{$orderID}-new for the order confirmation. Once you received the money, you would send an email with the ID com.example.shop-{$orderID}-paid with References and In-Reply-To set to com.example.shop-{$orderID}-new. The emails will automatically be grouped in the user’s mail program (if the user enabled threading)! Example:

    PHP
    $message->setMessageID('com.example.shop.'.$orderID.'-new');
    // getMessageID() returns the full Message-ID including the host
    $messageID = $message->getMessageID();
    // equivalent to:
    $messageID = '<com.example.shop.'.$orderID.'-new@'.Email::getHost().'>';
    // now save the messageID somewhere


    Later:

    PHP
    $message->setMessageID('com.example.shop.'.$orderID.'-paid');
    // this should be a *direct reply* to at least one other message
    $message->addInReplyTo($savedMessageID);
    // these should be all the message IDs of the current email thread
    $message->addReferences($savedMessageID);


    The background queue


    We’ve mentioned the background queue above and said that emails will automatically be retried if delivery failed. What exactly is this background queue? The name pretty much explains it purpose: It allows you to queue jobs for asynchronous execution. There are simply things which are not necessary to generate the reply to the current request, but which will take potentially a long time. Generally everything that touches the network will inevitable fail at times you need it least. Especially emails can be pretty flaky. For these things we introduced the background queue. You put jobs in and forget about them. WoltLab Suite Core will automatically execute them when they are due. What if a job fails? It will automatically be put into the queue again for later execution – up to a certain amount of times.


    En-queuing a job can be done with enqueueIn($jobs, $time = 0). The job will then be considered for execution in $time seconds, it may be executed at any later point in time. Similarly to the cronjobs the background queue is only cleared when a user accesses the page. There also is enqueueAt($jobs, $time) to set the earliest possible execution to the given UNIX timestamp. Before you can queue a job, you'll need a job. To create one you have to extend wcf\system\background\job\AbstractBackgroundJob and implement the perform() method. If wanted you can also customize the retryAfter() method that determines how long a job will have to wait before it is rescheduled.


    As example of such a such a job is the EmailDeliveryBackgroundJob which is returned by Message::getJobs():


    Note: The wcf1_background_job table is explicitly not part of the public API, do not touch and do not rely on it.

  • Good improvement. However, still missing: configurable email rate limits; e.g. x emails per hour or y emails per day / connection.

  • Hi

    Good improvement. However, still missing: configurable email rate limits; e.g. x emails per hour or y emails per day / connection.

    Currently the background queue executes only a single job per request. Thus in most cases there is a single email per connection.


    I'm fairly sure that the automated retrying will solve the rate limiting issues, though. At least unless the mail server rejects mails with a permanent error. Recipient / Domain based rate limiting is out of scope for WoltLab Suite Core, this is the job of the MTA as it is the only one with the complete view of the situation.

  • Well, as long as the WSC allows for user bulk proccessing (send email to users) or even 'Email users', which sends emails to all users, the described queueing will not solve the problem of exceeding email rate limits. Quite the opposite is true.
    Fact is, that WCF heavily contributes to resp. causes the problem (= blocked emails because rate limits are exceeded). By trying to re-send blocked emails WCF will aggravate the problem. And as far as I've seen from the codes above, the user will not be able to stop it without direct manipulation of wcf1_background_job.
    In comparison, it's safer to use WCF 2.x. It stupidly sends all pending emails at once, doesn't know that many of them were blocked and, therefore, won't try to re-send them.




    Recipient / Domain based rate limiting is out of scope for WoltLab Suite Core, this is the job of the MTA as it is the only one with the complete view of the situation.

    Admins typically (should) know the email rate limits of their hoster / provider. Therefore it's easy for them to configure the limit. And since there already is a queueing, it should be easy for WL to incorperate such a limit.
    Please remember, that probably most admins (here) do not have access to the MTA's configuration...

  • Hi

    Well, as long as the WSC allows for user bulk proccessing (send email to users) or even 'Email users', which sends emails to all users, the described queueing will not solve the problem of exceeding email rate limits.

    while the bulk mailer tries to send all the emails immediately the emails will be sent staggered once they enter the background queue, as the queue is cleared probabilistically and there is a back off. Once the third retry failed the email will be dropped.


    In any case: If the amount of emails grossly exceeds the imposed limit no amount of rate limiting imposed by WoltLab Suite Core will solve the problem. No one wants to receive mails from three days ago and once you exceed the limit by magnitudes you will “never” clear the queue. This is not a technical issue, but a human one.
    You can intercept mails before they are sent using a plugin and then manually queue them as you like, if you strongly feel this issue needs a solution.

    In comparison, it's safer to use WCF 2.x. It stupidly sends all pending emails at once, doesn't know that many of them were blocked and, therefore, won't try to re-send them.

    … and then the admin sees that the process did not finish and tries to manually “restart” it, sending the same mail twice or thrice to the same users.


    Admins typically (should) know the email rate limits of their hoster / provider.

    Maybe I misunderstood your initial post: I was talking about rate limits to certain recipients. As an example: Only 1 message per minute to Gmail, as otherwise you will end up in the spam filter. Or only 100 messages per minute, because the recipient's mail server otherwise can't keep up. As these are global limits for the sending IP address rate limiting for this case needs to be applied inside the MTA.

  • I use the wcf behind an existing webapplication. (own templates, jscript etc.) So no Woltlab javascript is ever touched. How can I execute the backgroud jobs? During testing emails were not send (debug) but I find them in the wcf1_background_job table.

  • It's basically an AJAX request to wcf\action\BackgroundQueuePerformAction, you can either mimic a regular AJAX call or just execute that action via a cronjob - just don't make it a synchronous call along with a regular request..

    Alexander Ebert
    Senior Developer WoltLab® GmbH

  • Thanks, I wrote a small script. I call it via ajax:



    PHP
    <?php
    require_once('global.php');
    $cronjobAction = new \wcf\data\cronjob\CronjobAction(array(), 'executeCronjobs');
    $cronjobAction->executeAction();
    \wcf\system\background\BackgroundQueueHandler::getInstance()->performNextJob();

    or should I call "wcf\action\BackgroundQueuePerformAction" ?

  • There is nothing wrong with the calling the method directly, just keep in mind that the queue can potentially grow large and therefore is checked every 10th request by default. Decoupling it from the cronjob scheduler might be a good idea, as it allows you to customize the interval of the background queue process. You could then use quite a low frequency and monitor wcf1_background_job, if you see some serious backlog building up, you can increase the frequency to have it running as often as necessary to get the job done. This is just an idea at optimizing the calls, not an actual requirement, do whatever you want ;)

    Alexander Ebert
    Senior Developer WoltLab® GmbH

  • Hi

    At last I'll call the script from cron.

    I recommend not to do that with the current script: When calling the script from cron you'll clear at most one item per minute. It may very well be possible that several items enter the queue at once, leading to a serious delay.


    Consider wrapping the call to performNextJob() in a:

    Code
    while (BackgroundQueueHandler::getInstance()->getRunnableCount() > 0)
  • Good improvement. However, still missing: configurable email rate limits; e.g. x emails per hour or y emails per day / connection.

    I agree with it, woltlab should have this option. So we can configure outgoing email hour/daily limit.

    In share hosting, they will suspend your account for exceeding this limit. So before woltlab implement this option, I unlikely to touch the bulk email feature, it's too risky.