Correction of Errors: Our Support Email Import Failed for 109 Days
For 109 days our WHMCS email to ticket import failed while outbound mail kept working. What broke, who it affected, and what we changed.
Between May 20 and September 6, 2026, email sent to our support address stopped becoming support tickets. The importer ran on schedule the whole time and failed on every single message. It wrote the same error about 6.8 million times, into a log that nothing was watching. Outbound email kept working perfectly, which is part of why this took us 109 days to find.
Fifty-three inbound emails were affected. If you wrote to us in that window and heard nothing back, that is on us. The next section tells you what to do.
Who this affected, and what to do#
The failure was limited to one path: the import that turns mail arriving at our support address into a ticket. Tickets opened inside MyNobull do not go through that import. If you emailed us instead, your message reached the mailbox and stayed there.
If you replied to an open ticket from your mail client during the window and the conversation went quiet, treat it the same way and open a fresh ticket.
What to do: if you wrote to us between May 15 and September 6 and never got a reply, open a ticket from MyNobull and say so. Do not re-send to the address that failed you. We are working through the affected messages from our side, but a ticket from you is faster than waiting for us to reach yours, and we would rather answer you twice than miss you again.
Our published first-response target is under 60 minutes. On these 53 messages we missed it by as much as 109 days. We are not going to dress that up.
Timeline#
| Date (2026) | What the logs show |
|---|---|
| May 15 | Last recorded successful import. Two messages converted into tickets. |
| May 16 | WHMCS is upgraded on the command line in non-interactive mode, going from 8.13.3 straight to 9.0.4 in a single Saturday evening run. This is where the importer breaks. |
| May 20 | First sustained parser failures, about 1,400 on the first day. |
| May 20 to Sep 5 | Zero successful imports for 109 consecutive days. Daily error volume climbs from about 1,400 to about 119,000. |
| Sep 6 | Cause identified. The stale script is replaced with the current release copy, and 56 messages import successfully the same day. |
The logs record no successful imports after May 15 and no parser failures until May 20. We have not established what accounts for the four-day gap between the upgrade and the first logged error.
What broke#
Our billing and ticketing system pulls mail from Microsoft 365 with a cron script called pop.php, which hands each raw message to a mail parser library and turns the result into a ticket. The copy of pop.php that was actually running had fallen behind the parser library it was calling. It passed one argument to a function that required two.
The mismatch was small and absolute. Every message hit the same wall, so every message failed. There was nothing partial or intermittent about it: from the first bad run to the last, the importer converted nothing at all.
How that script fell behind starts with a decision we would make again. Our crons directory sits in a private directory above the web root, which is what WHMCS recommends for self-hosted installations, because it prevents web-based access to those scripts. That is sound, and we are keeping it.
The obligation that comes with it is documented in the same place. When you move the crons directory, WHMCS states you must upload any updated crons files from the default location to your custom directory, and warns that failing to do so causes errors and issues with its automation. On the evening of Saturday May 16 we upgraded from 8.13.3 to 9.0.4, running the command line upgrader in non-interactive mode. That run applied every release from 9.0.0 through 9.0.4 in one pass and moved the parser library forward. We did not copy the updated crons files across afterwards.
So this was a documented step we missed, not a trap someone set for us. The upgrade reported success, because from its point of view it had succeeded. One thing worth knowing if you run the same stack: our command line integration compares files across a single point release, and this run spanned a major version and four point releases, so that output was never going to point at the problem either.
None of that is why it lasted 109 days. It lasted because we had never treated inbound ticket import as something that needed watching, so nothing existed to tell us the step had been missed.
Five whys#
- Why did inbound email stop becoming tickets?The import script called a parser function with one argument where the installed library required two, so every message failed to parse.
- Why was the script incompatible with the library?It was an older release than the library it called. The May 16 upgrade moved the library forward and left that copy of the script at its previous version.
- Why did the upgrade leave it behind?Our crons directory sits above the web root, which is WHMCS's own security recommendation. Files there are not maintained by the upgrader, and WHMCS documents that copying updated crons files across after an upgrade is the administrator's job.
- Why was that step missed?It was not on our upgrade verification checklist. The upgrade output did not raise it either: our command line integration compares files across a single point release, and this run spanned a major version and four point releases.
- Why did it then run broken for 109 days?Nothing monitored whether inbound import was succeeding. Outbound mail stayed healthy, our inbound volume is low enough that silence looks like a quiet week, and the error log was written but never alerted on.
The first answer is a bug and the fourth is a missed step. The fifth is why it lasted 109 days, and it is the one we are actually fixing: we had no independent check that inbound ticket import was working at all.
Why it took 109 days to find#
The fifth answer is what turned a one-line bug into a 109-day outage, and it deserves more than a line of its own. Three things covered for each other:
- Outbound email kept working. Mail from the billing system went out normally the whole time. Every mail signal we had said healthy, and none of them had anything to do with inbound.
- A quiet inbox looks like a calm week. Our inbound volume is a handful of messages on a busy day. Two imports on May 15 and none on May 21 is not a shape a person spots by eye, and there was no alert to spot it for us.
- The log was loud and unread. The importer recorded the failure about 6.8 million times. Daily error volume grew from roughly 1,400 on the first day to roughly 119,000 by early September, climbing for three and a half months in a file nothing alerted on.
Monitoring that only watches the direction which happens to be working is not monitoring. We publish our node load ceiling because we think a number nobody checks is not a commitment. We had plainly not held our own support pipeline to that standard.
What we changed#
| Action | Status |
|---|---|
| Replace the stale pop.php in the active cron directory with the current release copy | Done, September 6 |
| Reconcile the 53 affected emails against tickets created after recovery, and raise tickets for anything still missing | In progress |
| Alert on sustained mail parser failures in LibreNMS | Planned |
| Add a separate LibreNMS check for an import job that has stopped producing successes, so silence is itself an alarm | Planned |
| Add relocated cron files and an end-to-end email-to-ticket test to the upgrade verification checklist | Planned |
| Treat copying updated crons files to the custom directory as a required and verified step of every upgrade rather than an optional one | Planned |
The two monitoring items are deliberately separate. One catches a component that is failing loudly, which is the easy case. The other catches a component that has simply stopped, which is the case that beat us here and the one most worth having.
What we still owe you#
Reconciliation is not finished. Fifty-six messages imported successfully once the fix landed, and 53 were reported as missed during the outage. Those two counts came from different places and we have not yet matched them message by message, so we are not going to claim they cancel each other out. Until that work is done we are treating all 53 as potentially unanswered, and we would rather over-count our own failures than round them down.
We also do not know how many people gave up rather than following up. The logs cannot tell us that, and we are not going to estimate it.
Why we are publishing this#
A correction of errors is only useful if it is honest about the parts that are unflattering, and this one has several: a 109-day detection time, a log that recorded the same failure about 6.8 million times where nobody was watching, and a documented post-upgrade step that we simply did not carry out. None of that is subtle engineering. It is a gap in operational discipline, and the fix is discipline rather than cleverness.
We ask you to judge a host by what it tells you when things go wrong, which obliges us to tell you. Our services are unmanaged, which means the infrastructure layer is ours to run and ours to answer for. The path your message takes to reach an engineer is part of that layer, and for 109 days it was broken.
FAQ#
I emailed support during that window and never heard back. What should I do?
Open a ticket from MyNobull and mention that you wrote in during the outage, and we will pick it up from there. Please do not simply re-send the original email, because a fresh ticket carries your account context and gets to an engineer faster.
Was any of my email lost?
No. The failure was in converting messages into tickets, not in receiving them, so the mail stayed in the Microsoft 365 mailbox the entire time. That is how 56 messages were able to import successfully the moment the fix went in.
Was my data or my server at risk?
No. The failure was confined to the import that converts inbound mail into tickets. It involved no customer server, VPS, VDS, game server, or colocated system, and the affected mail stayed in our Microsoft 365 mailbox throughout.
Did this affect tickets I opened in MyNobull?
No. Tickets submitted through the client area are created directly and never touch the email importer, so they were created and answered normally throughout. Only mail arriving at the support address, including emailed replies to open tickets, was affected.
How did a WHMCS upgrade not catch this?
It was never supposed to. Our crons directory sits above the web root on WHMCS's own security advice, and their documentation is clear that copying updated crons files across after an upgrade is the administrator's job. We missed that step. Separately, the command line integration compares files across a single point release, so on a jump like 8.13.3 to 9.0.4 it will not flag the drift either.
How will you catch this next time?
Two LibreNMS checks, one for sustained parser failures and one for an import job that has stopped succeeding. The second matters more, because this outage was invisible precisely in the way a stopped job is invisible: nothing was wrong with what we were watching.
Where do I check whether something is broken right now?
status.nobullnetworks.com carries platform-wide incidents and scheduled maintenance. It is the right first stop before opening a ticket, and it is where an issue of this size would be posted today.