Category: Server

How to fix Mono crashing on Odroid XU4

Ryancrash, mono, odroid, radarr, sonarr

Recently I’ve been noticing my Sonarr and Radarr applications behaving erratic (sometimes not responsive, sometimes not performing tasks, not searching or adding content, but at other times behaving totally fine). A quick look at the logs told me the applications were crashing and being restarted by systemctl, after a few crashes they seemed to stabilize.
Today I had some time to dig deeper into the issue. I had already searched the Internet for general issues, but it didn’t seem to be a widespread problem. I assumed it might have to do with the ARM architecture, but Raspberry Pi users didn’t seem to be having these issues.

In the past I had difficulty reproducing the issue, but today I was in luck, every time I tried to kick off the “Process Monitored Downloads” task in Radarr, it would start working on the task and then crash and restart. The core issue turned out to be oddly specific to the Odroid XU4 hardware.

The XU4 has eight CPU cores, four A7 cores running at 1.4GHz, and four A15 cores running at 2GHz.

 

Whenever the mono process moved from an A7 to an A15 core (or vice versa) the process crashed.
Since both Sonarr and Radarr are Mono applications, they were both affected. Pinning the applications to either the A15 or the A7 CPUs resolved the problem.
taskset --cpu-list  can be used to change the CPU affinity of a process.

First look up the systemctl  service files for Radarr and Sonarr (e.g. via systemctl status radarr ). Edit the service files by prefixing the ExecStart  command with /usr/bin/taskset -c 0-3  (for the A7 cores) or /usr/bin/taskset -c 4-7  (for the A15 cores).
Then reload the systemctl files ( systemctl daemon-reload ) and restart the service.

Example:

If the service files are managed via the package manager, you may want to create an systemctl override instead of editing the service file, so that the package manager doesn’t overwrite your changes:

 

Now you probably want to see if it worked and want to know how to check which core a process is running on? There a re a few options:

htop: launch htop . Press <F2> , go to Columns , and add PROCESSOR  from Available Columns. The CPUs are numbered 1-8 in htop (as opposed to 0-7 by the system).

ps: the PSR column can display the core a process is on. The CPUs are numbered 0-7 in ps.

taskset: can display the current affinity of a process.

 

Statically served wordpress content

Ryanstatic content, wordpress

I’m currently still evaluating hugo and jekyll, themes and plugins, as an alternative to the current WordPress site. Until I decide what route to eventually go with, I had a look at WordPress plugins to generate static versions of a site.

Simply Static looked fine and I gave it a spin, it can easily crawl through the site and you can provide additional file/urls/directories to add to the static version (as well as exemptions).

The static version of the website is created regularly and stored locally, so I added a few ansible tasks to set up a periodic rsync of the files to my webserver that serves static content.

I have a HAProxy load balancer in front of my webservers that I have configured to serve the static version of the website first, and fall back to the wordpress server as a backup (that also gives me a nice redundancy, so I can update and reboot servers without causing a downtime).
HAProxy is also configured to always send certain requests (admin interface, search) to the WordPress server since they require PHP. This all happens transparently for the user.

I’m not going to bore with the details since it was all pretty standard stuff. It’s nothing fancy, but it looks reliable and does what it should.

I have this blog entry scheduled to go live in a few days, so we’ll see if all the automatisms work and the static version of the page generated and synced to the webserver.

Odroid HC1

Ryan

I finally got my new Odroid SBC and have migrated all the services from my Odriud XU4 over to the new HC1. The HC1 is based on the XU4 but drops multimedia interfaces like the HDMI port and instead adds a SATA connector and the black aluminium case fits a 2.5″ hard drive. The case is stackable and acts as a passive cooling heat sink. Since the HC1 is based on the XU4, software that runs on a XU4 will run on a HC1 without any fiddling.

It doesn’t have an eMMC connector, but you can assign the /  partition to the hard drive/SSD, only the /boot  partition needs to be on the SD Card.

Selfhosting email, and sending email to Microsoft.

Ryanemail, microsoft, MTA, SMTP

About once a year one of my outgoing email servers will magically pop up on Microsofts blacklist of email servers. The exact status in SNDS is “Blocked due to user complaints or other evidence of spamming“, which is a bit ironic since the weekly volume of emails that go to Microsoft controlled domains seldom is higher than 5 and they are all personal emails.
The error message the MTA recieves is “550 5.7.1 Unfortunately, messages from [xxx.xxx.xxx.xxx] weren’t sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.” in case anyone ends up here via google.

I’m not going to bitch and moan or attempt to guess why the server ends up on the list since it isn’t something I can influence
 (just submit a support request and wait), instead I’ll post some tips and pointers to useful tools you should be using:

SPF – Sender Policy Framework
Specify which IPs are allowed too send mail for your domains, also determine what should happen with mail that is recieved from unauthorized IPs.

DKIM – DomainKeys Identified Mail
Outgoing mail servers for a domain can sign the emails and thereby allow recipients to verify that the email really came from a valid mail server and is not from a spoofed sender.

DMARC – Domain-based Message Authentication, Reporting & Conformance
DMARC allows you to specify policies for domains or subdomains in regards to “what should happen if an email fails DKIM or SPF” and where/if to send reports for Emails recieved by other mail servers.

SNDS – Smart Network Data Service
Allows you to monitor the IPs of your mailservers as viewed by the Outlook.com system.

JMRP – Junk Email Reporting Program
Forwards the full message with headers of any email marked as “junk” or “phishing”  by Outlook.com users

MX Toolbox
MXtoolbox is a great website for testing your mail server settings, they also have a free_monitoring service for one IP,  so you can get alerted if your IP shows up on a Blacklist.

Last but not least I use the following small script to monitor the status of my servers according to Microsoft. I call the script via a daily cronjob, and if a server is blacklisted, it sends an alert. Go to Automated Data Access first to generate an API key.

And if all else fails and your IP does end up blacklisted in SNDS, you can go here to submit a ticket to get back off (although I’d suggest checking your mail server logs and the status of the IP on other RBLs first, just to make sure).

http://go.microsoft.com/fwlink/?LinkID=614866

 

Setting up multidomain DKIM with Exim

RyanDKIM, dmarc, exim, SMTP, SPF

Update November 2018:

A reader contacted me and pointed out that removing the {} around DKIM_DOMAIN solves the errors in the original example I found and had problems using. I’ve updated the code below (line 8) to reflect those changes in case anyone ends up here via google.

He also shared a nifty way to make selector rollovers easier by adding them to the filename:

And last but not least an elegant way to populate the  DKIM_DOMAIN variable: https://bugs.exim.org/show_bug.cgi?id=1019

Original Posting:

I was recently setting up SPF, DKIM and DMARC for multiple domains and was having trouble getting Exim to sign emails for the different domains. I found an article here explaining the steps. But I kept getting the following error in my exim logs:

failed to expand dkim_private_key: missing or misplaced { or }

The suggested configuration was the following:

I’m not quite sure why, but Exim was having trouble using the macros in the following macros, so I ended up changing it to the following snippet instead. If you don’t use DKIM_FILE you can omit it. Also you might want to set DKIM_STRICT to true if you published a DMARC policy that will reject or quarantine email failing the DKIM tests (unset, or “false” tells Exim to send the message unsigned if it ran into problems signing the email). The default setting for DKIM_CANON is “relaxed“, so it also can be omitted.

Other than that, just make sure the exim process has permissions to access the dkim directory and certificate files and everything should work nicely.