Just a shameless copy of this post for my own records. Thanks Paul!
openssl crl -inform PEM -text -noout -in mycrl.crlAs to the -inform, PEM is for text data, or DER for binary data.
Category Archives: sysadmin
Internet uplink and quota monitoring with HomeAssistant
I finally fell for the smart-home mania when I needed to read a few Zigbee climate sensors, and started using Home Assistant. There was no return from it, and I gradually grew the number of sensors and automations. This is all the easier thanks to a very active community site, offering many a recipe and troubleshooting advice. This is where I found a bandwidth monitor based on SNMP metrics that has been functional for a while.
My ISP, Internode (no longer the awesome service it used to be 10 years ago), has become increasingly flaky, silently dropping support for their Customer Tools API. This API was useful to track quota usage in a number of tools, including my own Munin plugin. Because of this, I unwittingly, and without warning, went beyond my monthly quota this month. I had to double my monthly bill to buy additional data blocks to tie me over.
It became obvious that I needed a new way to track my usage. What could be better than HomeAssistant, which was already ingesting SNMP data from the router? I posted my updated solution in the original thread, but thought that it might be worth duplicating here.
Public website with S3 and CloudFront
As I mentioned in a previous post, I am migrating a number of static websites from Apache on bare metal to an object store and a CDN in the cloud. Namely, this is AWS S3 and CloudFront. To avoid too much manual grooming of pet yaks, I also went directly for Infrastructure-as-Code with CloudFormation, with the objective of creating a relatively simple reusable web+CDN template.
This is not a new topic, and a number of resources already exist around the web. I, for example, started with this one, which does a fairly decent job. There are, however, a number of fine details which I have found were tricky to get right, could lead into incompatibilities, and for which accurate documentation was hard to find (even ChatGPT failed to provide a correct answer, though this is not entirely surprising).
The goal of this post is to call those out, and provide the CloudFormation template mentioned above for those looking for a base. The template will:
- create an S3 bucket for use as a website endpoint
- create a CloudFront distribution using that bucket as an Origin
- create a few DNS entries
- create a TLS certificate for the service
tl;dr:
- The S3 website endpoint behaves like a website, returning directory index documents, or HTML errors documents.
- The bucket needs to be public for this to work, and it is not possible to use either OAI nor OAC to limit access to the CloudFront distribution only. The solution is to control access based on the
Refererheader - Generating TLS certificates with CertificateManager will fail with no clear explanation if the CAA policy for the domain forbids it.
Render Apache Server-side Includes one last time, with Docker
I talk about restoring backups often recently. This is because the disk on my trusty bare-metal server died. This gave me the opportunity to reassess my hosting choices, and do the ground work to move from where it was to where I want it to be.
One of those changes is moving static website hosting away from a Apache HTTPd, running on an OS I administrate (read: “frequently broke”), to a more focused and hands-off system in the cloud, AWS S3 with a CloudFront CDN (more on this in a later post).
Unfortunately, decades of running Apache have left me with a number of static sites using some on-the-fly templating by relying on Server-side Includes (SSI). Headers, footers, geeky IPv6 and last-modified tags, … none of those work with a truly static host. I needed a solution to render those snippets into full pages.
At first, I thought I’d just write a simple parser in Python. I quickly gave up on the idea, however, when I realised I used included templates with parameters. Pretty nifty stuff, but also not trivial to write a parser for.
Then I realised I already had the perfect parser: Apache. All I needed was to let it render all the pages one last time, and publish those instead! This was packed quickly with a relatively simple Docker container, and the trusty wget. The busy person can find a Gist of the Dockerfile here.
Git restoration shenanigans
Backups. What a better time to test ’em than when you need ’em. Don’t lie. I know you’ve been there too. In an unfortunate turn of events, I had to restore a number of bare git repos from recent off-site copies (made with the handy rdiff-backup), but they needed a bit more work to be functional.
Once restored, I couldn’t pull or push from my existing working copies. I was greeted with cryptic error messages instead: fatal: git upload-pack: not our ref 0000000000000000000000000000000000000000 and ! [remote rejected] master -> master (missing necessary objects), respectively.
No amount of searching led to an adequate solution. So I simply leveraged git’s distributedness, and used one of the clone to recreate my bare repo. I was nonetheless a bit worried about having lost a few commits on the tip.
Playing in the bare repo later on led me to a more satisfying solution. Apparently, the refs/heads/master file was corrupted (empty), and editing it to contain the full sha-1 of the tip was enough to fix the issue. I found the sha-1 of the desired commit in the packed-refs file at the root of the bare repo. Once done, everything worked as before, and pre-existing working copies were able to pull and push without issue.
I learned two things:
- A bit more about git
- That I didn’t actually have any more commits there
Backups! Yay!
Continue readingAdvertise a metered network to Android devices with Router OS
Due to an unplanned outage of my main ISP, I had to get a mobile data SIM in a hurry, to use as an LTE backup uplink for my Mikrotik hAP ac3 (the whole setup of which I’ll describe one day). Given the price discrepancy of those, I wanted to make every transferred byte count: No unnecessary update fetching or immediate download of high-res sepia-toned photos of bulldogs in tutus.
Android can advertise itself as a metered network to its (Android) clients, so how do I do the same with Router OS 7?
- Make DHCP Option 43 (Vendor-Specific Option) contain the string
ANDROID_METERED; - The option should be sent even if not requested by the client (not standard compliant, but doesn’t hurt).
Split a multi-DB dump from MySQL/MariaDB
I recently had to restore databases from a rough mysqldump backup in a piecemeal fashion. One necessity is to SET the environment correctly, lest some weird encoding issues happen when restoring the data, leading to failures.
A sed one-liner can help for this.
DBNAME=mydb
sed -n "/^-- Server version/,/^-- Current Database/p;/^-- Current Database.*${DBNAME}\`/,/^-- Current Database/{p}" mysqldump.sql > ${DBNAME}.sql
This extracts SQL from the initial header, to the first database, which contains all the sessions SETs. It then captures statements any time the target database is the current one. Note that this doesn’t restore the GRANTs.
Befor blindly piping the output SQL into mysql, one would be well advised to review the contents of the file, to ensure only the desired modifications are included.
The ultimate bash startup logic
It’s always a bit confusing to reason about which startup files bash(1) is going to run in a given situation. It is documented, but the logic is not trivial.
So I made a chart (using PlantUML!). You’re welcome.
EDIT: There is also one for zsh(1) (PlantUML).
The main takeaway from this exercise is:
Continue readingIn
shmode,bashandzshdo slightly different things.
Scaling in the time of COVID: How we run load tests at Learnosity
I wrote this article for the Learnosity blog, where it originally appeared. I repost it here, with permission, for archival. With thanks, again, to Micheál Heffernan for countless editing passes.
In this series, I look at how we load test our platform to ensure platform stability during periods of heavy user traffic. For Learnosity, that’s typically during the back-to-school period. The year was different though, as COVID caused a dramatic global pivot to online assessment in education. Here is what the result of that looked like in terms of traffic.
We expect major growth every year but that kind of hockey stick curve is not something you can easily predict. But, because scalability is one of the cornerstones of our product offering, we were well-equipped to handle it.
This article series reveals how we prepared for that.
In part one (which was, incidentally, pre-COVID), I detailed how we actually created the load by writing a script using Locust. In this post, I’ll go through the process of running the load test. I’ll also look at some system bottlenecks it helped us find.
Let’s kick things off by looking at some important things a good load-testing method should do. Namely, it should
- Apply a realistic load, starting from known-supported levels.
- Determine whether the behaviour under load matches the requirements.
- If the behaviour is not as desired, you need to identify errors and fix them. These could be in
- the load-test code (not realistic enough)
- the load-test environment (unable to generate enough load)
- the system parameters
- the application code
- If the behaviour is as desired, then ramp up the load exponentially.
- If the behaviour is not as desired, you need to identify errors and fix them. These could be in
We used two separate tools for steps 1 above (as described in the first part of this series) and tracked the outcomes of step 2 in a spreadsheet.
TL;DR
- We used Locust to create the load, and a custom application to verify correct behaviour.
- We found a number of configuration-level issues, mainly around limits on file descriptors and open connections.
- Stuff we learned along the way:
- Record all parameters and their values, change one at a time;
- Be conscious of system limits, particularly on the allowed number of open files and sockets.
Removing spurious peaks from munin graphs
Every now and then, some spurious peaks show up on munin graphs. The peaks are order of magnitude higher than the expected range of the data. This particularly happens with DERIVE plugins, that are notably used for network interfaces.
One way to fix this, as suggested by Steve Schnepp (and in the faq), is to set the maximum straight into the RRD database, and then let it reprocess the data to honour this maximum.
Continue reading