About Olivier Mehani

Just another random geek.

Backups. What a better time to test ’em than when you need ’em. Don’t lie. I know you’ve been there too. In an unfortunate turn of events, I had to restore a number of bare git repos from recent off-site copies (made with the handy rdiff-backup), but they needed a bit more work to be functional.

Once restored, I couldn’t pull or push from my existing working copies. I was greeted with cryptic error messages instead: fatal: git upload-pack: not our ref 0000000000000000000000000000000000000000 and ! [remote rejected] master -> master (missing necessary objects), respectively.

No amount of searching led to an adequate solution. So I simply leveraged git’s distributedness, and used one of the clone to recreate my bare repo. I was nonetheless a bit worried about having lost a few commits on the tip.

Playing in the bare repo later on led me to a more satisfying solution. Apparently, the refs/heads/master file was corrupted (empty), and editing it to contain the full sha-1 of the tip was enough to fix the issue. I found the sha-1 of the desired commit in the packed-refs file at the root of the bare repo. Once done, everything worked as before, and pre-existing working copies were able to pull and push without issue.

I learned two things:

  • A bit more about git
  • That I didn’t actually have any more commits there

Backups! Yay!

Continue reading

Due to an unplanned outage of my main ISP, I had to get a mobile data SIM in a hurry, to use as an LTE backup uplink for my Mikrotik hAP ac3 (the whole setup of which I’ll describe one day). Given the price discrepancy of those, I wanted to make every transferred byte count: No unnecessary update fetching or immediate download of high-res sepia-toned photos of bulldogs in tutus.

Android can advertise itself as a metered network to its (Android) clients, so how do I do the same with Router OS 7?

(router agnostic) tl;dr:

  1. Make DHCP Option 43 (Vendor-Specific Option) contain the string ANDROID_METERED;
  2. The option should be sent even if not requested by the client (not standard compliant, but doesn’t hurt).
Continue reading

I recently had to restore databases from a rough mysqldump backup in a piecemeal fashion. One necessity is to SET the environment correctly, lest some weird encoding issues happen when restoring the data, leading to failures.

A sed one-liner can help for this.

DBNAME=mydb
sed -n "/^-- Server version/,/^-- Current Database/p;/^-- Current Database.*${DBNAME}\`/,/^-- Current Database/{p}" mysqldump.sql > ${DBNAME}.sql

This extracts SQL from the initial header, to the first database, which contains all the sessions SETs. It then captures statements any time the target database is the current one. Note that this doesn’t restore the GRANTs.

Befor blindly piping the output SQL into mysql, one would be well advised to review the contents of the file, to ensure only the desired modifications are included.

A Watchy Armadillonium with a Leatherman Tread LT watchband.

A few years back, I got a Pebble Steel. Some years later, I also got myself a Leatherman Tread LT. The two obviously needed to be put together, using the Tread as the watchband for the Pebble. Unfortunately, the Pebble had a weird band attachment, which led me to try to mush two Thingiverse designs (a Pebble NATO attachment and a Tread watch attachment) into something that almost worked. Ultimately the plastic proved too brittle, and I got distracted by other things.

Fast forward a few years, and my Pebble, quite sadly, is a bit unhealthy. As a replacement, I received a Watchy with an Armadillonium case. So the question reemerged. This time, I pushed back the not-invented-here syndrome, and looked around for existing solutions. I discovered ChronoLinks, which looked perfect, but I wasn’t sure whether they would fit my case. At the price tag, I didn’t want to risk it.

Ultimately, I resorted to searching on eBay, then AliBaba, and found something that looked like it would do the job, at a price that wouldn’t make me too sad if it didn’t.

tl;dr: It did! (mostly)

Continue reading

We’ve been having some fun with Click and Python decorators at work.

We had a situation where we wanted to

  1. transform any Exception to a click.ClickException, so they would be rendered nicely, and
  2. catch one particular exception, and retry the function that raised it with a different parameter value as a fallback.

We got the first behaviour quickly into a decorator. We then realised that the second could also be done nicely with a decorator, too.

Continue reading

GitHub now allows to expand/collapse all files in a PR diff at once (pressing Alt while clicking one of the toggles). Unfortunately, there is no similar feature to mark all files as viewed. This is handy after having reviewed meaningful changes to file, and automatically modified/generated files can be ignored.

So here goes a one-liner for the JS console.

Array.from(document.getElementsByClassName('js-reviewed-toggle')).forEach(c => c.getElementsByTagName('input')[0].checked || c.click())
Continue reading
Target metrics

I wrote this article for the Learnosity blog, where it originally appeared. I repost it here, with permission, for archival. With thanks, again, to Micheál Heffernan for countless editing passes.

In this series, I look at how we load test our platform to ensure platform stability during periods of heavy user traffic. For Learnosity, that’s typically during the back-to-school period. The year was different though, as COVID caused a dramatic global pivot to online assessment in education. Here is what the result of that looked like in terms of traffic.

Weekly Learnosity users comparison 2012--2020

We expect major growth every year but that kind of hockey stick curve is not something you can easily predict. But, because scalability is one of the cornerstones of our product offering, we were well-equipped to handle it.

This article series reveals how we prepared for that.

In part one (which was, incidentally, pre-COVID), I detailed how we actually created the load by writing a script using Locust. In this post, I’ll go through the process of running the load test. I’ll also look at some system bottlenecks it helped us find.

Let’s kick things off by looking at some important things a good load-testing method should do. Namely, it should

  1. Apply a realistic load, starting from known-supported levels.
  2. Determine whether the behaviour under load matches the requirements.
    • If the behaviour is not as desired, you need to identify errors and fix them. These could be in
      • the load-test code (not realistic enough)
      • the load-test environment (unable to generate enough load)
      • the system parameters
      • the application code
    • If the behaviour is as desired, then ramp up the load exponentially.

We used two separate tools for steps 1 above (as described in the first part of this series) and tracked the outcomes of step 2 in a spreadsheet.

TL;DR

  • We used Locust to create the load, and a custom application to verify correct behaviour.
  • We found a number of configuration-level issues, mainly around limits on file descriptors and open connections.
  • Stuff we learned along the way:
    • Record all parameters and their values, change one at a time;
    • Be conscious of system limits, particularly on the allowed number of open files and sockets.
Continue reading

Every now and then, some spurious peaks show up on munin graphs. The peaks are order of magnitude higher than the expected range of the data. This particularly happens with DERIVE plugins, that are notably used for network interfaces.

One way to fix this, as suggested by Steve Schnepp (and in the faq), is to set the maximum straight into the RRD database, and then let it reprocess the data to honour this maximum.

Continue reading