tip « Narf

Backups. What a better time to test ’em than when you need ’em. Don’t lie. I know you’ve been there too. In an unfortunate turn of events, I had to restore a number of bare git repos from recent off-site copies (made with the handy rdiff-backup), but they needed a bit more work to be functional.

Once restored, I couldn’t pull or push from my existing working copies. I was greeted with cryptic error messages instead: fatal: git upload-pack: not our ref 0000000000000000000000000000000000000000 and ! [remote rejected] master -> master (missing necessary objects), respectively.

No amount of searching led to an adequate solution. So I simply leveraged git’s distributedness, and used one of the clone to recreate my bare repo. I was nonetheless a bit worried about having lost a few commits on the tip.

Playing in the bare repo later on led me to a more satisfying solution. Apparently, the refs/heads/master file was corrupted (empty), and editing it to contain the full sha-1 of the tip was enough to fix the issue. I found the sha-1 of the desired commit in the packed-refs file at the root of the bare repo. Once done, everything worked as before, and pre-existing working copies were able to pull and push without issue.

I learned two things:

A bit more about git
That I didn’t actually have any more commits there

Backups! Yay!

Continue reading →

Due to an unplanned outage of my main ISP, I had to get a mobile data SIM in a hurry, to use as an LTE backup uplink for my Mikrotik hAP ac³ (the whole setup of which I’ll describe one day). Given the price discrepancy of those, I wanted to make every transferred byte count: No unnecessary update fetching or immediate download of high-res sepia-toned photos of bulldogs in tutus.

Android can advertise itself as a metered network to its (Android) clients, so how do I do the same with Router OS 7?

(router agnostic) tl;dr:

Make DHCP Option 43 (Vendor-Specific Option) contain the string ANDROID_METERED;
The option should be sent even if not requested by the client (not standard compliant, but doesn’t hurt).

Continue reading →

I recently had to restore databases from a rough mysqldump backup in a piecemeal fashion. One necessity is to SET the environment correctly, lest some weird encoding issues happen when restoring the data, leading to failures.

A sed one-liner can help for this.

DBNAME=mydb
sed -n "/^-- Server version/,/^-- Current Database/p;/^-- Current Database.*${DBNAME}\`/,/^-- Current Database/{p}" mysqldump.sql > ${DBNAME}.sql

This extracts SQL from the initial header, to the first database, which contains all the sessions SETs. It then captures statements any time the target database is the current one. Note that this doesn’t restore the GRANTs.

Befor blindly piping the output SQL into mysql, one would be well advised to review the contents of the file, to ensure only the desired modifications are included.

An activity diagram showing which startup files are processed by `bash` depending on context.

It’s always a bit confusing to reason about which startup files bash(1) is going to run in a given situation. It is documented, but the logic is not trivial.

So I made a chart (using PlantUML!). You’re welcome.

EDIT: There is also one for zsh(1) (PlantUML).

The main takeaway from this exercise is:

In sh mode, bash and zsh do slightly different things.

Continue reading →

We’ve been having some fun with Click and Python decorators at work.

We had a situation where we wanted to

transform any Exception to a click.ClickException, so they would be rendered nicely, and
catch one particular exception, and retry the function that raised it with a different parameter value as a fallback.

We got the first behaviour quickly into a decorator. We then realised that the second could also be done nicely with a decorator, too.

Continue reading →

If you have a Python string that contains the representation of escaped characters (say, a backslash \, an x, and two hexadecimal digits), and you want to decode those escapes back to the actual character they represent, you can use codecs.escape_decode.

Continue reading →

GitHub now allows to expand/collapse all files in a PR diff at once (pressing Alt while clicking one of the toggles). Unfortunately, there is no similar feature to mark all files as viewed. This is handy after having reviewed meaningful changes to file, and automatically modified/generated files can be ignored.

So here goes a one-liner for the JS console.

Array.from(document.getElementsByClassName('js-reviewed-toggle')).forEach(c => c.getElementsByTagName('input')[0].checked || c.click())

Continue reading →

Every now and then, some spurious peaks show up on munin graphs. The peaks are order of magnitude higher than the expected range of the data. This particularly happens with DERIVE plugins, that are notably used for network interfaces.

One way to fix this, as suggested by Steve Schnepp (and in the faq), is to set the maximum straight into the RRD database, and then let it reprocess the data to honour this maximum.

Continue reading →

When using syspatch on OpenBSD, the upgrade sometimes fails with

Relinking to create unique kernel... failed!
!!! "/usr/libexec/reorder_kernel" must be run manually to install the new kernel

This generally happens after a system upgrade, or an otherwise manual change of kernel. This fix is to update the kernel hash, before re-running reorder_kernel.

# sha256 /bsd > /var/db/kernel.SHA256
# /usr/libexec/reorder_kernel

I use Munin to monitor a few machines, and bubble up alerts when issues show up. It’s pretty good, easy to set up, and has a large number of contributed plugins to monitor pretty much everything. If still out of luck, it’s easy enough to write your own.

To ease the task of viewing the data, each machine runs munin-node, but only a couple of masters do the data collection with munin-update. This works reasonably well, except that machines monitored by more than one server need to work extra time to provide the same data to both.

Fortunately, Munin 2.0 introduced a proxy mode, allowing to decouple running the plugins to collect fresh data (with munin-asyncd) from giving that data to collection servers (via munin-async).

Setting this up is relatively easy, and the benefits show quickly, in the form of a reduced collection time, and fewer gaps in the data.

Reduced update time for the master, and no more gaps in the data.

Surprisingly it also showed as a substantially reduced load on low-power machines. But beware of the --fork parameter to munin-asyncd.

Continue reading →

Narf

The blagosphere got me…

Category Archives: tip

Git restoration shenanigans

Advertise a metered network to Android devices with Router OS

Split a multi-DB dump from MySQL/MariaDB

The ultimate bash startup logic

Fun with Python decorators

Resolve escapes in Python strings

GitHub Pull Request: mark all files as viewed

Removing spurious peaks from munin graphs

Relinking to create unique kernel… failed!

Using munin-async to reduce gaps in monitoring… and system load