What isn't so great, though, is the default power efficiency when using a standard Debian install with it. Now you might ask yourself why I went with Debian on new hardware. I just like the stability, and I argue that you can still run fairly recent kernel versions using the Debian backport repo. You can compile relevant software yourself, and you can install binary firmware blobs manually. If you don't use Debian Bookworm (12), then this post may not even be relevant for you. In any case, in this post, I want to outline what I did to minimize wattage, keep the laptop cool, and maximize battery life. Keep in mind - this post assumes that you're running recent (Zen3+) AMD hardware.

CPU scaling driver

First thing you should check which scaling driver you are using.

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver

This will probably give you acpi_cpufreq. If that's the case, then you should consider switching to amd_pstate_epp. Actually, there's also amd_pstate (no _eppsuffix) - the difference in my testing is large, and amd_pstate_epp is superior. The kernel documentation refers to this difference with different names (active, passive and guided). See the linked page, it explains the differences very nicely.

Anyhow, you'll want to configure the kernel to use the active mode. For this, I added a kernel parameter: initcall_blacklist=acpi_cpufreq_init amd_pstate=active. The blacklisting is needed, because otherwise the acpi_cpufreq driver may be started beforehand and stop the amd_pstate driver from taking over. The second paramter sets the mode to active. Keep in mind, to use the active mode you'll need a kernel newer than 6.5 (install from Backports).

After booting, the above sysfs file should return amd-pstate-epp. At this point, you should already notice a massive difference in terms of thermals and battery life, but we aren't done yet.

power-profiles-daemon

If it's not installed yet, you should at least install power-profiles-daemon. Essentially, it allows an easy way to switch between what's called "platform profiles". These profiles are defined within the ACPI spec. They massively influence Thinkpads computing performance and efficiency, since Lenovo uses this ACPI functionality to determine minimum/maximum frequency as well as frequency boost duration.

You can, of course, set this yourself - valid options are "low-power", "balanced" and "performance", and you can write them directly into /sys/firmware/acpi/platform_profile. But power-profiles-daemon also manages the CPU frequency governor for you.

Currently, power-profiles-daemon in Debian Bookworm is at version 0.12, which even for Debian is pretty ancient. You should update to the newest version - read the changelog if you're not convinced :).

Upgrade binary firmware manually

In order to fix bugs in newer iGPU firmware, you should update the firmware blobs. To do this, download the Linux firmware blobs, copy/overwrite /lib/firmware/amdgpu and re-create your initramfs with update-initramfs -u -k all.

After these points, your laptop should be much cooler, with longer battery life and better hardware decoding.

By the way - if you're using Waybar, I added an element to my Waybar which allows me to easily toggle the platform_profile. Maybe it's useful to you, here's a Github link.

Imagine a web application which consists of many different endpoints (as most do). You secured one of them with TLS client auth - also called mTLS.

Now, another web application on a different subdomain, requests (via GET) this resource on the client (browser) side. You anticipate this, and enable CORS headers on the protected endpoint.

add_header 'Access-Control-Allow-Origin' 'https://$URL' always;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';

Sweet, that should do it, right? Trembling with anticipation, you refresh the page of the web application which requests this URL, only for Firefox to greet you with:

HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 14 Apr 2024 18:02:33 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 146
Connection: keep-alive

No CORS headers, and a 403. What's up with that? Turns out (after a while of debugging, as usual) that the CORS "pre-flight" check failed. This is an OPTIONS request toward the URL in question, in which the browser checks for the headers above. The problem is that this request doesn't support mTLS authentication! As an added bonus, there's a ten year old Firefox bug about this behavior. To be fair, this seems to be an oversight in the CORS spec itself, and not in Firefox.

Anyhow, you can fix all of this by setting an about:config property network.cors_preflight.allow_client_cert to true.

After this, your CORS requests will use mTLS authenticated pre-flight requests to obtain the Access-Control-Allow-* headers, and Bob's your uncle.

Within the last few months, I had spontaneous hiccups within my infrastructure with regards to DNS lookups. Every now and then, at different points in time, DNS lookups for certain subdomains where different services are hosted on, timed out or only resolved with considerable delay.

I tried to reproduce this, which didn't work well, as the problem only reared its head now and then. I setup a cronjob to be run every hour, and logged the output. That worked, and led to this:

dig a nxsubdomain.lpcvoid.com

; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> a nxsubdomain.lpcvoid.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 54406
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 22 (No Reachable Authority): (at delegation duckdns.org.)
; EDE: 23 (Network Error): (3.97.58.28:53 timed out for lpcvoid.duckdns.org A)
;; QUESTION SECTION:
;nxsubdomain.lpcvoid.com.		IN	A

;; Query time: 4160 msec
;; SERVER: 192.168.1.1#53(192.168.1.1) (UDP)
;; WHEN: Sat Jan 27 22:01:27 CET 2024
;; MSG SIZE  rcvd: 132

Turns out, DuckDNS, the dyndns provider I use, was the root of these timeouts. Every now and then, it would time out randomly. Now, this isn't any attack on them, DuckDNS is a one-player show offering a service for free to everybody. I am incredibly grateful for their continued work, and these sort of problems are normal once you become as widely used as they are.

But I cannot live with this, since I use domains everywhere, even within my LAN. I could of course configure my DNS server to cache more aggressively, or use internal LAN addresses when the request comes from within the LAN (compared to my WAN IP when requests arrive via the WAN). But this would also entail maintaining a "split horizon" DNS. I would like to avoid that, and use my WAN IP internally within my LAN. This is called reflection in the NAT world. I really feel like I am approaching the point in time where an IPv6 setup would reduce my LAN complexity by a considerable amount. Some day...

In any case, I needed an alternative. There's an elegant solution to this problem, which is technically superior to a dedicated dyndns provider.

I already use Cloudflare for my domains DNS setup. Like all other DNS management platforms, Cloudflare allows you to configure various DNS zones which point somewhere. In the past, I had a DuckDNS subdomain (lpcvoid.duckdns.org) point to my WAN IP, and CNAME'ed a wildcard zone rule to point to the DuckDNS domain for the actual address resolution. This worked, but is inefficient: You need to first lookup the domain you're interested in (let's say blabla.lpcvoid.com). This info is returned by Cloudflare, and is pretty fast. Then, your browser will lookup lpcvoid.duckdns.org (which blabla.lpcvoid.com points to via the CNAME entry, remember) and that will in turn result in my WAN IP. So you have two lookups in total, and one of those (the DuckDNS one in my case) is potentially slow.

Ideally, Cloudflare would always know my WAN IP, and update DNS A records whenever my WAN IP changes. This would completely remove the DynDNS provider from the equation, and result in very fast, single lookups which Cloudflare can respond to instantly.

Thankfully, OpenWRT, which runs on my router, maintains a script which can do exactly this automatically as part of the ddns-scripts-cloudflare package. For this to work, you need a Cloudflare API key (which you can generate here). Generate a token which can only change DNS zones - you should in general be quite restrictive with Cloudflare API tokens, since they are a pretty high value target for attackers. Afterwards, use the token and the username "Bearer" (yes, capital "B"), along with the record you wish to update on WAN IP change. So if your subdomain is foobar.example.com, and the record is thus called "foobar", you configure your "domain" in the script as "foobar@example.com" in order for Cloudflare to resolve it properly.

After doing this, you should have an A record which points to your WAN IP. Now, if you're lazy like me, and have dozens of subdomains, you obviously want to have a wildcard record. The trick is to have your wildcard CNAME record (*) point to the A record you just configured. This lookup, by the way, will be handled internally via Cloudflare's resolvers, and not result in two DNS queries.

In the end, you should have something like this:

dig a nxsubdomain.lpcvoid.com

; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> a nxsubdomain.lpcvoid.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49990
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;nxsubdomain.lpcvoid.com.	IN	A

;; ANSWER SECTION:
nxsubdomain.lpcvoid.com. 600	IN	CNAME	homeserver.lpcvoid.com.
homeserver.lpcvoid.com.	600	IN	A	79.235.213.105

;; Query time: 20 msec
;; SERVER: 192.168.1.1#53(192.168.1.1) (UDP)
;; WHEN: Mon Jan 29 14:55:48 CET 2024
;; MSG SIZE  rcvd: 93

Beautiful, problem solved. DNS lookups are now snappy.

In my next post I guess I will write about my complete setup.

It turns out, dear reader, that I lied to you. I was never able to fix the glaring suspend/resume problem that I hinted at in the updates to the article. So, I did what no self-respecting Tinkerer would do after months of troubleshooting: I decided to give up and throw money at the problem, hoping it would go away.

Like the last time, I had a list of must-haves (I really dislike the word "requirements") that the new laptop should bring in order to contend:

Point 1 - AMD based

This was born out of the fact that I have a desktop machine running full AMD (5600x + 6700xt on an AM4 mainboard), which has been working absolutely flawlessly for about 3 years now. I bought it during the pandemic to play Valheim, and it's just been so insanely unproblematic (running Ubuntu) that I hope to mirror that experience for my laptop. The plan is to only have a single computer.

The no NVIDIA part should be obvious. 

Point 2 - Thunderbolt

I love how you can simply connect your laptop with a docking station and have a desktop replacement with a single wire. USB4 works also (Thunderbolt 4 is basically just USB4 with all features). There's unfortunately some ambiguity around PCIe passthrough in USB4 (vendors don't need to pass PCIe lanes to be USB4 conform). Thankfully, Microsoft (those two words in close proximity, wow) specified that every laptop that wants to be "Windows 11 ready" must expose PCIe lanes via USB4 (this is called "Thunderbolt 3 compatibility"). 

Also, I want to try out an eGPU setup. There's going to be a separate blog post about this, but as a small teaser, I am writing this blog post with a working, hotpluggable eGPU setup :)

Point 3 - powerful and fast

I want to replace my desktop with this laptop. My desktop isn't an insane speed machine by any stretch, but it does compile code pretty fast. Ideally, this laptop can also do that, and even chomp through some games. 

Point 4 - portable, ideally 13.3 inches

Yeah, I like small laptops. For some reason, even though I have been sitting in front of computers for over 20 years now, I still have pretty good eyesight. This means I don't need to go to 15" laptops just yet. This may change in the future when I grow older, but for now, I will enjoy smaller form factors for as long as I can.

Point 5 - no soldered RAM

Okay, at that point, I had to compromise. It turns out there's no LPDDR5-capable machine with SODIMM bars. One reason for that is the low number of pins that are present in SODIMM connectors. Imagine having a 2048-bit memory bus (like they are used on HBM modules) on a SODIMM interface; there's simply nobody who specified such a standard for pluggable SODIMM modules. The LP in LPDDR5 stands for "Low Power," which can't deal with the signal integrity losses of SODIMM connectors. It needs a nice, clean PCB trace without some noisy physical mounting connector. The upside of this is that LPDDR consumes a lot less power than the normal SODIMM-mounted DDR RAM found in other notebooks. By the way, the LPDDR standard is not just DDR with some added specs; it's a completely different specification from DDR and was developed independently.

The laptop I went for in the end has 64GB LPDDR5. I don't foresee me needing to upgrade RAM for a long time.

Point 6 - It shall be chargeable via USBC-PD

Pretty self-explaining, I guess. When traveling (and even at home, if I think about it), I want to have one charger for everything. USBC-PD makes life easy. Probably one of the best technological improvements for me is the unification of charging infrastructure that USBC made possible.

Point 7 - I want a touchscreen

Yeah, I know, this point is a bit weird. But I love how I can lay on the sofa and scroll through a website with the laptop on my chest. It's purely a personal opinion, but touchscreens are awesome. I learned to love them with my older Dell Latitude, which also had one. Once you have one, I am pretty sure you'll find that one situation that makes a touchscreen worth it. It's different for everybody, though.

Point 8 - FullHD (1080p/1200p) screen

I can't stand 4k screens on a tiny form factor such as 13.3". I see pixels only when I look very, very closely. The added strain on the GPU and battery just isn't worth it to me. Also, fractional scaling is nice within most desktop environments under Wayland, but it's not very nice in GRUB or other older textmode interfaces.

Point 9 - ANSI-US (QWERTY) keyboard layout

I love ANSI-US. Easy access to brackets, special characters, and no wasted keys on local language features such as the German umlauts I use those incredibly rarely.

Manufacturer

Frame.Work

I have read many positive things about the Frame.Work laptop, and when I heard that they would offer an AMD version, I was really interested. It is almost an ideal candidate. It's running Coreboot, has an AMD CPU, is repairable, extendable and upgradable (specifically, the "expansion card" system is really nice!). But they don't offer a touchscreen. I thought very long about accepting that drawback and going the Frame.work route - but I couldn't get myself to ignore that one drawback. If they ever offer a touchscreen, I may reconsider.

Dell

I liked my Dell Latitude very much. I looked for a laptop from Dell, but they wouldn't even offer a thin or light 13.3" laptop with 32+GB RAM and an AMD CPU. All their XPS13 are Intel. Added to that, the new XPS13 has some sort of touch-buttons (instead of using the fn toggle + Fx keys), and I like physical buttons. Maybe they work well in practice, but I didn't want to risk it.

For reference, this was the laptop I looked at, which seemed like the best fit for me: https://www.dell.com/de-de/shop/dell-notebooks/xps-13-plus-laptop/spd/xps-13-9320-laptop/cn93352cc

HP

Next manufacturer I looked at was HP. I owned a HP Spectre x360 some time ago and liked it a lot too. They don't exist with AMD CPUs, at least not at the time I looked. The HP Pro x360 435 looked like a potential match, however: https://www.hp.com/de-de/shop/product.aspx?id=816D9EA&opt=ABD&sel=NTB

The only problem was that HP simply doesn't allow configuration of the keyboard. In Germany, you're seemingly forced to buy a German layout. In the end, I decided against it and looked at the third manufacturer.

Lenovo

Now, I try not to buy Chinese. I am wary about the security implications of doing so. I don't like their human rights track record (and yes, I think the US is better in this regard). I think the chinese political system is a threat to the free world.

I even messaged the BSI via email and asked what their opinion is on Lenovo. They weren't very helpful, though, and replied back with a template that stated that I could check their security advisories. I mean, yeah, but still, I had hoped for a more nuanced opinion. Oh well.

Also, as far as I was able to research, all the big American companies have at least some supply chain in China (Even though that's changing). You can't really avoid it, unless you go back in time and use a device which can run Coreboot. Alternatively, you can also go for something like the Talos II - but that's very far outside of my budget (and it's no laptop :)).

If you search for this discussion online, there's a lot of people who aren't worried, or are worried about things I am not worried about. There's lots of argumentation going into the direction that people should just not use Windows and Chrome (I don't use either) if they are so worried, and that smartphones are so much worse (mine isn't, I run GrapheneOS). I feel like this discussion is misdirected; the real Lenovo backdoors (if there are any) are going to be in the embedded controller firmware onboard the laptop, or in the UEFI code. This is something I see discussed very rarely. The only "comforting" thought here is that I really doubt that ex-filtration of data via the internet would be easy to hide in properly firewalled corporate networks. It would make the news pretty fast, if some corporation figured out weird traffic originating from their fleet of Thinkpads. Maybe someday I will try to reverse engineer a Lenovo firmware blob and attempt to answer this question for myself.

In the end, I decided that it's worth at least checking what they are offering (especially since I was able to get Lenovo store access with ~20% discount). I am not complacent here - I still think that the laptop could be backdoor-ed in a way that's hard to detect. I just chose to live with that thought, since I fail to see clandestine extraction methods that wouldn't be noticed in various cooperate environments (perhaps even by me in my heavily segregated VLANs where I regularly check for unknown traffic).

Anyhow, it turns out, they offer a machine that 100% fits what I am after: The Thinkpad Z13 Gen 2.

I could configure it with 7840U, 64GB RAM, touchscreen, and ANSI-US layout (called "Backlit, Black/Arctic Grey with Fingerprint Reader - English (Euro) "). It was also about 300 Euros cheaper than Dell and HP devices, while offering 64GB RAM.

It took quite a while to arrive in Germany (around two weeks). Most of that time was UPS (they have horrible tracking; it's really insanely bad), and it seemed that the laptop was in stock in my configuration.

The laptop

Here are the exact specs:

H/W path Device Class Description ============================================================== system 21JVCTO1WW (LENOVO_MT_21JV_BU_Think_FM_ThinkPad Z13 Gen 2) /0 bus 21JVCTO1WW /0/0 memory 512KiB L1 cache /0/1 memory 8MiB L2 cache /0/2 memory 16MiB L3 cache /0/3 processor AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics /0/5 memory 64GiB System Memory /0/5/0 memory 16GiB Synchronous Unbuffered (Unregistered) 7500 MHz (0.1 ns) /0/5/1 memory 16GiB Synchronous Unbuffered (Unregistered) 7500 MHz (0.1 ns) /0/5/2 memory 16GiB Synchronous Unbuffered (Unregistered) 7500 MHz (0.1 ns) /0/5/3 memory 16GiB Synchronous Unbuffered (Unregistered) 7500 MHz (0.1 ns) /0/15 memory 128KiB BIOS /0/100 bridge Advanced Micro Devices, Inc. [AMD] /0/100/0.2 generic Advanced Micro Devices, Inc. [AMD] /0/100/2.2 bridge Advanced Micro Devices, Inc. [AMD] /0/100/2.2/0 wlp1s0 network MT7922 802.11ax PCI Express Wireless Network Adapter /0/100/2.4 bridge Advanced Micro Devices, Inc. [AMD] /0/100/2.4/0 /dev/nvme0 storage WD PC SN740 SDDQMQE-2T00-1201 /0/100/2.4/0/0 hwmon2 disk NVMe disk /0/100/2.4/0/2 /dev/ng0n1 disk NVMe disk /0/100/2.4/0/1 /dev/nvme0n1 disk 2048GB NVMe disk /0/100/2.4/0/1/1 volume 511MiB Windows FAT volume /0/100/2.4/0/1/2 /dev/nvme0n1p2 volume 488MiB EFI partition /0/100/2.4/0/1/3 /dev/nvme0n1p3 volume 1906GiB EFI partition /0/100/3.1 bridge Advanced Micro Devices, Inc. [AMD] /0/100/4.1 bridge Advanced Micro Devices, Inc. [AMD] /0/100/8.1 bridge Advanced Micro Devices, Inc. [AMD] /0/100/8.1/0 /dev/fb0 display Phoenix1 /0/100/8.1/0.1 card0 multimedia Rembrandt Radeon High Definition Audio Controller /0/100/8.1/0.1/0 input23 input HD-Audio Generic HDMI/DP,pcm=3 /0/100/8.1/0.1/1 input24 input HD-Audio Generic HDMI/DP,pcm=7 /0/100/8.1/0.1/2 input25 input HD-Audio Generic HDMI/DP,pcm=8 /0/100/8.1/0.1/3 input26 input HD-Audio Generic HDMI/DP,pcm=9 /0/100/8.1/0.2 generic Advanced Micro Devices, Inc. [AMD] /0/100/8.1/0.3 bus Advanced Micro Devices, Inc. [AMD] /0/100/8.1/0.3/0 usb1 bus xHCI Host Controller /0/100/8.1/0.3/0/3 communication Wireless_Device /0/100/8.1/0.3/0/5 generic Generic USB device /0/100/8.1/0.3/1 usb2 bus xHCI Host Controller /0/100/8.1/0.4 bus Advanced Micro Devices, Inc. [AMD] /0/100/8.1/0.4/0 usb3 bus xHCI Host Controller /0/100/8.1/0.4/0/1 input28 multimedia Integrated RGB Camera: Integrat /0/100/8.1/0.4/1 usb4 bus xHCI Host Controller /0/100/8.1/0.5 multimedia ACP/ACP3X/ACP6x Audio Coprocessor /0/100/8.1/0.6 card1 multimedia Family 17h/19h HD Audio Controller /0/100/8.1/0.6/0 input27 input HDA Digital PCBeep /0/100/8.1/0.6/1 input30 input HD-Audio Generic Mic /0/100/8.1/0.6/2 input31 input HD-Audio Generic Headphone /0/100/8.2 bridge Advanced Micro Devices, Inc. [AMD] /0/100/8.2/0 generic Advanced Micro Devices, Inc. [AMD] /0/100/8.2/0.1 generic Advanced Micro Devices, Inc. [AMD] /0/100/8.3 bridge Advanced Micro Devices, Inc. [AMD] /0/100/8.3/0 generic Advanced Micro Devices, Inc. [AMD] /0/100/8.3/0.3 bus Advanced Micro Devices, Inc. [AMD] /0/100/8.3/0.3/0 usb5 bus xHCI Host Controller /0/100/8.3/0.3/1 usb6 bus xHCI Host Controller /0/100/8.3/0.4 bus Advanced Micro Devices, Inc. [AMD] /0/100/8.3/0.4/0 usb7 bus xHCI Host Controller /0/100/8.3/0.4/1 usb8 bus xHCI Host Controller /0/100/8.3/0.5 bus Advanced Micro Devices, Inc. [AMD] /0/100/8.3/0.6 bus Advanced Micro Devices, Inc. [AMD] /0/100/14 bus FCH SMBus Controller /0/100/14.3 bridge FCH LPC Bridge /0/100/14.3/0 system PnP device PNP0c02 /0/100/14.3/1 system PnP device PNP0b00 /0/100/14.3/2 generic PnP device LEN0071 /0/100/14.3/3 generic PnP device LEN0316 /0/100/14.3/4 system PnP device PNP0c02 /0/100/14.3/5 system PnP device PNP0c01 /0/101 bridge Advanced Micro Devices, Inc. [AMD] /0/102 bridge Advanced Micro Devices, Inc. [AMD] /0/103 bridge Advanced Micro Devices, Inc. [AMD] /0/104 bridge Advanced Micro Devices, Inc. [AMD] /0/105 bridge Advanced Micro Devices, Inc. [AMD] /0/106 bridge Advanced Micro Devices, Inc. [AMD] /0/107 bridge Advanced Micro Devices, Inc. [AMD] /0/108 bridge Advanced Micro Devices, Inc. [AMD] /0/109 bridge Advanced Micro Devices, Inc. [AMD] /0/10a bridge Advanced Micro Devices, Inc. [AMD] /0/10b bridge Advanced Micro Devices, Inc. [AMD] /0/10c bridge Advanced Micro Devices, Inc. [AMD] /0/10d bridge Advanced Micro Devices, Inc. [AMD] /1 power 5B10W51881 /2 input0 input AT Translated Set 2 keyboard /3 input15 input Wacom HID 5385 Pen /4 input16 input Wacom HID 5385 Finger /5 input18 input PC Speaker /6 input19 input ThinkPad Extra Buttons /7 input2 input Power Button /8 input20 input SNSL0028:00 2C2F:0028 Mouse /9 input21 input SNSL0028:00 2C2F:0028 Touchpad /a input3 input Lid Switch /b input4 input TPPS/2 Elan TrackPoint /c input5 input Sleep Button /d input6 input Video Bus

(The output here comes from lshw -short -sanitize btw, since I was asked.) It's an absolute beast in a 13.3-inch form factor.

I read quite a few comments on this laptop before deciding to buy it, and many people weren't sure about the keyboard. I can confidently say that I love it. It's quiet, has very good travel (1.35mm on paper, but feels like much more). Very nice feedback, and it is full-sized (it stretches the complete width of the chassis). Also, no weird fn <> CTRL swapping is needed (sorry, old-school ThinkPad fans!). CTRL is where it's supposed to be—in the leftmost pinkie position.

The touchpad was a bit annoying at first (pinching and zooming sometimes weren't working). Having checked libinputs bug tracker for something similar, I found an issue from somebody who had the inverse problem. He found that kernel parameter psmouse.synaptics_intertouch=0 solved his problem. I booted with that option, and yeah. It really does solve all my problems with the touchpad. It is insane that the secondary bus for Synaptics devices is still such a problem. 

If you have this laptop, try it yourself: run libinput debug-events --verbose without the above kernel parameter, and try pinch-zooming a bit. You'll see a lot of times that libinput will detect only a POINTER_MOTION event even though you pinched, while this completely stops with the kernel parameter. I am not sure, but this seems like a bug in the device firmware, which would need to be fixed by Lenovo.

So, how does suspend/resume work now? Wasn't that the main reason for even buying this? To make a long story short, it works absolutely flawlessly on Debian 12 (6.1.0-16-amd64 #1 SMP PREEMPTDYNAMIC Debian 6.1.67-1 (2023-12-12) x8664 GNU/Linux). It instantly suspends and wakes up in about one second. Overnight power draw when suspended is ~4%, which is superb for me.

The only remaining problem for me is that hardware decode of VP9 currently doesn't work correctly - Youtube uses that a lot. There's a Mesa bug logged about this, and at time of writing there exists a preliminary firmware fix for it. Let's hope it finds its way into Debian at some point :). Also, I had a amdgpu hang/reset randomly when opening a webpage. I hope these sort of things get fixed as time goes by for this relatively new platform.

I will keep you informed if any of this changes.

So, a few weeks ago, I started looking for a laptop. I wanted something small-ish, max 14 inch. It shall have a 1080p display, since I can't stand the waste of energy and problems with nonstandard DPI with these high resolution but small displays. I wanted it to have at least DDR4 RAM, since I had some laying around. Also, I personally find Thunderbolt support really important, for those situations where I do want to connect it to my docking station. This also implies that I want USB-C power delivery, which makes the docking situation less painful, and also makes it possible to only carry around a small charger which can at the same time charge my phone. Oh, and I didn't want an Nvidia GPU, since they don't offer open source drivers and Sway doesn't support anything proprietary, which is a good thing. So, ideally Intel, because I never had any problems with i915.

So, with all these criterias set, long story short I settled on a Dell Latitude 7390. The decision was actually easy - I just searched through used laptops on Ebay, while ignoring the Thinkpad series (I cannot stand their swapping of fn and ctrl [I know you can remap it, but still]). Also, HP Elitebooks where way more expensive than Dell Latitudes. I paid 271 EUR for the following configuration:

H/W path Device Class Description ========================================================== system Latitude 7390 (081B) /0 bus 09386V /0/0 memory 64KiB BIOS /0/4d memory 16GiB System Memory /0/4d/0 memory 16GiB SODIMM DDR4 Synchronous Unbuffered (Unregistered) 2400 MHz (0,4 ns) /0/4d/1 memory [empty] /0/51 memory 256KiB L1 cache /0/52 memory 1MiB L2 cache /0/53 memory 6MiB L3 cache /0/54 processor Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz /0/100 bridge Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers /0/100/2 display UHD Graphics 620 /0/100/4 generic Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem /0/100/14 bus Sunrise Point-LP USB 3.0 xHCI Controller /0/100/14/0 usb1 bus xHCI Host Controller /0/100/14/0/4 communication N5321 gw /0/100/14/0/5 multimedia Integrated_Webcam_HD /0/100/14/0/7 communication Bluetooth wireless interface /0/100/14/0/8 input SiW HID Touch Controller /0/100/14/1 usb2 bus xHCI Host Controller /0/100/14.2 generic Sunrise Point-LP Thermal subsystem /0/100/15 generic Sunrise Point-LP Serial IO I2C Controller #0 /0/100/15.1 generic Sunrise Point-LP Serial IO I2C Controller #1 /0/100/15.2 generic Sunrise Point-LP Serial IO I2C Controller #2 /0/100/15.3 generic Sunrise Point-LP Serial IO I2C Controller #3 /0/100/16 communication Sunrise Point-LP CSME HECI #1 /0/100/1c bridge Sunrise Point-LP PCI Express Root Port #1 /0/100/1c/0 generic RTS525A PCI Express Card Reader /0/100/1c.2 bridge Sunrise Point-LP PCI Express Root Port #3 /0/100/1c.2/0 wlp2s0 network Wireless 8265 / 8275 /0/100/1c.4 bridge Sunrise Point-LP PCI Express Root Port #5 /0/100/1d bridge Sunrise Point-LP PCI Express Root Port #9 /0/100/1d/0 storage A2000 NVMe SSD /0/100/1f bridge Sunrise Point LPC Controller/eSPI Controller /0/100/1f.2 memory Memory controller /0/100/1f.3 multimedia Sunrise Point-LP HD Audio /0/100/1f.4 bus Sunrise Point-LP SMBus /0/100/1f.6 enp0s31f6 network Ethernet Connection (4) I219-LM /0/1 system PnP device PNP0c02 /0/2 system PnP device PNP0b00 /0/3 generic PnP device INT3f0d /0/4 input PnP device PNP0303 /0/5 system PnP device PNP0c02 /0/6 system PnP device PNP0c02 /0/7 system PnP device PNP0c02 /0/8 system PnP device PNP0c02 /1 power DELL DM3WC64 /2 wwx0215e0ec0100 network Ethernet interface

So, I got a 8. Gen i5, 16GB RAM, and Thunderbolt - and even a touchscreen. Pretty good deal! And I can report that eveything (Audio, Webcam, Touchscreen, USB ports, Thunderbolt) works with Debian 11 (Kernel 5.10).

Just one problem...

Keyboard swap

The Keyboard was the German QWERTZ layout. I can touch type ANSI-US QWERTY without really caring all too much about the layout, but it still annoyed the hell out of me. So I looked at other pictures of internals of the laptop, with the plan to do a swap of the keyboard. The good news first: yes, it's possible, but you need to take care which keyboard you buy.

You need two things: an upper palmrest, which is one large plastic frame with the correct keyboard cutout, and the keyboard itself.

The palmrest was the hardest part, and I was lucky - only a single seller was selling one. The keyboard seemed easier, loads of ANSI US keyboards on eBay. But I had to try two until I found one that actually works completely, as the Windows/Super key wasn't working on the first one. If you're ever in the same situation, here's the exact model I used:

It's a bit annoying to take it all apart, but be careful with the ribbons (and don't strip any screws!), and you'll be fine.

All in all...

... a very good laptop. Everything works, battery lasts forever and display is great. I can absolutely recommend this setup. It's rugged enough for throwing it into a bag, and beefy enough for compiling code.

Update 2023-10-30

It's been a solid daily driver on the 5.10 kernel. On Debian Bookworm's 6.10 though, suspend started being a problem. The laptop would "freeze" after resume, with the fan spinning, keyboard illumination being on, and the screen being off (no display illumination even). The only thing that "helped" was a hard-reset by power-button. I searched around for a solution for a fairly long time, until I stumbled on this AskUbuntu post.

I can confirm the problem disappears when I boot with snd_hda_intel.dmic_detect=0. /sys/power/mem_sleep was always on deep for me, so I didn't need to override that parameter as suggested additionally in the post. I tried looking in the source what the parameter actually does (link). It seems to disable some DSP firmware probe, which concerned me because I thought it may impact my audio or something, but nope.

Update 2023-10-31 (Happy Helloween!)

Of course, a day after writing an update that the above kernel parameter works, it stopped working. I spent another evening searching for a possible fix. I tried to reproduce the suspend problem by running systemctl suspend in a loop, as well as manually setting system state via echo mem > /sys/power/state to rule out any potential bug in systemd (probably not needed, in hindsight). I managed to reproduce it using this method, in about ~10% of cases. I can confirm that it happens when suspending, and not when waking up.

After some more searching, I found a method which actually, so far, works. I have run 90+ suspend-and-resume iterations now, and it's not a problem anymore.

The fix is yet another kernel parameter: intel_iommu=off. This disables the Intel Virtualization support within the Kernel. I messed a lot with this parameter in the past when I tried out iGPU passthrough within Proxmox, so it certainly rang a bell. I don't plan on using virtualization on this laptop, so it's fine for me - but be aware of this downside if you happen to stumble on these pages with the same problem.

I found a very interesting thread on the kernel bugzilla about this issue: link

I hope I don't have to write another update here :)

Sway

After some years of using GNOME as my primary desktop environment, I tried Sway. You know that feeling when you try something new, and it instantly hits that sweet spot of being both intuitive and feeling right? That's Sway for me. I love how lightweight it is, the fact that it's running on Wayland so multiple displays with different resoultions aren't painful anymore, and the best part: it's highly configurable. I'll upload my config somewhere soon.

I recommend running it together with swaylock and waybar.

SQLite

Whenever I have a situation within some project of mine where I need to save some data on disk, 9/10 times I use sqlite. It's fast, has bindings for every language under the sun, well documented (the documentation is interesting enough to be read like a book, IMHO) and battle tested.

Next time you need to save some data - consider joining the fan club.

fish

The first thing I do whenever I set up a new computer is install the fish shell. I have nothing against bash, in fact you can tune it to have a lot of the features that fish has out of the box, but that's the problem, I am lazy. Fish has a very good command history, and the built-in git support is amazing.

Thunderbird

I love Email in general. Open standard, widely supported, text first. Thunderbird brings a lot to the table - besides e-mail, it can also be used as an RSS reader, which is my primary method of consuming news (...along with a rather large list of RSS filters - never having to read something about Elon Musk again by itself is already worth it).

KDE Okular

Okular is a document viewer. I use it to read (and edit!) PDFs. It is snappy, doesn't annoy me, and is open source (okay, everything on this list is, but still!). Also, did you know that it's the first piece of software which is certified by "Blauer Engel" (a German Eco label) as being "Energy efficient"? Their testing methology looks decent. I bet we won't see any Electron stuff on that list anytime soon (and not on this list either).

Firefox

If this list where sorted by popularity, then Firefox would probably be a lot higher, but it isn't, so it's down here. Firefox is fantastic software. Mozilla, the parent company, wasn't always without missteps. We had the Mr. Robot marketing thing, which was a pretty big WTF moment for me. Or the funny graph about their CEO making more money while Firefox user numbers tanked. But all in all, I feel like they are doing their best in a hypercommercialized world to stay afloat, while building an absolutely amazing piece of very complex technology.

Rclone

I use Rclone for backing up data (using cron for managing backup times). It supports many backends, even though I only use it with sftp. I especially love that I can layer backends. For example, Rclone allows to layer a crypto backend onto an sftp backend, so that any file operation is automatically encrypted. If you're looking for some backup solution - seriously consider Rclone + cron.

In addition to usernames and passwords, all users require client certificate authentication. This means that their browser must send a certificate when connecting and establishing a TLS session. That looks like this, on Firefox/Ubuntu:

If they miss this certificate, their request does not even hit the backend, and instead a 403 error is sent. This allows an additional barrier, as people without this certificate can't even attempt to exploit weaknesses in the services being run. Another nice advantage is that this makes username/password leaks a lot less an issue, since there is not even a way to use stolen credentials without also stealing the certificate. Fortunately, on modern devices such as phones these certificates are stored in a secure hardware enclave, and are incredibly hard to extract, if even at all possible. Certainly not in the realm of a non-state actor. On desktop OSes, the certificates are easier to extract, but that's unfortunately not easily solvable short of a USB or smart card enclave.

Anyhow, in case you're interested in doing the same, the process is pretty simple. The following outlines the process. OpenSSL will ask you some details, just enter what you wish the certificate to say later on. If asked for passwords, make sure they are secure, and don't forget them - specially when it comes to the CA private key password, since you'll need that to sign new certificates!

First - generate a Certificate Authority. This is defined by a public/private keypair and a x509 certificate.

openssl genrsa -des3 -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt

I chose 10 years validity - you may be interested in less, but that's up to you. 4096 bits should be standard for new RSA keypairs. Next, we generate a keypair for the new user.

openssl genrsa -des3 -out USERNAME.key 4096

Following that, we generate a Certificate Signing Request (CSR).

openssl req -new -key USERNAME.key -out USERNAME.csr

Almost done - we now sign this CSR with the CA key you generated earlier.

openssl x509 -req -days 1460 -in USERNAME.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out USERNAME.crt

Here, I chose four years, so 1460 days. This is round about the lifetime that most of my users keep personal devices for, such as phones. We are a wasteful civilization :(. Also - the serial should be incremented each time you sign a new CSR.

Finally - bundle the signed certificate and keypair into a PKCS12 pfx file for easy import.

openssl pkcs12 -export -out USERNAME.pfx -inkey USERNAME.key -in USERNAME.crt -certfile ca.crt

Now, you have both *.pfx and a ca.crt files. The pfx can be imported into a browser or a phone. The ca.crt on the other hand gets used by your webserver. As I said, I use nginx - so we need to configure it properly. You just need to add the following to whatever server block you require the client to be certificate authenticated:

ssl_client_certificate /path/to/ca.crt;
ssl_verify_client optional;

Then, within a location block, add the following conditional:

if ($ssl_client_verify != SUCCESS) {
    return 403;
}

That's it - now you have one more barrier in place to protect services from outside access.

To make this a bit easier for you, here is a bash script. It expects a username and an optional parameter for the number of days the pfx shall be valid for. Default is one year. It makes no attempt to check edge cases though, such as if your CA cert is still valid or if the requested days are realistic.

#!/bin/bash

echo "CA/user certificate generation script"

# generate new CA keys and cert if they don't exist yet
if [ -f "ca.key" ] ; then
        echo "CA cert and keys found, using these for CSR"
else
        echo "Generating new CA private keys and certificate..."
        openssl genrsa -des3 -out ca.key 4096
        openssl req -new -x509 -days 3650 -key ca.key -out ca.crt
fi

if [ -z $2 ] ; then
        userdaysvalid=365
else
        userdaysvalid=$2
fi

username=$1
if [ -z $username ] ; then
        echo "Script expects a single username string as argument! Exiting."
        exit;
else
        echo "Generating pfx for username $username, valid for $userdaysvalid days."
        echo "Generating key..."
        openssl genrsa -des3 -out $username.key 4096
        echo "Generating CSR..."
        openssl req -new -key $username.key -out $username.csr
        echo "Signing the generated CSR..."
        openssl x509 -req -days $userdaysvalid -in $username.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out $username.crt
        echo "Generating pfx for import..."
        openssl pkcs12 -export -out $username.pfx -inkey $username.key -in $username.crt -certfile ca.crt
        echo "Done! You can now import $username.pfx into your cert store. You'll need ca.crt on the server side."
fi

So, I wish to dive into the world of home automation. I wish to avoid the buzzword "smart" as much as possible, so I will refer to it as "connected home" instead. Soon, I will be moving into my first house, and so I set off to research. This post will be updated over the course of the next few weeks, as I get my research/prototyping done.

There are a few points which are very important to me:

• I want to be able to see the current electrical load in Watts on certain wall sockets, and ideally also control their state.
• Analog to the sockets, I want to be able to see and control light state.
• It would be very cool to see which windows are open.
• I have not, do not and will never like the cloud. Whatever I do, I want it locally in my LAN and 100% under my control. No outside communication unless I make it so.
• I want to use only open source components. If I have to crack open some commercial device to reflash firmware via test points, that's fine too.
• I have no bus such as KNX built into my house. I would vastly perfer a bus to any RF technology, but I am not sure I can retrofit it easily. I do have empty tubing in the walls though (one per room, down the the power box). If anybody wants to chat about this, shoot me an email!

There seem to be three basic categories that you can buy into: Zigbee, Z-Wave and WLAN. TL;DR, I chose Z-Wave, and the following is my reasoning.

WLAN

This seems like the worst, so I'll get it out of the way first. There are absolute shitloads of "smart" devices that implement connectivity using off-the-shelf chips such as ESP-8266 or BK7231N. They all have in common that they will happily connect to your WLAN and phone home into the cloud. The allure of these is that even people with absolutely no technical abilities can use them. The downside is... everything else. They are privacy invasive, energy hungry and not under your control. Sometimes, you can flash open source firmware on them (see here as an example for a broad range of devices based off ESP-8266 and derivates), but this will still not solve another hard problem: use a dozen of these devices and you'll rapidly congest your 2.4GHz band. You (and your neighbors) won't like that. Finally, WLAN is energy hungry and it's not feasable to deploy small battery powered sensors, such as temperature or water sensors, that can run for years on a battery. So WLAN is out of the race.

Zigbee

Zigbee is a step into the right direction. It's a non-IP low energy protocol unfortunately based on ISM radio bands, which is 2.4GHz in most of the world. Its layers are defined in IEEE 802.15.4. It's a mesh net, meaning that each mesh member uses other members to reach the gateway, which is the root of the mesh network and actually communicates with the outside. Zigbee devices on their own never communicate directly with any device outside of their mesh. There are many different (often proprietary) gateways, but of course we want the open source solutions. Fortunately, there are cheap USB dongles which work just fine and can work with a wide variety of devices. Zigbee is incredibly cheap, very widespread, and free for anybody to implement. Unfortunately, this also leads to large discrepancies in manufacturers and their implementation. Zigbee provides a means to transport data, and nothing else. The application layer is defined by the manufacturer, which implies a lot of walled gardens. In essence, any Zigbee network by a manufacturer will be exclusive to that one manufacturer. It gets more extreme when people buy into multiple manufacturers, and have multiple mesh networks at the same time, further congesting the 2.4 GHz band. The gateways have to do the heavy lifting here and speak many different protocols for each company's devices, in order to expose them in a consistent manner to the gateway host. One product that claims to be able to do this is the Conbee II gateway, and I have ordered one to test.

Z-Wave

Z-Wave is, like Zigbee, also a mesh network which is composed of a bunch of clients (the sensors/actors and whatnot) and a gateway. Z-Wave is a proprietary standard and defined by Sigma Designs, which was then bought by Silicon Labs. In fact, it has quite a convoluted history, starting out as technology using an unlicenced frequency band in 1999. It really was the wild west back then it seems :). From an "how open is the system?" standpoint, it's unfortunate that Z-Wave only has a single vendor producing chips, even though this could change soon. It feels like vendor lockin, but maybe i'm wrong here and rigirous standardization and gatekeeping makes sense from a product perspective. In order to be granted a licence to use Z-Wave, products have to undergo large amounts of certification for interoperability. The upside is that it's supposedly guaranteed that devices will be interoperable with each other and with whatever gateway you want to operate. This is a sore point to me though, and I only accept it at all because I control the gateway and can verify nothing leaves my local mesh into the direction of the internet. Also - only the certification process is the proprietary part, most of the specifiation is open it seems.

Anyhow, the biggest advantage in my opinion is that it uses a 868 MHz frequency in Europe where I live, so no fighting with my computers over frequencies. The maximum device count for Z-Wave is 232, and this could be seen as a drawback, but I actually doubt I'll ever exceed even 50 devices. And maybe the limit is even good - Z-Wave imposes some limitations, such as hop count to gateway (4 hops maximum, Zigbee does not impose limits here, which could lead to a large RTT). A downside is the increased price - due to certification, manufacturers can't outprice Zigbee devices. This may hurt if you plan on using many sensors/actors.

Z-Wave setup

I went with an Aeotec Z-Stick Gen5 as a gateway, and using Domoticz on my little x86_64 server that's sitting in my pantry, running in a Docker container. Still waiting for sensors to arrive for preliminary testing.

...to be continued...

This post concerns the obsolete 2.7 version of Python, since that's what the target application embeds.

Python bytecode is pretty simple. You have one byte of opcode, followed by two optional bytes of argument (that's what cPython, the reference implementation, calls it). A list of opcodes is contained in opcode.h. Opcodes that are smaller than 90 do not expect any argument, and are thus only one byte in size.

There exist disassemblers and even decompilers for this bytecode, and they work well for standard bytecode that was not messed with. The one I am looking at has two obfusications (that I am aware of at this point) though. Let's look at them.

Insertion of junk instructions to confuse disassemblers

Normally, a disassembler would look at a set of instructions and decode them into some sort of text representation sequentially. For example, take the following bytecode:

6401006400006c00007d0000

This decodes to:

LOAD_CONST      1 (-1)
LOAD_CONST      0 (None)
IMPORT_NAME     0 (sys)
STORE_FAST      0 (sys)

Which is simple enough. The disassembler relies on the bytecode being correct though for a number of factors. For one, it decides how many bytes of argument to read by looking at the integer value of the opcode. If it's under 90, it treats the bytes after the opcode as a new opcode and not as argument data. This means that it's possible to write a program that can stop disassemblers by taking bytecode, inserting junk instructions or random bytes into it at some place, and then recalculate jumps and branches to account for this. The VM does not care much when executing this manipulated bytecode, as long as the obfuscator took care to direct it around the junk using jumps or other means of control flow. But a disassembler that reads code top down without dynamic analysis will run into these junk areas and produce completely wrong output, or even crash.

So, what can we do against this? I decided to write a small program which takes bytecode and recursively traces it, running along all branches and marking code paths which can actually be executed. This is easy because there aren't that many branching opcodes. In fact, these are the ones I identified:

JUMP_FORWARD            110
JUMP_IF_FALSE_OR_POP    111
JUMP_IF_TRUE_OR_POP     112
JUMP_ABSOLUTE           113
POP_JUMP_IF_FALSE       114
POP_JUMP_IF_TRUE        115
FOR_ITER                93

All non executed areas of code then get a nop (0x9), so the disassembler does not stumble over junk bytes anymore. This is simple, but actually does not solve another problem: What happens when the junk is located in an actually executable path? Most of the time, it is located behind a JUMP_ABSOLUTE instruction, and that's easy enough to detect with the static algorithm I wrote. Consider this though:

LOAD_CONST         3
POP_JUMP_IF_TRUE   42 
...junk bytes here...
IMPORT_NAME        1  #address 42

This code pushes the const at index 3 onto the stack. POP_JUMP_IF_TRUE checks the top of the stack, and jumps to the absolute bytecode offset if it is (after popping the top). In this case, as long as the const array has a true value at offset 3 (which is easy to guarantee), it will always jump, thus being equivalent to JUMP_ABSOLUTE. This is something my small static analyzer does not know though currently. I need to teach it to read const values like that and evaluate those jumps, then it can skip those junk instructions too.

Insertion of junk constants and local names

Another thing that this obfuscator does is insert random strings into both the constant and local names. It then goes through the bytecode and increments all LOAD_CONST and LOAD_NAME opcode arguments by the number of junk strings added to the respective arrays. This is a simple but effective obfuscation. Fortunately, it is also simple to revert if you can identify junk locals and global names by some shared attribute. In this case, the junk names include at least one space character.

Open questions

I wonder why they did not obfuscate opcodes, or even change the opcode mapping. That would have been a royal pain, since then I would have needed to figure out what function maps to what integer value. Or, why did they not obfuscate arguments or length of argument bytes? This too would have taken some time to reverse. Writing a junk code insertion routine is a lot more challenging than changing a bit of cPython code. So why did they not go for these low hanging fruit? Maybe I do not know the Python VM enough to understand this. If you know or want to wager a guess, I would be interested to hear.

WoWs uses Python within their engine to program their game logic. These Python scripts are obfusicated, so that programs like decompyle6 will not work out of the box. They actually modified a few functions of cpython to include an obfusication scheme.

First of all, a quick look at the files that come with WoWs reveals that there exist a scripts.zip within a subdirectory. It is a regular zip file and can be opened with a utility such as 7zip. Kind of surprised that they did not mangle the zip format to dissalow this, but that was maybe decided against when developing this feature because it will stop the devs themselves to quickly check their archives.

The archive contains a boatload (pun intended) of pyc files, which are python scripts compiled to a bytecode representation. This format, under normal circumstances, only allows for faster loading of files, since the loader can skip the tokenizing/parsing step and directly load the already processed bytecode for execution. In this case though, decompyle refuses to dissasemble the pyc files.

Time to fire up x64dbg and IDA.

It quickly becomes apparent that WoWs uses zipimport for processing that whole zip file at once.

As far as I can tell, they did not modify anything here, which makes sense because their obfusication regards Python internals. Fortunately for us, we are dealing with a Python function, which is thus open source. We can cheat here a bit by checking out zipimport source code.

Here, the function we are interested in is called zipimporter_load_module. This function loads an entire module and imports it into the current namespace. By placing a breakpoint on the PySys_WriteStderr function and writing a small x64dbg script, we can actually list the modules that are loaded by the game.

bp 0x01CC1425,1
start:
log "{s:[ebp-4]}"
run
goto start

The log tab then contains entires like this:

153.INT3 breakpoint at worldofwarships32.01CC1405 (01CC1405)!
154."mbad083ab"
155.INT3 breakpoint at worldofwarships32.01CC1405 (01CC1405)!
156."package_profiles_helpers"
157.INT3 breakpoint at worldofwarships32.01CC1405 (01CC1405)!
158."GameParams"
159.INT3 breakpoint at worldofwarships32.01CC1405 (01CC1405)!
160."mbd1b28d9"
161.INT3 breakpoint at worldofwarships32.01CC1405 (01CC1405)!
162."m15fb041b"
163.INT3 breakpoint at worldofwarships32.01CC1405 (01CC1405)!
164."ConsumablesConstants"

Which tells us not much, since these are actually just the files that we already saw within the zip, but at least we found an entry point to start digging around.

Upon digging around more, and following the control flow, I stumbled upon the unmarshal_code function from the zipimport cpython module. This looks exactly like the sort of thing we are looking for.

First off, a bit of information about the pyc file structure, it is very simple:

int32   magic;
int32   timestamp;
uint8[] data;

Since pyc files can be empty and still valid, it's not surprising that the function checks the size first:

.text:0104022B 014                 cmp     ebx, 8                  ; ebx contains size of pyc
.text:0104022E 014                 jge     short loc_1040271       ; check (size => 8); jump to loc_1040271 if true
.text:01040230 014                 mov     esi, dword_8F53DD4      ; 
.text:01040236 014                 mov     ecx, offset aBadPycData ; "bad pyc data"
.text:0104023B 014                 call    PyErr_SetString_0       ; document what's wrong using cpython PyErr_SetString

the next two checks are simple too, namely checks for the magic number and the timestamp. The magic number it expects is 0x0A0DF303, which corresponds to python 2.7. You are behind the times, Wargaming :)

Up to this point there where no surprises. Once all the trivial pyc checks are done, the game starts unmarshalling code.

.text:0104033C 014                 lea     edx, [ebx-8]    ; length of code, minus the 8 byte header
.text:0104033F 014                 lea     ecx, [edi+8]    ; marshalled code starts at 8. byte
.text:01040342 014                 call    PyMarshal_ReadObjectFromString

As far as I can see, PyMarshal_ReadObjectFromString does nothing out of the ordinary. It simply wraps r_object call which unmarshals a string value (string being a byte array here). But wait, what is that sneaky call below, a few bytes before ret?

Magic starts happening here. To understand what happens here, we need to look at the structure of a pyc file yet again.

The section I marked data in the struct is marshaled python bytecode. This marshaling is a recursive algorithm that wraps a lot of different data types (strings, longs, bytes, even complex types such as lists and tuples) within one large root code object. This code object has several different sub members. One of it is a section called co_code, which contains the raw bytecode. Another one is co_consts, which contains a list of indexed constants that opcodes such as LOAD_CONST use. Here, wargaming did something strange. It actually misuses the const section of the root code object as a data storage for storing encrypted data. This data then gets XORed bytewise with the content of the code section, using the size of the code section in modulus to prevent overflow.

...
.text:00F3DF76 02C                 cmp     esi, eax             ; exit if we are at end of const section
.text:00F3DF78 02C                 jge     short loc_F3DF8E
.text:00F3DF7A 02C                 mov     eax, esi             ; move index to eax for idiv
.text:00F3DF7C 02C                 mov     ecx, [ebp+var_C]     ; offset to code buffer
.text:00F3DF7F 02C                 cdq                          ; sign extension
.text:00F3DF80 02C                 idiv    ebx                  ; int div for mod operation
.text:00F3DF82 02C                 mov     al, [edx+ecx]        ; edx contains remainder, get next code byte      
.text:00F3DF85 02C                 mov     ecx, [ebp+var_14]    ; get const buffer offset
.text:00F3DF88 02C                 xor     [esi+ecx], al        ; xor code byte with const byte
.text:00F3DF8B 02C                 inc     esi                  ; increment index
.text:00F3DF8C 02C                 jmp     short loc_F3DF30     ; continue for next index
...

If we type this out (using python, how ironic), we get something like this:

f = open(sys.argv[1], "rb") #some pyc passed as parameter
magic = f.read(4)
moddate = f.read(4)
codeobj = marshal.loads(f.read()) #unmarshal the code object
size_code = len(codeobj.co_code)
size_const = len(codeobj.co_consts[3])
encrypted_code = codeobj.co_code[:]
decrypted_const = []
for x in range(size_const):
    decrypted_const.append(ord(codeobj.co_consts[3][x]) ^ ord(encrypted_code[x % size_code]))

After this part is done, we can see a long string, which looks suspiciously like base64. And it actually is! After decoding, we get this:

00000000  78 9C CD 59 0B 78 1C 55 15 3E B3 9B 36 EF F4 41  xœÍY.x.U.>³›6ïôA
00000010  D3 24 6D 69 A7 05 4A 48 9B B6 69 A1 49 6B 5B 4A  Ó$mi§.JH›¶i¡Ik[J
00000020  A9 08 02 05 B6 20 48 29 61 B3 33 9B CC 66 B3 9B  ©...¶ H)a³3›Ìf³›
....

Hold up, i've seen these two first bytes before! Indeed, they are zlib header bytes. They are the CMF and FLG bytes, defined in the standard. CMF means "Compression Method and flags" and describes both the method (such as deflate) and the compression window size. FLG are some flags regarding the compression. In short, 78 9C boil down to the default zlib compression with deflate.

After zlib decompression, we are greated with readable strings, yay!

000005A0  6D 6F 64 75 6C 65 74 08 00 00 00 5F 5F 66 69 6C  modulet....__fil
000005B0  65 5F 5F 74 08 00 00 00 67 43 50 4C 42 78 38 36  e__t....gCPLBx86
000005C0  74 0A 00 00 00 31 36 30 31 36 36 39 33 33 32 69  t....1601669332i
000005D0  00 00 00 00 74 0B 00 00 00 63 6F 6C 6C 65 63 74  ....t....collect
000005E0  69 6F 6E 73 74 09 00 00 00 75 74 66 38 5F 74 65  ionst....utf8_te
000005F0  73 74 74 08 00 00 00 63 6F 70 79 5F 72 65 67 69  stt....copy_regi

Now, lets add the 8 header bytes back to make a valid pyc file and try running the result through decompyle6.

ImportError: Ill-formed bytecode file test.pyc
<class 'KeyError'>; "marshaltype key '{' (dict) not implemented"  

Well that's a bummer. xdis actually did not implement dict unmarshalling, and I don't know why. I need to do more research about Python bytecode, maybe this is the result of more obfusication at some bytecode level. I'll get back here once I found out more.

I have been playing a video game called World of Tanks for quite some time now with my father. For a few weeks now, we have been going back and forth between World of Tanks, and Wargamings newer installment, World of Warships. Unfortunately, you cannot by default launch both game clients at the same time. It's time we fix this.

When you want to implement a restriction like that, there are pretty many options:

• You can create some file on start that gets deleted when your tools exits. Crusty, comes with many problems.
• You can search for the window title of your tool, or the name of your ELF/PE file. Better, but what if a user uses a program with the same name?
• You could use a mutex or an atom, or whatever global named object your OS offers that we can abuse for testing if something already exists.

The last point has many advantages for the mutex. First, a mutex actually gets closed by the system once your application terminates (or crashes), while an atom does not. This automatically prevents a whole class of bugs that would arise if we needed to manage the flags lifecycle ourselves. Second, the implementation is extremely simple, we only need two functions on Win32: CreateMutexW and OpenMutexW. Or the *A variants if you're inclined.

The tool checks for a mutex with a globally known (and unique) name. If it does not already exist, it creates it and continues execution. If it exists, the tool can exit with some sort of nag screen. Simple right?

Well, that's what the Wargaming launcher does. The API to this launcher lives in wgc_api.dll, which is loaded dynamically by every wargaming game. There, it tests for a mutex named wgc_running_games_mtx. Of course, there is lots of wrapping going on here, and the OpenMutexW function is called by 3 levels of indirection. It boils down to this:

.text:251C43C0    push    edi
.text:251C43C1    push    eax
.text:251C43C2    lea     eax, [ebp+var_C]
.text:251C43C5    mov     large fs:0, eax
.text:251C43CB    mov     [ebp+var_10], esp
.text:251C43CE    mov     esi, ecx
.text:251C43D0    cmp     dword ptr [esi+40h], 0
.text:251C43D4    lea     eax, [ebp+var_15]
.text:251C43D7    mov     [ebp+var_4], 0
.text:251C43DE    lea     ecx, [ebp+var_24]
.text:251C43E1    mov     [ebp+var_24], eax
.text:251C43E4    setnz   [ebp+var_15]
.text:251C43E8    lea     eax, [ebp+arg_0]
.text:251C43EB    mov     [ebp+var_1C], esi
.text:251C43EE    push    offset wgc_api.252C5E7C
.text:251C43F3    mov     [ebp+var_20], eax
.text:251C43F6    call    sr_test_mutex   ; nop this and set eax to 0
.text:251C43FB    test    eax, eax
.text:251C43FD    js      short loc_251C4433
.text:251C43FF    cmp     dword ptr [esi+38h], 0
.text:251C4403    jz      short loc_251C4433
.text:251C4405    cmp     dword ptr [esi+3Ch], 8
.text:251C4409    lea     eax, [esi+28h]
.text:251C440C    jb      short loc_251C4410

sr_test_mutex (my naming skills are sick) returns a non zero value if the mutex is found. This value is always static, and represents a static error code they must have definded in some header. Anyhow, we need to survive the test eax, eax. Fortunately, windows is a fan of __cdecl, which means the caller cleans up the stack and we can get away with nuking parametrized calls. So that's what we do.

.text:251C43F6    xor     eax,eax
.text:251C43F8    nop
.text:251C43F9    nop
.text:251C43FA    nop
.text:251C43FB    test    eax, eax
.text:251C43FD    js      wgc_api.251C4433   

Since js tests the sign flag, we want test eax,eax to never set this flag, and this in turn means eax must always be 0. Since we nuked the CALL instruction, we have 5 bytes to set eax to 0. The shortest way to do that is xoring eax with itself, which sets it to 0 by the nature of the xor operation. This consumes two bytes, so the three nops below are just padding and do nothing. That's it, we are done.

Here is a patch file for x64dbg, in case anybody wants it. The wgc_api.dll it works on has a sha256 sum of c9d48273984bc369a19b44b01b671c629acb6ec2886e52231f1854402d559258.

>wgc_api.dll
000043F6:E8->33
000043F7:65->C0
000043F8:00->90
000043F9:00->90
000043FA:00->90

The BEST2 instruction set was initially designed (as far as I can tell) by Softing. It is used by a virtual machine that is part of a runtime environment which can execute automotive diagnostic jobs. These jobs are written by engineers in a C like syntax and compiled to bytecode. The API is fairly rich, and includes functions for communication, large amounts of helper functions for boolean and integer logic, rich output formatting and data manipulation functions, shared memory and a small embedded database that operates on tables. The API functions are partly written directly in BEST2 assembler and linked by the compiler at compiletime.

The VM itself is a stack and register machine. Arguments to functions are passed by registers. There are 8 long (32 bit) registers, 16 string registers and 8 float registers. The long registers are (simularly to x86) stacked in a way that allows easy access to individual bytes/words.

struct bmwBESTVMStackedRegister {
    union {
        uint32 rl;
        struct {
            uint16 rw0;
            uint16 rw1;
        };
        struct {
            uint8 rb0;
            uint8 rb1;
            uint8 rb2;
            uint8 rb3;
        };
    };
};

The string registers are limited to 1024 bytes and initially empty. Operations can and will operate on whatever indices they please though, and the register is expected to grow to the index that was referenced. For example, consider following code:

clear S1
move S1[2], S2[RL1:RL3]

In the first line, we clear S1, which sets all bytes to 0. The length after this instruction is also 0x00. In the second line, we copy some bytes from S2 to S1 begining at offset 2. This means that the first 2 bytes of S1 are now 0x00, and the bytes afterwards are filled by the contents of S2 beginning from the value of RL1 (register long 1) and counting RL3 bytes. This was a bit of a pitfall, since I kind of expected some sort of initialization, but nope.

The VM has multiple flags, but nothing special - sign, carry, overflow and zero are pretty much what you would expect. There is a special one that acts as a trap register for catching or allowing errors, which some jobs use to great extent and which is a bit annoying.

Instructions are of a static length and encoded in two bytes. The first contains an opcode for a instruction, and the second contains information about the addressing mode for the operand. Each addressing mode is encoded in the upper respective lower nibble of the second byte, which effectively limits the operand count to two. Sane choice.

There are 15 operand addressing modes. The first four describe registers (regS, regAB, regI, regL). The next four address immediate values (imm8, imm16, imm32, immStr). The fun starts afterwards, where we get into the register indexing stuff. The indexing addressing scheme only operates on string registers, so if you ever find a long register referenced here, you have a decoding error. I'll address these operand schemes briefly.

IdxImm:

Immediate indexing. Used a lot when parsing xsend responses, which come from the car.

Example: move RS1, RS2[2]

IdxReg:

Indexing using the contents of another register.

Example: move RS1, RS2[RAB2]

IdxRegImm:

This one had me stumped for a bit, since I could not quite imagine why they would implement this. I would guess that it's an optimization for smaller code. Imagine you have a result of some operation in a long register RL2. You wish to use that to index a string register and copy data to RS2, but need to add a constant 3 to it (which is a common operation for processing diagnostic payloads). You would need to execute something like this:

move RL3, RL2
add RL3, 3
move RS2, RS1[RL3]

With IdxRegImm, you can do this:

move RS2, RS1[RL2+3]

This saves a few bytes and speeds things up.

IdxImmLenImm:

Indexing using two constants. The first one is the offset, and the second one is the lenght. This is comparable to a memcpy.

Example: move RS1, RS2[3:4]

This would copy 4 bytes to RS1, beginning at index 3 of RS2.

IdxImmLenReg, IdxRegLenImm, IdxRegLenReg

Same as IdxImmLenImm, but different sources of index.

After decoding the instructions and operands, the instruction is executed. This is the annoying part, as there are many opcodes that need to be implemented. See my VM for most of them, but I ommited a few because I never encountered them in the wild while dealing with Fxx and Exx vehicles.

The shared memory is accessed by shmset and shmget. Both opcodes expect some access key, which enables multiple shared memories at the same time. I have never seen this used though, and instead an empty string is passed. Other than that, shared memory behaves like a string register.

Tables are a central part of the VM, since a lot of data is stored there. This can include simple strings describing portions of a diagnostic payload, up to complex tables detailing how response values are to be manipulated to get a human readable response.

The jobs itself are located in .prg files, which are obfusicated in a very lazy fashion (why even bother). The file format is not very complicated, see my VM for a parser.

I will add more to this overview with time, or whenever somebody asks.

This is too easy. I mean, this time the internet noticed and there was a huge outcry. The problem here centers around distribution and gatekeeping. One of the main arguments Google and other proponents of walled gardens keep bringing up in favor of their stores is security. It's supposed to be harder to distribute malicious code over these stores due to a plethora of automatic analysis that they do. Except this time, it apparently didn't work and code got pushed successfully. If this doesn't work reliably, I'd much rather install my extensions without going over Google (or Mozilla). If the gatekeepers fail, why even bother. Uploading an extension to the Chrome webstore just requires a zip file with the extension code and assets bundled. As far as I know, there isn't even a method to link a repo or commit hash to the submission for verification. So what is it worth to host your extension on github?

Give me an open build service (which I could host myself) which builds extensions, lets me download them in packaged form and then install them manually. Updates can then be optionally enabled by giving the browser a way to automatically check the build service for new releases, and then asking the user for permission to update.

In the future, I'd also like stringent requirements for large extensions. For one, developers should be required to use some sort of U2F scheme to verify that they indeed did author commits. Just a login isn't sufficient here due to how easy it is to transfer.

I wanted one that was minimal, preferrably written in C, and which supports nice markup features along with not sticking Javascript everywhere. While I am sure that there are many such tools out there, my choice fell on blogc because of its incredibly well written documentation, along with its fast execution speed.

The initial port was trivial, as blogc is built pretty straight forward. You define a template, and then call blogc to compile html from this template. It can use plain text files in order to insert content into posts, which allows me to concentrate on the content when writing, instead of thinking about the way it looks later. The plaintext that was used to generate this post, for example, is located here.

Then I wanted to show the last three posts on the index page. First I thought I had run into a problem, since there is no way to limit the number of blocks within the templating language. I emailed the author, Rafael Martins, and he told me about the filtering mechanics that could be used for this purpose. See the Makefile, the magic parameters are FILTER_PER_PAGE and FILTER_PAGE. There is even documentation about these parameters, but I missed those for some reason, shame on me.

Anyhow, quite happy with this setup so far. Might need a few css tweaks, but time will tell. I can't recommend blogc enough.

I am not a fan of buying new phones every year. I have been using a HTC 10 since around 2016, and it has served me well so far thanks to custom roms that prolonged its life. The main problem with old phones is security related though, as most manufacturers simply stop patching their firmware at some point (HTC released the last firmware update for this phone in 2018 [version 2.41.401.41]).

I always kind of ignored the issue, since I convinced myself that my threat model does not involve state actors or drive by exploitation, but rather tracking and fingerprinting. I ran the phone on LineageOS and MicroG in order to get the Google spyware off it - but that has its own problems, namely:

• LineageOS uses userdebug builds, which means it also by default enables a lot of debugging attack surface.
• It exposes the root user via adb, which is exactly as scary as it sounds.
• It requires an unlocked bootloader in order to not brick your phone, which makes threat persistance easier.
• It does not offer any sort of rollback protection and as such it is possible to exploit vulnerabilities that have been dealt with by a certain patch level.

Note that this is not an attack on the guys at LineageOS - but I started wondering what a more ideal solution would look like. Thats when somebody on reddit mentioned GrapheneOS. I wont quote their security model here, but in short they take it very seriously, go check it out. GraphaneOS runs only on Pixel devices, which is the reason I got one a few days ago. I only use fdroid as an appstore, and everything I install must be open source. I'd recommend you do the same, it's worth it. Do not forget to re-lock your bootloader after install!

Ever since I made this website, I hosted it on a VPS for a few bucks a month running lighttpd. It was fast and worked well, but the server itself was for another project which I have since then frozen for the time being. I noticed that quite a few people had their personal websites on github and using their hosting feature for exposing it over https. I decided to try it.

The first thing I had to decide is if this move is going to make any reader that stumbles accross here more suspect to tracking than previously. I am an avid opponent of any sort of online tracking, and as such this website does its absolute best to not do this. No cookies, no serverside logging, just free html pages. Would the move to github hinder this? Do they employ tracking?

Prior to moving, I hosted on OVH. I've been doing this for around ten years, and they where always good to me. I have no way of knowing if they track user access to any of their servers, but for the sake of this argument I have to assume they do. Github also has insights avaliable to me, but it does not resolve IP addresses or any data that may be used to track an individual. They can probably do that with their non public logs, but OVH could too, which means it comes down to which entity you like more.

I breifly considered running the site on my server at home, which is exposed to the internet for purposes of having access to my stuff over ssh. But this comes with its own downsides, namely privacy (it would always expose my public IP), and that my residential connection is not a high availability line.

So, after a bit of consideration I moved to github pages. One less server to maintain, since I am lazy like that. An added bonus is that my commits are now public, and whoever reads this can laugh at my HTML/CSS mistakes.