Dev

Note to myself, but maybe you found this article via search engine… 🙂

Symptome: You are trying to post-process a Canon CR3 RAW image file with Darktable, but it doesn’t show any CR3 files.

Reason: Darktable uses LibRaw for reading CR3 files, which in turn uses exiv2. New file formats are not compiled into Fedora’s exiv2 though, presumably for licensing or patent reasons (see this bug report, tl;dr).

Solution: Uninstall Fedora’s Darktable version (either via dnf remove darktable or flatpak uninstall darktable), and install the flathub version via flatpak install darktable.

If you used the dnf version, you can copy the ~/.config/darktable directory to ~/.var/app/org.darktable.Darktable/darktable to keep your Darktable configuration.

Sometimes you cannot choose your project, but the project chooses you. This one is about sending the status of a Home Connect clothes washer to MQTT using LoRa radio communication.

The project can be found at GitHub.

The Problem

It all started when my clothes washer broke down. I replaced it with a new one, one that is also IoT capable by using the Home Connect network. I liked the idea because the machine is located in a shared laundry room at the basement of the building. If I knew about the progress and the remaining time, I could go to the basement and swap the laundry right on time, not too soon and not too late.

However, my WLAN does not reach the basement, so I couldn’t connect the washer to the Home Connect cloud. I tried PLC, but that made my DSL connection instable, so it wasn’t a solution either. I pondered about buying an LTE router, but the data tariff would cause monthly costs that I wasn’t really willing to pay.

Then I discovered LoRa, which is a radio communication technique that is specially designed for Long Range (hence its name) communication, with a range of up to several kilometers (on optimal conditions). It should be easy for LoRa to send data from the basement to my flat, and indeed, a first test was successful.

LoRa solves the problem of transporting the data. However, it comes with a price: The individual data packages are very small (about 50 bytes worst case), and in Europe there is also a 1% duty cycle restriction that needs to be respected. So it wasn’t possible to just connect the washer to the Home Connect cloud using LoRa as some kind of WLAN repeater.

Instead of that, I would have to connect to the washer directly, read its state and compress the information to what I actually need, before sending it. The problem is now that the connection between the appliance and the Home Connect cloud is proprietary and encrypted.

I found the solution to that problem in a blog post “hacking your dishwasher” by Trammell Hudson. By smart reverse engineering, Trammell was able to find a way to directly connect to his home appliances, without having to go through the Home Connect cloud. This was the last part of the puzzle that I needed.

Concept

With Trammell’s work, I was able to connect to my washer and read its current state. Basically, the washer is sending key-value pairs via JSON, where the key seems to be a 16 bit integer, and the value is mostly also an integer, but could also be a boolean or a string. This information can be easily compressed into small LoRa packages, as I mostly need to transport numeric key-value pairs.

So there is a LoRa “sender” at the basement. It spawns a WLAN access point that the washer connects to. It then communicates with the washer, retrieves its state change events, compresses them, and sends them out via LoRa.

In my flat, a LoRa “receiver” uncompresses the information. From it, JSON bodies are generated and sent to my home automation’s MQTT queue. The generated JSON bodies resemble those sent by Home Connect. A display that is connected to MQTT shows the current progress and the remaining time of the washer. I will also get a message on my phone when the washer is completed, or if an error has occured.

Implementation

For the implementation, I bought two Heltec LoRa32 V2 modules. They are based on an ESP32, with a LoRa module and an OLED on board. With a few modifications to the source, any other Semtech SX1276 based LoRa module can be used. For a proper housing, I created a 3D printed minimal Heltec LoRa32 V2 case.

Thanks to Trammell’s hcpy source code, it was surprisingly simple to write a C++ class for the ESP32 that opens a web socket connection to the washer and starts communicating with it.

As mentioned above, the washer is sending JSON messages that contain mostly integer based key-value pairs. To stuff as much information as possible into a single LoRa packet, I came up with a simple compression. The first byte is stating the type of information, followed by a 16-bit integer key, optionally follwed by the value. These are the possible types:

• 0: Represents the constant 0 (so no value needs to be transported)
• 1: Represents an unsigned 8-bit integer (so the value consumes 1 byte)
• 2: Represents a negative unsigned 8-bit integer (the positive value is transported, and then negated on the receiver side)
• 3,4: The same, but for 16-bit integers (the value consumes 2 bytes)
• 5,6: The same, but for 32-bit integers (the value consumes 4 bytes)
• 7: A boolean constant false (so no value needs to be transported)
• 8: A boolean constant true (so no value needs to be transported)
• 9: A string (followed by the null-terminated string as value)

These key-value pairs are collected until the LoRa package is full or the sender is flushed. A length byte is added that contains the total length of the pairs, so the receiver is able to unpack all of them again.

To secure the communication, a SHA256 based HMAC is generated. A random 16 bit package number is added as well, which is used by the receiver for acknowledgement. Finally, the package is encrypted using AES256.

The receiver side will unencrypt the package and generate an HMAC, using a shared secret. If the HMAC matches, an acknowledge package with the package number is sent back to the sender. After that, the payload is uncompressed and converted to JSON strings that are sent to MQTT.

It is important to know that the transport encryption is not state-of-the-art. There are several sacrifices that had to be made to keep the LoRa transport small and simple:

• Only the first 4 bytes of the MAC are used, for space reasons.
• The RSA256 encryption does not use a mode of operation, mainly because it would be hard to re-synchronize the LoRa connection if a package was lost. On the other hand, we are only sending the washer state. If someone would want to find out whether the washer is running or not, they could just check if a package has been sent within the past minute.
• The transport is not secured against replay attacks. The receiver should provide a random nonce, which is then used by the sender for the next package. This is something that should definitely be addressed.

So the LoRa connection provides an acceptable encryption, and is also protected against lost packages, since the sender will reattempt to send the package if there was no acknowledgement from the receiver.

Configuration

The trickiest part of the project is probably the configuration.

To directly connect to the Home Connect appliance, an encryption key and (depending on the protocol) an initialization vector is required. Both parts cannot be retrieved by the public Home Connect API, but you need to trick the API into thinking that you are connecting from the Home Connect app. This is where Trammell’s hcpy project comes into play. It will let you log into your Home Connect account, and then extract a profile of your appliance and writes it into a config.json file. This file is required for setting up my project.

The config-converter.py in my project will take this config.json file and extract all the necessary parts from it. It will print the appliance’s key and iv values for your sender/config.h. It will also create a new random shared secret for the LoRa encryption. And last but not least, it will create a receiver/mapping.cpp file, which is used to convert the integer keys and values to strings similar to the Home Connect API.

If you came this far, you made the hardest part. After that, the LoRa transceivers need to be configured. Unfortunately the parameters depend on the country where the sender is used, so there are no general default settings.

The following values are proposals and are only valid for countries of the EU. You are responsible to find the correct settings for your country. Failure to do so may result in legal problems, claims for damages, and even imprisonment.

• LORA_BAND: This is the frequency used for LoRa transmissions. For EU countries this is usually 867E6.
• LORA_POWER: The power of the LoRa sender, in dB. For EU countries this must be 14 or less.
• LORA_PABOOST: true for EU countries.
• LORA_SPREADING: The spreading factor. For EU countries, values between 7 and 12 are allowed. Higher values span longer distances, but also exhaust the permitted 1% duty cycle sooner. You should use the lowest possible value that gives a stable LoRa connection, and rather try to enhance reception by finding a better place for the LoRa devices or by using better antennas. The value should be 9 or less, as the duty cycle limit is likely to be exceeded with higher spreading factors.
• LORA_BANDWIDTH: The bandwidth, must be 125E3 in EU countries.
• LORA_SYNCWORD: A sync word. You can choose basically any values, or just use the default 0x12.

Make sure that the sender and the receiver are using the same settings, otherwise the transmission will fail.

The other settings are mainly about the WLAN access point for your appliance, the WLAN settings of your home network, and the credentials to access your MQTT server.

And that’s it! Actually it was quite a fun project, and I learned a lot about ESP32 programming and LoRa networks. I also spent way too much time with it, but maybe it will pay off because I get the laundry done sooner now.

On January 11 2021, Let’s Encrypt will change the default intermediate certificate from the cross-sign IdenTrust DST Root X3 certificate to their own ISRG Root X1 certificate.

The good news: The ISRG certificate is widely trusted by browsers by now, so the transition will be unnoticed by most users.

The bad news: The ISRG certificate is not included in Android devices before “Nougat” 7.1. These devices will show an error when trying to access sites that are signed with the new intermediate certificate. According to Let’s Encrypt, stunning 34% of the Android devices out there shall be affected.

To mitigate the problem, Let’s Encrypt provides an alternate certificate that is still cross-signed with the IdenTrust DST Root X3 certificate. If you have a web service that is accessed by a relevant number of older Android devices, you may want to use that alternate certificate. It will be available until September 29 2021. The IdenTrust DST Root X3 certificate itself will expire after that date, so this is a hard limit. Let’s hope that the problem is going to be solved on Android side in time.

As acme4j fully implements the RFC 8555, it is easy to change your code so it will use the alternate certificate. Based on the acme4j example, this code block will use the first alternate certificate if present, and falls back to the main certificate if not:

Certificate certificate = order.getCertificate();
certificate = certificate.getAlternateCertificates().stream()
        .findFirst()
        .orElse(certificate);

Remember to remove the workaround after September 29 2021 January 2024, so you won’t accidentally use other alternate certificates that may become available in the future.

PS: getAlternateCertificates() was added to the latest acme4j v2.11. If you have an older version, fear not: you just need to have a Login object, so you can bind the alternate certificate yourself. This is how it would look like in the example client:

Login login = session.login(acct.getLocation(), userKeyPair);

Certificate certificate = order.getCertificate();
certificate = certificate.getAlternates().stream()
        .map(login::bindCertificate)
        .findFirst()
        .orElse(certificate);

UPDATE: Let’s Encrypt found a way to extend the Android compatibility until January 2024. However, this extension may only work for Android devices. To quote the article:

The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors.

If you own a Synology NAS, you probably know the trouble that it frequently wakes up from hibernation without any obvious reason. Synology has added a more or less undocumented debugger for that issue. If enabled, it will log the reason for waking up from its beauty sleep, so one can take measures against it.

WARNING: This article is targeted at experienced Linux users. Even if you manage to enable the debug mode, you still have to analyze the log files, find a solution to your individual problem, and assess if it is safe to apply. If you make a mistake, you can cause a lot of damage to your system and your data. Please take this advice seriously.

In any case, make sure your data is properly backed-up. Remember that a RAID is not a backup!

Also note that the debug mechanism has changed from time to time, so it may be completely different on your NAS depending on the DSM version that is used.

You need to log into your NAS via ssh, and become root. (If you struggle at that, better stop here for your own good.)

Open the /etc/synoinfo.conf file, and locate these lines:

enable_hibernation_debug="no"
hibernation_debug_level="1"

To enable the debugger, change enable_hibernation_debug to yes.

The hibernation_debug_level line changes the debug level. The default 1 only logs the reason for your NAS to wake up, which should be sufficient. Level 2 also logs the interrupts that caused the wake-up.

After changing the configuration, invoke this command:

syno_hibernation_debug

The command itself gives no visual feedback, and also does not provide any kind of help or options. However, you can check the /tmp/hibernation.debug file. If the file is present, the debugger was successfully started, and the file contains the PID of the debugger process. If you would run that command again, it would start another debugger instance. It would interfere with hibernation and would also need to be killed manually later, so make sure there is only one debugger running.

All you need to do now is log out and wait for the NAS to wake up by itself. Then log in again, and check the log files /var/log/hibernation.log and /var/log/hibernationFull.log. They contain a hint to the reason of the insomnia.

As there are a lot of possible reasons, fixing the problem is out of scope of this article. There may even be no viable solution.

To turn off the debugger, just change /etc/synoinfo.conf back and invoke syno_hibernation_debug again.

I recently found an article about the AS3935 Franklin Lightning Sensor. As I am already recording some weather data, it immediately raised my interest.

The sensor module can be found at many online shops selling products from China. It is not really cheap, but still affordable. I decided to use an ESP8266 as microcontroller, so I can read the sensor data by WLAN. The sensor is connected to the ESP via SPI. There was also some space left for a SK6812 RGBW LED indicating the sensor status.

The result of this project can be found at GitHub. It’s called Kaminari (which is Japanese for lightning), and also comes with OpenSCAD files for a 3D printed, pyramid shaped case with illuminated tip. In this article I will explain a bit about how I developed the Kaminari firmware.

UPDATE: Meanwhile I have removed the pyramid case from the project. The reason is that in that case, the ESP and the sensor are too close together, so the ESP’s WiFi antenna was massively disturbing the sensor. There is no replacement case.

The first problem was the calibration. The sensor is roughly pre-calibrated, but must be fine-tuned to 500 kHz ±3.5% via the TUN_CAP register. For this purpose, the antenna frequency can be routed to the IRQ pin and then be measured by the ESP. I chose to prescale the frequency by a ratio of 128, giving an IRQ frequency of 3,906.25 Hz. For measurement, I’ve set an IRQ handler that is just counting up a variable on each interrupt. I then reset the counter, wait for 1000 ms, then read the counter, and get the IRQ frequency in Hz units. It’s not 100% accurate, but good enough for this purpose.

The TUN_CAP register offers 16 calibration steps. Just incrementing it until the frequency matches, would take up to 16 seconds. Instead I used an algorithm called successive approximation to find the correct calibration value in only 4 iterations, taking a quarter of the time.

To my disappointment, it turned out that the manufacturer of my module (CJMCU) has used nonstandard components, so my module could only reach a maximum frequency of about 491 kHz. I first suspected that the ESP might be too slow for this kind of measurement, but a scope connected to the IRQ pin confirmed the frequency. Well, it is still within the required tolerance, but it gives a suboptimal tuning result and renders the TUN_CAP register useless.

The next problem is finding a good noise floor level. This is some kind of background radio noise filter. If the level is too low, the sensor cannot operate properly because of interfering background noise. If it is set too high, the lightning detection quality declines.

The noise floor level cannot be calibrated just once at the start. Radio noise sources come and go, may it by turning on an electronic device or just by a change in the weather. I did some experiments, and the most promising solution is a kind of tug-of-war. When the AS3935 detects too much noise, it triggers an interrupt, and the noise floor level is raised to the next higher step. If the last level change was 10 minutes ago, the ESP attempts to reduce the level by one step.

In order to reduce the number of level changes, I have added a counter. Each noise interrupt increments the counter, and every 10 minutes the counter is decremented. The level is raised when the counter reaches 2, and lowered when the counter reaches -2.

Sometimes I noticed a “noise level runaway”, where the AS3935 triggers a lot of noise interrupts in a very short time, raising the noise floor level to its maximum value immediately. To stop that behavior, further noise interrupts are being ignored for one minute after a noise interrupt has been processed.

Now the noise floor level has settled to an average of 95 µVrms here. In the graph, one can see that the level is raised at some time, but then reduced again after a while. One can also see the frequent attempts to lower the level a bit further, immediately followed by a raise back to the average level. It seems that the AS3935 and the ESP have negotiated a good compromise. 😉

The AS3935 seems to be set up in an optimal way now, but I still get some false lightning events from time to time. There are a few registers left to experiment with, namely WDTH (watchdog threshold), SREJ (spike rejection) and MIN_NUM_LIGH (minimum number of lightning). I have raised the watchdog threshold to 2, and did not have a false lightning event since then.

Now I have to wait for some real lightnings… 😄