Peace Memorial Park, Hiroshima

Retro.Know-How

Open-Sourcing old Amiga software

The Yamaha YM3623B Digital Audio Interface Receiver on a MaestroPro sound board After 22 years, I released an update of a software I wrote in the end 1990's for the Amiga platform. It is a driver for the MacroSystem MaestroPro, a fully digital sound board. Along with the update, I have opened the source code of that library.

The sound driver itself is not that interesting, actually. I don't think there are many people on this world still using the sound board. What's more interesting is how I changed the project, to make it open-source and compilable under Linux and other modern platforms. That's what this article is about.

Versioning (or the lack of)

The first problem I faced was rather unexpected. Back in the good old Amiga days, I hadn't used revision control systems like CVS. Being just a hobby developer, I didn't knew about their existence nor their purpose. Instead, I frequently made backups of my source codes, so I wouldn't lose them in case of a harddisk failure, or after a botched code redesign. But besides that, programming was a surgery on the open heart of the source code, with no way to revert to a previous state that was known to be working.

As result, I found multiple different versions of the project on my Amiga harddisk, and I had to figure out what the latest one was. In this project I was lucky, because I had added a changelog to the main file of the source. I just had to find the copy with the most recent changelog.

Without a revision control system, the source of all older releases are lost, so I didn't even try to recreate a history from the backups. The last version on the Aminet was V41.40, but I couldn't find the source of that release any more. What I found instead was a V41.50 that was never released. I cannot remember why I decided not to release that version. Maybe the changes turned out to be a regression? Maybe I just lost interest in the Amiga, and didn't bother to release it any more?

Anyway, I could at least find the latest version of the source code. What to do with it? As I became a professional software developer since then, it was clear to me that I would not go on with just keeping the latest source code version (and some random backups), but I wanted to use a revision control system now.

I prefer to use git today. It would fit nicely into my development environment, and would permit to publish my source codes in my GitHub repository. But git was never ported to the classic Amiga, and probably never will due to its complexity.

Olaf Barthel did a port of subversion though. The last release was in 2009, and bases on a very old Subversion version 1.1.4. It wouldn't be much fun to use it, but it would be feasible.

There is also a CVS port made by Frank Wille, but I never really liked CVS, so this was no option for me.

So svn and git were the only candidates, with a strong preference to use git, but svn as the only option that would work on AmigaOS. The decision was connected to the next question.

Compilation

On what platform do I want to continue developing?

I could go on and develop the project on the Amiga, like I did in the 1990s. I had everything I needed there. I used GoldEd as editor, with customized macros for compiling my projects. I used PhxAss as assembler, and SAS/C as C compiler. None of this software is still maintained, and SAS/C was a commercial product that is not available any more. With these strict requirements, just a few people would be technically able to participate in the project.

Today, Amiga enthusiasts use the vbcc toolchain for development. It is still actively maintained. And it runs on AmigaOS, but also on all major operating systems. As editor, Visual Studio Code is a preferred choice because there is an Amiga Assembly Add-on available. It supports syntax highlighting, inline documentation, debugging, and much more.

These are the missing pieces of the puzzle. With vbcc, it is possible to build the project on Linux and other platforms, so almost every Amiga developer is able to participate. Developing on Linux also enables me to use git and all the other tools I got used to. But with just a few modifications to the makefile, the project could still be built on AmigaOS.

I decided to go the Linux way, but it's a decision that every retro developer has to do for themself. Cross-building an Amiga project on Linux would be comfortable (and fast), but is not really "retro". Building on AmigaOS would be the true retro spirit, but would leave me with an outdated and partially unmaintained toolset.

Porting

It was easy to copy the source files to my Linux file system, and initialize a git project there. The next problem I faced was that I had to port the makefile. It was tailor-made to my AmigaOS environment, with special assigns for include files and binaries.

I created a new makefile that was using env variables instead. AMIGA_NDK now points to the unpacked AmigaOS 3.2 NDK, while AMIGA_INCLUDES points to the include files of external dependencies (like MUI). I installed vbcc so all the commands were in the $PATH.

After that, I restructured and rearranged all the source code files. The project now only contains my own files that are absolutely necessary for building the project. An invocation of make then builds the project on my Linux machine.

I18n

There was an unexpected problem with the charset. While all modern operating systems use UTF-8, AmigaOS does not support it, but uses ISO-8859-1 instead. The result is that the repository contained an awful mix of both charsets. All files that are intended to be used by the git environment (like the README.md file) are stored in UTF-8. Other files that are AmigaOS related (like AmigaGuide files) must be stored in ISO-8859-1 instead.

I was hoping that I could define the correct encoding for each file type in an .editorconfig file. But sadly, Visual Studio Code ignores the charset settings and instead uses UTF-8 by default. It was too easy to accidentally destroy all special characters (like the German Umlaut in my family name) that way.

The only solution I found was to use UTF-8 or ISO-8859-1 only where absolutely necessary, but for most files I used ASCII as the lowest common denominator. An own make target make check checks all files for illegal characters, enforcing the proper usage of the encodings.

Testing

Of course I want to test (and run) the result on AmigaOS, either in UAE, or on a real Amiga.

On UAE, the created files can just be copied straight to the Amiga harddisk directory, and then immediately used in the emulated Amiga.

For the real Amiga, it's a bit more difficult though. One way is to create an ADF disk file using xdftool, and copy the files to it. That ADF file can then be copied to an USB stick, and read in the Amiga using a Gotek floppy drive emulator.

A better way is to use a simple NFS server that is mounted on both the Linux and Amiga machines. Files can be easily exchanged that way. Of couse it requires that the Amiga has a network connection.

Release

Back in the Amiga days, making a release was a fully manual process. For this purpose, I had a separate directory with a release template. I manually copied all compiled files to the proper places of that template, then packed it, and uploaded it to the Aminet.

Now I want the git project to be self-contained, so all the files of the release template are in the distribution directory. The make release target builds the entire project, then creates a fresh release directory, copies all the files to the correct places, and creates an lha package.

On a modern Linux machine, the entire process (from a clean checkout to the distribution package) takes less than a second. 🀩

And that's it. The source of the maestix.library is now open and available at GitHub. The first release that was built on the new environment, can be downloaded from the AmiNet.

CI/CD

You may have laughed now, but it's true: It is possible to do CI/CD with Amiga projects!

vamos is a virtual Amiga runtime environment that permits to run simple Amiga commands on Linux. It's just a CPU and API emulation, not a full-blown emulator like UAE, but it is sufficient to run unit test suites.

There are Docker images like docker4amigavbcc that, for example, permit to automatically build commits using GitLab CI.

And since it's easy to upload new packages to the AmiNet, even Continuous Deployment would be possible. Just create a version tag, and let your CI/CD chain do the rest. πŸ™‚

Altogether, it is possible to develop these retro projects in a state-of-the-art fashion, with a modern IDE, source versioning, platform neutral development, unit tests, and even CI/CD.

Amiga Assigns

AmigaOS was an operating system that was way ahead of its time. It had features that other home operating systems (like Windows or MacOS) were missing back then, like preemptive multitasking. There was also a feature that was called assignments. I truly miss that one on Linux!

An Amiga "assign" is a bit similar to the drive letter you may know from Windows. For example, the Windows path C:\example.txt refers to a file called example.txt in the root directory of the main partition, while the same file on the first floppy drive is called A:\example.txt.

On Amiga, the main partition is usually called DH0:, the second partition is called DH1: and so on, while the first floppy drive is called DF0:. A similar path on the Amiga would thus be DH0:example.txt or DF0:example.txt.

As you can see, on the Amiga a "drive letter" can actually consist of multiple characters and also numbers. This is called an assignment. The name DH0 is assigned to the main harddisk partition.

But wait, there is more!

You can have multiple assigns pointing to the same target. For example, the main partition usually contains the Amiga desktop environment, called Workbench. For this reason, the main partition also has a label like Workbench, and the file could also be accessed as Workbench:example.txt.

This is actually quite a smart concept, especially for exchangeable media. For example, let's imagine we have just started a game called shredzone, and it needs to access a file Music/Opening.mod on its installation disk (AmigaOS uses a slash as file separator, like all proper operating systems). It would open a file called shredzone:Music/Opening.mod.

AmigaOS would see that there is no assignment called shredzone, and would pop up a dialog like this:

From a user's perspective, I would now know that I have to find a medium called shredzone, and insert it into the computer. It does not matter if it's a floppy disk, a CD, or even a network mount. AmigaOS also does not command me to insert the medium into a certain drive. If it's a floppy disk, and I have multiple floppy drives, I can just pick the one I like to. AmigaOS will then detect that a medium with that name was inserted, would close the dialog, and grant the game access to the file.

On Linux, I would need to access that medium under a path like /run/media/shred/shredzone, which is a lot to type, contains my user name, and is harder to remember than just shredzone:.

But wait, there is still more! πŸ˜„

It is easy to add assignments to the system via command line. It's even possible to use subdirectories as assignment target. Let's stay with our shredzone game example. I got tired of having to insert the installation disk every time I want to play that game. So I create a directory called DH2:Games/Shredzone/Files on my hard drive, and I copy all files of that disk to that directory.

After that, I enter this command on the command line:

assign shredzone: DH2:Games/Shredzone/Files

Now, when I start the game, AmigaOS will see that there is a shredzone assign already existing, and will access the files there. So the song file shredzone:Music/Opening.mod would be accessed at DH2:Games/Shredzone/Files/Music/Opening.mod.

AmigaOS makes use of assigns itself. For example, there is a standard assignment called C:. It usually points to the C directory of the booting device, where all command line commands (like dir, copy, delete etc) are expected to be present. This assign is similar to what $PATH is to a Linux shell.

Imagine I have installed a set of development tools, like an assembler and a C compiler. The commands of this toolset can be found at DH1:Development/DevTools/C. On a Linux system, I would add this path to the $PATH env variable, so I can just type the command name to execute one of these commands.

On AmigaOS, I just add this path to an existing assignment:

assign add C: DH1:Development/DevTools/C

Now AmigaOS knows that when I enter a command in the command line, it has to look for it in Workbench:C, and if it's not found there, it will try to find it in DH1:Development/DevTools/C. I could even execute commands like dir C:, and see all files in both directories. Of course, it is not limited to two targets.

There is still more, like deferred assigns. But I only want to give you a general impression of what Amiga assigns are, and why I miss them on Linux.

Multiplication on a Z80 processor

The one thing computers are really good at is calculating. You might now expect that all CPUs are capable of the four basic arithmetic operations, but that isn't the case. The first 8 bit processors were only able to add and subtract numbers, and even the subtraction was performed by adding the negated subtrahend. Multiplication and division instructions first appeared on 16 bit processors, albeit they were still very slow in the first generation.

Simple multiplications and divisions by powers of two can be achieved by shifting a value bitwise to the left or right, respectively. This is, shifting a value by one bit to the left is the same as multiplying it by 2, while shifting by two bits to the left multiplies it by 4, and so on.

But how can we multiply any two numbers? It has to be done step by step, by using basic operations like addition or bit rotation. This article will explain how it works on a Z80 CPU.

Back at school, we have learned to multiply large numbers by long multiplication. Basically, we break up the problem by multiplying the multiplier with each digit of the multiplicand, and then summing the products. For example, if we want to compute the product of 27 and 12, we compute 27Γ—2 = 54 and 27Γ—1 = 27, and then sum the products 57+270 = 324.

  27 Γ— 12
β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”
       54
 +    27βˆ™
β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”
      324

We can use the same algorithm on a computer. But wait, wouldn't we still have to multiply, even if with smaller numbers? Actually, no! Since computers use binary digits, we only need to multiply either by 1, giving the value itself, or by 0, always giving 0.

This is the the same long multiplication of 27 (11011) and 12 (1100) with binary numbers:

  11011 Γ— 1100
β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”
         00000
        00000βˆ™
       11011βˆ™βˆ™
 +    11011βˆ™βˆ™βˆ™
β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”
     101000100

The steps can be executed in a loop. At the beginning, a result register is initialized with zero. If the rightmost bit of the multiplicand is 1, the multiplier is added to the result register. After that, the multiplicand is rotated to the right by one bit, and the multiplier is rotated to the left by one bit. This loop is repeated until the multiplicand is zero, because the result won't change after that anymore.

The following Z80 assembler code example multiplies the values in the BC and DE register pairs, and returns the product in the HL register pair. If an overflow occured during multiplication, the Carry flag will be set.

The multiplicant is kept in the BC register pair. To rotate it one bit to the right, we first use the srl b instruction. It rotates the B register, moving the value of bit 0 to the Carry flag, and inserting a 0 to bit 7, so the multiplicant is filled up with zeros with each rotation. After that, rr c rotates the C register and moves the content of the Carry flag to bit 7. Both instructions combined rotate the BC register pair one bit to the right, insert a 0 to the highest bit. The lowest bit is moved to the Carry flag, where it can be tested.

image/svg+xml0010010110011010Carry0BCCarrysrl brr c7654321076543210

We essentially do the same with the multiplier in the DE register pair, but in the opposite direction. As a rotation to the left by one bit essentially just doubles the value, we also could have used add de,de. Sadly the Z80 does not offer such an instruction.

multiply:	ld	hl, 0		; clear the result register
.loop:		ld	a, b		; is BC == 0?
		or	c		;   (also resets carry flag)
		ret     z		; then we're done!
		srl	b		; logical right shift of BC
		rr	c		; bit 0 goes to carry flag
		jr	nc, .zerobit	; unless bit 0 was 0
		add	hl, de		; add multiplier to result
		ret	c		;   return on overflow
.zerobit:	sla	e		; shift multiplier to the left
		rl	d		;   topmost bit goes to carry flag
		ret	c		;   return on overflow
		jr	.loop		; next iteration

The example only multiplies positive integers. To multiply negative integers, we first need to change all factors to positive numbers and do the multiplication. The result then needs to be negated if one of the factors was negative, but not both.

Reading Amiga Harddisks with Linux

While cleaning up the cellar, I found my Amiga 500 and also a GVP Impact Series II SCSI host adapter. Inside, there was a Fujitsu M2611SA harddisk. After about 25 years, I had totally forgotten about it, and I wondered what was stored on it. So let me take you on the adventure trip of how to salvage old Amiga harddisks on modern Linux machines.

The Amiga ecosystem has always been very SCSI friendly. Commodore broke this tradition only with the final AGA models, where they switched to the IDE bus to reduce costs. The Amiga community never approved this change, and many accelerator cards that were sold for these machines also came with a SCSI host adapter. The SCSI bus was a lot faster than the IDE bus. Also a single ribbon cable could connect up to seven SCSI devices, where the IDE bus only permitted two devices.

Today this SCSI affinity turns out to be a problem though. SCSI was never a topic on consumer PCs, so there are no SCSI-to-USB adapters on the market (I wish they were), and SCSI cards for the PCIe bus are very expensive. I'm still having an Adaptec SCSI card in my cupboard that I bought many years ago, but it is for the old-style PCI bus. Luckily there are PCI-to-PCIe adapters available on the market, so I could reuse this old card in my computer. The card stack looks adventurous, but it will do for a few hours of operation to backup the data.

The big question is: Can a modern Linux machine even read Amiga formatted harddisks?

Mounting Amiga Harddisks

Yes, it can. It seems that there are a lot of Amiga fans among the Linux kernel developers. The Amiga uses a different partition table scheme than PCs, but if you're lucky, your Linux will still detect the Amiga partitions and offer them as e.g. /dev/sdg1. Then all you need to do is to mount the partition via mount.

It didn't work on Fedora though, so I had to do some more typing. First I had to find out the offsets of the individual partitions. GNU Parted can be used for that, as it is able to decode Amiga partition tables:

# parted /dev/sdg
GNU Parted 3.3
Using /dev/sdg
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) u
Unit?  [compact]? b
(parted) p
Model: FUJITSU M2611S (scsi)
Disk /dev/sdg: 45078528B
Sector size (logical/physical): 512B/512B
Partition Table: amiga
Disk Flags:

Number  Start   End        Size       File system  Name  Flags
 1      52224B  45078015B  45025792B  affs1        DH0   boot

(parted) q

So there is only one partition on the HD. It starts at offset 52224 and is Amiga FFS formatted. Luckily most Linux distributions are able to mount this file system out of the box. The start offset is needed to mount the partition. I also mount it read-only to make sure that I won't accidentally change or delete my precious old data.

mount -o ro,offset=52224 -t affs /dev/sdg /mnt/

Et voilΓ :

# ll /mnt/
drwx------. 1 root root    0 Apr 16  1997 C
drwx------. 1 root root    0 Jun 11  1994 Devs
-rw-------. 1 root root 1233 Apr 16  1997 Devs.info
drwx------. 1 root root    0 Apr 16  1997 Fonts
drwx------. 1 root root    0 Apr 16  1997 L
drwx------. 1 root root    0 Apr 16  1997 Libs
drwx------. 1 root root    0 Feb 27  1992 Locale
drwx------. 1 root root    0 Apr 16  1997 Prefs
-rw-------. 1 root root 1238 Apr 16  1997 Prefs.info
drwx------. 1 root root    0 Apr 16  1997 S
drwx------. 1 root root    0 Apr 16  1997 Storage
-rw-------. 1 root root 1233 Apr 16  1997 Storage.info
drwx------. 1 root root    0 Jan  4  1992 System
-rw-------. 1 root root 1233 Apr 16  1997 System.info
drwx------. 1 root root    0 Feb 27  1992 Tools
-rw-------. 1 root root 1233 Apr 16  1997 Tools.info
drwx------. 1 root root    0 Jan  4  1992 Trashcan
-rw-------. 1 root root 1588 Apr 16  1997 Trashcan.info
drwx------. 1 root root    0 Feb  3  1992 Utilities
-rw-------. 1 root root 1233 Apr 16  1997 Utilities.info
drwx------. 1 root root    0 Apr 16  1997 WBStartup
-rw-------. 1 root root 1233 Apr 16  1997 WBStartup.info

Disk Dumps

As old harddisks are quite noisy, it might be a good idea to dump the entire content first, and salvage the partitions later. dd is the classic tool for creating a dump:

dd if=/dev/sdg of=amiga-hd.dd bs=512 status=progress

Later a loop device will simulate a real harddisk device:

losetup /dev/loop1 amiga-hd.dd

/dev/loop1 can now be used for parted and for mount.

To remove the loop device again:

losetup -d /dev/loop1

Smart File System

Back in the Amiga days, the Smart File System was very popular as an alternative to the original Fast File System. It was freeware, it was a lot faster than FFS, and it even had a stateless defragmentation that ran in the background.

The Linux kernel does not support SFS out of the box. However, Marek Szyprowski implemented a kernel module in 2003, which (sadly) never left the experimental stage and thus never found its way into the official set of supported Linux file systems.

To use it, you first need to set up a Linux with a 2.6.27 kernel, for example Fedora 10. After that, download the kernel patch and compile it to a kernel module. If you managed that, you can also mount Amiga SFS partitions. I was able to recover all files from an SFS partition that way, though it wasn't much fun.

PS: Sadly the harddisk I've found didn't contain forgotten source codes or other secrets. It just had a standard Amiga Workbench on it, and a copy of the game Scorched Tanks.

Continue reading...
#Advertisement? This blog is free of ads. All shown products have been paid by myself.
Saturday, January 16, 2021
R Tape loading error, Part 2

In the first part I showed how the Sinclair ZX Spectrum stored data on tape. This second part explains what is stored, and what causes a tape loading error.

The ZX Spectrum BASIC offers a SAVE command for saving all kind of data. It can be used to save a BASIC program, variable arrays, but also arbitrary parts of memory. These files are always saved in two separate blocks. The first block is called header. It contains the file name, data type, and other meta information. The second block follows about a second later and contains the data itself.

The internal structure of each block is identical. The first byte distinguishes between header ($00) and data blocks ($FF). The final byte is a parity checksum. Everything between these two bytes is the payload.

A header block always contains a payload of 17 bytes. The first byte identifies the file type, followed by the file name (10 characters), followed by the length of the data block, and closed by two optional parameters that have different meanings depending on the file type. The length and the two parameters consume two bytes each, with the lower byte coming first because the Z80 CPU is little endian.

This is an example header block of a screenshot:

00$00 = Header
0003$03 = Binary file (Code or SCREEN$)
0153S
0268h
0372r
0465e
0564d
062E.
077Az
086Fo
096En
1065e
11001BLength: 6912 bytes ($1B00)
130040Parameter 1, here: starting address ($4000)
150000Parameter 2, here: unused
20Parity

A screenshot is actually just a memory dump that starts at address $4000 (which is the starting address of the screen buffer) and is exactly 6912 bytes long (the ZX Spectrum has a resolution of 256Γ—192 monochrome pixels plus 32Γ—24 bytes color attributes, giving a screen buffer size of 6912 bytes).

For other file types, the two optional parameters have different meanings. For example, a BASIC program file stores the line number to start at after loading.

The final byte is the parity. It is used for error detection, and computed just by XOR-ing all the bytes that have been read. The result must be $00, otherwise a "R Tape loading error" is reported.

This kind of error detection is rather weak. Due to the nature of the XOR operation, two wrongs give a right. This means that when the block contains an even number of bad bits at the same position, they will be undetected. It is also not possible to correct reading errors, as the XOR operation only allows to identify the position of the bad bit, but not the actual byte that contained the error. More sophisticated error correction algorithms would have slowed down the loading process, though.

The parity is computed as a final step, after all the bytes have been read from the block on tape. For that reason, the loader can only decide at the end of the recording whether the loading was successful or not.

But then, why does the tape loading error sometimes appear while the block is still loading? Well, in the first part I have explained that the loading routine just reads an unknown number of bytes. It ends when waiting for a pulse change took to long. Now, if there is an audio gap on tape, the signal seems to end just in the middle of the block. It is then very likely that the parity checksum is wrong because there are still bytes missing.

Some simple copy protections made use of the way the Spectrum loads data from tape. A very common way were “headerless” files, where the header block was left out and only the data block was recorded on tape. The BASIC LOAD command was unable to read those files because of the missing header.