Introduction

Last time I've written about the UnIC project, I described how the T80 core worked for most cases, but not for all: CP/M would boot, games, even SymbOS would work just fine. Yet, PIP hang the system.

Last time I've written about this, it was unclear to me what was going on. I suspected something around interrupts, but I had no real clue as to what the root cause was.

Last time I've written about this, I described how I created special boards to ferret out the problem and how I have found all manners of other issues, yet I didn't get any closer to the original problem.

In this post, I will describe what the problem actually was and how I arrived at the that conclusion. For the impatient: the root cause was a bug in the T80 that allowed NMI handling to start without clearing the interrupt-enable bit. This would allow an interrupt to - well - interrupt the NMI handler, which should not happen. To trigger the problem, very unfortunate timing of the NMI signal was necessary but when it happened, not only the maskable interrupt took precedence over the NMI vector, but due to the confusion that ensued in the T80 core, the NMI handler was never invoked. Thus, the title: this is indeed the mother of all priority inversions.

To get there though was a rough road, probably one of my hardest bugs-hunts. My lab-notes for the process grew to 2000 lines and more than 100k characters. I took me more than two months to get to the bottom of this. Don't worry though, I'm not going to drag you through all that. Just keep in mind: reality was way more complicated then I make it out to be here. With that, let's dive in!

Previously on UnIC

Last time, we left off the project with Shadow Tracer being a big help. It is a combination of an FPGA wrapper and a PCB that allows the T80 core to 'shadow' the execution of the Z80 CPU, while comparing it's visible state with that of the Z80. Any deviation in address, data or control signals can be detected and signaled. These signals can be used to trigger a logic analyzer (also built into the FPGA) to capture the events that led to the disagreement. Such a setup is great for quickly checking cycle-to-cycle compatibility, but as I have discovered in that post, it has its limitations: namely, the FDC controller in the PCW8256 changes the state on its status bit mid-register-read. This is as clear a timing violation as it gets, but there's nothing I can do about that (more on that in a second). The result is that the T80 and the Z80 disagree on the presented bus value and their execution diverges.

Ignoring diverging code-paths

I needed a way to get further progress even in the face of this timing violation. There are a few things that help:

First, the check for this status-bit change always happens in a tight loop, only the T80 wishes to go around it one more time due to it sampling the signal a bit earlier. This means that the state of the two processors should be the same prior to this mismatch.

Second, the T80 can't really influence anything; it's bus signals are not actually connected to the outside world. In practice it means that it is force-fed the instruction stream that the Z80 executes, there's no real divergence in that sense. The PC - the instruction pointer - becomes different, but the instruction stream doesn't. This means that the cores re-converge relatively quickly, at the place of the first absolute branch; which, luckily happens rather quickly for the code in question.

To make further progress, I needed to ignore this brief period of disagreement. I contemplated doing a surgical method of minimal impact, but ended up reaching for the big hammer: simply ignoring all control- or address-signal mismatches. This works of course, but I'm left with only catching data-bus differences. My thinking was that I can go back to a more targeted filter once the runs are clean with regards to data-bus matches.

More async signals

My runs were not though. The next set of problems were this:

What you see here is an NMI hitting the CPU. What should happen is that the CPU to drop what it's doing (by saving the PC to the stack) then jumping to address 0x0066.

What you see is that this is happening, but the Z80 (A[15:0]) is jumping there a few cycles after the T80 (A_i[15:0]) is making the jump.

Side-bar: The PCW8256 series has a weird quirk that is important to understand for what is coming. These machines can connect the FDC interrupt to either the NMI or the INT input of the CPU. Furthermore, the PCW8256 doesn't use DMA for floppy accesses. This means that it needs to capture every single byte in SW as they come off of the floppy controller. This in turn means rather strict timing and quick response requirements. The response times for interrupt and NMI are similar if they are both enabled. However the FDC is not the only source for an interrupt: chiefly, the vertical retrace also generates an interrupts 60 times a second (at least on this side of the pond). While the processor is handling that interrupt, no other interrupt can be services (easily at least). So what happens if the FDC needs urgent attention? I'm guessing this was the reason the engineers decided to connect the FDC interrupt to the NMI input. What decision led to the ability to route the interrupt either to the NMI or the INT line is much more unclear.

I have to stress here: this is an unusual design choice. Z80 systems don't commonly use the NMI line. The ZX spectrum doesn't. The Enterprise 128 didn't. The TRS80 didn't (except for some funky reset). The MSX systems didn't. Heck, even the CPC664 and co didn't. The ZX81 did, but the ZX81 is about as weird as it gets.

Some faffing around showed the reason: the NMI line is asserted asynchronously to the Z80 clock. This of course is another potential setup violation, but one that is way more understandable then the previous one on the status register: it is generally very hard to get external events synchronized to the CPU clock.

Understandable or not, the end result is that - again - the Z80 and T80 disagree on when exactly the assertion occurs.

Similar problems exist in the interrupt line as well. I tried various hacks to get the sampling points lined up with the Z80 to no avail. I couldn't figure out any way of fixing this within the FPGA, which left me with solutions without. In particular: I needed a way to synchronize the falling edge of NMI and INT lines to the CPU clock in such a way that both the Z80 and T80 see them at the same clock cycle. The circuit isn't terribly complicated of course:

The problem is that this needs to be inserted in between the PCW motherboard and Shadow Tracer. The inverter is needed to make sure we capture these signals on the falling edge of the clock, giving plenty of time for the CPU to get a clean signal on the subsequent rising edge.

After building this, I captured its effect on a scope:

Here, I set persistence to infinity. The top trace shows the incoming interrupt line, the middle one shows the outgoing one. The bottom one is the CPU clock. You can see how the outgoing signal is very stable relative to the clock, but the incoming one is all over the place: the synchronizer is both necessary and working.

The moral of the story is this: input signals to Shadow Tracer can be asynchronous and when they are, there are natural mismatches. These signals need to be re-synchronized to some sort of clock before letting them through to either the reference or the replica. Unfortunately there is no universal way of dealing with this; the logic will have to be project specific. It's the price of doing business I guess.

The upshot though is that a similar system can be envisioned for the data-bus as well which would fix the uPD765 misbehaving. This is more involved as the data-bus is bi-directional and it is sampled sometimes on the falling, in others the rising edge of the clock, but can be done.

Strange interrupts

I was hoping that this change would get rid of the mismatches during interrupt handler invocations, but no luck: I still had differences. However, these differences were weird.

Here you see something strange: the marker is set where the T80 processor seems to accept an interrupt: it generates the requisite interrupt acknowledge bus-cycle with M1 and IORQ both asserted. This is in obvious response to the INT line going active a few cycles prior. You can also see the T80 jumping to the interrupt handler address 0x38 later in the trace. The Z80 however? Nothing! It's like the interrupt didn't happen. Or to be more precise, as if the interrupts were disabled. Looking at the execution stream (by matching the address and data-lines during M1 cycles to the disassembly of the memory image), I realized we're in this piece of code:

l1d8ch:
	in a,(000h)		;1d8c	db 00 	. .
	add a,a			;1d8e	87 	.
	jr nc,l1d8ch		;1d8f	30 fb 	0 .
	and 040h		;1d91	e6 40 	. @
	jr z,l1da0h		;1d93	28 0b 	( .
	ini			;1d95	ed a2 	. .
	jr nz,l1d8ch		;1d97	20 f3 	  .
	dec d			;1d99	15 	.

You don't know this, because I've cut out so much of my journey through the wilderness, but this was familiar code. I had seen it before! This is part of the NMI handler. So we should not, under any circumstances take an interrupt here. Interrupts should be disabled.

So the plot thickens. Why would it be possible for the T80 (my replica) to take an interrupt during handling an NMI? That would only be possible if someone intentionally enables interrupts in the NMI handler (using the 'EI' instruction), but I know for a fact that doesn't happen.

The smoking gun

Eventually I chanced upon the following failure:

Here you can see something even more weird then above: an NMI hit, then (at the cursor) both the Z80 and the T80 starts executing the NMI handler by jumping to 0x0066. However, immediately after doing so, the T80 attempts an interrupt handler execution as evidenced by it's address changing to 0x0038 (and failing at it). This should really really not, under any circumstances happen.

But look! What was the instruction interrupted? It's instruction code 0xFB, which is ‘EI'. That is the instruction to enable interrupts. This is spooky: we are just about to enable interrupts when an NMI occurs (we have a pending interrupt already). What should happen is that the NMI to take precedence and the interrupt to wait until after the RETN instruction of the NMI handler. Instead, what the T80 is doing is to interrupt the NMI handler and trying to handle the interrupt instead.

And this is when we arrive at the title of the article: the mother of all priority inversions. A pedestrian interrupt getting higher priority handling then an non-maskable interrupt.

I rigged up my simulation environment with a short test program to verify the behavior. Indeed, the T80 behaved in this peculiar way. So, not only I have a smoking gun, but a repro.

The fix

Of course at this point, finding the problem and implementing the fix wasn't terribly difficult. What was going on was this: the ‘EI' instruction doesn't immediately enable interrupts, it is delayed a little. In effect, interrupts are enabled only after the next instruction. If, during this delay slot an NMI happens, the update of IFF (interrupt status flags) gets screwed up. The NMI gets handled, but the IFF1 flag still gets its delayed update to ‘1', enabling interrupts. This, coupled with a pending interrupt would result in the behavior above. Similar shenanigans happen with the ‘RETN' instruction.

With all this knowledge, the fix was easy: simply block the pending update of the IFF1 flag if an NMI is in the process of being handled. The original code was this:

	if TState = 2 then
		if SetEI = '1' then
			IntE_FF1 <= '1';
			IntE_FF2 <= '1';
		end if;
		if I_RETN = '1' then
			IntE_FF1 <= IntE_FF2;
		end if;
	end if;

The new one:

	if TState = 2 then
		if SetEI = '1' then
			IntE_FF1 <= not NMICycle;
			IntE_FF2 <= '1';
		end if;
		if I_RETN = '1' then
			IntE_FF1 <= IntE_FF2 and not NMICycle;
		end if;
	end if;

This is how several weeks of headaches and hair-loss resolve: in two one-line changes.

But does it work?

Yes, yes it does. I can boot into CP/M with PIP completing it's process perfectly:

If you are wondering whether this picture is the same I've shown before, yes, yes it is. But that's the point of a replica: if done right, you should not be able to tell the difference.

Gazing into the future

I still suspect there to be some differences between the T80 and the Z80 that I would like to track down. I will design a circuit that can re-sample all asynchronous signals (including the data-bus) to completely solve the sampling uncertainties within Shadow Tracer. Then, re-enable all checks, not just data and track down the remaining differences. If I'm lucky, there won't be any. If I'm not, there will be a lot more to write about on these pages.

That though is a different story for a different day.

A word from our sponsor

This project would not have been possible without the services of PCBWay. They are helping me with free PCB manufacturing services for which I'm eternally thankful. Not only that, but their production quality is great for my purposes and their turn-around times are fantastic.

They do way more than just PCB manufacturing. They can make stencils for you, even manage the whole PCBA assembly process. They have 3D printers, injection molding or CNC machines as well to complete your project. They provide quick online price estimators to gauge your expenses. If you are here, you are probably tinkering with something at home like I do, so check them out!