x86 PICs as devices #1457

ehem · 2024-10-09T01:10:29Z

A major reason behind non-x86 PICs being devices is to utilize the module and newbus infrastructure. This does make x86 the odd one out, so turning the x86 PICs into full devices will likely aid converging the interrupt systems.

Having PICs for all architectures be proper devices allows the interrupt event core use kobj methods for interrupt callbacks. This shrinks the interrupt event data structures as everything can be done via a single pointer, rather than a pointer for each callback.

If interrupt events could be embedded inside the architecture interrupt structures, struct intr_event could be further shrunk by removing another pointer. The result would make the architecture core <=> event core relationship similar to the PIC <=> architecture core relationship.

Note there aren't nearly as many commits as the raw count suggests. With some massaging, the x86 PICs => devices and interrupt event => kobj methods steps can be reordered. I've got both visible here since both work. Likely around half of the commits will disappear in the end as they're duplicates of each other.

ehem · 2024-10-14T20:36:43Z

This has ended up going rather further than I had expected before terminating.

The old Marvell GPIO driver has needed a severe update for some time. It has become so rotten I can no longer work around the problems.

The x86 local-APIC and IO-APIC drivers have been tested. The AT PIC driver may work, but I haven't yet gotten a good recipe for qemu. The impact on x86/Xen and x86/MSI drivers is small enough for them to likely still be working, but they're not yet tested.

ehem · 2024-10-16T23:17:21Z

This is certainly going to need some adjustment, but I need advice as to which direction to go with this. There are 3 places I see where this series could be reasonably terminated.

First, it could be terminated after converting the x86 PICs to devices. This was the main goal. This fails to pick up most of the benefits, but does make the interrupt infrastructures consistent in PICs being devices.

Second, it could terminate after the conversion to int intr_event_create_device(struct intr_event **event, device_t pic, interrupt_t *source, u_int irq, int flags, const char *fmt, ...). This shrinks the structures for hardware interrupts by 24 bytes (3 fewer pointers) and enlarges software interrupts by 16 bytes. Many systems will have >100 hardware interrupts and with 3-7 software interrupts being the maximum, this is a notable benefit. This requires conversion to #include <machine/interrupt.h> in many places.

Third, it could terminate after converting to int intr_event_init(struct intr_event *ie, device_t pic, u_int irq, int flags, const char *fmt, ...). I feel having the PIC <=> arch core interface being similar to the arch core <=> shared interface is a good thing. I've found it rather bizarre how the PIC structure wraps the architecture core, but the architecture core then has a pointer to the shared core. Not having memory allocation in intr_event_init() allows the architectures to initialize their intr_event structures sooner as they can be setup while the appropriate mutex is held. This also reduces memory usage for hardware interrupts by 48 bytes and enlarges software interrupts by 8 bytes.

Overall this seems a rather good idea. Yet now I'm waiting on reviewers...

bsdimp · 2024-10-22T22:41:34Z

This change is too large. It has a lot of gratuitous changes that aren't strictly necessary. 49 changes and 244 files is simply too much to review. A quick survey of the changes shows lots of header file renames (which aren't needed) and kobj function name renaming (also need needed). In it's current form, it's not possible to review this.

ehem · 2024-10-24T00:49:24Z

That is what happens if you look at a squash of 49 commits. Take almost any sequence of 49 commits in FreeBSD's repository, look at the delta and it will be confusing. There is also some infrastructure which has been waiting 2.5 years and had that been brought in in a timely fashion it wouldn't need to be included here.

For some overview of the earlier ones, 5a4854b ended up getting in via other means. It is here as I was expecting it to be brought in with this, but then unexpectedly it came in via other means.

The next 3 are 2 bits of cleanup and one thing I've got an unpleasant suspicion will explode in the not too distant future.

Commit #5 is a key commit. Without that the approach is impossible. Notably x86 presently initializes PICs at SI_SUB_INTR while newbus was doing all initialization at SI_SUB_DRIVERS. Initializing the root bus earlier works fine, but I'm unsure whether that will be accepted.

Advice on the question asked immediately above might allow abandoning some pieces. Might be clearer to point to the primary commit for intr_event_create_device(). The point is the arguments to intr_event_create() are rarely independent. For any given PIC, the functions will always be the same. As such if the interface moves to intr_event there is substantial savings on hardware interrupts since only a single pointer is used.

There isn't a good commit to point to for intr_event_init(), though perhaps f2abf73 can give some idea. I've long been bothered by how INTRNG and x86 PICs wrap their data around the architecture interrupt type, then cast back and forth. Whereas both INTRNG and x86 include a struct intr_event *<prefix>_event; inside their respective structures.

Why is one boundary embedding the structure inside the other, while the other is handled via pointer? These two relationships are similar, so shouldn't their interfaces be similar? Embedding the structure results in 16 bytes of savings since two pointers are removed (pointer to intr_event and there is no longer any need for ->ie_source).

sys/kern/subr_bus.c

sys/sys/bus.h

sys/kern/subr_bus.c

This allows the interrupt framework to introduce hooks into device function tables. Some functionality may be common and the framework may provide standard functionality. Certain PIC tasks may also go through the function tables.

The intr_event structure is a baseline for what all architectures support. This isn't truly separate from the infrastructures and is always allocated. As such merging seems best. Adjust allocation and release as needed for full functionality.

…ndling As the interrupt event interface is friendly to all INTRNG drivers, switch to that. Remove the INTRNG version of the interface as it is now redundant. Yet to be introduced drivers are likely to need updates. Leave base class implementation behind for debugging.

Always allocating `intr_event` means we have a better place to store the interrupt number. Implement using this in all cases.

Now that the value is on the event, there is no need for INTRNG to to duplicate the value on the intr_irqsrc structure.

FreeBSD/ARM is now always INTRNG, therefore these are no longer needed. Simple cleanup which ideally would have been done some time ago.

These were apparently used in the past, but they're no longer referenced. Purge as unused. Simple cleanup.

This check should be impossible to occur. While possibly useful in debug kernels, this is very low value in normal kernels. Simply do an assert instead of checking in non-debug kernels.

Update the local declaration of fbd_driver. The fbd_driver is only used by arm/nvidia/drm2/tegra_host1x.c. If the JETSON-TK1 and TEGRA124 kernel configurations are removed, both of these will be unused.

Yet another instance of kobj not declared in header. As such remove the extern declaration from the main source files.

Global declarations should almost always be in headers. Once that is done, also use the DECLARE_CLASS() macro instead of overt extern. May this help the next person.

This makes KOBJOPLOOKUP() rather more suitable for purposes outside the macros generated by `makeobjops.awk`. The magic variable name "_m" isn't suitable for anything besides the auto-generated files. Whereas returning a function pointer is suitable for explicit lookup on a known parent class, or caching the function pointer in a location outside the lookup cache. Differential Revision: https://reviews.freebsd.org/D48408

Convert large numbers of instances of overt driver_t structures to using the DEFINE_CLASSN() macro. Greater consistency should reduce future errors. Convert several uses of '{ 0, 0 }' driver method list ends to DEVMETHOD_END macro. Small amounts of similar cleanup to similar macros. The opal_sensor module had omitted the sentinal from the table. That had likely been functioning due to dumb luck for some time. Differential Revision: https://reviews.freebsd.org/D47546

Spotted while examing this driver. Remove some end of line spaces/tabs. Replace some spaces which should instead be tabs.

Using kobj allows implementing most PIC functions merely by inheriting from the parent class. As there are multiple OpenPIC implementations, this ensures all common hooks go through by default.

This finishes removing OpenPIC functions from the global namespace. Instead PICs based on OpenPIC should use `openpic_class` for explicitly calling OpenPIC functions. Filter out a few end of line spaces while at it.

The correct behavior for cascading PICs is to pass the parent trapframe. This is partially inspired be lessons learned with af610ca and f750dce.

This allows the interrupt framework to introduce hooks into device function tables. Some functionality may be common and the framework may provide standard functionality. Certain PIC tasks may also go through the function tables. In the case of PowerPC there need to be two classes. The core includes extra functionality not present on other platforms. As such "pic_hw_class" matches the existing use which assumes hardware PICs. In particular the core handles level/edge, low/high and passes those to the PIC. The PIC then has a private area pointer. Whereas other architectures the PIC handles the interrupt parameters for itself. Also include a "pic_base_class" for a future where this was split off.

Create an IPI device for handling IPI interrupts. While modifying the internals of an interrupt event is concerning, this does avoid the present troubles from 0c50edf. ->ie_pic may well replace the architecture counterparts on machine interrupt source structures, at which point the value would be non-removable anyway.

…tructures The intr_event structure is a baseline for what all architectures support. This isn't truly separate from the infrastructures and is always allocated. As such merging seems best. As `intr_event_init()` does not sleep, it can be called while locks are held. This allows cleanup of `intr_lookup()` as the event can be initialized while the lock is held. The removes the lazy event allocation, pushing architectures closer together.

Now that the value is on the event, there is no need for PowerPC to duplicate the value on the powerpc_intr structure.

This was added at b4dbc59 likely in the belief most adb bus devices children of the class. At this point very few classes are children of adb_driver, but at least one is. Finding that was quite difficult. Explicitly mark the kobj public.

The event compatibility implementation is targeted for compatibility for source not converted to devices. Notably architectures being maintained outside the main tree. SWIs can have lower overhead by removing the full compatibility.

Hopefully nothing had ever been modifying the lists of base classes, but now mark them as constant. Something outside of FreeBSD's repository could still explicitly create a writable list, yet this is too dangerous to leave alone. Mostly simple cleanup, unlikely to effect very many things.

There is no mention in the commit message, but the commentted `DEFINE_CLASS_0()` paired with `static driver_t` is unusual. A simple explanation is the original author of db1fda1 lacked a handy means to make the kobj local. In turn this seems pretty strong evidence there has long been a need for PUBLIC/PRIVATE.

Each architecture can convert to kobj interrupt interface independently. This is the final result of all architectures fully implementing use of the kobj interrupt interface.

As all FreeBSD architectures provide a PIC base class, move the declaration to the common header.

ehem · 2025-03-17T19:42:12Z

Presently commit 7018d81 is the end of the x86-centric path. If someone with a commit bit has only a little bit of time, the commits through 169dab8 are pretty quick standalones.

The main course of the interrupt event => kobj method series is fdd1f88. INTRNG runs through 38660f7, PowerPC goes to a909236.

ehem force-pushed the x86picdev branch 6 times, most recently from de5428d to 7ae23df Compare October 14, 2024 20:19

ehem closed this Oct 14, 2024

ehem deleted the x86picdev branch October 14, 2024 20:19

ehem restored the x86picdev branch October 14, 2024 20:19

ehem reopened this Oct 14, 2024

ehem marked this pull request as ready for review October 14, 2024 20:31

ehem requested review from bsdimp and bsdjhb as code owners October 14, 2024 20:31

ehem force-pushed the x86picdev branch 3 times, most recently from c233398 to b7da32e Compare October 31, 2024 05:13