Matt Dillon noticed in a thread on the freebsd-hackers mailing list (see here for original post, and look for articles following that have [CHECKER] in the subject) that the M_NOWAIT command is being used incorrectly in a number of places, both in DragonFly and in FreeBSD.

I’m just pasting the rest of his post, as it’s not something that boils down easily:

“In a nutshell, most of the M_NOWAIT uses are incorrect. In particular, much of the CAM code runs from an interrupt or a software interrupt and almost universally uses M_NOWAIT for its allocations, with direconsequences if/when allocations fail.

I think I’ve come up with a solution! The reason M_NOWAIT is used is primarily because an interrupt thread may be preempting a normal thread and cannot safely manipulate ‘cache’ pages in the VM page queues. This is because reusing a cache page requires messing around with the VM Object the cache page resides in. So M_NOWAIT causes kmem_malloc() to only pull pages out of the VM ‘free’ page queue. At the same time it is allowed to actually exhaust the free page queue whereas normal allocations are not allowed to completely exhaust the free page queue.

But in DragonFly it is utterly trivial to (A) test to see if we are preempting someone and (B) to switch away and when we get scheduled again we will no longer be preempting anyone and can use a more exhaustive allocation algorithm.

This means that in DragonFly we do not have to key interrupt-time allocations off of M_NOWAIT. Instead we can simply test to see if we are preempting someone and if we are only allocate from the VM page ‘free’ queue… and if that allocation fails all we need to do is lwkt_switch() away, and when we get cpu back we will no longer be preempting anyone and can safely access the other queues.

This in turn (whew) means that with very few changes to the low level kmem_malloc() code we can allow interrupt code to use M_WAITOK for something like 90% of the cases where it currently uses M_NOWAIT.”