MSI(-X) interface RFC (was: MSI prototype)

["Chuck Tuffli" <ctuffli@xxxxxxxxx>]

Given the feedback from the MSI prototype, I'd like to propose the
following semantics for bus_alloc_resource() and either define a MSI
specific setup function or modify the parameters to bus_setup_intr().


struct resource *
bus_alloc_resource(device_t dev, int type, int *rid, u_long start,
	                     u_long end, u_long count, u_int flags);

DESCRIPTION
The parameters for bus_alloc_resource() remain the same, and the
proposal serves only to define their meaning in the context of
MSI(-X).

The arguments are as follows:

type is still the type of resource to allocate. The implementation
would add two new types: SYS_RES_MSI and SYS_RES_MSIX that would
respectively allocate MSI or MSI-X vectors.

rid for MSI(-X) will be the value of the first vector allocated if the
function successfully allocates a resource. Drivers need not, but may,
initialize rid prior to calling this function.

start and end define the range of vector values from which to allocate
vectors. Typically drivers will use the default values (0 and ~0) to
specify the entire range of valid vectors, but this interface allows
drivers to ask for vectors in a particular range. As an example, IA
defines that vectors with a higher values have a higher priority.
Unless the driver specifies RF_RANGE_STRICT (see below), the
implementation treats the range as a recommendation and not a
requirement.

count indicates how many vectors to allocate. The implementation will
only allocate contiguous vector values (e.g. 22, 23, 24 ...). If the
driver specifies SYS_RES_MSI, the starting vector value will also be
aligned to the number of vectors requested, rounded up to the nearest
power of two if necessary (e.g. starting vectors may be 4, 8, 12, 16,
. .. if count requests 3 vectors). Note that the value should not
exceed what is advertised in the device's Multiple Message Capable
field in the Message Control for MSI register or the Table Size field
in the Message Control for MSI-X register.

flags sets the flags for the resource. The implementation adds
RF_RANGE_STRICT to indicate that if there are no vectors available in
the range specified by start/end, the implementation should not try to
find vectors outside this range. Also, although it is permissible to
share MSI(-X) vectors, it is typically avoided. Therefore, it is not
recommended to use RF_SHAREABLE with MSI(-X).

RETURN VALUES
Same as before with a couple of comments. There are a number of
reasons the implementation may return NULL in spite of having
unallocated vectors. Some examples include:
- The implementation may not be able to satisfy MSI starting alignment
restrictions. Drivers may be able to recover from this failure by
requesting fewer vectors or increasing the range of allowable vectors.
- Separate from the alignment restrictions, there may not be enough
contiguous vectors to fulfill a request. Drivers may be able to
recover from this failure by requesting fewer vectors or increasing
the range of allowable vectors.
- The driver may have specified a range that does not have enough
available vectors. Drivers may be able to recover from this failure by
increasing the range defined by start/end or not setting the
RF_RANGE_STRICT flag.



Unfortunately, I wasn't as lucky with bus_setup_intr() and couldn't
map the necessary functionality into the current interface. We can
either define a new interface (bus_setup_intr_vec?) that supports the
extra arguments, or we can modify bus_setup_intr(). I don't have a
preference and can see good arguments for both approaches.

int
bus_setup_intr*(device_t dev, struct resource *r, int flags,
                        driver_intr_t handler, void *arg, int vec, int entry,
                        void **cookiep, lwkt_serialize_t serializer);

DESCRIPTION
The majority of the parameters are identical in meaning and function
to their counter parts in bus_setup_intr(). The vec and entry
parameters exist to assign an allocated vector to a table entry. As an
additional benefit, this approach allows vector aliasing (i.e. filling
multiple table entries with the same vector value). Note that the PCI
3.0 spec specifically suggests aliasing as a way to handle vector
shortages.

The arguments are as follows:

vec corresponds to a vector in the range returned by bus_alloc_resource().

entry specifies which slot in the device's vector table to fill. Note
that for SYS_RES_IRQ and SYS_RES_MSI only a value of zero may be used.



Some other notes:
- Drivers are not allowed to modify the values in the Message Control
register including the MSI Enable bit.
- Enabling MSI via bus_setup_intr_vec() will also clear bit 10
(Interrupt Disable) of the Device Control register as well as bit 15
(MSI-X Enable) of the Message Control for MSI-X register. Only one
type of interrupt may be active at any one time.
- Enabling MSI-X via bus_setup_intr_vec() will also clear bit 10
(Interrupt Disable) of the Device Control register as well as bit 0
(MSI Enable) of the Message Control for MSI register. Only one type of
interrupt may be active at any one time.
- Device drivers need to be aware of the memory region associated with
the device's MSI-X table and not map it. The bus driver will take care
of that task.


TIA for your feedback.

---chuck