Skip to content

Commit ccd5d1b

Browse files
torvaldstorvalds
committed
Merge tag 'ntb-4.13' of git://github.com/jonmason/ntb
Pull NTB updates from Jon Mason: "The major change in the series is a rework of the NTB infrastructure to all for IDT hardware to be supported (and resulting fallout from that). There are also a few clean-ups, etc. New IDT NTB driver and changes to the NTB infrastructure to allow for this different kind of NTB HW, some style fixes (per Greg KH recommendation), and some ntb_test tweaks" * tag 'ntb-4.13' of git://github.com/jonmason/ntb: ntb_netdev: set the net_device's parent ntb: Add error path/handling to Debug FS entry creation ntb: Add more debugfs support for ntb_perf testing options ntb: Remove debug-fs variables from the context structure ntb: Add a module option to control affinity of DMA channels NTB: Add IDT 89HPESxNTx PCIe-switches support ntb_hw_intel: Style fixes: open code macros that just obfuscate code ntb_hw_amd: Style fixes: open code macros that just obfuscate code NTB: Add ntb.h comments NTB: Add PCIe Gen4 link speed NTB: Add new Memory Windows API documentation NTB: Add Messaging NTB API NTB: Alter Scratchpads API to support multi-ports devices NTB: Alter MW API to support multi-ports devices NTB: Alter link-state API to support multi-port devices NTB: Add indexed ports NTB API NTB: Make link-state API being declared first NTB: ntb_test: add parameter for doorbell bitmask NTB: ntb_test: modprobe on remote host
2 parents 4d25ec1 + 854b1dd commit ccd5d1b

20 files changed

Lines changed: 5120 additions & 360 deletions

File tree

‎Documentation/ntb.txt‎

Lines changed: 91 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
11
# NTB Drivers
22

33
NTB (Non-Transparent Bridge) is a type of PCI-Express bridge chip that connects
4-
the separate memory systems of two computers to the same PCI-Express fabric.
5-
Existing NTB hardware supports a common feature set, including scratchpad
6-
registers, doorbell registers, and memory translation windows. Scratchpad
7-
registers are read-and-writable registers that are accessible from either side
8-
of the device, so that peers can exchange a small amount of information at a
9-
fixed address. Doorbell registers provide a way for peers to send interrupt
10-
events. Memory windows allow translated read and write access to the peer
11-
memory.
4+
the separate memory systems of two or more computers to the same PCI-Express
5+
fabric. Existing NTB hardware supports a common feature set: doorbell
6+
registers and memory translation windows, as well as non common features like
7+
scratchpad and message registers. Scratchpad registers are read-and-writable
8+
registers that are accessible from either side of the device, so that peers can
9+
exchange a small amount of information at a fixed address. Message registers can
10+
be utilized for the same purpose. Additionally they are provided with with
11+
special status bits to make sure the information isn't rewritten by another
12+
peer. Doorbell registers provide a way for peers to send interrupt events.
13+
Memory windows allow translated read and write access to the peer memory.
1214

1315
## NTB Core Driver (ntb)
1416

@@ -26,6 +28,87 @@ as ntb hardware, or hardware drivers, are inserted and removed. The
2628
registration uses the Linux Device framework, so it should feel familiar to
2729
anyone who has written a pci driver.
2830

31+
### NTB Typical client driver implementation
32+
33+
Primary purpose of NTB is to share some peace of memory between at least two
34+
systems. So the NTB device features like Scratchpad/Message registers are
35+
mainly used to perform the proper memory window initialization. Typically
36+
there are two types of memory window interfaces supported by the NTB API:
37+
inbound translation configured on the local ntb port and outbound translation
38+
configured by the peer, on the peer ntb port. The first type is
39+
depicted on the next figure
40+
41+
Inbound translation:
42+
Memory: Local NTB Port: Peer NTB Port: Peer MMIO:
43+
____________
44+
| dma-mapped |-ntb_mw_set_trans(addr) |
45+
| memory | _v____________ | ______________
46+
| (addr) |<======| MW xlat addr |<====| MW base addr |<== memory-mapped IO
47+
|------------| |--------------| | |--------------|
48+
49+
So typical scenario of the first type memory window initialization looks:
50+
1) allocate a memory region, 2) put translated address to NTB config,
51+
3) somehow notify a peer device of performed initialization, 4) peer device
52+
maps corresponding outbound memory window so to have access to the shared
53+
memory region.
54+
55+
The second type of interface, that implies the shared windows being
56+
initialized by a peer device, is depicted on the figure:
57+
58+
Outbound translation:
59+
Memory: Local NTB Port: Peer NTB Port: Peer MMIO:
60+
____________ ______________
61+
| dma-mapped | | | MW base addr |<== memory-mapped IO
62+
| memory | | |--------------|
63+
| (addr) |<===================| MW xlat addr |<-ntb_peer_mw_set_trans(addr)
64+
|------------| | |--------------|
65+
66+
Typical scenario of the second type interface initialization would be:
67+
1) allocate a memory region, 2) somehow deliver a translated address to a peer
68+
device, 3) peer puts the translated address to NTB config, 4) peer device maps
69+
outbound memory window so to have access to the shared memory region.
70+
71+
As one can see the described scenarios can be combined in one portable
72+
algorithm.
73+
Local device:
74+
1) Allocate memory for a shared window
75+
2) Initialize memory window by translated address of the allocated region
76+
(it may fail if local memory window initialization is unsupported)
77+
3) Send the translated address and memory window index to a peer device
78+
Peer device:
79+
1) Initialize memory window with retrieved address of the allocated
80+
by another device memory region (it may fail if peer memory window
81+
initialization is unsupported)
82+
2) Map outbound memory window
83+
84+
In accordance with this scenario, the NTB Memory Window API can be used as
85+
follows:
86+
Local device:
87+
1) ntb_mw_count(pidx) - retrieve number of memory ranges, which can
88+
be allocated for memory windows between local device and peer device
89+
of port with specified index.
90+
2) ntb_get_align(pidx, midx) - retrieve parameters restricting the
91+
shared memory region alignment and size. Then memory can be properly
92+
allocated.
93+
3) Allocate physically contiguous memory region in compliance with
94+
restrictions retrieved in 2).
95+
4) ntb_mw_set_trans(pidx, midx) - try to set translation address of
96+
the memory window with specified index for the defined peer device
97+
(it may fail if local translated address setting is not supported)
98+
5) Send translated base address (usually together with memory window
99+
number) to the peer device using, for instance, scratchpad or message
100+
registers.
101+
Peer device:
102+
1) ntb_peer_mw_set_trans(pidx, midx) - try to set received from other
103+
device (related to pidx) translated address for specified memory
104+
window. It may fail if retrieved address, for instance, exceeds
105+
maximum possible address or isn't properly aligned.
106+
2) ntb_peer_mw_get_addr(widx) - retrieve MMIO address to map the memory
107+
window so to have an access to the shared memory.
108+
109+
Also it is worth to note, that method ntb_mw_count(pidx) should return the
110+
same value as ntb_peer_mw_count() on the peer with port index - pidx.
111+
29112
### NTB Transport Client (ntb\_transport) and NTB Netdev (ntb\_netdev)
30113

31114
The primary client for NTB is the Transport client, used in tandem with NTB

‎MAINTAINERS‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9381,6 +9381,12 @@ F: include/linux/ntb.h
93819381
F: include/linux/ntb_transport.h
93829382
F: tools/testing/selftests/ntb/
93839383

9384+
NTB IDT DRIVER
9385+
M: Serge Semin <[email protected]>
9386+
L: [email protected]
9387+
S: Supported
9388+
F: drivers/ntb/hw/idt/
9389+
93849390
NTB INTEL DRIVER
93859391
M: Jon Mason <[email protected]>
93869392
M: Dave Jiang <[email protected]>

‎drivers/net/ntb_netdev.c‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -418,6 +418,8 @@ static int ntb_netdev_probe(struct device *client_dev)
418418
if (!ndev)
419419
return -ENOMEM;
420420

421+
SET_NETDEV_DEV(ndev, client_dev);
422+
421423
dev = netdev_priv(ndev);
422424
dev->ndev = ndev;
423425
dev->pdev = pdev;

‎drivers/ntb/hw/Kconfig‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
source "drivers/ntb/hw/amd/Kconfig"
2+
source "drivers/ntb/hw/idt/Kconfig"
23
source "drivers/ntb/hw/intel/Kconfig"

‎drivers/ntb/hw/Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
obj-$(CONFIG_NTB_AMD) += amd/
2+
obj-$(CONFIG_NTB_IDT) += idt/
23
obj-$(CONFIG_NTB_INTEL) += intel/

0 commit comments

Comments
 (0)