rxrpc.rst

   1.. SPDX-License-Identifier: GPL-2.0
   2
   3======================
   4RxRPC Network Protocol
   5======================
   6
   7The RxRPC protocol driver provides a reliable two-phase transport on top of UDP
   8that can be used to perform RxRPC remote operations.  This is done over sockets
   9of AF_RXRPC family, using sendmsg() and recvmsg() with control data to send and
  10receive data, aborts and errors.
  11
  12Contents of this document:
  13
  14 (#) Overview.
  15
  16 (#) RxRPC protocol summary.
  17
  18 (#) AF_RXRPC driver model.
  19
  20 (#) Control messages.
  21
  22 (#) Socket options.
  23
  24 (#) Security.
  25
  26 (#) Example client usage.
  27
  28 (#) Example server usage.
  29
  30 (#) AF_RXRPC kernel interface.
  31
  32 (#) Configurable parameters.
  33
  34
  35Overview
  36========
  37
  38RxRPC is a two-layer protocol.  There is a session layer which provides
  39reliable virtual connections using UDP over IPv4 (or IPv6) as the transport
  40layer, but implements a real network protocol; and there's the presentation
  41layer which renders structured data to binary blobs and back again using XDR
  42(as does SunRPC)::
  43
  44		+-------------+
  45		| Application |
  46		+-------------+
  47		|     XDR     |		Presentation
  48		+-------------+
  49		|    RxRPC    |		Session
  50		+-------------+
  51		|     UDP     |		Transport
  52		+-------------+
  53
  54
  55AF_RXRPC provides:
  56
  57 (1) Part of an RxRPC facility for both kernel and userspace applications by
  58     making the session part of it a Linux network protocol (AF_RXRPC).
  59
  60 (2) A two-phase protocol.  The client transmits a blob (the request) and then
  61     receives a blob (the reply), and the server receives the request and then
  62     transmits the reply.
  63
  64 (3) Retention of the reusable bits of the transport system set up for one call
  65     to speed up subsequent calls.
  66
  67 (4) A secure protocol, using the Linux kernel's key retention facility to
  68     manage security on the client end.  The server end must of necessity be
  69     more active in security negotiations.
  70
  71AF_RXRPC does not provide XDR marshalling/presentation facilities.  That is
  72left to the application.  AF_RXRPC only deals in blobs.  Even the operation ID
  73is just the first four bytes of the request blob, and as such is beyond the
  74kernel's interest.
  75
  76
  77Sockets of AF_RXRPC family are:
  78
  79 (1) created as type SOCK_DGRAM;
  80
  81 (2) provided with a protocol of the type of underlying transport they're going
  82     to use - currently only PF_INET is supported.
  83
  84
  85The Andrew File System (AFS) is an example of an application that uses this and
  86that has both kernel (filesystem) and userspace (utility) components.
  87
  88
  89RxRPC Protocol Summary
  90======================
  91
  92An overview of the RxRPC protocol:
  93
  94 (#) RxRPC sits on top of another networking protocol (UDP is the only option
  95     currently), and uses this to provide network transport.  UDP ports, for
  96     example, provide transport endpoints.
  97
  98 (#) RxRPC supports multiple virtual "connections" from any given transport
  99     endpoint, thus allowing the endpoints to be shared, even to the same
 100     remote endpoint.
 101
 102 (#) Each connection goes to a particular "service".  A connection may not go
 103     to multiple services.  A service may be considered the RxRPC equivalent of
 104     a port number.  AF_RXRPC permits multiple services to share an endpoint.
 105
 106 (#) Client-originating packets are marked, thus a transport endpoint can be
 107     shared between client and server connections (connections have a
 108     direction).
 109
 110 (#) Up to a billion connections may be supported concurrently between one
 111     local transport endpoint and one service on one remote endpoint.  An RxRPC
 112     connection is described by seven numbers::
 113
 114	Local address	}
 115	Local port	} Transport (UDP) address
 116	Remote address	}
 117	Remote port	}
 118	Direction
 119	Connection ID
 120	Service ID
 121
 122 (#) Each RxRPC operation is a "call".  A connection may make up to four
 123     billion calls, but only up to four calls may be in progress on a
 124     connection at any one time.
 125
 126 (#) Calls are two-phase and asymmetric: the client sends its request data,
 127     which the service receives; then the service transmits the reply data
 128     which the client receives.
 129
 130 (#) The data blobs are of indefinite size, the end of a phase is marked with a
 131     flag in the packet.  The number of packets of data making up one blob may
 132     not exceed 4 billion, however, as this would cause the sequence number to
 133     wrap.
 134
 135 (#) The first four bytes of the request data are the service operation ID.
 136
 137 (#) Security is negotiated on a per-connection basis.  The connection is
 138     initiated by the first data packet on it arriving.  If security is
 139     requested, the server then issues a "challenge" and then the client
 140     replies with a "response".  If the response is successful, the security is
 141     set for the lifetime of that connection, and all subsequent calls made
 142     upon it use that same security.  In the event that the server lets a
 143     connection lapse before the client, the security will be renegotiated if
 144     the client uses the connection again.
 145
 146 (#) Calls use ACK packets to handle reliability.  Data packets are also
 147     explicitly sequenced per call.
 148
 149 (#) There are two types of positive acknowledgment: hard-ACKs and soft-ACKs.
 150     A hard-ACK indicates to the far side that all the data received to a point
 151     has been received and processed; a soft-ACK indicates that the data has
 152     been received but may yet be discarded and re-requested.  The sender may
 153     not discard any transmittable packets until they've been hard-ACK'd.
 154
 155 (#) Reception of a reply data packet implicitly hard-ACK's all the data
 156     packets that make up the request.
 157
 158 (#) An call is complete when the request has been sent, the reply has been
 159     received and the final hard-ACK on the last packet of the reply has
 160     reached the server.
 161
 162 (#) An call may be aborted by either end at any time up to its completion.
 163
 164
 165AF_RXRPC Driver Model
 166=====================
 167
 168About the AF_RXRPC driver:
 169
 170 (#) The AF_RXRPC protocol transparently uses internal sockets of the transport
 171     protocol to represent transport endpoints.
 172
 173 (#) AF_RXRPC sockets map onto RxRPC connection bundles.  Actual RxRPC
 174     connections are handled transparently.  One client socket may be used to
 175     make multiple simultaneous calls to the same service.  One server socket
 176     may handle calls from many clients.
 177
 178 (#) Additional parallel client connections will be initiated to support extra
 179     concurrent calls, up to a tunable limit.
 180
 181 (#) Each connection is retained for a certain amount of time [tunable] after
 182     the last call currently using it has completed in case a new call is made
 183     that could reuse it.
 184
 185 (#) Each internal UDP socket is retained [tunable] for a certain amount of
 186     time [tunable] after the last connection using it discarded, in case a new
 187     connection is made that could use it.
 188
 189 (#) A client-side connection is only shared between calls if they have
 190     the same key struct describing their security (and assuming the calls
 191     would otherwise share the connection).  Non-secured calls would also be
 192     able to share connections with each other.
 193
 194 (#) A server-side connection is shared if the client says it is.
 195
 196 (#) ACK'ing is handled by the protocol driver automatically, including ping
 197     replying.
 198
 199 (#) SO_KEEPALIVE automatically pings the other side to keep the connection
 200     alive [TODO].
 201
 202 (#) If an ICMP error is received, all calls affected by that error will be
 203     aborted with an appropriate network error passed through recvmsg().
 204
 205
 206Interaction with the user of the RxRPC socket:
 207
 208 (#) A socket is made into a server socket by binding an address with a
 209     non-zero service ID.
 210
 211 (#) In the client, sending a request is achieved with one or more sendmsgs,
 212     followed by the reply being received with one or more recvmsgs.
 213
 214 (#) The first sendmsg for a request to be sent from a client contains a tag to
 215     be used in all other sendmsgs or recvmsgs associated with that call.  The
 216     tag is carried in the control data.
 217
 218 (#) connect() is used to supply a default destination address for a client
 219     socket.  This may be overridden by supplying an alternate address to the
 220     first sendmsg() of a call (struct msghdr::msg_name).
 221
 222 (#) If connect() is called on an unbound client, a random local port will
 223     bound before the operation takes place.
 224
 225 (#) A server socket may also be used to make client calls.  To do this, the
 226     first sendmsg() of the call must specify the target address.  The server's
 227     transport endpoint is used to send the packets.
 228
 229 (#) Once the application has received the last message associated with a call,
 230     the tag is guaranteed not to be seen again, and so it can be used to pin
 231     client resources.  A new call can then be initiated with the same tag
 232     without fear of interference.
 233
 234 (#) In the server, a request is received with one or more recvmsgs, then the
 235     the reply is transmitted with one or more sendmsgs, and then the final ACK
 236     is received with a last recvmsg.
 237
 238 (#) When sending data for a call, sendmsg is given MSG_MORE if there's more
 239     data to come on that call.
 240
 241 (#) When receiving data for a call, recvmsg flags MSG_MORE if there's more
 242     data to come for that call.
 243
 244 (#) When receiving data or messages for a call, MSG_EOR is flagged by recvmsg
 245     to indicate the terminal message for that call.
 246
 247 (#) A call may be aborted by adding an abort control message to the control
 248     data.  Issuing an abort terminates the kernel's use of that call's tag.
 249     Any messages waiting in the receive queue for that call will be discarded.
 250
 251 (#) Aborts, busy notifications and challenge packets are delivered by recvmsg,
 252     and control data messages will be set to indicate the context.  Receiving
 253     an abort or a busy message terminates the kernel's use of that call's tag.
 254
 255 (#) The control data part of the msghdr struct is used for a number of things:
 256
 257     (#) The tag of the intended or affected call.
 258
 259     (#) Sending or receiving errors, aborts and busy notifications.
 260
 261     (#) Notifications of incoming calls.
 262
 263     (#) Sending debug requests and receiving debug replies [TODO].
 264
 265 (#) When the kernel has received and set up an incoming call, it sends a
 266     message to server application to let it know there's a new call awaiting
 267     its acceptance [recvmsg reports a special control message].  The server
 268     application then uses sendmsg to assign a tag to the new call.  Once that
 269     is done, the first part of the request data will be delivered by recvmsg.
 270
 271 (#) The server application has to provide the server socket with a keyring of
 272     secret keys corresponding to the security types it permits.  When a secure
 273     connection is being set up, the kernel looks up the appropriate secret key
 274     in the keyring and then sends a challenge packet to the client and
 275     receives a response packet.  The kernel then checks the authorisation of
 276     the packet and either aborts the connection or sets up the security.
 277
 278 (#) The name of the key a client will use to secure its communications is
 279     nominated by a socket option.
 280
 281
 282Notes on sendmsg:
 283
 284 (#) MSG_WAITALL can be set to tell sendmsg to ignore signals if the peer is
 285     making progress at accepting packets within a reasonable time such that we
 286     manage to queue up all the data for transmission.  This requires the
 287     client to accept at least one packet per 2*RTT time period.
 288
 289     If this isn't set, sendmsg() will return immediately, either returning
 290     EINTR/ERESTARTSYS if nothing was consumed or returning the amount of data
 291     consumed.
 292
 293
 294Notes on recvmsg:
 295
 296 (#) If there's a sequence of data messages belonging to a particular call on
 297     the receive queue, then recvmsg will keep working through them until:
 298
 299     (a) it meets the end of that call's received data,
 300
 301     (b) it meets a non-data message,
 302
 303     (c) it meets a message belonging to a different call, or
 304
 305     (d) it fills the user buffer.
 306
 307     If recvmsg is called in blocking mode, it will keep sleeping, awaiting the
 308     reception of further data, until one of the above four conditions is met.
 309
 310 (2) MSG_PEEK operates similarly, but will return immediately if it has put any
 311     data in the buffer rather than sleeping until it can fill the buffer.
 312
 313 (3) If a data message is only partially consumed in filling a user buffer,
 314     then the remainder of that message will be left on the front of the queue
 315     for the next taker.  MSG_TRUNC will never be flagged.
 316
 317 (4) If there is more data to be had on a call (it hasn't copied the last byte
 318     of the last data message in that phase yet), then MSG_MORE will be
 319     flagged.
 320
 321
 322Control Messages
 323================
 324
 325AF_RXRPC makes use of control messages in sendmsg() and recvmsg() to multiplex
 326calls, to invoke certain actions and to report certain conditions.  These are:
 327
 328	=======================	=== ===========	===============================
 329	MESSAGE ID		SRT DATA	MEANING
 330	=======================	=== ===========	===============================
 331	RXRPC_USER_CALL_ID	sr- User ID	App's call specifier
 332	RXRPC_ABORT		srt Abort code	Abort code to issue/received
 333	RXRPC_ACK		-rt n/a		Final ACK received
 334	RXRPC_NET_ERROR		-rt error num	Network error on call
 335	RXRPC_BUSY		-rt n/a		Call rejected (server busy)
 336	RXRPC_LOCAL_ERROR	-rt error num	Local error encountered
 337	RXRPC_NEW_CALL		-r- n/a		New call received
 338	RXRPC_ACCEPT		s-- n/a		Accept new call
 339	RXRPC_EXCLUSIVE_CALL	s-- n/a		Make an exclusive client call
 340	RXRPC_UPGRADE_SERVICE	s-- n/a		Client call can be upgraded
 341	RXRPC_TX_LENGTH		s-- data len	Total length of Tx data
 342	=======================	=== ===========	===============================
 343
 344	(SRT = usable in Sendmsg / delivered by Recvmsg / Terminal message)
 345
 346 (#) RXRPC_USER_CALL_ID
 347
 348     This is used to indicate the application's call ID.  It's an unsigned long
 349     that the app specifies in the client by attaching it to the first data
 350     message or in the server by passing it in association with an RXRPC_ACCEPT
 351     message.  recvmsg() passes it in conjunction with all messages except
 352     those of the RXRPC_NEW_CALL message.
 353
 354 (#) RXRPC_ABORT
 355
 356     This is can be used by an application to abort a call by passing it to
 357     sendmsg, or it can be delivered by recvmsg to indicate a remote abort was
 358     received.  Either way, it must be associated with an RXRPC_USER_CALL_ID to
 359     specify the call affected.  If an abort is being sent, then error EBADSLT
 360     will be returned if there is no call with that user ID.
 361
 362 (#) RXRPC_ACK
 363
 364     This is delivered to a server application to indicate that the final ACK
 365     of a call was received from the client.  It will be associated with an
 366     RXRPC_USER_CALL_ID to indicate the call that's now complete.
 367
 368 (#) RXRPC_NET_ERROR
 369
 370     This is delivered to an application to indicate that an ICMP error message
 371     was encountered in the process of trying to talk to the peer.  An
 372     errno-class integer value will be included in the control message data
 373     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
 374     affected.
 375
 376 (#) RXRPC_BUSY
 377
 378     This is delivered to a client application to indicate that a call was
 379     rejected by the server due to the server being busy.  It will be
 380     associated with an RXRPC_USER_CALL_ID to indicate the rejected call.
 381
 382 (#) RXRPC_LOCAL_ERROR
 383
 384     This is delivered to an application to indicate that a local error was
 385     encountered and that a call has been aborted because of it.  An
 386     errno-class integer value will be included in the control message data
 387     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
 388     affected.
 389
 390 (#) RXRPC_NEW_CALL
 391
 392     This is delivered to indicate to a server application that a new call has
 393     arrived and is awaiting acceptance.  No user ID is associated with this,
 394     as a user ID must subsequently be assigned by doing an RXRPC_ACCEPT.
 395
 396 (#) RXRPC_ACCEPT
 397
 398     This is used by a server application to attempt to accept a call and
 399     assign it a user ID.  It should be associated with an RXRPC_USER_CALL_ID
 400     to indicate the user ID to be assigned.  If there is no call to be
 401     accepted (it may have timed out, been aborted, etc.), then sendmsg will
 402     return error ENODATA.  If the user ID is already in use by another call,
 403     then error EBADSLT will be returned.
 404
 405 (#) RXRPC_EXCLUSIVE_CALL
 406
 407     This is used to indicate that a client call should be made on a one-off
 408     connection.  The connection is discarded once the call has terminated.
 409
 410 (#) RXRPC_UPGRADE_SERVICE
 411
 412     This is used to make a client call to probe if the specified service ID
 413     may be upgraded by the server.  The caller must check msg_name returned to
 414     recvmsg() for the service ID actually in use.  The operation probed must
 415     be one that takes the same arguments in both services.
 416
 417     Once this has been used to establish the upgrade capability (or lack
 418     thereof) of the server, the service ID returned should be used for all
 419     future communication to that server and RXRPC_UPGRADE_SERVICE should no
 420     longer be set.
 421
 422 (#) RXRPC_TX_LENGTH
 423
 424     This is used to inform the kernel of the total amount of data that is
 425     going to be transmitted by a call (whether in a client request or a
 426     service response).  If given, it allows the kernel to encrypt from the
 427     userspace buffer directly to the packet buffers, rather than copying into
 428     the buffer and then encrypting in place.  This may only be given with the
 429     first sendmsg() providing data for a call.  EMSGSIZE will be generated if
 430     the amount of data actually given is different.
 431
 432     This takes a parameter of __s64 type that indicates how much will be
 433     transmitted.  This may not be less than zero.
 434
 435The symbol RXRPC__SUPPORTED is defined as one more than the highest control
 436message type supported.  At run time this can be queried by means of the
 437RXRPC_SUPPORTED_CMSG socket option (see below).
 438
 439
 440Socket Options
 441==============
 442
 443AF_RXRPC sockets support a few socket options at the SOL_RXRPC level:
 444
 445 (#) RXRPC_SECURITY_KEY
 446
 447     This is used to specify the description of the key to be used.  The key is
 448     extracted from the calling process's keyrings with request_key() and
 449     should be of "rxrpc" type.
 450
 451     The optval pointer points to the description string, and optlen indicates
 452     how long the string is, without the NUL terminator.
 453
 454 (#) RXRPC_SECURITY_KEYRING
 455
 456     Similar to above but specifies a keyring of server secret keys to use (key
 457     type "keyring").  See the "Security" section.
 458
 459 (#) RXRPC_EXCLUSIVE_CONNECTION
 460
 461     This is used to request that new connections should be used for each call
 462     made subsequently on this socket.  optval should be NULL and optlen 0.
 463
 464 (#) RXRPC_MIN_SECURITY_LEVEL
 465
 466     This is used to specify the minimum security level required for calls on
 467     this socket.  optval must point to an int containing one of the following
 468     values:
 469
 470     (a) RXRPC_SECURITY_PLAIN
 471
 472	 Encrypted checksum only.
 473
 474     (b) RXRPC_SECURITY_AUTH
 475
 476	 Encrypted checksum plus packet padded and first eight bytes of packet
 477	 encrypted - which includes the actual packet length.
 478
 479     (c) RXRPC_SECURITY_ENCRYPT
 480
 481	 Encrypted checksum plus entire packet padded and encrypted, including
 482	 actual packet length.
 483
 484 (#) RXRPC_UPGRADEABLE_SERVICE
 485
 486     This is used to indicate that a service socket with two bindings may
 487     upgrade one bound service to the other if requested by the client.  optval
 488     must point to an array of two unsigned short ints.  The first is the
 489     service ID to upgrade from and the second the service ID to upgrade to.
 490
 491 (#) RXRPC_SUPPORTED_CMSG
 492
 493     This is a read-only option that writes an int into the buffer indicating
 494     the highest control message type supported.
 495
 496
 497Security
 498========
 499
 500Currently, only the kerberos 4 equivalent protocol has been implemented
 501(security index 2 - rxkad).  This requires the rxkad module to be loaded and,
 502on the client, tickets of the appropriate type to be obtained from the AFS
 503kaserver or the kerberos server and installed as "rxrpc" type keys.  This is
 504normally done using the klog program.  An example simple klog program can be
 505found at:
 506
 507	http://people.redhat.com/~dhowells/rxrpc/klog.c
 508
 509The payload provided to add_key() on the client should be of the following
 510form::
 511
 512	struct rxrpc_key_sec2_v1 {
 513		uint16_t	security_index;	/* 2 */
 514		uint16_t	ticket_length;	/* length of ticket[] */
 515		uint32_t	expiry;		/* time at which expires */
 516		uint8_t		kvno;		/* key version number */
 517		uint8_t		__pad[3];
 518		uint8_t		session_key[8];	/* DES session key */
 519		uint8_t		ticket[0];	/* the encrypted ticket */
 520	};
 521
 522Where the ticket blob is just appended to the above structure.
 523
 524
 525For the server, keys of type "rxrpc_s" must be made available to the server.
 526They have a description of "<serviceID>:<securityIndex>" (eg: "52:2" for an
 527rxkad key for the AFS VL service).  When such a key is created, it should be
 528given the server's secret key as the instantiation data (see the example
 529below).
 530
 531	add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
 532
 533A keyring is passed to the server socket by naming it in a sockopt.  The server
 534socket then looks the server secret keys up in this keyring when secure
 535incoming connections are made.  This can be seen in an example program that can
 536be found at:
 537
 538	http://people.redhat.com/~dhowells/rxrpc/listen.c
 539
 540
 541Example Client Usage
 542====================
 543
 544A client would issue an operation by:
 545
 546 (1) An RxRPC socket is set up by::
 547
 548	client = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
 549
 550     Where the third parameter indicates the protocol family of the transport
 551     socket used - usually IPv4 but it can also be IPv6 [TODO].
 552
 553 (2) A local address can optionally be bound::
 554
 555	struct sockaddr_rxrpc srx = {
 556		.srx_family	= AF_RXRPC,
 557		.srx_service	= 0,  /* we're a client */
 558		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
 559		.transport.sin_family	= AF_INET,
 560		.transport.sin_port	= htons(7000), /* AFS callback */
 561		.transport.sin_address	= 0,  /* all local interfaces */
 562	};
 563	bind(client, &srx, sizeof(srx));
 564
 565     This specifies the local UDP port to be used.  If not given, a random
 566     non-privileged port will be used.  A UDP port may be shared between
 567     several unrelated RxRPC sockets.  Security is handled on a basis of
 568     per-RxRPC virtual connection.
 569
 570 (3) The security is set::
 571
 572	const char *key = "AFS:cambridge.redhat.com";
 573	setsockopt(client, SOL_RXRPC, RXRPC_SECURITY_KEY, key, strlen(key));
 574
 575     This issues a request_key() to get the key representing the security
 576     context.  The minimum security level can be set::
 577
 578	unsigned int sec = RXRPC_SECURITY_ENCRYPT;
 579	setsockopt(client, SOL_RXRPC, RXRPC_MIN_SECURITY_LEVEL,
 580		   &sec, sizeof(sec));
 581
 582 (4) The server to be contacted can then be specified (alternatively this can
 583     be done through sendmsg)::
 584
 585	struct sockaddr_rxrpc srx = {
 586		.srx_family	= AF_RXRPC,
 587		.srx_service	= VL_SERVICE_ID,
 588		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
 589		.transport.sin_family	= AF_INET,
 590		.transport.sin_port	= htons(7005), /* AFS volume manager */
 591		.transport.sin_address	= ...,
 592	};
 593	connect(client, &srx, sizeof(srx));
 594
 595 (5) The request data should then be posted to the server socket using a series
 596     of sendmsg() calls, each with the following control message attached:
 597
 598	==================	===================================
 599	RXRPC_USER_CALL_ID	specifies the user ID for this call
 600	==================	===================================
 601
 602     MSG_MORE should be set in msghdr::msg_flags on all but the last part of
 603     the request.  Multiple requests may be made simultaneously.
 604
 605     An RXRPC_TX_LENGTH control message can also be specified on the first
 606     sendmsg() call.
 607
 608     If a call is intended to go to a destination other than the default
 609     specified through connect(), then msghdr::msg_name should be set on the
 610     first request message of that call.
 611
 612 (6) The reply data will then be posted to the server socket for recvmsg() to
 613     pick up.  MSG_MORE will be flagged by recvmsg() if there's more reply data
 614     for a particular call to be read.  MSG_EOR will be set on the terminal
 615     read for a call.
 616
 617     All data will be delivered with the following control message attached:
 618
 619	RXRPC_USER_CALL_ID	- specifies the user ID for this call
 620
 621     If an abort or error occurred, this will be returned in the control data
 622     buffer instead, and MSG_EOR will be flagged to indicate the end of that
 623     call.
 624
 625A client may ask for a service ID it knows and ask that this be upgraded to a
 626better service if one is available by supplying RXRPC_UPGRADE_SERVICE on the
 627first sendmsg() of a call.  The client should then check srx_service in the
 628msg_name filled in by recvmsg() when collecting the result.  srx_service will
 629hold the same value as given to sendmsg() if the upgrade request was ignored by
 630the service - otherwise it will be altered to indicate the service ID the
 631server upgraded to.  Note that the upgraded service ID is chosen by the server.
 632The caller has to wait until it sees the service ID in the reply before sending
 633any more calls (further calls to the same destination will be blocked until the
 634probe is concluded).
 635
 636
 637Example Server Usage
 638====================
 639
 640A server would be set up to accept operations in the following manner:
 641
 642 (1) An RxRPC socket is created by::
 643
 644	server = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
 645
 646     Where the third parameter indicates the address type of the transport
 647     socket used - usually IPv4.
 648
 649 (2) Security is set up if desired by giving the socket a keyring with server
 650     secret keys in it::
 651
 652	keyring = add_key("keyring", "AFSkeys", NULL, 0,
 653			  KEY_SPEC_PROCESS_KEYRING);
 654
 655	const char secret_key[8] = {
 656		0xa7, 0x83, 0x8a, 0xcb, 0xc7, 0x83, 0xec, 0x94 };
 657	add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
 658
 659	setsockopt(server, SOL_RXRPC, RXRPC_SECURITY_KEYRING, "AFSkeys", 7);
 660
 661     The keyring can be manipulated after it has been given to the socket. This
 662     permits the server to add more keys, replace keys, etc. while it is live.
 663
 664 (3) A local address must then be bound::
 665
 666	struct sockaddr_rxrpc srx = {
 667		.srx_family	= AF_RXRPC,
 668		.srx_service	= VL_SERVICE_ID, /* RxRPC service ID */
 669		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
 670		.transport.sin_family	= AF_INET,
 671		.transport.sin_port	= htons(7000), /* AFS callback */
 672		.transport.sin_address	= 0,  /* all local interfaces */
 673	};
 674	bind(server, &srx, sizeof(srx));
 675
 676     More than one service ID may be bound to a socket, provided the transport
 677     parameters are the same.  The limit is currently two.  To do this, bind()
 678     should be called twice.
 679
 680 (4) If service upgrading is required, first two service IDs must have been
 681     bound and then the following option must be set::
 682
 683	unsigned short service_ids[2] = { from_ID, to_ID };
 684	setsockopt(server, SOL_RXRPC, RXRPC_UPGRADEABLE_SERVICE,
 685		   service_ids, sizeof(service_ids));
 686
 687     This will automatically upgrade connections on service from_ID to service
 688     to_ID if they request it.  This will be reflected in msg_name obtained
 689     through recvmsg() when the request data is delivered to userspace.
 690
 691 (5) The server is then set to listen out for incoming calls::
 692
 693	listen(server, 100);
 694
 695 (6) The kernel notifies the server of pending incoming connections by sending
 696     it a message for each.  This is received with recvmsg() on the server
 697     socket.  It has no data, and has a single dataless control message
 698     attached::
 699
 700	RXRPC_NEW_CALL
 701
 702     The address that can be passed back by recvmsg() at this point should be
 703     ignored since the call for which the message was posted may have gone by
 704     the time it is accepted - in which case the first call still on the queue
 705     will be accepted.
 706
 707 (7) The server then accepts the new call by issuing a sendmsg() with two
 708     pieces of control data and no actual data:
 709
 710	==================	==============================
 711	RXRPC_ACCEPT		indicate connection acceptance
 712	RXRPC_USER_CALL_ID	specify user ID for this call
 713	==================	==============================
 714
 715 (8) The first request data packet will then be posted to the server socket for
 716     recvmsg() to pick up.  At that point, the RxRPC address for the call can
 717     be read from the address fields in the msghdr struct.
 718
 719     Subsequent request data will be posted to the server socket for recvmsg()
 720     to collect as it arrives.  All but the last piece of the request data will
 721     be delivered with MSG_MORE flagged.
 722
 723     All data will be delivered with the following control message attached:
 724
 725
 726	==================	===================================
 727	RXRPC_USER_CALL_ID	specifies the user ID for this call
 728	==================	===================================
 729
 730 (9) The reply data should then be posted to the server socket using a series
 731     of sendmsg() calls, each with the following control messages attached:
 732
 733	==================	===================================
 734	RXRPC_USER_CALL_ID	specifies the user ID for this call
 735	==================	===================================
 736
 737     MSG_MORE should be set in msghdr::msg_flags on all but the last message
 738     for a particular call.
 739
 740(10) The final ACK from the client will be posted for retrieval by recvmsg()
 741     when it is received.  It will take the form of a dataless message with two
 742     control messages attached:
 743
 744	==================	===================================
 745	RXRPC_USER_CALL_ID	specifies the user ID for this call
 746	RXRPC_ACK		indicates final ACK (no data)
 747	==================	===================================
 748
 749     MSG_EOR will be flagged to indicate that this is the final message for
 750     this call.
 751
 752(11) Up to the point the final packet of reply data is sent, the call can be
 753     aborted by calling sendmsg() with a dataless message with the following
 754     control messages attached:
 755
 756	==================	===================================
 757	RXRPC_USER_CALL_ID	specifies the user ID for this call
 758	RXRPC_ABORT		indicates abort code (4 byte data)
 759	==================	===================================
 760
 761     Any packets waiting in the socket's receive queue will be discarded if
 762     this is issued.
 763
 764Note that all the communications for a particular service take place through
 765the one server socket, using control messages on sendmsg() and recvmsg() to
 766determine the call affected.
 767
 768
 769AF_RXRPC Kernel Interface
 770=========================
 771
 772The AF_RXRPC module also provides an interface for use by in-kernel utilities
 773such as the AFS filesystem.  This permits such a utility to:
 774
 775 (1) Use different keys directly on individual client calls on one socket
 776     rather than having to open a whole slew of sockets, one for each key it
 777     might want to use.
 778
 779 (2) Avoid having RxRPC call request_key() at the point of issue of a call or
 780     opening of a socket.  Instead the utility is responsible for requesting a
 781     key at the appropriate point.  AFS, for instance, would do this during VFS
 782     operations such as open() or unlink().  The key is then handed through
 783     when the call is initiated.
 784
 785 (3) Request the use of something other than GFP_KERNEL to allocate memory.
 786
 787 (4) Avoid the overhead of using the recvmsg() call.  RxRPC messages can be
 788     intercepted before they get put into the socket Rx queue and the socket
 789     buffers manipulated directly.
 790
 791To use the RxRPC facility, a kernel utility must still open an AF_RXRPC socket,
 792bind an address as appropriate and listen if it's to be a server socket, but
 793then it passes this to the kernel interface functions.
 794
 795The kernel interface functions are as follows:
 796
 797 (#) Begin a new client call::
 798
 799	struct rxrpc_call *
 800	rxrpc_kernel_begin_call(struct socket *sock,
 801				struct sockaddr_rxrpc *srx,
 802				struct key *key,
 803				unsigned long user_call_ID,
 804				s64 tx_total_len,
 805				gfp_t gfp,
 806				rxrpc_notify_rx_t notify_rx,
 807				bool upgrade,
 808				bool intr,
 809				unsigned int debug_id);
 810
 811     This allocates the infrastructure to make a new RxRPC call and assigns
 812     call and connection numbers.  The call will be made on the UDP port that
 813     the socket is bound to.  The call will go to the destination address of a
 814     connected client socket unless an alternative is supplied (srx is
 815     non-NULL).
 816
 817     If a key is supplied then this will be used to secure the call instead of
 818     the key bound to the socket with the RXRPC_SECURITY_KEY sockopt.  Calls
 819     secured in this way will still share connections if at all possible.
 820
 821     The user_call_ID is equivalent to that supplied to sendmsg() in the
 822     control data buffer.  It is entirely feasible to use this to point to a
 823     kernel data structure.
 824
 825     tx_total_len is the amount of data the caller is intending to transmit
 826     with this call (or -1 if unknown at this point).  Setting the data size
 827     allows the kernel to encrypt directly to the packet buffers, thereby
 828     saving a copy.  The value may not be less than -1.
 829
 830     notify_rx is a pointer to a function to be called when events such as
 831     incoming data packets or remote aborts happen.
 832
 833     upgrade should be set to true if a client operation should request that
 834     the server upgrade the service to a better one.  The resultant service ID
 835     is returned by rxrpc_kernel_recv_data().
 836
 837     intr should be set to true if the call should be interruptible.  If this
 838     is not set, this function may not return until a channel has been
 839     allocated; if it is set, the function may return -ERESTARTSYS.
 840
 841     debug_id is the call debugging ID to be used for tracing.  This can be
 842     obtained by atomically incrementing rxrpc_debug_id.
 843
 844     If this function is successful, an opaque reference to the RxRPC call is
 845     returned.  The caller now holds a reference on this and it must be
 846     properly ended.
 847
 848 (#) Shut down a client call::
 849
 850	void rxrpc_kernel_shutdown_call(struct socket *sock,
 851					struct rxrpc_call *call);
 852
 853     This is used to shut down a previously begun call.  The user_call_ID is
 854     expunged from AF_RXRPC's knowledge and will not be seen again in
 855     association with the specified call.
 856
 857 (#) Release the ref on a client call::
 858
 859	void rxrpc_kernel_put_call(struct socket *sock,
 860				   struct rxrpc_call *call);
 861
 862     This is used to release the caller's ref on an rxrpc call.
 863
 864 (#) Send data through a call::
 865
 866	typedef void (*rxrpc_notify_end_tx_t)(struct sock *sk,
 867					      unsigned long user_call_ID,
 868					      struct sk_buff *skb);
 869
 870	int rxrpc_kernel_send_data(struct socket *sock,
 871				   struct rxrpc_call *call,
 872				   struct msghdr *msg,
 873				   size_t len,
 874				   rxrpc_notify_end_tx_t notify_end_rx);
 875
 876     This is used to supply either the request part of a client call or the
 877     reply part of a server call.  msg.msg_iovlen and msg.msg_iov specify the
 878     data buffers to be used.  msg_iov may not be NULL and must point
 879     exclusively to in-kernel virtual addresses.  msg.msg_flags may be given
 880     MSG_MORE if there will be subsequent data sends for this call.
 881
 882     The msg must not specify a destination address, control data or any flags
 883     other than MSG_MORE.  len is the total amount of data to transmit.
 884
 885     notify_end_rx can be NULL or it can be used to specify a function to be
 886     called when the call changes state to end the Tx phase.  This function is
 887     called with a spinlock held to prevent the last DATA packet from being
 888     transmitted until the function returns.
 889
 890 (#) Receive data from a call::
 891
 892	int rxrpc_kernel_recv_data(struct socket *sock,
 893				   struct rxrpc_call *call,
 894				   void *buf,
 895				   size_t size,
 896				   size_t *_offset,
 897				   bool want_more,
 898				   u32 *_abort,
 899				   u16 *_service)
 900
 901      This is used to receive data from either the reply part of a client call
 902      or the request part of a service call.  buf and size specify how much
 903      data is desired and where to store it.  *_offset is added on to buf and
 904      subtracted from size internally; the amount copied into the buffer is
 905      added to *_offset before returning.
 906
 907      want_more should be true if further data will be required after this is
 908      satisfied and false if this is the last item of the receive phase.
 909
 910      There are three normal returns: 0 if the buffer was filled and want_more
 911      was true; 1 if the buffer was filled, the last DATA packet has been
 912      emptied and want_more was false; and -EAGAIN if the function needs to be
 913      called again.
 914
 915      If the last DATA packet is processed but the buffer contains less than
 916      the amount requested, EBADMSG is returned.  If want_more wasn't set, but
 917      more data was available, EMSGSIZE is returned.
 918
 919      If a remote ABORT is detected, the abort code received will be stored in
 920      ``*_abort`` and ECONNABORTED will be returned.
 921
 922      The service ID that the call ended up with is returned into *_service.
 923      This can be used to see if a call got a service upgrade.
 924
 925 (#) Abort a call??
 926
 927     ::
 928
 929	void rxrpc_kernel_abort_call(struct socket *sock,
 930				     struct rxrpc_call *call,
 931				     u32 abort_code);
 932
 933     This is used to abort a call if it's still in an abortable state.  The
 934     abort code specified will be placed in the ABORT message sent.
 935
 936 (#) Intercept received RxRPC messages::
 937
 938	typedef void (*rxrpc_interceptor_t)(struct sock *sk,
 939					    unsigned long user_call_ID,
 940					    struct sk_buff *skb);
 941
 942	void
 943	rxrpc_kernel_intercept_rx_messages(struct socket *sock,
 944					   rxrpc_interceptor_t interceptor);
 945
 946     This installs an interceptor function on the specified AF_RXRPC socket.
 947     All messages that would otherwise wind up in the socket's Rx queue are
 948     then diverted to this function.  Note that care must be taken to process
 949     the messages in the right order to maintain DATA message sequentiality.
 950
 951     The interceptor function itself is provided with the address of the socket
 952     and handling the incoming message, the ID assigned by the kernel utility
 953     to the call and the socket buffer containing the message.
 954
 955     The skb->mark field indicates the type of message:
 956
 957	===============================	=======================================
 958	Mark				Meaning
 959	===============================	=======================================
 960	RXRPC_SKB_MARK_DATA		Data message
 961	RXRPC_SKB_MARK_FINAL_ACK	Final ACK received for an incoming call
 962	RXRPC_SKB_MARK_BUSY		Client call rejected as server busy
 963	RXRPC_SKB_MARK_REMOTE_ABORT	Call aborted by peer
 964	RXRPC_SKB_MARK_NET_ERROR	Network error detected
 965	RXRPC_SKB_MARK_LOCAL_ERROR	Local error encountered
 966	RXRPC_SKB_MARK_NEW_CALL		New incoming call awaiting acceptance
 967	===============================	=======================================
 968
 969     The remote abort message can be probed with rxrpc_kernel_get_abort_code().
 970     The two error messages can be probed with rxrpc_kernel_get_error_number().
 971     A new call can be accepted with rxrpc_kernel_accept_call().
 972
 973     Data messages can have their contents extracted with the usual bunch of
 974     socket buffer manipulation functions.  A data message can be determined to
 975     be the last one in a sequence with rxrpc_kernel_is_data_last().  When a
 976     data message has been used up, rxrpc_kernel_data_consumed() should be
 977     called on it.
 978
 979     Messages should be handled to rxrpc_kernel_free_skb() to dispose of.  It
 980     is possible to get extra refs on all types of message for later freeing,
 981     but this may pin the state of a call until the message is finally freed.
 982
 983 (#) Accept an incoming call::
 984
 985	struct rxrpc_call *
 986	rxrpc_kernel_accept_call(struct socket *sock,
 987				 unsigned long user_call_ID);
 988
 989     This is used to accept an incoming call and to assign it a call ID.  This
 990     function is similar to rxrpc_kernel_begin_call() and calls accepted must
 991     be ended in the same way.
 992
 993     If this function is successful, an opaque reference to the RxRPC call is
 994     returned.  The caller now holds a reference on this and it must be
 995     properly ended.
 996
 997 (#) Reject an incoming call::
 998
 999	int rxrpc_kernel_reject_call(struct socket *sock);
1000
1001     This is used to reject the first incoming call on the socket's queue with
1002     a BUSY message.  -ENODATA is returned if there were no incoming calls.
1003     Other errors may be returned if the call had been aborted (-ECONNABORTED)
1004     or had timed out (-ETIME).
1005
1006 (#) Allocate a null key for doing anonymous security::
1007
1008	struct key *rxrpc_get_null_key(const char *keyname);
1009
1010     This is used to allocate a null RxRPC key that can be used to indicate
1011     anonymous security for a particular domain.
1012
1013 (#) Get the peer address of a call::
1014
1015	void rxrpc_kernel_get_peer(struct socket *sock, struct rxrpc_call *call,
1016				   struct sockaddr_rxrpc *_srx);
1017
1018     This is used to find the remote peer address of a call.
1019
1020 (#) Set the total transmit data size on a call::
1021
1022	void rxrpc_kernel_set_tx_length(struct socket *sock,
1023					struct rxrpc_call *call,
1024					s64 tx_total_len);
1025
1026     This sets the amount of data that the caller is intending to transmit on a
1027     call.  It's intended to be used for setting the reply size as the request
1028     size should be set when the call is begun.  tx_total_len may not be less
1029     than zero.
1030
1031 (#) Get call RTT::
1032
1033	u64 rxrpc_kernel_get_rtt(struct socket *sock, struct rxrpc_call *call);
1034
1035     Get the RTT time to the peer in use by a call.  The value returned is in
1036     nanoseconds.
1037
1038 (#) Check call still alive::
1039
1040	bool rxrpc_kernel_check_life(struct socket *sock,
1041				     struct rxrpc_call *call,
1042				     u32 *_life);
1043	void rxrpc_kernel_probe_life(struct socket *sock,
1044				     struct rxrpc_call *call);
1045
1046     The first function passes back in ``*_life`` a number that is updated when
1047     ACKs are received from the peer (notably including PING RESPONSE ACKs
1048     which we can elicit by sending PING ACKs to see if the call still exists
1049     on the server).  The caller should compare the numbers of two calls to see
1050     if the call is still alive after waiting for a suitable interval.  It also
1051     returns true as long as the call hasn't yet reached the completed state.
1052
1053     This allows the caller to work out if the server is still contactable and
1054     if the call is still alive on the server while waiting for the server to
1055     process a client operation.
1056
1057     The second function causes a ping ACK to be transmitted to try to provoke
1058     the peer into responding, which would then cause the value returned by the
1059     first function to change.  Note that this must be called in TASK_RUNNING
1060     state.
1061
1062 (#) Apply the RXRPC_MIN_SECURITY_LEVEL sockopt to a socket from within in the
1063     kernel::
1064
1065       int rxrpc_sock_set_min_security_level(struct sock *sk,
1066					     unsigned int val);
1067
1068     This specifies the minimum security level required for calls on this
1069     socket.
1070
1071
1072Configurable Parameters
1073=======================
1074
1075The RxRPC protocol driver has a number of configurable parameters that can be
1076adjusted through sysctls in /proc/net/rxrpc/:
1077
1078 (#) req_ack_delay
1079
1080     The amount of time in milliseconds after receiving a packet with the
1081     request-ack flag set before we honour the flag and actually send the
1082     requested ack.
1083
1084     Usually the other side won't stop sending packets until the advertised
1085     reception window is full (to a maximum of 255 packets), so delaying the
1086     ACK permits several packets to be ACK'd in one go.
1087
1088 (#) soft_ack_delay
1089
1090     The amount of time in milliseconds after receiving a new packet before we
1091     generate a soft-ACK to tell the sender that it doesn't need to resend.
1092
1093 (#) idle_ack_delay
1094
1095     The amount of time in milliseconds after all the packets currently in the
1096     received queue have been consumed before we generate a hard-ACK to tell
1097     the sender it can free its buffers, assuming no other reason occurs that
1098     we would send an ACK.
1099
1100 (#) resend_timeout
1101
1102     The amount of time in milliseconds after transmitting a packet before we
1103     transmit it again, assuming no ACK is received from the receiver telling
1104     us they got it.
1105
1106 (#) max_call_lifetime
1107
1108     The maximum amount of time in seconds that a call may be in progress
1109     before we preemptively kill it.
1110
1111 (#) dead_call_expiry
1112
1113     The amount of time in seconds before we remove a dead call from the call
1114     list.  Dead calls are kept around for a little while for the purpose of
1115     repeating ACK and ABORT packets.
1116
1117 (#) connection_expiry
1118
1119     The amount of time in seconds after a connection was last used before we
1120     remove it from the connection list.  While a connection is in existence,
1121     it serves as a placeholder for negotiated security; when it is deleted,
1122     the security must be renegotiated.
1123
1124 (#) transport_expiry
1125
1126     The amount of time in seconds after a transport was last used before we
1127     remove it from the transport list.  While a transport is in existence, it
1128     serves to anchor the peer data and keeps the connection ID counter.
1129
1130 (#) rxrpc_rx_window_size
1131
1132     The size of the receive window in packets.  This is the maximum number of
1133     unconsumed received packets we're willing to hold in memory for any
1134     particular call.
1135
1136 (#) rxrpc_rx_mtu
1137
1138     The maximum packet MTU size that we're willing to receive in bytes.  This
1139     indicates to the peer whether we're willing to accept jumbo packets.
1140
1141 (#) rxrpc_rx_jumbo_max
1142
1143     The maximum number of packets that we're willing to accept in a jumbo
1144     packet.  Non-terminal packets in a jumbo packet must contain a four byte
1145     header plus exactly 1412 bytes of data.  The terminal packet must contain
1146     a four byte header plus any amount of data.  In any event, a jumbo packet
1147     may not exceed rxrpc_rx_mtu in size.
1148
1149
1150API Function Reference
1151======================
1152
1153.. kernel-doc:: net/rxrpc/af_rxrpc.c
1154.. kernel-doc:: net/rxrpc/call_object.c
1155.. kernel-doc:: net/rxrpc/key.c
1156.. kernel-doc:: net/rxrpc/oob.c
1157.. kernel-doc:: net/rxrpc/peer_object.c
1158.. kernel-doc:: net/rxrpc/recvmsg.c
1159.. kernel-doc:: net/rxrpc/rxgk.c
1160.. kernel-doc:: net/rxrpc/rxkad.c
1161.. kernel-doc:: net/rxrpc/sendmsg.c
1162.. kernel-doc:: net/rxrpc/server_key.c