FILE
bfq-iosched.c
File preview with syntax highlighting.
linux-6.19.14/block/bfq-iosched.c
1// SPDX-License-Identifier: GPL-2.0-or-later 2/* 3 * Budget Fair Queueing (BFQ) I/O scheduler. 4 * 5 * Based on ideas and code from CFQ: 6 * Copyright (C) 2003 Jens Axboe <axboe@kernel.dk> 7 * 8 * Copyright (C) 2008 Fabio Checconi <fabio@gandalf.sssup.it> 9 * Paolo Valente <paolo.valente@unimore.it> 10 * 11 * Copyright (C) 2010 Paolo Valente <paolo.valente@unimore.it> 12 * Arianna Avanzini <avanzini@google.com> 13 * 14 * Copyright (C) 2017 Paolo Valente <paolo.valente@linaro.org> 15 * 16 * BFQ is a proportional-share I/O scheduler, with some extra 17 * low-latency capabilities. BFQ also supports full hierarchical 18 * scheduling through cgroups. Next paragraphs provide an introduction 19 * on BFQ inner workings. Details on BFQ benefits, usage and 20 * limitations can be found in Documentation/block/bfq-iosched.rst. 21 * 22 * BFQ is a proportional-share storage-I/O scheduling algorithm based 23 * on the slice-by-slice service scheme of CFQ. But BFQ assigns 24 * budgets, measured in number of sectors, to processes instead of 25 * time slices. The device is not granted to the in-service process 26 * for a given time slice, but until it has exhausted its assigned 27 * budget. This change from the time to the service domain enables BFQ 28 * to distribute the device throughput among processes as desired, 29 * without any distortion due to throughput fluctuations, or to device 30 * internal queueing. BFQ uses an ad hoc internal scheduler, called 31 * B-WF2Q+, to schedule processes according to their budgets. More 32 * precisely, BFQ schedules queues associated with processes. Each 33 * process/queue is assigned a user-configurable weight, and B-WF2Q+ 34 * guarantees that each queue receives a fraction of the throughput 35 * proportional to its weight. Thanks to the accurate policy of 36 * B-WF2Q+, BFQ can afford to assign high budgets to I/O-bound 37 * processes issuing sequential requests (to boost the throughput), 38 * and yet guarantee a low latency to interactive and soft real-time 39 * applications. 40 * 41 * In particular, to provide these low-latency guarantees, BFQ 42 * explicitly privileges the I/O of two classes of time-sensitive 43 * applications: interactive and soft real-time. In more detail, BFQ 44 * behaves this way if the low_latency parameter is set (default 45 * configuration). This feature enables BFQ to provide applications in 46 * these classes with a very low latency. 47 * 48 * To implement this feature, BFQ constantly tries to detect whether 49 * the I/O requests in a bfq_queue come from an interactive or a soft 50 * real-time application. For brevity, in these cases, the queue is 51 * said to be interactive or soft real-time. In both cases, BFQ 52 * privileges the service of the queue, over that of non-interactive 53 * and non-soft-real-time queues. This privileging is performed, 54 * mainly, by raising the weight of the queue. So, for brevity, we 55 * call just weight-raising periods the time periods during which a 56 * queue is privileged, because deemed interactive or soft real-time. 57 * 58 * The detection of soft real-time queues/applications is described in 59 * detail in the comments on the function 60 * bfq_bfqq_softrt_next_start. On the other hand, the detection of an 61 * interactive queue works as follows: a queue is deemed interactive 62 * if it is constantly non empty only for a limited time interval, 63 * after which it does become empty. The queue may be deemed 64 * interactive again (for a limited time), if it restarts being 65 * constantly non empty, provided that this happens only after the 66 * queue has remained empty for a given minimum idle time. 67 * 68 * By default, BFQ computes automatically the above maximum time 69 * interval, i.e., the time interval after which a constantly 70 * non-empty queue stops being deemed interactive. Since a queue is 71 * weight-raised while it is deemed interactive, this maximum time 72 * interval happens to coincide with the (maximum) duration of the 73 * weight-raising for interactive queues. 74 * 75 * Finally, BFQ also features additional heuristics for 76 * preserving both a low latency and a high throughput on NCQ-capable, 77 * rotational or flash-based devices, and to get the job done quickly 78 * for applications consisting in many I/O-bound processes. 79 * 80 * NOTE: if the main or only goal, with a given device, is to achieve 81 * the maximum-possible throughput at all times, then do switch off 82 * all low-latency heuristics for that device, by setting low_latency 83 * to 0. 84 * 85 * BFQ is described in [1], where also a reference to the initial, 86 * more theoretical paper on BFQ can be found. The interested reader 87 * can find in the latter paper full details on the main algorithm, as 88 * well as formulas of the guarantees and formal proofs of all the 89 * properties. With respect to the version of BFQ presented in these 90 * papers, this implementation adds a few more heuristics, such as the 91 * ones that guarantee a low latency to interactive and soft real-time 92 * applications, and a hierarchical extension based on H-WF2Q+. 93 * 94 * B-WF2Q+ is based on WF2Q+, which is described in [2], together with 95 * H-WF2Q+, while the augmented tree used here to implement B-WF2Q+ 96 * with O(log N) complexity derives from the one introduced with EEVDF 97 * in [3]. 98 * 99 * [1] P. Valente, A. Avanzini, "Evolution of the BFQ Storage I/O 100 * Scheduler", Proceedings of the First Workshop on Mobile System 101 * Technologies (MST-2015), May 2015. 102 * http://algogroup.unimore.it/people/paolo/disk_sched/mst-2015.pdf 103 * 104 * [2] Jon C.R. Bennett and H. Zhang, "Hierarchical Packet Fair Queueing 105 * Algorithms", IEEE/ACM Transactions on Networking, 5(5):675-689, 106 * Oct 1997. 107 * 108 * http://www.cs.cmu.edu/~hzhang/papers/TON-97-Oct.ps.gz 109 * 110 * [3] I. Stoica and H. Abdel-Wahab, "Earliest Eligible Virtual Deadline 111 * First: A Flexible and Accurate Mechanism for Proportional Share 112 * Resource Allocation", technical report. 113 * 114 * http://www.cs.berkeley.edu/~istoica/papers/eevdf-tr-95.pdf 115 */ 116#include <linux/module.h> 117#include <linux/slab.h> 118#include <linux/blkdev.h> 119#include <linux/cgroup.h> 120#include <linux/ktime.h> 121#include <linux/rbtree.h> 122#include <linux/ioprio.h> 123#include <linux/sbitmap.h> 124#include <linux/delay.h> 125#include <linux/backing-dev.h> 126 127#include <trace/events/block.h> 128 129#include "elevator.h" 130#include "blk.h" 131#include "blk-mq.h" 132#include "blk-mq-sched.h" 133#include "bfq-iosched.h" 134#include "blk-wbt.h" 135 136#define BFQ_BFQQ_FNS(name) \ 137void bfq_mark_bfqq_##name(struct bfq_queue *bfqq) \ 138{ \ 139 __set_bit(BFQQF_##name, &(bfqq)->flags); \ 140} \ 141void bfq_clear_bfqq_##name(struct bfq_queue *bfqq) \ 142{ \ 143 __clear_bit(BFQQF_##name, &(bfqq)->flags); \ 144} \ 145int bfq_bfqq_##name(const struct bfq_queue *bfqq) \ 146{ \ 147 return test_bit(BFQQF_##name, &(bfqq)->flags); \ 148} 149 150BFQ_BFQQ_FNS(just_created); 151BFQ_BFQQ_FNS(busy); 152BFQ_BFQQ_FNS(wait_request); 153BFQ_BFQQ_FNS(non_blocking_wait_rq); 154BFQ_BFQQ_FNS(fifo_expire); 155BFQ_BFQQ_FNS(has_short_ttime); 156BFQ_BFQQ_FNS(sync); 157BFQ_BFQQ_FNS(IO_bound); 158BFQ_BFQQ_FNS(in_large_burst); 159BFQ_BFQQ_FNS(coop); 160BFQ_BFQQ_FNS(split_coop); 161BFQ_BFQQ_FNS(softrt_update); 162#undef BFQ_BFQQ_FNS \ 163 164/* Expiration time of async (0) and sync (1) requests, in ns. */ 165static const u64 bfq_fifo_expire[2] = { NSEC_PER_SEC / 4, NSEC_PER_SEC / 8 }; 166 167/* Maximum backwards seek (magic number lifted from CFQ), in KiB. */ 168static const int bfq_back_max = 16 * 1024; 169 170/* Penalty of a backwards seek, in number of sectors. */ 171static const int bfq_back_penalty = 2; 172 173/* Idling period duration, in ns. */ 174static u64 bfq_slice_idle = NSEC_PER_SEC / 125; 175 176/* Minimum number of assigned budgets for which stats are safe to compute. */ 177static const int bfq_stats_min_budgets = 194; 178 179/* Default maximum budget values, in sectors and number of requests. */ 180static const int bfq_default_max_budget = 16 * 1024; 181 182/* 183 * When a sync request is dispatched, the queue that contains that 184 * request, and all the ancestor entities of that queue, are charged 185 * with the number of sectors of the request. In contrast, if the 186 * request is async, then the queue and its ancestor entities are 187 * charged with the number of sectors of the request, multiplied by 188 * the factor below. This throttles the bandwidth for async I/O, 189 * w.r.t. to sync I/O, and it is done to counter the tendency of async 190 * writes to steal I/O throughput to reads. 191 * 192 * The current value of this parameter is the result of a tuning with 193 * several hardware and software configurations. We tried to find the 194 * lowest value for which writes do not cause noticeable problems to 195 * reads. In fact, the lower this parameter, the stabler I/O control, 196 * in the following respect. The lower this parameter is, the less 197 * the bandwidth enjoyed by a group decreases 198 * - when the group does writes, w.r.t. to when it does reads; 199 * - when other groups do reads, w.r.t. to when they do writes. 200 */ 201static const int bfq_async_charge_factor = 3; 202 203/* Default timeout values, in jiffies, approximating CFQ defaults. */ 204const int bfq_timeout = HZ / 8; 205 206/* 207 * Time limit for merging (see comments in bfq_setup_cooperator). Set 208 * to the slowest value that, in our tests, proved to be effective in 209 * removing false positives, while not causing true positives to miss 210 * queue merging. 211 * 212 * As can be deduced from the low time limit below, queue merging, if 213 * successful, happens at the very beginning of the I/O of the involved 214 * cooperating processes, as a consequence of the arrival of the very 215 * first requests from each cooperator. After that, there is very 216 * little chance to find cooperators. 217 */ 218static const unsigned long bfq_merge_time_limit = HZ/10; 219 220static struct kmem_cache *bfq_pool; 221 222/* Below this threshold (in ns), we consider thinktime immediate. */ 223#define BFQ_MIN_TT (2 * NSEC_PER_MSEC) 224 225/* hw_tag detection: parallel requests threshold and min samples needed. */ 226#define BFQ_HW_QUEUE_THRESHOLD 3 227#define BFQ_HW_QUEUE_SAMPLES 32 228 229#define BFQQ_SEEK_THR (sector_t)(8 * 100) 230#define BFQQ_SECT_THR_NONROT (sector_t)(2 * 32) 231#define BFQ_RQ_SEEKY(bfqd, last_pos, rq) \ 232 (get_sdist(last_pos, rq) > \ 233 BFQQ_SEEK_THR && \ 234 (!blk_queue_nonrot(bfqd->queue) || \ 235 blk_rq_sectors(rq) < BFQQ_SECT_THR_NONROT)) 236#define BFQQ_CLOSE_THR (sector_t)(8 * 1024) 237#define BFQQ_SEEKY(bfqq) (hweight32(bfqq->seek_history) > 19) 238/* 239 * Sync random I/O is likely to be confused with soft real-time I/O, 240 * because it is characterized by limited throughput and apparently 241 * isochronous arrival pattern. To avoid false positives, queues 242 * containing only random (seeky) I/O are prevented from being tagged 243 * as soft real-time. 244 */ 245#define BFQQ_TOTALLY_SEEKY(bfqq) (bfqq->seek_history == -1) 246 247/* Min number of samples required to perform peak-rate update */ 248#define BFQ_RATE_MIN_SAMPLES 32 249/* Min observation time interval required to perform a peak-rate update (ns) */ 250#define BFQ_RATE_MIN_INTERVAL (300*NSEC_PER_MSEC) 251/* Target observation time interval for a peak-rate update (ns) */ 252#define BFQ_RATE_REF_INTERVAL NSEC_PER_SEC 253 254/* 255 * Shift used for peak-rate fixed precision calculations. 256 * With 257 * - the current shift: 16 positions 258 * - the current type used to store rate: u32 259 * - the current unit of measure for rate: [sectors/usec], or, more precisely, 260 * [(sectors/usec) / 2^BFQ_RATE_SHIFT] to take into account the shift, 261 * the range of rates that can be stored is 262 * [1 / 2^BFQ_RATE_SHIFT, 2^(32 - BFQ_RATE_SHIFT)] sectors/usec = 263 * [1 / 2^16, 2^16] sectors/usec = [15e-6, 65536] sectors/usec = 264 * [15, 65G] sectors/sec 265 * Which, assuming a sector size of 512B, corresponds to a range of 266 * [7.5K, 33T] B/sec 267 */ 268#define BFQ_RATE_SHIFT 16 269 270/* 271 * When configured for computing the duration of the weight-raising 272 * for interactive queues automatically (see the comments at the 273 * beginning of this file), BFQ does it using the following formula: 274 * duration = (ref_rate / r) * ref_wr_duration, 275 * where r is the peak rate of the device, and ref_rate and 276 * ref_wr_duration are two reference parameters. In particular, 277 * ref_rate is the peak rate of the reference storage device (see 278 * below), and ref_wr_duration is about the maximum time needed, with 279 * BFQ and while reading two files in parallel, to load typical large 280 * applications on the reference device (see the comments on 281 * max_service_from_wr below, for more details on how ref_wr_duration 282 * is obtained). In practice, the slower/faster the device at hand 283 * is, the more/less it takes to load applications with respect to the 284 * reference device. Accordingly, the longer/shorter BFQ grants 285 * weight raising to interactive applications. 286 * 287 * BFQ uses two different reference pairs (ref_rate, ref_wr_duration), 288 * depending on whether the device is rotational or non-rotational. 289 * 290 * In the following definitions, ref_rate[0] and ref_wr_duration[0] 291 * are the reference values for a rotational device, whereas 292 * ref_rate[1] and ref_wr_duration[1] are the reference values for a 293 * non-rotational device. The reference rates are not the actual peak 294 * rates of the devices used as a reference, but slightly lower 295 * values. The reason for using slightly lower values is that the 296 * peak-rate estimator tends to yield slightly lower values than the 297 * actual peak rate (it can yield the actual peak rate only if there 298 * is only one process doing I/O, and the process does sequential 299 * I/O). 300 * 301 * The reference peak rates are measured in sectors/usec, left-shifted 302 * by BFQ_RATE_SHIFT. 303 */ 304static int ref_rate[2] = {14000, 33000}; 305/* 306 * To improve readability, a conversion function is used to initialize 307 * the following array, which entails that the array can be 308 * initialized only in a function. 309 */ 310static int ref_wr_duration[2]; 311 312/* 313 * BFQ uses the above-detailed, time-based weight-raising mechanism to 314 * privilege interactive tasks. This mechanism is vulnerable to the 315 * following false positives: I/O-bound applications that will go on 316 * doing I/O for much longer than the duration of weight 317 * raising. These applications have basically no benefit from being 318 * weight-raised at the beginning of their I/O. On the opposite end, 319 * while being weight-raised, these applications 320 * a) unjustly steal throughput to applications that may actually need 321 * low latency; 322 * b) make BFQ uselessly perform device idling; device idling results 323 * in loss of device throughput with most flash-based storage, and may 324 * increase latencies when used purposelessly. 325 * 326 * BFQ tries to reduce these problems, by adopting the following 327 * countermeasure. To introduce this countermeasure, we need first to 328 * finish explaining how the duration of weight-raising for 329 * interactive tasks is computed. 330 * 331 * For a bfq_queue deemed as interactive, the duration of weight 332 * raising is dynamically adjusted, as a function of the estimated 333 * peak rate of the device, so as to be equal to the time needed to 334 * execute the 'largest' interactive task we benchmarked so far. By 335 * largest task, we mean the task for which each involved process has 336 * to do more I/O than for any of the other tasks we benchmarked. This 337 * reference interactive task is the start-up of LibreOffice Writer, 338 * and in this task each process/bfq_queue needs to have at most ~110K 339 * sectors transferred. 340 * 341 * This last piece of information enables BFQ to reduce the actual 342 * duration of weight-raising for at least one class of I/O-bound 343 * applications: those doing sequential or quasi-sequential I/O. An 344 * example is file copy. In fact, once started, the main I/O-bound 345 * processes of these applications usually consume the above 110K 346 * sectors in much less time than the processes of an application that 347 * is starting, because these I/O-bound processes will greedily devote 348 * almost all their CPU cycles only to their target, 349 * throughput-friendly I/O operations. This is even more true if BFQ 350 * happens to be underestimating the device peak rate, and thus 351 * overestimating the duration of weight raising. But, according to 352 * our measurements, once transferred 110K sectors, these processes 353 * have no right to be weight-raised any longer. 354 * 355 * Basing on the last consideration, BFQ ends weight-raising for a 356 * bfq_queue if the latter happens to have received an amount of 357 * service at least equal to the following constant. The constant is 358 * set to slightly more than 110K, to have a minimum safety margin. 359 * 360 * This early ending of weight-raising reduces the amount of time 361 * during which interactive false positives cause the two problems 362 * described at the beginning of these comments. 363 */ 364static const unsigned long max_service_from_wr = 120000; 365 366/* 367 * Maximum time between the creation of two queues, for stable merge 368 * to be activated (in ms) 369 */ 370static const unsigned long bfq_activation_stable_merging = 600; 371/* 372 * Minimum time to be waited before evaluating delayed stable merge (in ms) 373 */ 374static const unsigned long bfq_late_stable_merging = 600; 375 376#define RQ_BIC(rq) ((struct bfq_io_cq *)((rq)->elv.priv[0])) 377#define RQ_BFQQ(rq) ((rq)->elv.priv[1]) 378 379struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync, 380 unsigned int actuator_idx) 381{ 382 if (is_sync) 383 return bic->bfqq[1][actuator_idx]; 384 385 return bic->bfqq[0][actuator_idx]; 386} 387 388static void bfq_put_stable_ref(struct bfq_queue *bfqq); 389 390void bic_set_bfqq(struct bfq_io_cq *bic, 391 struct bfq_queue *bfqq, 392 bool is_sync, 393 unsigned int actuator_idx) 394{ 395 struct bfq_queue *old_bfqq = bic->bfqq[is_sync][actuator_idx]; 396 397 /* 398 * If bfqq != NULL, then a non-stable queue merge between 399 * bic->bfqq and bfqq is happening here. This causes troubles 400 * in the following case: bic->bfqq has also been scheduled 401 * for a possible stable merge with bic->stable_merge_bfqq, 402 * and bic->stable_merge_bfqq == bfqq happens to 403 * hold. Troubles occur because bfqq may then undergo a split, 404 * thereby becoming eligible for a stable merge. Yet, if 405 * bic->stable_merge_bfqq points exactly to bfqq, then bfqq 406 * would be stably merged with itself. To avoid this anomaly, 407 * we cancel the stable merge if 408 * bic->stable_merge_bfqq == bfqq. 409 */ 410 struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[actuator_idx]; 411 412 /* Clear bic pointer if bfqq is detached from this bic */ 413 if (old_bfqq && old_bfqq->bic == bic) 414 old_bfqq->bic = NULL; 415 416 if (is_sync) 417 bic->bfqq[1][actuator_idx] = bfqq; 418 else 419 bic->bfqq[0][actuator_idx] = bfqq; 420 421 if (bfqq && bfqq_data->stable_merge_bfqq == bfqq) { 422 /* 423 * Actually, these same instructions are executed also 424 * in bfq_setup_cooperator, in case of abort or actual 425 * execution of a stable merge. We could avoid 426 * repeating these instructions there too, but if we 427 * did so, we would nest even more complexity in this 428 * function. 429 */ 430 bfq_put_stable_ref(bfqq_data->stable_merge_bfqq); 431 432 bfqq_data->stable_merge_bfqq = NULL; 433 } 434} 435 436struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic) 437{ 438 return bic->icq.q->elevator->elevator_data; 439} 440 441/** 442 * icq_to_bic - convert iocontext queue structure to bfq_io_cq. 443 * @icq: the iocontext queue. 444 */ 445static struct bfq_io_cq *icq_to_bic(struct io_cq *icq) 446{ 447 /* bic->icq is the first member, %NULL will convert to %NULL */ 448 return container_of(icq, struct bfq_io_cq, icq); 449} 450 451/** 452 * bfq_bic_lookup - search into @ioc a bic associated to @bfqd. 453 * @q: the request queue. 454 */ 455static struct bfq_io_cq *bfq_bic_lookup(struct request_queue *q) 456{ 457 if (!current->io_context) 458 return NULL; 459 460 return icq_to_bic(ioc_lookup_icq(q)); 461} 462 463/* 464 * Scheduler run of queue, if there are requests pending and no one in the 465 * driver that will restart queueing. 466 */ 467void bfq_schedule_dispatch(struct bfq_data *bfqd) 468{ 469 lockdep_assert_held(&bfqd->lock); 470 471 if (bfqd->queued != 0) { 472 bfq_log(bfqd, "schedule dispatch"); 473 blk_mq_run_hw_queues(bfqd->queue, true); 474 } 475} 476 477#define bfq_class_idle(bfqq) ((bfqq)->ioprio_class == IOPRIO_CLASS_IDLE) 478 479#define bfq_sample_valid(samples) ((samples) > 80) 480 481/* 482 * Lifted from AS - choose which of rq1 and rq2 that is best served now. 483 * We choose the request that is closer to the head right now. Distance 484 * behind the head is penalized and only allowed to a certain extent. 485 */ 486static struct request *bfq_choose_req(struct bfq_data *bfqd, 487 struct request *rq1, 488 struct request *rq2, 489 sector_t last) 490{ 491 sector_t s1, s2, d1 = 0, d2 = 0; 492 unsigned long back_max; 493#define BFQ_RQ1_WRAP 0x01 /* request 1 wraps */ 494#define BFQ_RQ2_WRAP 0x02 /* request 2 wraps */ 495 unsigned int wrap = 0; /* bit mask: requests behind the disk head? */ 496 497 if (!rq1 || rq1 == rq2) 498 return rq2; 499 if (!rq2) 500 return rq1; 501 502 if (rq_is_sync(rq1) && !rq_is_sync(rq2)) 503 return rq1; 504 else if (rq_is_sync(rq2) && !rq_is_sync(rq1)) 505 return rq2; 506 if ((rq1->cmd_flags & REQ_META) && !(rq2->cmd_flags & REQ_META)) 507 return rq1; 508 else if ((rq2->cmd_flags & REQ_META) && !(rq1->cmd_flags & REQ_META)) 509 return rq2; 510 511 s1 = blk_rq_pos(rq1); 512 s2 = blk_rq_pos(rq2); 513 514 /* 515 * By definition, 1KiB is 2 sectors. 516 */ 517 back_max = bfqd->bfq_back_max * 2; 518 519 /* 520 * Strict one way elevator _except_ in the case where we allow 521 * short backward seeks which are biased as twice the cost of a 522 * similar forward seek. 523 */ 524 if (s1 >= last) 525 d1 = s1 - last; 526 else if (s1 + back_max >= last) 527 d1 = (last - s1) * bfqd->bfq_back_penalty; 528 else 529 wrap |= BFQ_RQ1_WRAP; 530 531 if (s2 >= last) 532 d2 = s2 - last; 533 else if (s2 + back_max >= last) 534 d2 = (last - s2) * bfqd->bfq_back_penalty; 535 else 536 wrap |= BFQ_RQ2_WRAP; 537 538 /* Found required data */ 539 540 /* 541 * By doing switch() on the bit mask "wrap" we avoid having to 542 * check two variables for all permutations: --> faster! 543 */ 544 switch (wrap) { 545 case 0: /* common case for CFQ: rq1 and rq2 not wrapped */ 546 if (d1 < d2) 547 return rq1; 548 else if (d2 < d1) 549 return rq2; 550 551 if (s1 >= s2) 552 return rq1; 553 else 554 return rq2; 555 556 case BFQ_RQ2_WRAP: 557 return rq1; 558 case BFQ_RQ1_WRAP: 559 return rq2; 560 case BFQ_RQ1_WRAP|BFQ_RQ2_WRAP: /* both rqs wrapped */ 561 default: 562 /* 563 * Since both rqs are wrapped, 564 * start with the one that's further behind head 565 * (--> only *one* back seek required), 566 * since back seek takes more time than forward. 567 */ 568 if (s1 <= s2) 569 return rq1; 570 else 571 return rq2; 572 } 573} 574 575#define BFQ_LIMIT_INLINE_DEPTH 16 576 577#ifdef CONFIG_BFQ_GROUP_IOSCHED 578static bool bfqq_request_over_limit(struct bfq_data *bfqd, 579 struct bfq_io_cq *bic, blk_opf_t opf, 580 unsigned int act_idx, int limit) 581{ 582 struct bfq_entity *inline_entities[BFQ_LIMIT_INLINE_DEPTH]; 583 struct bfq_entity **entities = inline_entities; 584 int alloc_depth = BFQ_LIMIT_INLINE_DEPTH; 585 struct bfq_sched_data *sched_data; 586 struct bfq_entity *entity; 587 struct bfq_queue *bfqq; 588 unsigned long wsum; 589 bool ret = false; 590 int depth; 591 int level; 592 593retry: 594 spin_lock_irq(&bfqd->lock); 595 bfqq = bic_to_bfqq(bic, op_is_sync(opf), act_idx); 596 if (!bfqq) 597 goto out; 598 599 entity = &bfqq->entity; 600 if (!entity->on_st_or_in_serv) 601 goto out; 602 603 /* +1 for bfqq entity, root cgroup not included */ 604 depth = bfqg_to_blkg(bfqq_group(bfqq))->blkcg->css.cgroup->level + 1; 605 if (depth > alloc_depth) { 606 spin_unlock_irq(&bfqd->lock); 607 if (entities != inline_entities) 608 kfree(entities); 609 entities = kmalloc_array(depth, sizeof(*entities), GFP_NOIO); 610 if (!entities) 611 return false; 612 alloc_depth = depth; 613 goto retry; 614 } 615 616 sched_data = entity->sched_data; 617 /* Gather our ancestors as we need to traverse them in reverse order */ 618 level = 0; 619 for_each_entity(entity) { 620 /* 621 * If at some level entity is not even active, allow request 622 * queueing so that BFQ knows there's work to do and activate 623 * entities. 624 */ 625 if (!entity->on_st_or_in_serv) 626 goto out; 627 /* Uh, more parents than cgroup subsystem thinks? */ 628 if (WARN_ON_ONCE(level >= depth)) 629 break; 630 entities[level++] = entity; 631 } 632 WARN_ON_ONCE(level != depth); 633 for (level--; level >= 0; level--) { 634 entity = entities[level]; 635 if (level > 0) { 636 wsum = bfq_entity_service_tree(entity)->wsum; 637 } else { 638 int i; 639 /* 640 * For bfqq itself we take into account service trees 641 * of all higher priority classes and multiply their 642 * weights so that low prio queue from higher class 643 * gets more requests than high prio queue from lower 644 * class. 645 */ 646 wsum = 0; 647 for (i = 0; i <= bfqq->ioprio_class - 1; i++) { 648 wsum = wsum * IOPRIO_BE_NR + 649 sched_data->service_tree[i].wsum; 650 } 651 } 652 if (!wsum) 653 continue; 654 limit = DIV_ROUND_CLOSEST(limit * entity->weight, wsum); 655 if (entity->allocated >= limit) { 656 bfq_log_bfqq(bfqq->bfqd, bfqq, 657 "too many requests: allocated %d limit %d level %d", 658 entity->allocated, limit, level); 659 ret = true; 660 break; 661 } 662 } 663out: 664 spin_unlock_irq(&bfqd->lock); 665 if (entities != inline_entities) 666 kfree(entities); 667 return ret; 668} 669#else 670static bool bfqq_request_over_limit(struct bfq_data *bfqd, 671 struct bfq_io_cq *bic, blk_opf_t opf, 672 unsigned int act_idx, int limit) 673{ 674 return false; 675} 676#endif 677 678/* 679 * Async I/O can easily starve sync I/O (both sync reads and sync 680 * writes), by consuming all tags. Similarly, storms of sync writes, 681 * such as those that sync(2) may trigger, can starve sync reads. 682 * Limit depths of async I/O and sync writes so as to counter both 683 * problems. 684 * 685 * Also if a bfq queue or its parent cgroup consume more tags than would be 686 * appropriate for their weight, we trim the available tag depth to 1. This 687 * avoids a situation where one cgroup can starve another cgroup from tags and 688 * thus block service differentiation among cgroups. Note that because the 689 * queue / cgroup already has many requests allocated and queued, this does not 690 * significantly affect service guarantees coming from the BFQ scheduling 691 * algorithm. 692 */ 693static void bfq_limit_depth(blk_opf_t opf, struct blk_mq_alloc_data *data) 694{ 695 struct bfq_data *bfqd = data->q->elevator->elevator_data; 696 struct bfq_io_cq *bic = bfq_bic_lookup(data->q); 697 unsigned int limit, act_idx; 698 699 /* Sync reads have full depth available */ 700 if (blk_mq_is_sync_read(opf)) 701 limit = data->q->nr_requests; 702 else 703 limit = bfqd->async_depths[!!bfqd->wr_busy_queues][op_is_sync(opf)]; 704 705 for (act_idx = 0; bic && act_idx < bfqd->num_actuators; act_idx++) { 706 /* Fast path to check if bfqq is already allocated. */ 707 if (!bic_to_bfqq(bic, op_is_sync(opf), act_idx)) 708 continue; 709 710 /* 711 * Does queue (or any parent entity) exceed number of 712 * requests that should be available to it? Heavily 713 * limit depth so that it cannot consume more 714 * available requests and thus starve other entities. 715 */ 716 if (bfqq_request_over_limit(bfqd, bic, opf, act_idx, limit)) { 717 limit = 1; 718 break; 719 } 720 } 721 722 bfq_log(bfqd, "[%s] wr_busy %d sync %d depth %u", 723 __func__, bfqd->wr_busy_queues, op_is_sync(opf), limit); 724 725 if (limit < data->q->nr_requests) 726 data->shallow_depth = limit; 727} 728 729static struct bfq_queue * 730bfq_rq_pos_tree_lookup(struct bfq_data *bfqd, struct rb_root *root, 731 sector_t sector, struct rb_node **ret_parent, 732 struct rb_node ***rb_link) 733{ 734 struct rb_node **p, *parent; 735 struct bfq_queue *bfqq = NULL; 736 737 parent = NULL; 738 p = &root->rb_node; 739 while (*p) { 740 struct rb_node **n; 741 742 parent = *p; 743 bfqq = rb_entry(parent, struct bfq_queue, pos_node); 744 745 /* 746 * Sort strictly based on sector. Smallest to the left, 747 * largest to the right. 748 */ 749 if (sector > blk_rq_pos(bfqq->next_rq)) 750 n = &(*p)->rb_right; 751 else if (sector < blk_rq_pos(bfqq->next_rq)) 752 n = &(*p)->rb_left; 753 else 754 break; 755 p = n; 756 bfqq = NULL; 757 } 758 759 *ret_parent = parent; 760 if (rb_link) 761 *rb_link = p; 762 763 bfq_log(bfqd, "rq_pos_tree_lookup %llu: returning %d", 764 (unsigned long long)sector, 765 bfqq ? bfqq->pid : 0); 766 767 return bfqq; 768} 769 770static bool bfq_too_late_for_merging(struct bfq_queue *bfqq) 771{ 772 return bfqq->service_from_backlogged > 0 && 773 time_is_before_jiffies(bfqq->first_IO_time + 774 bfq_merge_time_limit); 775} 776 777/* 778 * The following function is not marked as __cold because it is 779 * actually cold, but for the same performance goal described in the 780 * comments on the likely() at the beginning of 781 * bfq_setup_cooperator(). Unexpectedly, to reach an even lower 782 * execution time for the case where this function is not invoked, we 783 * had to add an unlikely() in each involved if(). 784 */ 785void __cold 786bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq) 787{ 788 struct rb_node **p, *parent; 789 struct bfq_queue *__bfqq; 790 791 if (bfqq->pos_root) { 792 rb_erase(&bfqq->pos_node, bfqq->pos_root); 793 bfqq->pos_root = NULL; 794 } 795 796 /* oom_bfqq does not participate in queue merging */ 797 if (bfqq == &bfqd->oom_bfqq) 798 return; 799 800 /* 801 * bfqq cannot be merged any longer (see comments in 802 * bfq_setup_cooperator): no point in adding bfqq into the 803 * position tree. 804 */ 805 if (bfq_too_late_for_merging(bfqq)) 806 return; 807 808 if (bfq_class_idle(bfqq)) 809 return; 810 if (!bfqq->next_rq) 811 return; 812 813 bfqq->pos_root = &bfqq_group(bfqq)->rq_pos_tree; 814 __bfqq = bfq_rq_pos_tree_lookup(bfqd, bfqq->pos_root, 815 blk_rq_pos(bfqq->next_rq), &parent, &p); 816 if (!__bfqq) { 817 rb_link_node(&bfqq->pos_node, parent, p); 818 rb_insert_color(&bfqq->pos_node, bfqq->pos_root); 819 } else 820 bfqq->pos_root = NULL; 821} 822 823/* 824 * The following function returns false either if every active queue 825 * must receive the same share of the throughput (symmetric scenario), 826 * or, as a special case, if bfqq must receive a share of the 827 * throughput lower than or equal to the share that every other active 828 * queue must receive. If bfqq does sync I/O, then these are the only 829 * two cases where bfqq happens to be guaranteed its share of the 830 * throughput even if I/O dispatching is not plugged when bfqq remains 831 * temporarily empty (for more details, see the comments in the 832 * function bfq_better_to_idle()). For this reason, the return value 833 * of this function is used to check whether I/O-dispatch plugging can 834 * be avoided. 835 * 836 * The above first case (symmetric scenario) occurs when: 837 * 1) all active queues have the same weight, 838 * 2) all active queues belong to the same I/O-priority class, 839 * 3) all active groups at the same level in the groups tree have the same 840 * weight, 841 * 4) all active groups at the same level in the groups tree have the same 842 * number of children. 843 * 844 * Unfortunately, keeping the necessary state for evaluating exactly 845 * the last two symmetry sub-conditions above would be quite complex 846 * and time consuming. Therefore this function evaluates, instead, 847 * only the following stronger three sub-conditions, for which it is 848 * much easier to maintain the needed state: 849 * 1) all active queues have the same weight, 850 * 2) all active queues belong to the same I/O-priority class, 851 * 3) there is at most one active group. 852 * In particular, the last condition is always true if hierarchical 853 * support or the cgroups interface are not enabled, thus no state 854 * needs to be maintained in this case. 855 */ 856static bool bfq_asymmetric_scenario(struct bfq_data *bfqd, 857 struct bfq_queue *bfqq) 858{ 859 bool smallest_weight = bfqq && 860 bfqq->weight_counter && 861 bfqq->weight_counter == 862 container_of( 863 rb_first_cached(&bfqd->queue_weights_tree), 864 struct bfq_weight_counter, 865 weights_node); 866 867 /* 868 * For queue weights to differ, queue_weights_tree must contain 869 * at least two nodes. 870 */ 871 bool varied_queue_weights = !smallest_weight && 872 !RB_EMPTY_ROOT(&bfqd->queue_weights_tree.rb_root) && 873 (bfqd->queue_weights_tree.rb_root.rb_node->rb_left || 874 bfqd->queue_weights_tree.rb_root.rb_node->rb_right); 875 876 bool multiple_classes_busy = 877 (bfqd->busy_queues[0] && bfqd->busy_queues[1]) || 878 (bfqd->busy_queues[0] && bfqd->busy_queues[2]) || 879 (bfqd->busy_queues[1] && bfqd->busy_queues[2]); 880 881 return varied_queue_weights || multiple_classes_busy 882#ifdef CONFIG_BFQ_GROUP_IOSCHED 883 || bfqd->num_groups_with_pending_reqs > 1 884#endif 885 ; 886} 887 888/* 889 * If the weight-counter tree passed as input contains no counter for 890 * the weight of the input queue, then add that counter; otherwise just 891 * increment the existing counter. 892 * 893 * Note that weight-counter trees contain few nodes in mostly symmetric 894 * scenarios. For example, if all queues have the same weight, then the 895 * weight-counter tree for the queues may contain at most one node. 896 * This holds even if low_latency is on, because weight-raised queues 897 * are not inserted in the tree. 898 * In most scenarios, the rate at which nodes are created/destroyed 899 * should be low too. 900 */ 901void bfq_weights_tree_add(struct bfq_queue *bfqq) 902{ 903 struct rb_root_cached *root = &bfqq->bfqd->queue_weights_tree; 904 struct bfq_entity *entity = &bfqq->entity; 905 struct rb_node **new = &(root->rb_root.rb_node), *parent = NULL; 906 bool leftmost = true; 907 908 /* 909 * Do not insert if the queue is already associated with a 910 * counter, which happens if: 911 * 1) a request arrival has caused the queue to become both 912 * non-weight-raised, and hence change its weight, and 913 * backlogged; in this respect, each of the two events 914 * causes an invocation of this function, 915 * 2) this is the invocation of this function caused by the 916 * second event. This second invocation is actually useless, 917 * and we handle this fact by exiting immediately. More 918 * efficient or clearer solutions might possibly be adopted. 919 */ 920 if (bfqq->weight_counter) 921 return; 922 923 while (*new) { 924 struct bfq_weight_counter *__counter = container_of(*new, 925 struct bfq_weight_counter, 926 weights_node); 927 parent = *new; 928 929 if (entity->weight == __counter->weight) { 930 bfqq->weight_counter = __counter; 931 goto inc_counter; 932 } 933 if (entity->weight < __counter->weight) 934 new = &((*new)->rb_left); 935 else { 936 new = &((*new)->rb_right); 937 leftmost = false; 938 } 939 } 940 941 bfqq->weight_counter = kzalloc(sizeof(struct bfq_weight_counter), 942 GFP_ATOMIC); 943 944 /* 945 * In the unlucky event of an allocation failure, we just 946 * exit. This will cause the weight of queue to not be 947 * considered in bfq_asymmetric_scenario, which, in its turn, 948 * causes the scenario to be deemed wrongly symmetric in case 949 * bfqq's weight would have been the only weight making the 950 * scenario asymmetric. On the bright side, no unbalance will 951 * however occur when bfqq becomes inactive again (the 952 * invocation of this function is triggered by an activation 953 * of queue). In fact, bfq_weights_tree_remove does nothing 954 * if !bfqq->weight_counter. 955 */ 956 if (unlikely(!bfqq->weight_counter)) 957 return; 958 959 bfqq->weight_counter->weight = entity->weight; 960 rb_link_node(&bfqq->weight_counter->weights_node, parent, new); 961 rb_insert_color_cached(&bfqq->weight_counter->weights_node, root, 962 leftmost); 963 964inc_counter: 965 bfqq->weight_counter->num_active++; 966 bfqq->ref++; 967} 968 969/* 970 * Decrement the weight counter associated with the queue, and, if the 971 * counter reaches 0, remove the counter from the tree. 972 * See the comments to the function bfq_weights_tree_add() for considerations 973 * about overhead. 974 */ 975void bfq_weights_tree_remove(struct bfq_queue *bfqq) 976{ 977 struct rb_root_cached *root; 978 979 if (!bfqq->weight_counter) 980 return; 981 982 root = &bfqq->bfqd->queue_weights_tree; 983 bfqq->weight_counter->num_active--; 984 if (bfqq->weight_counter->num_active > 0) 985 goto reset_entity_pointer; 986 987 rb_erase_cached(&bfqq->weight_counter->weights_node, root); 988 kfree(bfqq->weight_counter); 989 990reset_entity_pointer: 991 bfqq->weight_counter = NULL; 992 bfq_put_queue(bfqq); 993} 994 995/* 996 * Return expired entry, or NULL to just start from scratch in rbtree. 997 */ 998static struct request *bfq_check_fifo(struct bfq_queue *bfqq, 999 struct request *last) 1000{ 1001 struct request *rq; 1002 1003 if (bfq_bfqq_fifo_expire(bfqq)) 1004 return NULL; 1005 1006 bfq_mark_bfqq_fifo_expire(bfqq); 1007 1008 rq = rq_entry_fifo(bfqq->fifo.next); 1009 1010 if (rq == last || blk_time_get_ns() < rq->fifo_time) 1011 return NULL; 1012 1013 bfq_log_bfqq(bfqq->bfqd, bfqq, "check_fifo: returned %p", rq); 1014 return rq; 1015} 1016 1017static struct request *bfq_find_next_rq(struct bfq_data *bfqd, 1018 struct bfq_queue *bfqq, 1019 struct request *last) 1020{ 1021 struct rb_node *rbnext = rb_next(&last->rb_node); 1022 struct rb_node *rbprev = rb_prev(&last->rb_node); 1023 struct request *next, *prev = NULL; 1024 1025 /* Follow expired path, else get first next available. */ 1026 next = bfq_check_fifo(bfqq, last); 1027 if (next) 1028 return next; 1029 1030 if (rbprev) 1031 prev = rb_entry_rq(rbprev); 1032 1033 if (rbnext) 1034 next = rb_entry_rq(rbnext); 1035 else { 1036 rbnext = rb_first(&bfqq->sort_list); 1037 if (rbnext && rbnext != &last->rb_node) 1038 next = rb_entry_rq(rbnext); 1039 } 1040 1041 return bfq_choose_req(bfqd, next, prev, blk_rq_pos(last)); 1042} 1043 1044/* see the definition of bfq_async_charge_factor for details */ 1045static unsigned long bfq_serv_to_charge(struct request *rq, 1046 struct bfq_queue *bfqq) 1047{ 1048 if (bfq_bfqq_sync(bfqq) || bfqq->wr_coeff > 1 || 1049 bfq_asymmetric_scenario(bfqq->bfqd, bfqq)) 1050 return blk_rq_sectors(rq); 1051 1052 return blk_rq_sectors(rq) * bfq_async_charge_factor; 1053} 1054 1055/** 1056 * bfq_updated_next_req - update the queue after a new next_rq selection. 1057 * @bfqd: the device data the queue belongs to. 1058 * @bfqq: the queue to update. 1059 * 1060 * If the first request of a queue changes we make sure that the queue 1061 * has enough budget to serve at least its first request (if the 1062 * request has grown). We do this because if the queue has not enough 1063 * budget for its first request, it has to go through two dispatch 1064 * rounds to actually get it dispatched. 1065 */ 1066static void bfq_updated_next_req(struct bfq_data *bfqd, 1067 struct bfq_queue *bfqq) 1068{ 1069 struct bfq_entity *entity = &bfqq->entity; 1070 struct request *next_rq = bfqq->next_rq; 1071 unsigned long new_budget; 1072 1073 if (!next_rq) 1074 return; 1075 1076 if (bfqq == bfqd->in_service_queue) 1077 /* 1078 * In order not to break guarantees, budgets cannot be 1079 * changed after an entity has been selected. 1080 */ 1081 return; 1082 1083 new_budget = max_t(unsigned long, 1084 max_t(unsigned long, bfqq->max_budget, 1085 bfq_serv_to_charge(next_rq, bfqq)), 1086 entity->service); 1087 if (entity->budget != new_budget) { 1088 entity->budget = new_budget; 1089 bfq_log_bfqq(bfqd, bfqq, "updated next rq: new budget %lu", 1090 new_budget); 1091 bfq_requeue_bfqq(bfqd, bfqq, false); 1092 } 1093} 1094 1095static unsigned int bfq_wr_duration(struct bfq_data *bfqd) 1096{ 1097 u64 dur; 1098 1099 dur = bfqd->rate_dur_prod; 1100 do_div(dur, bfqd->peak_rate); 1101 1102 /* 1103 * Limit duration between 3 and 25 seconds. The upper limit 1104 * has been conservatively set after the following worst case: 1105 * on a QEMU/KVM virtual machine 1106 * - running in a slow PC 1107 * - with a virtual disk stacked on a slow low-end 5400rpm HDD 1108 * - serving a heavy I/O workload, such as the sequential reading 1109 * of several files 1110 * mplayer took 23 seconds to start, if constantly weight-raised. 1111 * 1112 * As for higher values than that accommodating the above bad 1113 * scenario, tests show that higher values would often yield 1114 * the opposite of the desired result, i.e., would worsen 1115 * responsiveness by allowing non-interactive applications to 1116 * preserve weight raising for too long. 1117 * 1118 * On the other end, lower values than 3 seconds make it 1119 * difficult for most interactive tasks to complete their jobs 1120 * before weight-raising finishes. 1121 */ 1122 return clamp_val(dur, msecs_to_jiffies(3000), msecs_to_jiffies(25000)); 1123} 1124 1125/* switch back from soft real-time to interactive weight raising */ 1126static void switch_back_to_interactive_wr(struct bfq_queue *bfqq, 1127 struct bfq_data *bfqd) 1128{ 1129 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 1130 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 1131 bfqq->last_wr_start_finish = bfqq->wr_start_at_switch_to_srt; 1132} 1133 1134static void 1135bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, 1136 struct bfq_io_cq *bic, bool bfq_already_existing) 1137{ 1138 unsigned int old_wr_coeff = 1; 1139 bool busy = bfq_already_existing && bfq_bfqq_busy(bfqq); 1140 unsigned int a_idx = bfqq->actuator_idx; 1141 struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; 1142 1143 if (bfqq_data->saved_has_short_ttime) 1144 bfq_mark_bfqq_has_short_ttime(bfqq); 1145 else 1146 bfq_clear_bfqq_has_short_ttime(bfqq); 1147 1148 if (bfqq_data->saved_IO_bound) 1149 bfq_mark_bfqq_IO_bound(bfqq); 1150 else 1151 bfq_clear_bfqq_IO_bound(bfqq); 1152 1153 bfqq->last_serv_time_ns = bfqq_data->saved_last_serv_time_ns; 1154 bfqq->inject_limit = bfqq_data->saved_inject_limit; 1155 bfqq->decrease_time_jif = bfqq_data->saved_decrease_time_jif; 1156 1157 bfqq->entity.new_weight = bfqq_data->saved_weight; 1158 bfqq->ttime = bfqq_data->saved_ttime; 1159 bfqq->io_start_time = bfqq_data->saved_io_start_time; 1160 bfqq->tot_idle_time = bfqq_data->saved_tot_idle_time; 1161 /* 1162 * Restore weight coefficient only if low_latency is on 1163 */ 1164 if (bfqd->low_latency) { 1165 old_wr_coeff = bfqq->wr_coeff; 1166 bfqq->wr_coeff = bfqq_data->saved_wr_coeff; 1167 } 1168 bfqq->service_from_wr = bfqq_data->saved_service_from_wr; 1169 bfqq->wr_start_at_switch_to_srt = 1170 bfqq_data->saved_wr_start_at_switch_to_srt; 1171 bfqq->last_wr_start_finish = bfqq_data->saved_last_wr_start_finish; 1172 bfqq->wr_cur_max_time = bfqq_data->saved_wr_cur_max_time; 1173 1174 if (bfqq->wr_coeff > 1 && (bfq_bfqq_in_large_burst(bfqq) || 1175 time_is_before_jiffies(bfqq->last_wr_start_finish + 1176 bfqq->wr_cur_max_time))) { 1177 if (bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time && 1178 !bfq_bfqq_in_large_burst(bfqq) && 1179 time_is_after_eq_jiffies(bfqq->wr_start_at_switch_to_srt + 1180 bfq_wr_duration(bfqd))) { 1181 switch_back_to_interactive_wr(bfqq, bfqd); 1182 } else { 1183 bfqq->wr_coeff = 1; 1184 bfq_log_bfqq(bfqq->bfqd, bfqq, 1185 "resume state: switching off wr"); 1186 } 1187 } 1188 1189 /* make sure weight will be updated, however we got here */ 1190 bfqq->entity.prio_changed = 1; 1191 1192 if (likely(!busy)) 1193 return; 1194 1195 if (old_wr_coeff == 1 && bfqq->wr_coeff > 1) 1196 bfqd->wr_busy_queues++; 1197 else if (old_wr_coeff > 1 && bfqq->wr_coeff == 1) 1198 bfqd->wr_busy_queues--; 1199} 1200 1201static int bfqq_process_refs(struct bfq_queue *bfqq) 1202{ 1203 return bfqq->ref - bfqq->entity.allocated - 1204 bfqq->entity.on_st_or_in_serv - 1205 (bfqq->weight_counter != NULL) - bfqq->stable_ref; 1206} 1207 1208/* Empty burst list and add just bfqq (see comments on bfq_handle_burst) */ 1209static void bfq_reset_burst_list(struct bfq_data *bfqd, struct bfq_queue *bfqq) 1210{ 1211 struct bfq_queue *item; 1212 struct hlist_node *n; 1213 1214 hlist_for_each_entry_safe(item, n, &bfqd->burst_list, burst_list_node) 1215 hlist_del_init(&item->burst_list_node); 1216 1217 /* 1218 * Start the creation of a new burst list only if there is no 1219 * active queue. See comments on the conditional invocation of 1220 * bfq_handle_burst(). 1221 */ 1222 if (bfq_tot_busy_queues(bfqd) == 0) { 1223 hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list); 1224 bfqd->burst_size = 1; 1225 } else 1226 bfqd->burst_size = 0; 1227 1228 bfqd->burst_parent_entity = bfqq->entity.parent; 1229} 1230 1231/* Add bfqq to the list of queues in current burst (see bfq_handle_burst) */ 1232static void bfq_add_to_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq) 1233{ 1234 /* Increment burst size to take into account also bfqq */ 1235 bfqd->burst_size++; 1236 1237 if (bfqd->burst_size == bfqd->bfq_large_burst_thresh) { 1238 struct bfq_queue *pos, *bfqq_item; 1239 struct hlist_node *n; 1240 1241 /* 1242 * Enough queues have been activated shortly after each 1243 * other to consider this burst as large. 1244 */ 1245 bfqd->large_burst = true; 1246 1247 /* 1248 * We can now mark all queues in the burst list as 1249 * belonging to a large burst. 1250 */ 1251 hlist_for_each_entry(bfqq_item, &bfqd->burst_list, 1252 burst_list_node) 1253 bfq_mark_bfqq_in_large_burst(bfqq_item); 1254 bfq_mark_bfqq_in_large_burst(bfqq); 1255 1256 /* 1257 * From now on, and until the current burst finishes, any 1258 * new queue being activated shortly after the last queue 1259 * was inserted in the burst can be immediately marked as 1260 * belonging to a large burst. So the burst list is not 1261 * needed any more. Remove it. 1262 */ 1263 hlist_for_each_entry_safe(pos, n, &bfqd->burst_list, 1264 burst_list_node) 1265 hlist_del_init(&pos->burst_list_node); 1266 } else /* 1267 * Burst not yet large: add bfqq to the burst list. Do 1268 * not increment the ref counter for bfqq, because bfqq 1269 * is removed from the burst list before freeing bfqq 1270 * in put_queue. 1271 */ 1272 hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list); 1273} 1274 1275/* 1276 * If many queues belonging to the same group happen to be created 1277 * shortly after each other, then the processes associated with these 1278 * queues have typically a common goal. In particular, bursts of queue 1279 * creations are usually caused by services or applications that spawn 1280 * many parallel threads/processes. Examples are systemd during boot, 1281 * or git grep. To help these processes get their job done as soon as 1282 * possible, it is usually better to not grant either weight-raising 1283 * or device idling to their queues, unless these queues must be 1284 * protected from the I/O flowing through other active queues. 1285 * 1286 * In this comment we describe, firstly, the reasons why this fact 1287 * holds, and, secondly, the next function, which implements the main 1288 * steps needed to properly mark these queues so that they can then be 1289 * treated in a different way. 1290 * 1291 * The above services or applications benefit mostly from a high 1292 * throughput: the quicker the requests of the activated queues are 1293 * cumulatively served, the sooner the target job of these queues gets 1294 * completed. As a consequence, weight-raising any of these queues, 1295 * which also implies idling the device for it, is almost always 1296 * counterproductive, unless there are other active queues to isolate 1297 * these new queues from. If there no other active queues, then 1298 * weight-raising these new queues just lowers throughput in most 1299 * cases. 1300 * 1301 * On the other hand, a burst of queue creations may be caused also by 1302 * the start of an application that does not consist of a lot of 1303 * parallel I/O-bound threads. In fact, with a complex application, 1304 * several short processes may need to be executed to start-up the 1305 * application. In this respect, to start an application as quickly as 1306 * possible, the best thing to do is in any case to privilege the I/O 1307 * related to the application with respect to all other 1308 * I/O. Therefore, the best strategy to start as quickly as possible 1309 * an application that causes a burst of queue creations is to 1310 * weight-raise all the queues created during the burst. This is the 1311 * exact opposite of the best strategy for the other type of bursts. 1312 * 1313 * In the end, to take the best action for each of the two cases, the 1314 * two types of bursts need to be distinguished. Fortunately, this 1315 * seems relatively easy, by looking at the sizes of the bursts. In 1316 * particular, we found a threshold such that only bursts with a 1317 * larger size than that threshold are apparently caused by 1318 * services or commands such as systemd or git grep. For brevity, 1319 * hereafter we call just 'large' these bursts. BFQ *does not* 1320 * weight-raise queues whose creation occurs in a large burst. In 1321 * addition, for each of these queues BFQ performs or does not perform 1322 * idling depending on which choice boosts the throughput more. The 1323 * exact choice depends on the device and request pattern at 1324 * hand. 1325 * 1326 * Unfortunately, false positives may occur while an interactive task 1327 * is starting (e.g., an application is being started). The 1328 * consequence is that the queues associated with the task do not 1329 * enjoy weight raising as expected. Fortunately these false positives 1330 * are very rare. They typically occur if some service happens to 1331 * start doing I/O exactly when the interactive task starts. 1332 * 1333 * Turning back to the next function, it is invoked only if there are 1334 * no active queues (apart from active queues that would belong to the 1335 * same, possible burst bfqq would belong to), and it implements all 1336 * the steps needed to detect the occurrence of a large burst and to 1337 * properly mark all the queues belonging to it (so that they can then 1338 * be treated in a different way). This goal is achieved by 1339 * maintaining a "burst list" that holds, temporarily, the queues that 1340 * belong to the burst in progress. The list is then used to mark 1341 * these queues as belonging to a large burst if the burst does become 1342 * large. The main steps are the following. 1343 * 1344 * . when the very first queue is created, the queue is inserted into the 1345 * list (as it could be the first queue in a possible burst) 1346 * 1347 * . if the current burst has not yet become large, and a queue Q that does 1348 * not yet belong to the burst is activated shortly after the last time 1349 * at which a new queue entered the burst list, then the function appends 1350 * Q to the burst list 1351 * 1352 * . if, as a consequence of the previous step, the burst size reaches 1353 * the large-burst threshold, then 1354 * 1355 * . all the queues in the burst list are marked as belonging to a 1356 * large burst 1357 * 1358 * . the burst list is deleted; in fact, the burst list already served 1359 * its purpose (keeping temporarily track of the queues in a burst, 1360 * so as to be able to mark them as belonging to a large burst in the 1361 * previous sub-step), and now is not needed any more 1362 * 1363 * . the device enters a large-burst mode 1364 * 1365 * . if a queue Q that does not belong to the burst is created while 1366 * the device is in large-burst mode and shortly after the last time 1367 * at which a queue either entered the burst list or was marked as 1368 * belonging to the current large burst, then Q is immediately marked 1369 * as belonging to a large burst. 1370 * 1371 * . if a queue Q that does not belong to the burst is created a while 1372 * later, i.e., not shortly after, than the last time at which a queue 1373 * either entered the burst list or was marked as belonging to the 1374 * current large burst, then the current burst is deemed as finished and: 1375 * 1376 * . the large-burst mode is reset if set 1377 * 1378 * . the burst list is emptied 1379 * 1380 * . Q is inserted in the burst list, as Q may be the first queue 1381 * in a possible new burst (then the burst list contains just Q 1382 * after this step). 1383 */ 1384static void bfq_handle_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq) 1385{ 1386 /* 1387 * If bfqq is already in the burst list or is part of a large 1388 * burst, or finally has just been split, then there is 1389 * nothing else to do. 1390 */ 1391 if (!hlist_unhashed(&bfqq->burst_list_node) || 1392 bfq_bfqq_in_large_burst(bfqq) || 1393 time_is_after_eq_jiffies(bfqq->split_time + 1394 msecs_to_jiffies(10))) 1395 return; 1396 1397 /* 1398 * If bfqq's creation happens late enough, or bfqq belongs to 1399 * a different group than the burst group, then the current 1400 * burst is finished, and related data structures must be 1401 * reset. 1402 * 1403 * In this respect, consider the special case where bfqq is 1404 * the very first queue created after BFQ is selected for this 1405 * device. In this case, last_ins_in_burst and 1406 * burst_parent_entity are not yet significant when we get 1407 * here. But it is easy to verify that, whether or not the 1408 * following condition is true, bfqq will end up being 1409 * inserted into the burst list. In particular the list will 1410 * happen to contain only bfqq. And this is exactly what has 1411 * to happen, as bfqq may be the first queue of the first 1412 * burst. 1413 */ 1414 if (time_is_before_jiffies(bfqd->last_ins_in_burst + 1415 bfqd->bfq_burst_interval) || 1416 bfqq->entity.parent != bfqd->burst_parent_entity) { 1417 bfqd->large_burst = false; 1418 bfq_reset_burst_list(bfqd, bfqq); 1419 goto end; 1420 } 1421 1422 /* 1423 * If we get here, then bfqq is being activated shortly after the 1424 * last queue. So, if the current burst is also large, we can mark 1425 * bfqq as belonging to this large burst immediately. 1426 */ 1427 if (bfqd->large_burst) { 1428 bfq_mark_bfqq_in_large_burst(bfqq); 1429 goto end; 1430 } 1431 1432 /* 1433 * If we get here, then a large-burst state has not yet been 1434 * reached, but bfqq is being activated shortly after the last 1435 * queue. Then we add bfqq to the burst. 1436 */ 1437 bfq_add_to_burst(bfqd, bfqq); 1438end: 1439 /* 1440 * At this point, bfqq either has been added to the current 1441 * burst or has caused the current burst to terminate and a 1442 * possible new burst to start. In particular, in the second 1443 * case, bfqq has become the first queue in the possible new 1444 * burst. In both cases last_ins_in_burst needs to be moved 1445 * forward. 1446 */ 1447 bfqd->last_ins_in_burst = jiffies; 1448} 1449 1450static int bfq_bfqq_budget_left(struct bfq_queue *bfqq) 1451{ 1452 struct bfq_entity *entity = &bfqq->entity; 1453 1454 return entity->budget - entity->service; 1455} 1456 1457/* 1458 * If enough samples have been computed, return the current max budget 1459 * stored in bfqd, which is dynamically updated according to the 1460 * estimated disk peak rate; otherwise return the default max budget 1461 */ 1462static int bfq_max_budget(struct bfq_data *bfqd) 1463{ 1464 if (bfqd->budgets_assigned < bfq_stats_min_budgets) 1465 return bfq_default_max_budget; 1466 else 1467 return bfqd->bfq_max_budget; 1468} 1469 1470/* 1471 * Return min budget, which is a fraction of the current or default 1472 * max budget (trying with 1/32) 1473 */ 1474static int bfq_min_budget(struct bfq_data *bfqd) 1475{ 1476 if (bfqd->budgets_assigned < bfq_stats_min_budgets) 1477 return bfq_default_max_budget / 32; 1478 else 1479 return bfqd->bfq_max_budget / 32; 1480} 1481 1482/* 1483 * The next function, invoked after the input queue bfqq switches from 1484 * idle to busy, updates the budget of bfqq. The function also tells 1485 * whether the in-service queue should be expired, by returning 1486 * true. The purpose of expiring the in-service queue is to give bfqq 1487 * the chance to possibly preempt the in-service queue, and the reason 1488 * for preempting the in-service queue is to achieve one of the two 1489 * goals below. 1490 * 1491 * 1. Guarantee to bfqq its reserved bandwidth even if bfqq has 1492 * expired because it has remained idle. In particular, bfqq may have 1493 * expired for one of the following two reasons: 1494 * 1495 * - BFQQE_NO_MORE_REQUESTS bfqq did not enjoy any device idling 1496 * and did not make it to issue a new request before its last 1497 * request was served; 1498 * 1499 * - BFQQE_TOO_IDLE bfqq did enjoy device idling, but did not issue 1500 * a new request before the expiration of the idling-time. 1501 * 1502 * Even if bfqq has expired for one of the above reasons, the process 1503 * associated with the queue may be however issuing requests greedily, 1504 * and thus be sensitive to the bandwidth it receives (bfqq may have 1505 * remained idle for other reasons: CPU high load, bfqq not enjoying 1506 * idling, I/O throttling somewhere in the path from the process to 1507 * the I/O scheduler, ...). But if, after every expiration for one of 1508 * the above two reasons, bfqq has to wait for the service of at least 1509 * one full budget of another queue before being served again, then 1510 * bfqq is likely to get a much lower bandwidth or resource time than 1511 * its reserved ones. To address this issue, two countermeasures need 1512 * to be taken. 1513 * 1514 * First, the budget and the timestamps of bfqq need to be updated in 1515 * a special way on bfqq reactivation: they need to be updated as if 1516 * bfqq did not remain idle and did not expire. In fact, if they are 1517 * computed as if bfqq expired and remained idle until reactivation, 1518 * then the process associated with bfqq is treated as if, instead of 1519 * being greedy, it stopped issuing requests when bfqq remained idle, 1520 * and restarts issuing requests only on this reactivation. In other 1521 * words, the scheduler does not help the process recover the "service 1522 * hole" between bfqq expiration and reactivation. As a consequence, 1523 * the process receives a lower bandwidth than its reserved one. In 1524 * contrast, to recover this hole, the budget must be updated as if 1525 * bfqq was not expired at all before this reactivation, i.e., it must 1526 * be set to the value of the remaining budget when bfqq was 1527 * expired. Along the same line, timestamps need to be assigned the 1528 * value they had the last time bfqq was selected for service, i.e., 1529 * before last expiration. Thus timestamps need to be back-shifted 1530 * with respect to their normal computation (see [1] for more details 1531 * on this tricky aspect). 1532 * 1533 * Secondly, to allow the process to recover the hole, the in-service 1534 * queue must be expired too, to give bfqq the chance to preempt it 1535 * immediately. In fact, if bfqq has to wait for a full budget of the 1536 * in-service queue to be completed, then it may become impossible to 1537 * let the process recover the hole, even if the back-shifted 1538 * timestamps of bfqq are lower than those of the in-service queue. If 1539 * this happens for most or all of the holes, then the process may not 1540 * receive its reserved bandwidth. In this respect, it is worth noting 1541 * that, being the service of outstanding requests unpreemptible, a 1542 * little fraction of the holes may however be unrecoverable, thereby 1543 * causing a little loss of bandwidth. 1544 * 1545 * The last important point is detecting whether bfqq does need this 1546 * bandwidth recovery. In this respect, the next function deems the 1547 * process associated with bfqq greedy, and thus allows it to recover 1548 * the hole, if: 1) the process is waiting for the arrival of a new 1549 * request (which implies that bfqq expired for one of the above two 1550 * reasons), and 2) such a request has arrived soon. The first 1551 * condition is controlled through the flag non_blocking_wait_rq, 1552 * while the second through the flag arrived_in_time. If both 1553 * conditions hold, then the function computes the budget in the 1554 * above-described special way, and signals that the in-service queue 1555 * should be expired. Timestamp back-shifting is done later in 1556 * __bfq_activate_entity. 1557 * 1558 * 2. Reduce latency. Even if timestamps are not backshifted to let 1559 * the process associated with bfqq recover a service hole, bfqq may 1560 * however happen to have, after being (re)activated, a lower finish 1561 * timestamp than the in-service queue. That is, the next budget of 1562 * bfqq may have to be completed before the one of the in-service 1563 * queue. If this is the case, then preempting the in-service queue 1564 * allows this goal to be achieved, apart from the unpreemptible, 1565 * outstanding requests mentioned above. 1566 * 1567 * Unfortunately, regardless of which of the above two goals one wants 1568 * to achieve, service trees need first to be updated to know whether 1569 * the in-service queue must be preempted. To have service trees 1570 * correctly updated, the in-service queue must be expired and 1571 * rescheduled, and bfqq must be scheduled too. This is one of the 1572 * most costly operations (in future versions, the scheduling 1573 * mechanism may be re-designed in such a way to make it possible to 1574 * know whether preemption is needed without needing to update service 1575 * trees). In addition, queue preemptions almost always cause random 1576 * I/O, which may in turn cause loss of throughput. Finally, there may 1577 * even be no in-service queue when the next function is invoked (so, 1578 * no queue to compare timestamps with). Because of these facts, the 1579 * next function adopts the following simple scheme to avoid costly 1580 * operations, too frequent preemptions and too many dependencies on 1581 * the state of the scheduler: it requests the expiration of the 1582 * in-service queue (unconditionally) only for queues that need to 1583 * recover a hole. Then it delegates to other parts of the code the 1584 * responsibility of handling the above case 2. 1585 */ 1586static bool bfq_bfqq_update_budg_for_activation(struct bfq_data *bfqd, 1587 struct bfq_queue *bfqq, 1588 bool arrived_in_time) 1589{ 1590 struct bfq_entity *entity = &bfqq->entity; 1591 1592 /* 1593 * In the next compound condition, we check also whether there 1594 * is some budget left, because otherwise there is no point in 1595 * trying to go on serving bfqq with this same budget: bfqq 1596 * would be expired immediately after being selected for 1597 * service. This would only cause useless overhead. 1598 */ 1599 if (bfq_bfqq_non_blocking_wait_rq(bfqq) && arrived_in_time && 1600 bfq_bfqq_budget_left(bfqq) > 0) { 1601 /* 1602 * We do not clear the flag non_blocking_wait_rq here, as 1603 * the latter is used in bfq_activate_bfqq to signal 1604 * that timestamps need to be back-shifted (and is 1605 * cleared right after). 1606 */ 1607 1608 /* 1609 * In next assignment we rely on that either 1610 * entity->service or entity->budget are not updated 1611 * on expiration if bfqq is empty (see 1612 * __bfq_bfqq_recalc_budget). Thus both quantities 1613 * remain unchanged after such an expiration, and the 1614 * following statement therefore assigns to 1615 * entity->budget the remaining budget on such an 1616 * expiration. 1617 */ 1618 entity->budget = min_t(unsigned long, 1619 bfq_bfqq_budget_left(bfqq), 1620 bfqq->max_budget); 1621 1622 /* 1623 * At this point, we have used entity->service to get 1624 * the budget left (needed for updating 1625 * entity->budget). Thus we finally can, and have to, 1626 * reset entity->service. The latter must be reset 1627 * because bfqq would otherwise be charged again for 1628 * the service it has received during its previous 1629 * service slot(s). 1630 */ 1631 entity->service = 0; 1632 1633 return true; 1634 } 1635 1636 /* 1637 * We can finally complete expiration, by setting service to 0. 1638 */ 1639 entity->service = 0; 1640 entity->budget = max_t(unsigned long, bfqq->max_budget, 1641 bfq_serv_to_charge(bfqq->next_rq, bfqq)); 1642 bfq_clear_bfqq_non_blocking_wait_rq(bfqq); 1643 return false; 1644} 1645 1646/* 1647 * Return the farthest past time instant according to jiffies 1648 * macros. 1649 */ 1650static unsigned long bfq_smallest_from_now(void) 1651{ 1652 return jiffies - MAX_JIFFY_OFFSET; 1653} 1654 1655static void bfq_update_bfqq_wr_on_rq_arrival(struct bfq_data *bfqd, 1656 struct bfq_queue *bfqq, 1657 unsigned int old_wr_coeff, 1658 bool wr_or_deserves_wr, 1659 bool interactive, 1660 bool in_burst, 1661 bool soft_rt) 1662{ 1663 if (old_wr_coeff == 1 && wr_or_deserves_wr) { 1664 /* start a weight-raising period */ 1665 if (interactive) { 1666 bfqq->service_from_wr = 0; 1667 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 1668 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 1669 } else { 1670 /* 1671 * No interactive weight raising in progress 1672 * here: assign minus infinity to 1673 * wr_start_at_switch_to_srt, to make sure 1674 * that, at the end of the soft-real-time 1675 * weight raising periods that is starting 1676 * now, no interactive weight-raising period 1677 * may be wrongly considered as still in 1678 * progress (and thus actually started by 1679 * mistake). 1680 */ 1681 bfqq->wr_start_at_switch_to_srt = 1682 bfq_smallest_from_now(); 1683 bfqq->wr_coeff = bfqd->bfq_wr_coeff * 1684 BFQ_SOFTRT_WEIGHT_FACTOR; 1685 bfqq->wr_cur_max_time = 1686 bfqd->bfq_wr_rt_max_time; 1687 } 1688 1689 /* 1690 * If needed, further reduce budget to make sure it is 1691 * close to bfqq's backlog, so as to reduce the 1692 * scheduling-error component due to a too large 1693 * budget. Do not care about throughput consequences, 1694 * but only about latency. Finally, do not assign a 1695 * too small budget either, to avoid increasing 1696 * latency by causing too frequent expirations. 1697 */ 1698 bfqq->entity.budget = min_t(unsigned long, 1699 bfqq->entity.budget, 1700 2 * bfq_min_budget(bfqd)); 1701 } else if (old_wr_coeff > 1) { 1702 if (interactive) { /* update wr coeff and duration */ 1703 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 1704 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 1705 } else if (in_burst) 1706 bfqq->wr_coeff = 1; 1707 else if (soft_rt) { 1708 /* 1709 * The application is now or still meeting the 1710 * requirements for being deemed soft rt. We 1711 * can then correctly and safely (re)charge 1712 * the weight-raising duration for the 1713 * application with the weight-raising 1714 * duration for soft rt applications. 1715 * 1716 * In particular, doing this recharge now, i.e., 1717 * before the weight-raising period for the 1718 * application finishes, reduces the probability 1719 * of the following negative scenario: 1720 * 1) the weight of a soft rt application is 1721 * raised at startup (as for any newly 1722 * created application), 1723 * 2) since the application is not interactive, 1724 * at a certain time weight-raising is 1725 * stopped for the application, 1726 * 3) at that time the application happens to 1727 * still have pending requests, and hence 1728 * is destined to not have a chance to be 1729 * deemed soft rt before these requests are 1730 * completed (see the comments to the 1731 * function bfq_bfqq_softrt_next_start() 1732 * for details on soft rt detection), 1733 * 4) these pending requests experience a high 1734 * latency because the application is not 1735 * weight-raised while they are pending. 1736 */ 1737 if (bfqq->wr_cur_max_time != 1738 bfqd->bfq_wr_rt_max_time) { 1739 bfqq->wr_start_at_switch_to_srt = 1740 bfqq->last_wr_start_finish; 1741 1742 bfqq->wr_cur_max_time = 1743 bfqd->bfq_wr_rt_max_time; 1744 bfqq->wr_coeff = bfqd->bfq_wr_coeff * 1745 BFQ_SOFTRT_WEIGHT_FACTOR; 1746 } 1747 bfqq->last_wr_start_finish = jiffies; 1748 } 1749 } 1750} 1751 1752static bool bfq_bfqq_idle_for_long_time(struct bfq_data *bfqd, 1753 struct bfq_queue *bfqq) 1754{ 1755 return bfqq->dispatched == 0 && 1756 time_is_before_jiffies( 1757 bfqq->budget_timeout + 1758 bfqd->bfq_wr_min_idle_time); 1759} 1760 1761 1762/* 1763 * Return true if bfqq is in a higher priority class, or has a higher 1764 * weight than the in-service queue. 1765 */ 1766static bool bfq_bfqq_higher_class_or_weight(struct bfq_queue *bfqq, 1767 struct bfq_queue *in_serv_bfqq) 1768{ 1769 int bfqq_weight, in_serv_weight; 1770 1771 if (bfqq->ioprio_class < in_serv_bfqq->ioprio_class) 1772 return true; 1773 1774 if (in_serv_bfqq->entity.parent == bfqq->entity.parent) { 1775 bfqq_weight = bfqq->entity.weight; 1776 in_serv_weight = in_serv_bfqq->entity.weight; 1777 } else { 1778 if (bfqq->entity.parent) 1779 bfqq_weight = bfqq->entity.parent->weight; 1780 else 1781 bfqq_weight = bfqq->entity.weight; 1782 if (in_serv_bfqq->entity.parent) 1783 in_serv_weight = in_serv_bfqq->entity.parent->weight; 1784 else 1785 in_serv_weight = in_serv_bfqq->entity.weight; 1786 } 1787 1788 return bfqq_weight > in_serv_weight; 1789} 1790 1791/* 1792 * Get the index of the actuator that will serve bio. 1793 */ 1794static unsigned int bfq_actuator_index(struct bfq_data *bfqd, struct bio *bio) 1795{ 1796 unsigned int i; 1797 sector_t end; 1798 1799 /* no search needed if one or zero ranges present */ 1800 if (bfqd->num_actuators == 1) 1801 return 0; 1802 1803 /* bio_end_sector(bio) gives the sector after the last one */ 1804 end = bio_end_sector(bio) - 1; 1805 1806 for (i = 0; i < bfqd->num_actuators; i++) { 1807 if (end >= bfqd->sector[i] && 1808 end < bfqd->sector[i] + bfqd->nr_sectors[i]) 1809 return i; 1810 } 1811 1812 WARN_ONCE(true, 1813 "bfq_actuator_index: bio sector out of ranges: end=%llu\n", 1814 end); 1815 return 0; 1816} 1817 1818static bool bfq_better_to_idle(struct bfq_queue *bfqq); 1819 1820static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, 1821 struct bfq_queue *bfqq, 1822 int old_wr_coeff, 1823 struct request *rq, 1824 bool *interactive) 1825{ 1826 bool soft_rt, in_burst, wr_or_deserves_wr, 1827 bfqq_wants_to_preempt, 1828 idle_for_long_time = bfq_bfqq_idle_for_long_time(bfqd, bfqq), 1829 /* 1830 * See the comments on 1831 * bfq_bfqq_update_budg_for_activation for 1832 * details on the usage of the next variable. 1833 */ 1834 arrived_in_time = blk_time_get_ns() <= 1835 bfqq->ttime.last_end_request + 1836 bfqd->bfq_slice_idle * 3; 1837 unsigned int act_idx = bfq_actuator_index(bfqd, rq->bio); 1838 bool bfqq_non_merged_or_stably_merged = 1839 bfqq->bic || RQ_BIC(rq)->bfqq_data[act_idx].stably_merged; 1840 1841 /* 1842 * bfqq deserves to be weight-raised if: 1843 * - it is sync, 1844 * - it does not belong to a large burst, 1845 * - it has been idle for enough time or is soft real-time, 1846 * - is linked to a bfq_io_cq (it is not shared in any sense), 1847 * - has a default weight (otherwise we assume the user wanted 1848 * to control its weight explicitly) 1849 */ 1850 in_burst = bfq_bfqq_in_large_burst(bfqq); 1851 soft_rt = bfqd->bfq_wr_max_softrt_rate > 0 && 1852 !BFQQ_TOTALLY_SEEKY(bfqq) && 1853 !in_burst && 1854 time_is_before_jiffies(bfqq->soft_rt_next_start) && 1855 bfqq->dispatched == 0 && 1856 bfqq->entity.new_weight == 40; 1857 *interactive = !in_burst && idle_for_long_time && 1858 bfqq->entity.new_weight == 40; 1859 /* 1860 * Merged bfq_queues are kept out of weight-raising 1861 * (low-latency) mechanisms. The reason is that these queues 1862 * are usually created for non-interactive and 1863 * non-soft-real-time tasks. Yet this is not the case for 1864 * stably-merged queues. These queues are merged just because 1865 * they are created shortly after each other. So they may 1866 * easily serve the I/O of an interactive or soft-real time 1867 * application, if the application happens to spawn multiple 1868 * processes. So let also stably-merged queued enjoy weight 1869 * raising. 1870 */ 1871 wr_or_deserves_wr = bfqd->low_latency && 1872 (bfqq->wr_coeff > 1 || 1873 (bfq_bfqq_sync(bfqq) && bfqq_non_merged_or_stably_merged && 1874 (*interactive || soft_rt))); 1875 1876 /* 1877 * Using the last flag, update budget and check whether bfqq 1878 * may want to preempt the in-service queue. 1879 */ 1880 bfqq_wants_to_preempt = 1881 bfq_bfqq_update_budg_for_activation(bfqd, bfqq, 1882 arrived_in_time); 1883 1884 /* 1885 * If bfqq happened to be activated in a burst, but has been 1886 * idle for much more than an interactive queue, then we 1887 * assume that, in the overall I/O initiated in the burst, the 1888 * I/O associated with bfqq is finished. So bfqq does not need 1889 * to be treated as a queue belonging to a burst 1890 * anymore. Accordingly, we reset bfqq's in_large_burst flag 1891 * if set, and remove bfqq from the burst list if it's 1892 * there. We do not decrement burst_size, because the fact 1893 * that bfqq does not need to belong to the burst list any 1894 * more does not invalidate the fact that bfqq was created in 1895 * a burst. 1896 */ 1897 if (likely(!bfq_bfqq_just_created(bfqq)) && 1898 idle_for_long_time && 1899 time_is_before_jiffies( 1900 bfqq->budget_timeout + 1901 msecs_to_jiffies(10000))) { 1902 hlist_del_init(&bfqq->burst_list_node); 1903 bfq_clear_bfqq_in_large_burst(bfqq); 1904 } 1905 1906 bfq_clear_bfqq_just_created(bfqq); 1907 1908 if (bfqd->low_latency) { 1909 if (unlikely(time_is_after_jiffies(bfqq->split_time))) 1910 /* wraparound */ 1911 bfqq->split_time = 1912 jiffies - bfqd->bfq_wr_min_idle_time - 1; 1913 1914 if (time_is_before_jiffies(bfqq->split_time + 1915 bfqd->bfq_wr_min_idle_time)) { 1916 bfq_update_bfqq_wr_on_rq_arrival(bfqd, bfqq, 1917 old_wr_coeff, 1918 wr_or_deserves_wr, 1919 *interactive, 1920 in_burst, 1921 soft_rt); 1922 1923 if (old_wr_coeff != bfqq->wr_coeff) 1924 bfqq->entity.prio_changed = 1; 1925 } 1926 } 1927 1928 bfqq->last_idle_bklogged = jiffies; 1929 bfqq->service_from_backlogged = 0; 1930 bfq_clear_bfqq_softrt_update(bfqq); 1931 1932 bfq_add_bfqq_busy(bfqq); 1933 1934 /* 1935 * Expire in-service queue if preemption may be needed for 1936 * guarantees or throughput. As for guarantees, we care 1937 * explicitly about two cases. The first is that bfqq has to 1938 * recover a service hole, as explained in the comments on 1939 * bfq_bfqq_update_budg_for_activation(), i.e., that 1940 * bfqq_wants_to_preempt is true. However, if bfqq does not 1941 * carry time-critical I/O, then bfqq's bandwidth is less 1942 * important than that of queues that carry time-critical I/O. 1943 * So, as a further constraint, we consider this case only if 1944 * bfqq is at least as weight-raised, i.e., at least as time 1945 * critical, as the in-service queue. 1946 * 1947 * The second case is that bfqq is in a higher priority class, 1948 * or has a higher weight than the in-service queue. If this 1949 * condition does not hold, we don't care because, even if 1950 * bfqq does not start to be served immediately, the resulting 1951 * delay for bfqq's I/O is however lower or much lower than 1952 * the ideal completion time to be guaranteed to bfqq's I/O. 1953 * 1954 * In both cases, preemption is needed only if, according to 1955 * the timestamps of both bfqq and of the in-service queue, 1956 * bfqq actually is the next queue to serve. So, to reduce 1957 * useless preemptions, the return value of 1958 * next_queue_may_preempt() is considered in the next compound 1959 * condition too. Yet next_queue_may_preempt() just checks a 1960 * simple, necessary condition for bfqq to be the next queue 1961 * to serve. In fact, to evaluate a sufficient condition, the 1962 * timestamps of the in-service queue would need to be 1963 * updated, and this operation is quite costly (see the 1964 * comments on bfq_bfqq_update_budg_for_activation()). 1965 * 1966 * As for throughput, we ask bfq_better_to_idle() whether we 1967 * still need to plug I/O dispatching. If bfq_better_to_idle() 1968 * says no, then plugging is not needed any longer, either to 1969 * boost throughput or to perserve service guarantees. Then 1970 * the best option is to stop plugging I/O, as not doing so 1971 * would certainly lower throughput. We may end up in this 1972 * case if: (1) upon a dispatch attempt, we detected that it 1973 * was better to plug I/O dispatch, and to wait for a new 1974 * request to arrive for the currently in-service queue, but 1975 * (2) this switch of bfqq to busy changes the scenario. 1976 */ 1977 if (bfqd->in_service_queue && 1978 ((bfqq_wants_to_preempt && 1979 bfqq->wr_coeff >= bfqd->in_service_queue->wr_coeff) || 1980 bfq_bfqq_higher_class_or_weight(bfqq, bfqd->in_service_queue) || 1981 !bfq_better_to_idle(bfqd->in_service_queue)) && 1982 next_queue_may_preempt(bfqd)) 1983 bfq_bfqq_expire(bfqd, bfqd->in_service_queue, 1984 false, BFQQE_PREEMPTED); 1985} 1986 1987static void bfq_reset_inject_limit(struct bfq_data *bfqd, 1988 struct bfq_queue *bfqq) 1989{ 1990 /* invalidate baseline total service time */ 1991 bfqq->last_serv_time_ns = 0; 1992 1993 /* 1994 * Reset pointer in case we are waiting for 1995 * some request completion. 1996 */ 1997 bfqd->waited_rq = NULL; 1998 1999 /* 2000 * If bfqq has a short think time, then start by setting the 2001 * inject limit to 0 prudentially, because the service time of 2002 * an injected I/O request may be higher than the think time 2003 * of bfqq, and therefore, if one request was injected when 2004 * bfqq remains empty, this injected request might delay the 2005 * service of the next I/O request for bfqq significantly. In 2006 * case bfqq can actually tolerate some injection, then the 2007 * adaptive update will however raise the limit soon. This 2008 * lucky circumstance holds exactly because bfqq has a short 2009 * think time, and thus, after remaining empty, is likely to 2010 * get new I/O enqueued---and then completed---before being 2011 * expired. This is the very pattern that gives the 2012 * limit-update algorithm the chance to measure the effect of 2013 * injection on request service times, and then to update the 2014 * limit accordingly. 2015 * 2016 * However, in the following special case, the inject limit is 2017 * left to 1 even if the think time is short: bfqq's I/O is 2018 * synchronized with that of some other queue, i.e., bfqq may 2019 * receive new I/O only after the I/O of the other queue is 2020 * completed. Keeping the inject limit to 1 allows the 2021 * blocking I/O to be served while bfqq is in service. And 2022 * this is very convenient both for bfqq and for overall 2023 * throughput, as explained in detail in the comments in 2024 * bfq_update_has_short_ttime(). 2025 * 2026 * On the opposite end, if bfqq has a long think time, then 2027 * start directly by 1, because: 2028 * a) on the bright side, keeping at most one request in 2029 * service in the drive is unlikely to cause any harm to the 2030 * latency of bfqq's requests, as the service time of a single 2031 * request is likely to be lower than the think time of bfqq; 2032 * b) on the downside, after becoming empty, bfqq is likely to 2033 * expire before getting its next request. With this request 2034 * arrival pattern, it is very hard to sample total service 2035 * times and update the inject limit accordingly (see comments 2036 * on bfq_update_inject_limit()). So the limit is likely to be 2037 * never, or at least seldom, updated. As a consequence, by 2038 * setting the limit to 1, we avoid that no injection ever 2039 * occurs with bfqq. On the downside, this proactive step 2040 * further reduces chances to actually compute the baseline 2041 * total service time. Thus it reduces chances to execute the 2042 * limit-update algorithm and possibly raise the limit to more 2043 * than 1. 2044 */ 2045 if (bfq_bfqq_has_short_ttime(bfqq)) 2046 bfqq->inject_limit = 0; 2047 else 2048 bfqq->inject_limit = 1; 2049 2050 bfqq->decrease_time_jif = jiffies; 2051} 2052 2053static void bfq_update_io_intensity(struct bfq_queue *bfqq, u64 now_ns) 2054{ 2055 u64 tot_io_time = now_ns - bfqq->io_start_time; 2056 2057 if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfqq->dispatched == 0) 2058 bfqq->tot_idle_time += 2059 now_ns - bfqq->ttime.last_end_request; 2060 2061 if (unlikely(bfq_bfqq_just_created(bfqq))) 2062 return; 2063 2064 /* 2065 * Must be busy for at least about 80% of the time to be 2066 * considered I/O bound. 2067 */ 2068 if (bfqq->tot_idle_time * 5 > tot_io_time) 2069 bfq_clear_bfqq_IO_bound(bfqq); 2070 else 2071 bfq_mark_bfqq_IO_bound(bfqq); 2072 2073 /* 2074 * Keep an observation window of at most 200 ms in the past 2075 * from now. 2076 */ 2077 if (tot_io_time > 200 * NSEC_PER_MSEC) { 2078 bfqq->io_start_time = now_ns - (tot_io_time>>1); 2079 bfqq->tot_idle_time >>= 1; 2080 } 2081} 2082 2083/* 2084 * Detect whether bfqq's I/O seems synchronized with that of some 2085 * other queue, i.e., whether bfqq, after remaining empty, happens to 2086 * receive new I/O only right after some I/O request of the other 2087 * queue has been completed. We call waker queue the other queue, and 2088 * we assume, for simplicity, that bfqq may have at most one waker 2089 * queue. 2090 * 2091 * A remarkable throughput boost can be reached by unconditionally 2092 * injecting the I/O of the waker queue, every time a new 2093 * bfq_dispatch_request happens to be invoked while I/O is being 2094 * plugged for bfqq. In addition to boosting throughput, this 2095 * unblocks bfqq's I/O, thereby improving bandwidth and latency for 2096 * bfqq. Note that these same results may be achieved with the general 2097 * injection mechanism, but less effectively. For details on this 2098 * aspect, see the comments on the choice of the queue for injection 2099 * in bfq_select_queue(). 2100 * 2101 * Turning back to the detection of a waker queue, a queue Q is deemed as a 2102 * waker queue for bfqq if, for three consecutive times, bfqq happens to become 2103 * non empty right after a request of Q has been completed within given 2104 * timeout. In this respect, even if bfqq is empty, we do not check for a waker 2105 * if it still has some in-flight I/O. In fact, in this case bfqq is actually 2106 * still being served by the drive, and may receive new I/O on the completion 2107 * of some of the in-flight requests. In particular, on the first time, Q is 2108 * tentatively set as a candidate waker queue, while on the third consecutive 2109 * time that Q is detected, the field waker_bfqq is set to Q, to confirm that Q 2110 * is a waker queue for bfqq. These detection steps are performed only if bfqq 2111 * has a long think time, so as to make it more likely that bfqq's I/O is 2112 * actually being blocked by a synchronization. This last filter, plus the 2113 * above three-times requirement and time limit for detection, make false 2114 * positives less likely. 2115 * 2116 * NOTE 2117 * 2118 * The sooner a waker queue is detected, the sooner throughput can be 2119 * boosted by injecting I/O from the waker queue. Fortunately, 2120 * detection is likely to be actually fast, for the following 2121 * reasons. While blocked by synchronization, bfqq has a long think 2122 * time. This implies that bfqq's inject limit is at least equal to 1 2123 * (see the comments in bfq_update_inject_limit()). So, thanks to 2124 * injection, the waker queue is likely to be served during the very 2125 * first I/O-plugging time interval for bfqq. This triggers the first 2126 * step of the detection mechanism. Thanks again to injection, the 2127 * candidate waker queue is then likely to be confirmed no later than 2128 * during the next I/O-plugging interval for bfqq. 2129 * 2130 * ISSUE 2131 * 2132 * On queue merging all waker information is lost. 2133 */ 2134static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq, 2135 u64 now_ns) 2136{ 2137 char waker_name[MAX_BFQQ_NAME_LENGTH]; 2138 2139 if (!bfqd->last_completed_rq_bfqq || 2140 bfqd->last_completed_rq_bfqq == bfqq || 2141 bfq_bfqq_has_short_ttime(bfqq) || 2142 now_ns - bfqd->last_completion >= 4 * NSEC_PER_MSEC || 2143 bfqd->last_completed_rq_bfqq == &bfqd->oom_bfqq || 2144 bfqq == &bfqd->oom_bfqq) 2145 return; 2146 2147 /* 2148 * We reset waker detection logic also if too much time has passed 2149 * since the first detection. If wakeups are rare, pointless idling 2150 * doesn't hurt throughput that much. The condition below makes sure 2151 * we do not uselessly idle blocking waker in more than 1/64 cases. 2152 */ 2153 if (bfqd->last_completed_rq_bfqq != 2154 bfqq->tentative_waker_bfqq || 2155 now_ns > bfqq->waker_detection_started + 2156 128 * (u64)bfqd->bfq_slice_idle) { 2157 /* 2158 * First synchronization detected with a 2159 * candidate waker queue, or with a different 2160 * candidate waker queue from the current one. 2161 */ 2162 bfqq->tentative_waker_bfqq = 2163 bfqd->last_completed_rq_bfqq; 2164 bfqq->num_waker_detections = 1; 2165 bfqq->waker_detection_started = now_ns; 2166 bfq_bfqq_name(bfqq->tentative_waker_bfqq, waker_name, 2167 MAX_BFQQ_NAME_LENGTH); 2168 bfq_log_bfqq(bfqd, bfqq, "set tentative waker %s", waker_name); 2169 } else /* Same tentative waker queue detected again */ 2170 bfqq->num_waker_detections++; 2171 2172 if (bfqq->num_waker_detections == 3) { 2173 bfqq->waker_bfqq = bfqd->last_completed_rq_bfqq; 2174 bfqq->tentative_waker_bfqq = NULL; 2175 bfq_bfqq_name(bfqq->waker_bfqq, waker_name, 2176 MAX_BFQQ_NAME_LENGTH); 2177 bfq_log_bfqq(bfqd, bfqq, "set waker %s", waker_name); 2178 2179 /* 2180 * If the waker queue disappears, then 2181 * bfqq->waker_bfqq must be reset. To 2182 * this goal, we maintain in each 2183 * waker queue a list, woken_list, of 2184 * all the queues that reference the 2185 * waker queue through their 2186 * waker_bfqq pointer. When the waker 2187 * queue exits, the waker_bfqq pointer 2188 * of all the queues in the woken_list 2189 * is reset. 2190 * 2191 * In addition, if bfqq is already in 2192 * the woken_list of a waker queue, 2193 * then, before being inserted into 2194 * the woken_list of a new waker 2195 * queue, bfqq must be removed from 2196 * the woken_list of the old waker 2197 * queue. 2198 */ 2199 if (!hlist_unhashed(&bfqq->woken_list_node)) 2200 hlist_del_init(&bfqq->woken_list_node); 2201 hlist_add_head(&bfqq->woken_list_node, 2202 &bfqd->last_completed_rq_bfqq->woken_list); 2203 } 2204} 2205 2206static void bfq_add_request(struct request *rq) 2207{ 2208 struct bfq_queue *bfqq = RQ_BFQQ(rq); 2209 struct bfq_data *bfqd = bfqq->bfqd; 2210 struct request *next_rq, *prev; 2211 unsigned int old_wr_coeff = bfqq->wr_coeff; 2212 bool interactive = false; 2213 u64 now_ns = blk_time_get_ns(); 2214 2215 bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq)); 2216 bfqq->queued[rq_is_sync(rq)]++; 2217 /* 2218 * Updating of 'bfqd->queued' is protected by 'bfqd->lock', however, it 2219 * may be read without holding the lock in bfq_has_work(). 2220 */ 2221 WRITE_ONCE(bfqd->queued, bfqd->queued + 1); 2222 2223 if (bfq_bfqq_sync(bfqq) && RQ_BIC(rq)->requests <= 1) { 2224 bfq_check_waker(bfqd, bfqq, now_ns); 2225 2226 /* 2227 * Periodically reset inject limit, to make sure that 2228 * the latter eventually drops in case workload 2229 * changes, see step (3) in the comments on 2230 * bfq_update_inject_limit(). 2231 */ 2232 if (time_is_before_eq_jiffies(bfqq->decrease_time_jif + 2233 msecs_to_jiffies(1000))) 2234 bfq_reset_inject_limit(bfqd, bfqq); 2235 2236 /* 2237 * The following conditions must hold to setup a new 2238 * sampling of total service time, and then a new 2239 * update of the inject limit: 2240 * - bfqq is in service, because the total service 2241 * time is evaluated only for the I/O requests of 2242 * the queues in service; 2243 * - this is the right occasion to compute or to 2244 * lower the baseline total service time, because 2245 * there are actually no requests in the drive, 2246 * or 2247 * the baseline total service time is available, and 2248 * this is the right occasion to compute the other 2249 * quantity needed to update the inject limit, i.e., 2250 * the total service time caused by the amount of 2251 * injection allowed by the current value of the 2252 * limit. It is the right occasion because injection 2253 * has actually been performed during the service 2254 * hole, and there are still in-flight requests, 2255 * which are very likely to be exactly the injected 2256 * requests, or part of them; 2257 * - the minimum interval for sampling the total 2258 * service time and updating the inject limit has 2259 * elapsed. 2260 */ 2261 if (bfqq == bfqd->in_service_queue && 2262 (bfqd->tot_rq_in_driver == 0 || 2263 (bfqq->last_serv_time_ns > 0 && 2264 bfqd->rqs_injected && bfqd->tot_rq_in_driver > 0)) && 2265 time_is_before_eq_jiffies(bfqq->decrease_time_jif + 2266 msecs_to_jiffies(10))) { 2267 bfqd->last_empty_occupied_ns = blk_time_get_ns(); 2268 /* 2269 * Start the state machine for measuring the 2270 * total service time of rq: setting 2271 * wait_dispatch will cause bfqd->waited_rq to 2272 * be set when rq will be dispatched. 2273 */ 2274 bfqd->wait_dispatch = true; 2275 /* 2276 * If there is no I/O in service in the drive, 2277 * then possible injection occurred before the 2278 * arrival of rq will not affect the total 2279 * service time of rq. So the injection limit 2280 * must not be updated as a function of such 2281 * total service time, unless new injection 2282 * occurs before rq is completed. To have the 2283 * injection limit updated only in the latter 2284 * case, reset rqs_injected here (rqs_injected 2285 * will be set in case injection is performed 2286 * on bfqq before rq is completed). 2287 */ 2288 if (bfqd->tot_rq_in_driver == 0) 2289 bfqd->rqs_injected = false; 2290 } 2291 } 2292 2293 if (bfq_bfqq_sync(bfqq)) 2294 bfq_update_io_intensity(bfqq, now_ns); 2295 2296 elv_rb_add(&bfqq->sort_list, rq); 2297 2298 /* 2299 * Check if this request is a better next-serve candidate. 2300 */ 2301 prev = bfqq->next_rq; 2302 next_rq = bfq_choose_req(bfqd, bfqq->next_rq, rq, bfqd->last_position); 2303 bfqq->next_rq = next_rq; 2304 2305 /* 2306 * Adjust priority tree position, if next_rq changes. 2307 * See comments on bfq_pos_tree_add_move() for the unlikely(). 2308 */ 2309 if (unlikely(!bfqd->nonrot_with_queueing && prev != bfqq->next_rq)) 2310 bfq_pos_tree_add_move(bfqd, bfqq); 2311 2312 if (!bfq_bfqq_busy(bfqq)) /* switching to busy ... */ 2313 bfq_bfqq_handle_idle_busy_switch(bfqd, bfqq, old_wr_coeff, 2314 rq, &interactive); 2315 else { 2316 if (bfqd->low_latency && old_wr_coeff == 1 && !rq_is_sync(rq) && 2317 time_is_before_jiffies( 2318 bfqq->last_wr_start_finish + 2319 bfqd->bfq_wr_min_inter_arr_async)) { 2320 bfqq->wr_coeff = bfqd->bfq_wr_coeff; 2321 bfqq->wr_cur_max_time = bfq_wr_duration(bfqd); 2322 2323 bfqd->wr_busy_queues++; 2324 bfqq->entity.prio_changed = 1; 2325 } 2326 if (prev != bfqq->next_rq) 2327 bfq_updated_next_req(bfqd, bfqq); 2328 } 2329 2330 /* 2331 * Assign jiffies to last_wr_start_finish in the following 2332 * cases: 2333 * 2334 * . if bfqq is not going to be weight-raised, because, for 2335 * non weight-raised queues, last_wr_start_finish stores the 2336 * arrival time of the last request; as of now, this piece 2337 * of information is used only for deciding whether to 2338 * weight-raise async queues 2339 * 2340 * . if bfqq is not weight-raised, because, if bfqq is now 2341 * switching to weight-raised, then last_wr_start_finish 2342 * stores the time when weight-raising starts 2343 * 2344 * . if bfqq is interactive, because, regardless of whether 2345 * bfqq is currently weight-raised, the weight-raising 2346 * period must start or restart (this case is considered 2347 * separately because it is not detected by the above 2348 * conditions, if bfqq is already weight-raised) 2349 * 2350 * last_wr_start_finish has to be updated also if bfqq is soft 2351 * real-time, because the weight-raising period is constantly 2352 * restarted on idle-to-busy transitions for these queues, but 2353 * this is already done in bfq_bfqq_handle_idle_busy_switch if 2354 * needed. 2355 */ 2356 if (bfqd->low_latency && 2357 (old_wr_coeff == 1 || bfqq->wr_coeff == 1 || interactive)) 2358 bfqq->last_wr_start_finish = jiffies; 2359} 2360 2361static struct request *bfq_find_rq_fmerge(struct bfq_data *bfqd, 2362 struct bio *bio, 2363 struct request_queue *q) 2364{ 2365 struct bfq_queue *bfqq = bfqd->bio_bfqq; 2366 2367 2368 if (bfqq) 2369 return elv_rb_find(&bfqq->sort_list, bio_end_sector(bio)); 2370 2371 return NULL; 2372} 2373 2374static sector_t get_sdist(sector_t last_pos, struct request *rq) 2375{ 2376 if (last_pos) 2377 return abs(blk_rq_pos(rq) - last_pos); 2378 2379 return 0; 2380} 2381 2382static void bfq_remove_request(struct request_queue *q, 2383 struct request *rq) 2384{ 2385 struct bfq_queue *bfqq = RQ_BFQQ(rq); 2386 struct bfq_data *bfqd = bfqq->bfqd; 2387 const int sync = rq_is_sync(rq); 2388 2389 if (bfqq->next_rq == rq) { 2390 bfqq->next_rq = bfq_find_next_rq(bfqd, bfqq, rq); 2391 bfq_updated_next_req(bfqd, bfqq); 2392 } 2393 2394 if (rq->queuelist.prev != &rq->queuelist) 2395 list_del_init(&rq->queuelist); 2396 bfqq->queued[sync]--; 2397 /* 2398 * Updating of 'bfqd->queued' is protected by 'bfqd->lock', however, it 2399 * may be read without holding the lock in bfq_has_work(). 2400 */ 2401 WRITE_ONCE(bfqd->queued, bfqd->queued - 1); 2402 elv_rb_del(&bfqq->sort_list, rq); 2403 2404 elv_rqhash_del(q, rq); 2405 if (q->last_merge == rq) 2406 q->last_merge = NULL; 2407 2408 if (RB_EMPTY_ROOT(&bfqq->sort_list)) { 2409 bfqq->next_rq = NULL; 2410 2411 if (bfq_bfqq_busy(bfqq) && bfqq != bfqd->in_service_queue) { 2412 bfq_del_bfqq_busy(bfqq, false); 2413 /* 2414 * bfqq emptied. In normal operation, when 2415 * bfqq is empty, bfqq->entity.service and 2416 * bfqq->entity.budget must contain, 2417 * respectively, the service received and the 2418 * budget used last time bfqq emptied. These 2419 * facts do not hold in this case, as at least 2420 * this last removal occurred while bfqq is 2421 * not in service. To avoid inconsistencies, 2422 * reset both bfqq->entity.service and 2423 * bfqq->entity.budget, if bfqq has still a 2424 * process that may issue I/O requests to it. 2425 */ 2426 bfqq->entity.budget = bfqq->entity.service = 0; 2427 } 2428 2429 /* 2430 * Remove queue from request-position tree as it is empty. 2431 */ 2432 if (bfqq->pos_root) { 2433 rb_erase(&bfqq->pos_node, bfqq->pos_root); 2434 bfqq->pos_root = NULL; 2435 } 2436 } else { 2437 /* see comments on bfq_pos_tree_add_move() for the unlikely() */ 2438 if (unlikely(!bfqd->nonrot_with_queueing)) 2439 bfq_pos_tree_add_move(bfqd, bfqq); 2440 } 2441 2442 if (rq->cmd_flags & REQ_META) 2443 bfqq->meta_pending--; 2444 2445} 2446 2447static bool bfq_bio_merge(struct request_queue *q, struct bio *bio, 2448 unsigned int nr_segs) 2449{ 2450 struct bfq_data *bfqd = q->elevator->elevator_data; 2451 struct bfq_io_cq *bic = bfq_bic_lookup(q); 2452 struct request *free = NULL; 2453 bool ret; 2454 2455 spin_lock_irq(&bfqd->lock); 2456 2457 if (bic) { 2458 /* 2459 * Make sure cgroup info is uptodate for current process before 2460 * considering the merge. 2461 */ 2462 bfq_bic_update_cgroup(bic, bio); 2463 2464 bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf), 2465 bfq_actuator_index(bfqd, bio)); 2466 } else { 2467 bfqd->bio_bfqq = NULL; 2468 } 2469 bfqd->bio_bic = bic; 2470 2471 ret = blk_mq_sched_try_merge(q, bio, nr_segs, &free); 2472 2473 spin_unlock_irq(&bfqd->lock); 2474 if (free) 2475 blk_mq_free_request(free); 2476 2477 return ret; 2478} 2479 2480static int bfq_request_merge(struct request_queue *q, struct request **req, 2481 struct bio *bio) 2482{ 2483 struct bfq_data *bfqd = q->elevator->elevator_data; 2484 struct request *__rq; 2485 2486 __rq = bfq_find_rq_fmerge(bfqd, bio, q); 2487 if (__rq && elv_bio_merge_ok(__rq, bio)) { 2488 *req = __rq; 2489 2490 if (blk_discard_mergable(__rq)) 2491 return ELEVATOR_DISCARD_MERGE; 2492 return ELEVATOR_FRONT_MERGE; 2493 } 2494 2495 return ELEVATOR_NO_MERGE; 2496} 2497 2498static void bfq_request_merged(struct request_queue *q, struct request *req, 2499 enum elv_merge type) 2500{ 2501 if (type == ELEVATOR_FRONT_MERGE && 2502 rb_prev(&req->rb_node) && 2503 blk_rq_pos(req) < 2504 blk_rq_pos(container_of(rb_prev(&req->rb_node), 2505 struct request, rb_node))) { 2506 struct bfq_queue *bfqq = RQ_BFQQ(req); 2507 struct bfq_data *bfqd; 2508 struct request *prev, *next_rq; 2509 2510 if (!bfqq) 2511 return; 2512 2513 bfqd = bfqq->bfqd; 2514 2515 /* Reposition request in its sort_list */ 2516 elv_rb_del(&bfqq->sort_list, req); 2517 elv_rb_add(&bfqq->sort_list, req); 2518 2519 /* Choose next request to be served for bfqq */ 2520 prev = bfqq->next_rq; 2521 next_rq = bfq_choose_req(bfqd, bfqq->next_rq, req, 2522 bfqd->last_position); 2523 bfqq->next_rq = next_rq; 2524 /* 2525 * If next_rq changes, update both the queue's budget to 2526 * fit the new request and the queue's position in its 2527 * rq_pos_tree. 2528 */ 2529 if (prev != bfqq->next_rq) { 2530 bfq_updated_next_req(bfqd, bfqq); 2531 /* 2532 * See comments on bfq_pos_tree_add_move() for 2533 * the unlikely(). 2534 */ 2535 if (unlikely(!bfqd->nonrot_with_queueing)) 2536 bfq_pos_tree_add_move(bfqd, bfqq); 2537 } 2538 } 2539} 2540 2541/* 2542 * This function is called to notify the scheduler that the requests 2543 * rq and 'next' have been merged, with 'next' going away. BFQ 2544 * exploits this hook to address the following issue: if 'next' has a 2545 * fifo_time lower that rq, then the fifo_time of rq must be set to 2546 * the value of 'next', to not forget the greater age of 'next'. 2547 * 2548 * NOTE: in this function we assume that rq is in a bfq_queue, basing 2549 * on that rq is picked from the hash table q->elevator->hash, which, 2550 * in its turn, is filled only with I/O requests present in 2551 * bfq_queues, while BFQ is in use for the request queue q. In fact, 2552 * the function that fills this hash table (elv_rqhash_add) is called 2553 * only by bfq_insert_request. 2554 */ 2555static void bfq_requests_merged(struct request_queue *q, struct request *rq, 2556 struct request *next) 2557{ 2558 struct bfq_queue *bfqq = RQ_BFQQ(rq), 2559 *next_bfqq = RQ_BFQQ(next); 2560 2561 if (!bfqq) 2562 goto remove; 2563 2564 /* 2565 * If next and rq belong to the same bfq_queue and next is older 2566 * than rq, then reposition rq in the fifo (by substituting next 2567 * with rq). Otherwise, if next and rq belong to different 2568 * bfq_queues, never reposition rq: in fact, we would have to 2569 * reposition it with respect to next's position in its own fifo, 2570 * which would most certainly be too expensive with respect to 2571 * the benefits. 2572 */ 2573 if (bfqq == next_bfqq && 2574 !list_empty(&rq->queuelist) && !list_empty(&next->queuelist) && 2575 next->fifo_time < rq->fifo_time) { 2576 list_del_init(&rq->queuelist); 2577 list_replace_init(&next->queuelist, &rq->queuelist); 2578 rq->fifo_time = next->fifo_time; 2579 } 2580 2581 if (bfqq->next_rq == next) 2582 bfqq->next_rq = rq; 2583 2584 bfqg_stats_update_io_merged(bfqq_group(bfqq), next->cmd_flags); 2585remove: 2586 /* Merged request may be in the IO scheduler. Remove it. */ 2587 if (!RB_EMPTY_NODE(&next->rb_node)) { 2588 bfq_remove_request(next->q, next); 2589 if (next_bfqq) 2590 bfqg_stats_update_io_remove(bfqq_group(next_bfqq), 2591 next->cmd_flags); 2592 } 2593} 2594 2595/* Must be called with bfqq != NULL */ 2596static void bfq_bfqq_end_wr(struct bfq_queue *bfqq) 2597{ 2598 /* 2599 * If bfqq has been enjoying interactive weight-raising, then 2600 * reset soft_rt_next_start. We do it for the following 2601 * reason. bfqq may have been conveying the I/O needed to load 2602 * a soft real-time application. Such an application actually 2603 * exhibits a soft real-time I/O pattern after it finishes 2604 * loading, and finally starts doing its job. But, if bfqq has 2605 * been receiving a lot of bandwidth so far (likely to happen 2606 * on a fast device), then soft_rt_next_start now contains a 2607 * high value that. So, without this reset, bfqq would be 2608 * prevented from being possibly considered as soft_rt for a 2609 * very long time. 2610 */ 2611 2612 if (bfqq->wr_cur_max_time != 2613 bfqq->bfqd->bfq_wr_rt_max_time) 2614 bfqq->soft_rt_next_start = jiffies; 2615 2616 if (bfq_bfqq_busy(bfqq)) 2617 bfqq->bfqd->wr_busy_queues--; 2618 bfqq->wr_coeff = 1; 2619 bfqq->wr_cur_max_time = 0; 2620 bfqq->last_wr_start_finish = jiffies; 2621 /* 2622 * Trigger a weight change on the next invocation of 2623 * __bfq_entity_update_weight_prio. 2624 */ 2625 bfqq->entity.prio_changed = 1; 2626} 2627 2628void bfq_end_wr_async_queues(struct bfq_data *bfqd, 2629 struct bfq_group *bfqg) 2630{ 2631 int i, j, k; 2632 2633 for (k = 0; k < bfqd->num_actuators; k++) { 2634 for (i = 0; i < 2; i++) 2635 for (j = 0; j < IOPRIO_NR_LEVELS; j++) 2636 if (bfqg->async_bfqq[i][j][k]) 2637 bfq_bfqq_end_wr(bfqg->async_bfqq[i][j][k]); 2638 if (bfqg->async_idle_bfqq[k]) 2639 bfq_bfqq_end_wr(bfqg->async_idle_bfqq[k]); 2640 } 2641} 2642 2643static void bfq_end_wr(struct bfq_data *bfqd) 2644{ 2645 struct bfq_queue *bfqq; 2646 int i; 2647 2648 spin_lock_irq(&bfqd->lock); 2649 2650 for (i = 0; i < bfqd->num_actuators; i++) { 2651 list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list) 2652 bfq_bfqq_end_wr(bfqq); 2653 } 2654 list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list) 2655 bfq_bfqq_end_wr(bfqq); 2656 bfq_end_wr_async(bfqd); 2657 2658 spin_unlock_irq(&bfqd->lock); 2659} 2660 2661static sector_t bfq_io_struct_pos(void *io_struct, bool request) 2662{ 2663 if (request) 2664 return blk_rq_pos(io_struct); 2665 else 2666 return ((struct bio *)io_struct)->bi_iter.bi_sector; 2667} 2668 2669static int bfq_rq_close_to_sector(void *io_struct, bool request, 2670 sector_t sector) 2671{ 2672 return abs(bfq_io_struct_pos(io_struct, request) - sector) <= 2673 BFQQ_CLOSE_THR; 2674} 2675 2676static struct bfq_queue *bfqq_find_close(struct bfq_data *bfqd, 2677 struct bfq_queue *bfqq, 2678 sector_t sector) 2679{ 2680 struct rb_root *root = &bfqq_group(bfqq)->rq_pos_tree; 2681 struct rb_node *parent, *node; 2682 struct bfq_queue *__bfqq; 2683 2684 if (RB_EMPTY_ROOT(root)) 2685 return NULL; 2686 2687 /* 2688 * First, if we find a request starting at the end of the last 2689 * request, choose it. 2690 */ 2691 __bfqq = bfq_rq_pos_tree_lookup(bfqd, root, sector, &parent, NULL); 2692 if (__bfqq) 2693 return __bfqq; 2694 2695 /* 2696 * If the exact sector wasn't found, the parent of the NULL leaf 2697 * will contain the closest sector (rq_pos_tree sorted by 2698 * next_request position). 2699 */ 2700 __bfqq = rb_entry(parent, struct bfq_queue, pos_node); 2701 if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector)) 2702 return __bfqq; 2703 2704 if (blk_rq_pos(__bfqq->next_rq) < sector) 2705 node = rb_next(&__bfqq->pos_node); 2706 else 2707 node = rb_prev(&__bfqq->pos_node); 2708 if (!node) 2709 return NULL; 2710 2711 __bfqq = rb_entry(node, struct bfq_queue, pos_node); 2712 if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector)) 2713 return __bfqq; 2714 2715 return NULL; 2716} 2717 2718static struct bfq_queue *bfq_find_close_cooperator(struct bfq_data *bfqd, 2719 struct bfq_queue *cur_bfqq, 2720 sector_t sector) 2721{ 2722 struct bfq_queue *bfqq; 2723 2724 /* 2725 * We shall notice if some of the queues are cooperating, 2726 * e.g., working closely on the same area of the device. In 2727 * that case, we can group them together and: 1) don't waste 2728 * time idling, and 2) serve the union of their requests in 2729 * the best possible order for throughput. 2730 */ 2731 bfqq = bfqq_find_close(bfqd, cur_bfqq, sector); 2732 if (!bfqq || bfqq == cur_bfqq) 2733 return NULL; 2734 2735 return bfqq; 2736} 2737 2738static struct bfq_queue * 2739bfq_setup_merge(struct bfq_queue *bfqq, struct bfq_queue *new_bfqq) 2740{ 2741 int process_refs, new_process_refs; 2742 struct bfq_queue *__bfqq; 2743 2744 /* 2745 * If there are no process references on the new_bfqq, then it is 2746 * unsafe to follow the ->new_bfqq chain as other bfqq's in the chain 2747 * may have dropped their last reference (not just their last process 2748 * reference). 2749 */ 2750 if (!bfqq_process_refs(new_bfqq)) 2751 return NULL; 2752 2753 /* Avoid a circular list and skip interim queue merges. */ 2754 while ((__bfqq = new_bfqq->new_bfqq)) { 2755 if (__bfqq == bfqq) 2756 return NULL; 2757 new_bfqq = __bfqq; 2758 } 2759 2760 process_refs = bfqq_process_refs(bfqq); 2761 new_process_refs = bfqq_process_refs(new_bfqq); 2762 /* 2763 * If the process for the bfqq has gone away, there is no 2764 * sense in merging the queues. 2765 */ 2766 if (process_refs == 0 || new_process_refs == 0) 2767 return NULL; 2768 2769 /* 2770 * Make sure merged queues belong to the same parent. Parents could 2771 * have changed since the time we decided the two queues are suitable 2772 * for merging. 2773 */ 2774 if (new_bfqq->entity.parent != bfqq->entity.parent) 2775 return NULL; 2776 2777 bfq_log_bfqq(bfqq->bfqd, bfqq, "scheduling merge with queue %d", 2778 new_bfqq->pid); 2779 2780 /* 2781 * Merging is just a redirection: the requests of the process 2782 * owning one of the two queues are redirected to the other queue. 2783 * The latter queue, in its turn, is set as shared if this is the 2784 * first time that the requests of some process are redirected to 2785 * it. 2786 * 2787 * We redirect bfqq to new_bfqq and not the opposite, because 2788 * we are in the context of the process owning bfqq, thus we 2789 * have the io_cq of this process. So we can immediately 2790 * configure this io_cq to redirect the requests of the 2791 * process to new_bfqq. In contrast, the io_cq of new_bfqq is 2792 * not available any more (new_bfqq->bic == NULL). 2793 * 2794 * Anyway, even in case new_bfqq coincides with the in-service 2795 * queue, redirecting requests the in-service queue is the 2796 * best option, as we feed the in-service queue with new 2797 * requests close to the last request served and, by doing so, 2798 * are likely to increase the throughput. 2799 */ 2800 bfqq->new_bfqq = new_bfqq; 2801 /* 2802 * The above assignment schedules the following redirections: 2803 * each time some I/O for bfqq arrives, the process that 2804 * generated that I/O is disassociated from bfqq and 2805 * associated with new_bfqq. Here we increases new_bfqq->ref 2806 * in advance, adding the number of processes that are 2807 * expected to be associated with new_bfqq as they happen to 2808 * issue I/O. 2809 */ 2810 new_bfqq->ref += process_refs; 2811 return new_bfqq; 2812} 2813 2814static bool bfq_may_be_close_cooperator(struct bfq_queue *bfqq, 2815 struct bfq_queue *new_bfqq) 2816{ 2817 if (bfq_too_late_for_merging(new_bfqq)) 2818 return false; 2819 2820 if (bfq_class_idle(bfqq) || bfq_class_idle(new_bfqq) || 2821 (bfqq->ioprio_class != new_bfqq->ioprio_class)) 2822 return false; 2823 2824 /* 2825 * If either of the queues has already been detected as seeky, 2826 * then merging it with the other queue is unlikely to lead to 2827 * sequential I/O. 2828 */ 2829 if (BFQQ_SEEKY(bfqq) || BFQQ_SEEKY(new_bfqq)) 2830 return false; 2831 2832 /* 2833 * Interleaved I/O is known to be done by (some) applications 2834 * only for reads, so it does not make sense to merge async 2835 * queues. 2836 */ 2837 if (!bfq_bfqq_sync(bfqq) || !bfq_bfqq_sync(new_bfqq)) 2838 return false; 2839 2840 return true; 2841} 2842 2843static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, 2844 struct bfq_queue *bfqq); 2845 2846static struct bfq_queue * 2847bfq_setup_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, 2848 struct bfq_queue *stable_merge_bfqq, 2849 struct bfq_iocq_bfqq_data *bfqq_data) 2850{ 2851 int proc_ref = min(bfqq_process_refs(bfqq), 2852 bfqq_process_refs(stable_merge_bfqq)); 2853 struct bfq_queue *new_bfqq = NULL; 2854 2855 bfqq_data->stable_merge_bfqq = NULL; 2856 if (idling_boosts_thr_without_issues(bfqd, bfqq) || proc_ref == 0) 2857 goto out; 2858 2859 /* next function will take at least one ref */ 2860 new_bfqq = bfq_setup_merge(bfqq, stable_merge_bfqq); 2861 2862 if (new_bfqq) { 2863 bfqq_data->stably_merged = true; 2864 if (new_bfqq->bic) { 2865 unsigned int new_a_idx = new_bfqq->actuator_idx; 2866 struct bfq_iocq_bfqq_data *new_bfqq_data = 2867 &new_bfqq->bic->bfqq_data[new_a_idx]; 2868 2869 new_bfqq_data->stably_merged = true; 2870 } 2871 } 2872 2873out: 2874 /* deschedule stable merge, because done or aborted here */ 2875 bfq_put_stable_ref(stable_merge_bfqq); 2876 2877 return new_bfqq; 2878} 2879 2880/* 2881 * Attempt to schedule a merge of bfqq with the currently in-service 2882 * queue or with a close queue among the scheduled queues. Return 2883 * NULL if no merge was scheduled, a pointer to the shared bfq_queue 2884 * structure otherwise. 2885 * 2886 * The OOM queue is not allowed to participate to cooperation: in fact, since 2887 * the requests temporarily redirected to the OOM queue could be redirected 2888 * again to dedicated queues at any time, the state needed to correctly 2889 * handle merging with the OOM queue would be quite complex and expensive 2890 * to maintain. Besides, in such a critical condition as an out of memory, 2891 * the benefits of queue merging may be little relevant, or even negligible. 2892 * 2893 * WARNING: queue merging may impair fairness among non-weight raised 2894 * queues, for at least two reasons: 1) the original weight of a 2895 * merged queue may change during the merged state, 2) even being the 2896 * weight the same, a merged queue may be bloated with many more 2897 * requests than the ones produced by its originally-associated 2898 * process. 2899 */ 2900static struct bfq_queue * 2901bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, 2902 void *io_struct, bool request, struct bfq_io_cq *bic) 2903{ 2904 struct bfq_queue *in_service_bfqq, *new_bfqq; 2905 unsigned int a_idx = bfqq->actuator_idx; 2906 struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; 2907 2908 /* if a merge has already been setup, then proceed with that first */ 2909 new_bfqq = bfqq->new_bfqq; 2910 if (new_bfqq) { 2911 while (new_bfqq->new_bfqq) 2912 new_bfqq = new_bfqq->new_bfqq; 2913 return new_bfqq; 2914 } 2915 2916 /* 2917 * Check delayed stable merge for rotational or non-queueing 2918 * devs. For this branch to be executed, bfqq must not be 2919 * currently merged with some other queue (i.e., bfqq->bic 2920 * must be non null). If we considered also merged queues, 2921 * then we should also check whether bfqq has already been 2922 * merged with bic->stable_merge_bfqq. But this would be 2923 * costly and complicated. 2924 */ 2925 if (unlikely(!bfqd->nonrot_with_queueing)) { 2926 /* 2927 * Make sure also that bfqq is sync, because 2928 * bic->stable_merge_bfqq may point to some queue (for 2929 * stable merging) also if bic is associated with a 2930 * sync queue, but this bfqq is async 2931 */ 2932 if (bfq_bfqq_sync(bfqq) && bfqq_data->stable_merge_bfqq && 2933 !bfq_bfqq_just_created(bfqq) && 2934 time_is_before_jiffies(bfqq->split_time + 2935 msecs_to_jiffies(bfq_late_stable_merging)) && 2936 time_is_before_jiffies(bfqq->creation_time + 2937 msecs_to_jiffies(bfq_late_stable_merging))) { 2938 struct bfq_queue *stable_merge_bfqq = 2939 bfqq_data->stable_merge_bfqq; 2940 2941 return bfq_setup_stable_merge(bfqd, bfqq, 2942 stable_merge_bfqq, 2943 bfqq_data); 2944 } 2945 } 2946 2947 /* 2948 * Do not perform queue merging if the device is non 2949 * rotational and performs internal queueing. In fact, such a 2950 * device reaches a high speed through internal parallelism 2951 * and pipelining. This means that, to reach a high 2952 * throughput, it must have many requests enqueued at the same 2953 * time. But, in this configuration, the internal scheduling 2954 * algorithm of the device does exactly the job of queue 2955 * merging: it reorders requests so as to obtain as much as 2956 * possible a sequential I/O pattern. As a consequence, with 2957 * the workload generated by processes doing interleaved I/O, 2958 * the throughput reached by the device is likely to be the 2959 * same, with and without queue merging. 2960 * 2961 * Disabling merging also provides a remarkable benefit in 2962 * terms of throughput. Merging tends to make many workloads 2963 * artificially more uneven, because of shared queues 2964 * remaining non empty for incomparably more time than 2965 * non-merged queues. This may accentuate workload 2966 * asymmetries. For example, if one of the queues in a set of 2967 * merged queues has a higher weight than a normal queue, then 2968 * the shared queue may inherit such a high weight and, by 2969 * staying almost always active, may force BFQ to perform I/O 2970 * plugging most of the time. This evidently makes it harder 2971 * for BFQ to let the device reach a high throughput. 2972 * 2973 * Finally, the likely() macro below is not used because one 2974 * of the two branches is more likely than the other, but to 2975 * have the code path after the following if() executed as 2976 * fast as possible for the case of a non rotational device 2977 * with queueing. We want it because this is the fastest kind 2978 * of device. On the opposite end, the likely() may lengthen 2979 * the execution time of BFQ for the case of slower devices 2980 * (rotational or at least without queueing). But in this case 2981 * the execution time of BFQ matters very little, if not at 2982 * all. 2983 */ 2984 if (likely(bfqd->nonrot_with_queueing)) 2985 return NULL; 2986 2987 /* 2988 * Prevent bfqq from being merged if it has been created too 2989 * long ago. The idea is that true cooperating processes, and 2990 * thus their associated bfq_queues, are supposed to be 2991 * created shortly after each other. This is the case, e.g., 2992 * for KVM/QEMU and dump I/O threads. Basing on this 2993 * assumption, the following filtering greatly reduces the 2994 * probability that two non-cooperating processes, which just 2995 * happen to do close I/O for some short time interval, have 2996 * their queues merged by mistake. 2997 */ 2998 if (bfq_too_late_for_merging(bfqq)) 2999 return NULL; 3000 3001 if (!io_struct || unlikely(bfqq == &bfqd->oom_bfqq)) 3002 return NULL; 3003 3004 /* If there is only one backlogged queue, don't search. */ 3005 if (bfq_tot_busy_queues(bfqd) == 1) 3006 return NULL; 3007 3008 in_service_bfqq = bfqd->in_service_queue; 3009 3010 if (in_service_bfqq && in_service_bfqq != bfqq && 3011 likely(in_service_bfqq != &bfqd->oom_bfqq) && 3012 bfq_rq_close_to_sector(io_struct, request, 3013 bfqd->in_serv_last_pos) && 3014 bfqq->entity.parent == in_service_bfqq->entity.parent && 3015 bfq_may_be_close_cooperator(bfqq, in_service_bfqq)) { 3016 new_bfqq = bfq_setup_merge(bfqq, in_service_bfqq); 3017 if (new_bfqq) 3018 return new_bfqq; 3019 } 3020 /* 3021 * Check whether there is a cooperator among currently scheduled 3022 * queues. The only thing we need is that the bio/request is not 3023 * NULL, as we need it to establish whether a cooperator exists. 3024 */ 3025 new_bfqq = bfq_find_close_cooperator(bfqd, bfqq, 3026 bfq_io_struct_pos(io_struct, request)); 3027 3028 if (new_bfqq && likely(new_bfqq != &bfqd->oom_bfqq) && 3029 bfq_may_be_close_cooperator(bfqq, new_bfqq)) 3030 return bfq_setup_merge(bfqq, new_bfqq); 3031 3032 return NULL; 3033} 3034 3035static void bfq_bfqq_save_state(struct bfq_queue *bfqq) 3036{ 3037 struct bfq_io_cq *bic = bfqq->bic; 3038 unsigned int a_idx = bfqq->actuator_idx; 3039 struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[a_idx]; 3040 3041 /* 3042 * If !bfqq->bic, the queue is already shared or its requests 3043 * have already been redirected to a shared queue; both idle window 3044 * and weight raising state have already been saved. Do nothing. 3045 */ 3046 if (!bic) 3047 return; 3048 3049 bfqq_data->saved_last_serv_time_ns = bfqq->last_serv_time_ns; 3050 bfqq_data->saved_inject_limit = bfqq->inject_limit; 3051 bfqq_data->saved_decrease_time_jif = bfqq->decrease_time_jif; 3052 3053 bfqq_data->saved_weight = bfqq->entity.orig_weight; 3054 bfqq_data->saved_ttime = bfqq->ttime; 3055 bfqq_data->saved_has_short_ttime = 3056 bfq_bfqq_has_short_ttime(bfqq); 3057 bfqq_data->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); 3058 bfqq_data->saved_io_start_time = bfqq->io_start_time; 3059 bfqq_data->saved_tot_idle_time = bfqq->tot_idle_time; 3060 bfqq_data->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq); 3061 bfqq_data->was_in_burst_list = 3062 !hlist_unhashed(&bfqq->burst_list_node); 3063 3064 if (unlikely(bfq_bfqq_just_created(bfqq) && 3065 !bfq_bfqq_in_large_burst(bfqq) && 3066 bfqq->bfqd->low_latency)) { 3067 /* 3068 * bfqq being merged right after being created: bfqq 3069 * would have deserved interactive weight raising, but 3070 * did not make it to be set in a weight-raised state, 3071 * because of this early merge. Store directly the 3072 * weight-raising state that would have been assigned 3073 * to bfqq, so that to avoid that bfqq unjustly fails 3074 * to enjoy weight raising if split soon. 3075 */ 3076 bfqq_data->saved_wr_coeff = bfqq->bfqd->bfq_wr_coeff; 3077 bfqq_data->saved_wr_start_at_switch_to_srt = 3078 bfq_smallest_from_now(); 3079 bfqq_data->saved_wr_cur_max_time = 3080 bfq_wr_duration(bfqq->bfqd); 3081 bfqq_data->saved_last_wr_start_finish = jiffies; 3082 } else { 3083 bfqq_data->saved_wr_coeff = bfqq->wr_coeff; 3084 bfqq_data->saved_wr_start_at_switch_to_srt = 3085 bfqq->wr_start_at_switch_to_srt; 3086 bfqq_data->saved_service_from_wr = 3087 bfqq->service_from_wr; 3088 bfqq_data->saved_last_wr_start_finish = 3089 bfqq->last_wr_start_finish; 3090 bfqq_data->saved_wr_cur_max_time = bfqq->wr_cur_max_time; 3091 } 3092} 3093 3094 3095void bfq_reassign_last_bfqq(struct bfq_queue *cur_bfqq, 3096 struct bfq_queue *new_bfqq) 3097{ 3098 if (cur_bfqq->entity.parent && 3099 cur_bfqq->entity.parent->last_bfqq_created == cur_bfqq) 3100 cur_bfqq->entity.parent->last_bfqq_created = new_bfqq; 3101 else if (cur_bfqq->bfqd && cur_bfqq->bfqd->last_bfqq_created == cur_bfqq) 3102 cur_bfqq->bfqd->last_bfqq_created = new_bfqq; 3103} 3104 3105void bfq_release_process_ref(struct bfq_data *bfqd, struct bfq_queue *bfqq) 3106{ 3107 /* 3108 * To prevent bfqq's service guarantees from being violated, 3109 * bfqq may be left busy, i.e., queued for service, even if 3110 * empty (see comments in __bfq_bfqq_expire() for 3111 * details). But, if no process will send requests to bfqq any 3112 * longer, then there is no point in keeping bfqq queued for 3113 * service. In addition, keeping bfqq queued for service, but 3114 * with no process ref any longer, may have caused bfqq to be 3115 * freed when dequeued from service. But this is assumed to 3116 * never happen. 3117 */ 3118 if (bfq_bfqq_busy(bfqq) && RB_EMPTY_ROOT(&bfqq->sort_list) && 3119 bfqq != bfqd->in_service_queue) 3120 bfq_del_bfqq_busy(bfqq, false); 3121 3122 bfq_reassign_last_bfqq(bfqq, NULL); 3123 3124 bfq_put_queue(bfqq); 3125} 3126 3127static struct bfq_queue *bfq_merge_bfqqs(struct bfq_data *bfqd, 3128 struct bfq_io_cq *bic, 3129 struct bfq_queue *bfqq) 3130{ 3131 struct bfq_queue *new_bfqq = bfqq->new_bfqq; 3132 3133 bfq_log_bfqq(bfqd, bfqq, "merging with queue %lu", 3134 (unsigned long)new_bfqq->pid); 3135 /* Save weight raising and idle window of the merged queues */ 3136 bfq_bfqq_save_state(bfqq); 3137 bfq_bfqq_save_state(new_bfqq); 3138 if (bfq_bfqq_IO_bound(bfqq)) 3139 bfq_mark_bfqq_IO_bound(new_bfqq); 3140 bfq_clear_bfqq_IO_bound(bfqq); 3141 3142 /* 3143 * The processes associated with bfqq are cooperators of the 3144 * processes associated with new_bfqq. So, if bfqq has a 3145 * waker, then assume that all these processes will be happy 3146 * to let bfqq's waker freely inject I/O when they have no 3147 * I/O. 3148 */ 3149 if (bfqq->waker_bfqq && !new_bfqq->waker_bfqq && 3150 bfqq->waker_bfqq != new_bfqq) { 3151 new_bfqq->waker_bfqq = bfqq->waker_bfqq; 3152 new_bfqq->tentative_waker_bfqq = NULL; 3153 3154 /* 3155 * If the waker queue disappears, then 3156 * new_bfqq->waker_bfqq must be reset. So insert 3157 * new_bfqq into the woken_list of the waker. See 3158 * bfq_check_waker for details. 3159 */ 3160 hlist_add_head(&new_bfqq->woken_list_node, 3161 &new_bfqq->waker_bfqq->woken_list); 3162 3163 } 3164 3165 /* 3166 * If bfqq is weight-raised, then let new_bfqq inherit 3167 * weight-raising. To reduce false positives, neglect the case 3168 * where bfqq has just been created, but has not yet made it 3169 * to be weight-raised (which may happen because EQM may merge 3170 * bfqq even before bfq_add_request is executed for the first 3171 * time for bfqq). Handling this case would however be very 3172 * easy, thanks to the flag just_created. 3173 */ 3174 if (new_bfqq->wr_coeff == 1 && bfqq->wr_coeff > 1) { 3175 new_bfqq->wr_coeff = bfqq->wr_coeff; 3176 new_bfqq->wr_cur_max_time = bfqq->wr_cur_max_time; 3177 new_bfqq->last_wr_start_finish = bfqq->last_wr_start_finish; 3178 new_bfqq->wr_start_at_switch_to_srt = 3179 bfqq->wr_start_at_switch_to_srt; 3180 if (bfq_bfqq_busy(new_bfqq)) 3181 bfqd->wr_busy_queues++; 3182 new_bfqq->entity.prio_changed = 1; 3183 } 3184 3185 if (bfqq->wr_coeff > 1) { /* bfqq has given its wr to new_bfqq */ 3186 bfqq->wr_coeff = 1; 3187 bfqq->entity.prio_changed = 1; 3188 if (bfq_bfqq_busy(bfqq)) 3189 bfqd->wr_busy_queues--; 3190 } 3191 3192 bfq_log_bfqq(bfqd, new_bfqq, "merge_bfqqs: wr_busy %d", 3193 bfqd->wr_busy_queues); 3194 3195 /* 3196 * Merge queues (that is, let bic redirect its requests to new_bfqq) 3197 */ 3198 bic_set_bfqq(bic, new_bfqq, true, bfqq->actuator_idx); 3199 bfq_mark_bfqq_coop(new_bfqq); 3200 /* 3201 * new_bfqq now belongs to at least two bics (it is a shared queue): 3202 * set new_bfqq->bic to NULL. bfqq either: 3203 * - does not belong to any bic any more, and hence bfqq->bic must 3204 * be set to NULL, or 3205 * - is a queue whose owning bics have already been redirected to a 3206 * different queue, hence the queue is destined to not belong to 3207 * any bic soon and bfqq->bic is already NULL (therefore the next 3208 * assignment causes no harm). 3209 */ 3210 new_bfqq->bic = NULL; 3211 /* 3212 * If the queue is shared, the pid is the pid of one of the associated 3213 * processes. Which pid depends on the exact sequence of merge events 3214 * the queue underwent. So printing such a pid is useless and confusing 3215 * because it reports a random pid between those of the associated 3216 * processes. 3217 * We mark such a queue with a pid -1, and then print SHARED instead of 3218 * a pid in logging messages. 3219 */ 3220 new_bfqq->pid = -1; 3221 bfqq->bic = NULL; 3222 3223 bfq_reassign_last_bfqq(bfqq, new_bfqq); 3224 3225 bfq_release_process_ref(bfqd, bfqq); 3226 3227 return new_bfqq; 3228} 3229 3230static bool bfq_allow_bio_merge(struct request_queue *q, struct request *rq, 3231 struct bio *bio) 3232{ 3233 struct bfq_data *bfqd = q->elevator->elevator_data; 3234 bool is_sync = op_is_sync(bio->bi_opf); 3235 struct bfq_queue *bfqq = bfqd->bio_bfqq, *new_bfqq; 3236 3237 /* 3238 * Disallow merge of a sync bio into an async request. 3239 */ 3240 if (is_sync && !rq_is_sync(rq)) 3241 return false; 3242 3243 /* 3244 * Lookup the bfqq that this bio will be queued with. Allow 3245 * merge only if rq is queued there. 3246 */ 3247 if (!bfqq) 3248 return false; 3249 3250 /* 3251 * We take advantage of this function to perform an early merge 3252 * of the queues of possible cooperating processes. 3253 */ 3254 new_bfqq = bfq_setup_cooperator(bfqd, bfqq, bio, false, bfqd->bio_bic); 3255 if (new_bfqq) { 3256 /* 3257 * bic still points to bfqq, then it has not yet been 3258 * redirected to some other bfq_queue, and a queue 3259 * merge between bfqq and new_bfqq can be safely 3260 * fulfilled, i.e., bic can be redirected to new_bfqq 3261 * and bfqq can be put. 3262 */ 3263 while (bfqq != new_bfqq) 3264 bfqq = bfq_merge_bfqqs(bfqd, bfqd->bio_bic, bfqq); 3265 3266 /* 3267 * Change also bqfd->bio_bfqq, as 3268 * bfqd->bio_bic now points to new_bfqq, and 3269 * this function may be invoked again (and then may 3270 * use again bqfd->bio_bfqq). 3271 */ 3272 bfqd->bio_bfqq = bfqq; 3273 } 3274 3275 return bfqq == RQ_BFQQ(rq); 3276} 3277 3278/* 3279 * Set the maximum time for the in-service queue to consume its 3280 * budget. This prevents seeky processes from lowering the throughput. 3281 * In practice, a time-slice service scheme is used with seeky 3282 * processes. 3283 */ 3284static void bfq_set_budget_timeout(struct bfq_data *bfqd, 3285 struct bfq_queue *bfqq) 3286{ 3287 unsigned int timeout_coeff; 3288 3289 if (bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time) 3290 timeout_coeff = 1; 3291 else 3292 timeout_coeff = bfqq->entity.weight / bfqq->entity.orig_weight; 3293 3294 bfqd->last_budget_start = blk_time_get(); 3295 3296 bfqq->budget_timeout = jiffies + 3297 bfqd->bfq_timeout * timeout_coeff; 3298} 3299 3300static void __bfq_set_in_service_queue(struct bfq_data *bfqd, 3301 struct bfq_queue *bfqq) 3302{ 3303 if (bfqq) { 3304 bfq_clear_bfqq_fifo_expire(bfqq); 3305 3306 bfqd->budgets_assigned = (bfqd->budgets_assigned * 7 + 256) / 8; 3307 3308 if (time_is_before_jiffies(bfqq->last_wr_start_finish) && 3309 bfqq->wr_coeff > 1 && 3310 bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time && 3311 time_is_before_jiffies(bfqq->budget_timeout)) { 3312 /* 3313 * For soft real-time queues, move the start 3314 * of the weight-raising period forward by the 3315 * time the queue has not received any 3316 * service. Otherwise, a relatively long 3317 * service delay is likely to cause the 3318 * weight-raising period of the queue to end, 3319 * because of the short duration of the 3320 * weight-raising period of a soft real-time 3321 * queue. It is worth noting that this move 3322 * is not so dangerous for the other queues, 3323 * because soft real-time queues are not 3324 * greedy. 3325 * 3326 * To not add a further variable, we use the 3327 * overloaded field budget_timeout to 3328 * determine for how long the queue has not 3329 * received service, i.e., how much time has 3330 * elapsed since the queue expired. However, 3331 * this is a little imprecise, because 3332 * budget_timeout is set to jiffies if bfqq 3333 * not only expires, but also remains with no 3334 * request. 3335 */ 3336 if (time_after(bfqq->budget_timeout, 3337 bfqq->last_wr_start_finish)) 3338 bfqq->last_wr_start_finish += 3339 jiffies - bfqq->budget_timeout; 3340 else 3341 bfqq->last_wr_start_finish = jiffies; 3342 } 3343 3344 bfq_set_budget_timeout(bfqd, bfqq); 3345 bfq_log_bfqq(bfqd, bfqq, 3346 "set_in_service_queue, cur-budget = %d", 3347 bfqq->entity.budget); 3348 } 3349 3350 bfqd->in_service_queue = bfqq; 3351 bfqd->in_serv_last_pos = 0; 3352} 3353 3354/* 3355 * Get and set a new queue for service. 3356 */ 3357static struct bfq_queue *bfq_set_in_service_queue(struct bfq_data *bfqd) 3358{ 3359 struct bfq_queue *bfqq = bfq_get_next_queue(bfqd); 3360 3361 __bfq_set_in_service_queue(bfqd, bfqq); 3362 return bfqq; 3363} 3364 3365static void bfq_arm_slice_timer(struct bfq_data *bfqd) 3366{ 3367 struct bfq_queue *bfqq = bfqd->in_service_queue; 3368 u32 sl; 3369 3370 bfq_mark_bfqq_wait_request(bfqq); 3371 3372 /* 3373 * We don't want to idle for seeks, but we do want to allow 3374 * fair distribution of slice time for a process doing back-to-back 3375 * seeks. So allow a little bit of time for him to submit a new rq. 3376 */ 3377 sl = bfqd->bfq_slice_idle; 3378 /* 3379 * Unless the queue is being weight-raised or the scenario is 3380 * asymmetric, grant only minimum idle time if the queue 3381 * is seeky. A long idling is preserved for a weight-raised 3382 * queue, or, more in general, in an asymmetric scenario, 3383 * because a long idling is needed for guaranteeing to a queue 3384 * its reserved share of the throughput (in particular, it is 3385 * needed if the queue has a higher weight than some other 3386 * queue). 3387 */ 3388 if (BFQQ_SEEKY(bfqq) && bfqq->wr_coeff == 1 && 3389 !bfq_asymmetric_scenario(bfqd, bfqq)) 3390 sl = min_t(u64, sl, BFQ_MIN_TT); 3391 else if (bfqq->wr_coeff > 1) 3392 sl = max_t(u32, sl, 20ULL * NSEC_PER_MSEC); 3393 3394 bfqd->last_idling_start = blk_time_get(); 3395 bfqd->last_idling_start_jiffies = jiffies; 3396 3397 hrtimer_start(&bfqd->idle_slice_timer, ns_to_ktime(sl), 3398 HRTIMER_MODE_REL); 3399 bfqg_stats_set_start_idle_time(bfqq_group(bfqq)); 3400} 3401 3402/* 3403 * In autotuning mode, max_budget is dynamically recomputed as the 3404 * amount of sectors transferred in timeout at the estimated peak 3405 * rate. This enables BFQ to utilize a full timeslice with a full 3406 * budget, even if the in-service queue is served at peak rate. And 3407 * this maximises throughput with sequential workloads. 3408 */ 3409static unsigned long bfq_calc_max_budget(struct bfq_data *bfqd) 3410{ 3411 return (u64)bfqd->peak_rate * USEC_PER_MSEC * 3412 jiffies_to_msecs(bfqd->bfq_timeout)>>BFQ_RATE_SHIFT; 3413} 3414 3415/* 3416 * Update parameters related to throughput and responsiveness, as a 3417 * function of the estimated peak rate. See comments on 3418 * bfq_calc_max_budget(), and on the ref_wr_duration array. 3419 */ 3420static void update_thr_responsiveness_params(struct bfq_data *bfqd) 3421{ 3422 if (bfqd->bfq_user_max_budget == 0) { 3423 bfqd->bfq_max_budget = 3424 bfq_calc_max_budget(bfqd); 3425 bfq_log(bfqd, "new max_budget = %d", bfqd->bfq_max_budget); 3426 } 3427} 3428 3429static void bfq_reset_rate_computation(struct bfq_data *bfqd, 3430 struct request *rq) 3431{ 3432 if (rq != NULL) { /* new rq dispatch now, reset accordingly */ 3433 bfqd->last_dispatch = bfqd->first_dispatch = blk_time_get_ns(); 3434 bfqd->peak_rate_samples = 1; 3435 bfqd->sequential_samples = 0; 3436 bfqd->tot_sectors_dispatched = bfqd->last_rq_max_size = 3437 blk_rq_sectors(rq); 3438 } else /* no new rq dispatched, just reset the number of samples */ 3439 bfqd->peak_rate_samples = 0; /* full re-init on next disp. */ 3440 3441 bfq_log(bfqd, 3442 "reset_rate_computation at end, sample %u/%u tot_sects %llu", 3443 bfqd->peak_rate_samples, bfqd->sequential_samples, 3444 bfqd->tot_sectors_dispatched); 3445} 3446 3447static void bfq_update_rate_reset(struct bfq_data *bfqd, struct request *rq) 3448{ 3449 u32 rate, weight, divisor; 3450 3451 /* 3452 * For the convergence property to hold (see comments on 3453 * bfq_update_peak_rate()) and for the assessment to be 3454 * reliable, a minimum number of samples must be present, and 3455 * a minimum amount of time must have elapsed. If not so, do 3456 * not compute new rate. Just reset parameters, to get ready 3457 * for a new evaluation attempt. 3458 */ 3459 if (bfqd->peak_rate_samples < BFQ_RATE_MIN_SAMPLES || 3460 bfqd->delta_from_first < BFQ_RATE_MIN_INTERVAL) 3461 goto reset_computation; 3462 3463 /* 3464 * If a new request completion has occurred after last 3465 * dispatch, then, to approximate the rate at which requests 3466 * have been served by the device, it is more precise to 3467 * extend the observation interval to the last completion. 3468 */ 3469 bfqd->delta_from_first = 3470 max_t(u64, bfqd->delta_from_first, 3471 bfqd->last_completion - bfqd->first_dispatch); 3472 3473 /* 3474 * Rate computed in sects/usec, and not sects/nsec, for 3475 * precision issues. 3476 */ 3477 rate = div64_ul(bfqd->tot_sectors_dispatched<<BFQ_RATE_SHIFT, 3478 div_u64(bfqd->delta_from_first, NSEC_PER_USEC)); 3479 3480 /* 3481 * Peak rate not updated if: 3482 * - the percentage of sequential dispatches is below 3/4 of the 3483 * total, and rate is below the current estimated peak rate 3484 * - rate is unreasonably high (> 20M sectors/sec) 3485 */ 3486 if ((bfqd->sequential_samples < (3 * bfqd->peak_rate_samples)>>2 && 3487 rate <= bfqd->peak_rate) || 3488 rate > 20<<BFQ_RATE_SHIFT) 3489 goto reset_computation; 3490 3491 /* 3492 * We have to update the peak rate, at last! To this purpose, 3493 * we use a low-pass filter. We compute the smoothing constant 3494 * of the filter as a function of the 'weight' of the new 3495 * measured rate. 3496 * 3497 * As can be seen in next formulas, we define this weight as a 3498 * quantity proportional to how sequential the workload is, 3499 * and to how long the observation time interval is. 3500 * 3501 * The weight runs from 0 to 8. The maximum value of the 3502 * weight, 8, yields the minimum value for the smoothing 3503 * constant. At this minimum value for the smoothing constant, 3504 * the measured rate contributes for half of the next value of 3505 * the estimated peak rate. 3506 * 3507 * So, the first step is to compute the weight as a function 3508 * of how sequential the workload is. Note that the weight 3509 * cannot reach 9, because bfqd->sequential_samples cannot 3510 * become equal to bfqd->peak_rate_samples, which, in its 3511 * turn, holds true because bfqd->sequential_samples is not 3512 * incremented for the first sample. 3513 */ 3514 weight = (9 * bfqd->sequential_samples) / bfqd->peak_rate_samples; 3515 3516 /* 3517 * Second step: further refine the weight as a function of the 3518 * duration of the observation interval. 3519 */ 3520 weight = min_t(u32, 8, 3521 div_u64(weight * bfqd->delta_from_first, 3522 BFQ_RATE_REF_INTERVAL)); 3523 3524 /* 3525 * Divisor ranging from 10, for minimum weight, to 2, for 3526 * maximum weight. 3527 */ 3528 divisor = 10 - weight; 3529 3530 /* 3531 * Finally, update peak rate: 3532 * 3533 * peak_rate = peak_rate * (divisor-1) / divisor + rate / divisor 3534 */ 3535 bfqd->peak_rate *= divisor-1; 3536 bfqd->peak_rate /= divisor; 3537 rate /= divisor; /* smoothing constant alpha = 1/divisor */ 3538 3539 bfqd->peak_rate += rate; 3540 3541 /* 3542 * For a very slow device, bfqd->peak_rate can reach 0 (see 3543 * the minimum representable values reported in the comments 3544 * on BFQ_RATE_SHIFT). Push to 1 if this happens, to avoid 3545 * divisions by zero where bfqd->peak_rate is used as a 3546 * divisor. 3547 */ 3548 bfqd->peak_rate = max_t(u32, 1, bfqd->peak_rate); 3549 3550 update_thr_responsiveness_params(bfqd); 3551 3552reset_computation: 3553 bfq_reset_rate_computation(bfqd, rq); 3554} 3555 3556/* 3557 * Update the read/write peak rate (the main quantity used for 3558 * auto-tuning, see update_thr_responsiveness_params()). 3559 * 3560 * It is not trivial to estimate the peak rate (correctly): because of 3561 * the presence of sw and hw queues between the scheduler and the 3562 * device components that finally serve I/O requests, it is hard to 3563 * say exactly when a given dispatched request is served inside the 3564 * device, and for how long. As a consequence, it is hard to know 3565 * precisely at what rate a given set of requests is actually served 3566 * by the device. 3567 * 3568 * On the opposite end, the dispatch time of any request is trivially 3569 * available, and, from this piece of information, the "dispatch rate" 3570 * of requests can be immediately computed. So, the idea in the next 3571 * function is to use what is known, namely request dispatch times 3572 * (plus, when useful, request completion times), to estimate what is 3573 * unknown, namely in-device request service rate. 3574 * 3575 * The main issue is that, because of the above facts, the rate at 3576 * which a certain set of requests is dispatched over a certain time 3577 * interval can vary greatly with respect to the rate at which the 3578 * same requests are then served. But, since the size of any 3579 * intermediate queue is limited, and the service scheme is lossless 3580 * (no request is silently dropped), the following obvious convergence 3581 * property holds: the number of requests dispatched MUST become 3582 * closer and closer to the number of requests completed as the 3583 * observation interval grows. This is the key property used in 3584 * the next function to estimate the peak service rate as a function 3585 * of the observed dispatch rate. The function assumes to be invoked 3586 * on every request dispatch. 3587 */ 3588static void bfq_update_peak_rate(struct bfq_data *bfqd, struct request *rq) 3589{ 3590 u64 now_ns = blk_time_get_ns(); 3591 3592 if (bfqd->peak_rate_samples == 0) { /* first dispatch */ 3593 bfq_log(bfqd, "update_peak_rate: goto reset, samples %d", 3594 bfqd->peak_rate_samples); 3595 bfq_reset_rate_computation(bfqd, rq); 3596 goto update_last_values; /* will add one sample */ 3597 } 3598 3599 /* 3600 * Device idle for very long: the observation interval lasting 3601 * up to this dispatch cannot be a valid observation interval 3602 * for computing a new peak rate (similarly to the late- 3603 * completion event in bfq_completed_request()). Go to 3604 * update_rate_and_reset to have the following three steps 3605 * taken: 3606 * - close the observation interval at the last (previous) 3607 * request dispatch or completion 3608 * - compute rate, if possible, for that observation interval 3609 * - start a new observation interval with this dispatch 3610 */ 3611 if (now_ns - bfqd->last_dispatch > 100*NSEC_PER_MSEC && 3612 bfqd->tot_rq_in_driver == 0) 3613 goto update_rate_and_reset; 3614 3615 /* Update sampling information */ 3616 bfqd->peak_rate_samples++; 3617 3618 if ((bfqd->tot_rq_in_driver > 0 || 3619 now_ns - bfqd->last_completion < BFQ_MIN_TT) 3620 && !BFQ_RQ_SEEKY(bfqd, bfqd->last_position, rq)) 3621 bfqd->sequential_samples++; 3622 3623 bfqd->tot_sectors_dispatched += blk_rq_sectors(rq); 3624 3625 /* Reset max observed rq size every 32 dispatches */ 3626 if (likely(bfqd->peak_rate_samples % 32)) 3627 bfqd->last_rq_max_size = 3628 max_t(u32, blk_rq_sectors(rq), bfqd->last_rq_max_size); 3629 else 3630 bfqd->last_rq_max_size = blk_rq_sectors(rq); 3631 3632 bfqd->delta_from_first = now_ns - bfqd->first_dispatch; 3633 3634 /* Target observation interval not yet reached, go on sampling */ 3635 if (bfqd->delta_from_first < BFQ_RATE_REF_INTERVAL) 3636 goto update_last_values; 3637 3638update_rate_and_reset: 3639 bfq_update_rate_reset(bfqd, rq); 3640update_last_values: 3641 bfqd->last_position = blk_rq_pos(rq) + blk_rq_sectors(rq); 3642 if (RQ_BFQQ(rq) == bfqd->in_service_queue) 3643 bfqd->in_serv_last_pos = bfqd->last_position; 3644 bfqd->last_dispatch = now_ns; 3645} 3646 3647/* 3648 * Remove request from internal lists. 3649 */ 3650static void bfq_dispatch_remove(struct request_queue *q, struct request *rq) 3651{ 3652 struct bfq_queue *bfqq = RQ_BFQQ(rq); 3653 3654 /* 3655 * For consistency, the next instruction should have been 3656 * executed after removing the request from the queue and 3657 * dispatching it. We execute instead this instruction before 3658 * bfq_remove_request() (and hence introduce a temporary 3659 * inconsistency), for efficiency. In fact, should this 3660 * dispatch occur for a non in-service bfqq, this anticipated 3661 * increment prevents two counters related to bfqq->dispatched 3662 * from risking to be, first, uselessly decremented, and then 3663 * incremented again when the (new) value of bfqq->dispatched 3664 * happens to be taken into account. 3665 */ 3666 bfqq->dispatched++; 3667 bfq_update_peak_rate(q->elevator->elevator_data, rq); 3668 3669 bfq_remove_request(q, rq); 3670} 3671 3672/* 3673 * There is a case where idling does not have to be performed for 3674 * throughput concerns, but to preserve the throughput share of 3675 * the process associated with bfqq. 3676 * 3677 * To introduce this case, we can note that allowing the drive 3678 * to enqueue more than one request at a time, and hence 3679 * delegating de facto final scheduling decisions to the 3680 * drive's internal scheduler, entails loss of control on the 3681 * actual request service order. In particular, the critical 3682 * situation is when requests from different processes happen 3683 * to be present, at the same time, in the internal queue(s) 3684 * of the drive. In such a situation, the drive, by deciding 3685 * the service order of the internally-queued requests, does 3686 * determine also the actual throughput distribution among 3687 * these processes. But the drive typically has no notion or 3688 * concern about per-process throughput distribution, and 3689 * makes its decisions only on a per-request basis. Therefore, 3690 * the service distribution enforced by the drive's internal 3691 * scheduler is likely to coincide with the desired throughput 3692 * distribution only in a completely symmetric, or favorably 3693 * skewed scenario where: 3694 * (i-a) each of these processes must get the same throughput as 3695 * the others, 3696 * (i-b) in case (i-a) does not hold, it holds that the process 3697 * associated with bfqq must receive a lower or equal 3698 * throughput than any of the other processes; 3699 * (ii) the I/O of each process has the same properties, in 3700 * terms of locality (sequential or random), direction 3701 * (reads or writes), request sizes, greediness 3702 * (from I/O-bound to sporadic), and so on; 3703 3704 * In fact, in such a scenario, the drive tends to treat the requests 3705 * of each process in about the same way as the requests of the 3706 * others, and thus to provide each of these processes with about the 3707 * same throughput. This is exactly the desired throughput 3708 * distribution if (i-a) holds, or, if (i-b) holds instead, this is an 3709 * even more convenient distribution for (the process associated with) 3710 * bfqq. 3711 * 3712 * In contrast, in any asymmetric or unfavorable scenario, device 3713 * idling (I/O-dispatch plugging) is certainly needed to guarantee 3714 * that bfqq receives its assigned fraction of the device throughput 3715 * (see [1] for details). 3716 * 3717 * The problem is that idling may significantly reduce throughput with 3718 * certain combinations of types of I/O and devices. An important 3719 * example is sync random I/O on flash storage with command 3720 * queueing. So, unless bfqq falls in cases where idling also boosts 3721 * throughput, it is important to check conditions (i-a), i(-b) and 3722 * (ii) accurately, so as to avoid idling when not strictly needed for 3723 * service guarantees. 3724 * 3725 * Unfortunately, it is extremely difficult to thoroughly check 3726 * condition (ii). And, in case there are active groups, it becomes 3727 * very difficult to check conditions (i-a) and (i-b) too. In fact, 3728 * if there are active groups, then, for conditions (i-a) or (i-b) to 3729 * become false 'indirectly', it is enough that an active group 3730 * contains more active processes or sub-groups than some other active 3731 * group. More precisely, for conditions (i-a) or (i-b) to become 3732 * false because of such a group, it is not even necessary that the 3733 * group is (still) active: it is sufficient that, even if the group 3734 * has become inactive, some of its descendant processes still have 3735 * some request already dispatched but still waiting for 3736 * completion. In fact, requests have still to be guaranteed their 3737 * share of the throughput even after being dispatched. In this 3738 * respect, it is easy to show that, if a group frequently becomes 3739 * inactive while still having in-flight requests, and if, when this 3740 * happens, the group is not considered in the calculation of whether 3741 * the scenario is asymmetric, then the group may fail to be 3742 * guaranteed its fair share of the throughput (basically because 3743 * idling may not be performed for the descendant processes of the 3744 * group, but it had to be). We address this issue with the following 3745 * bi-modal behavior, implemented in the function 3746 * bfq_asymmetric_scenario(). 3747 * 3748 * If there are groups with requests waiting for completion 3749 * (as commented above, some of these groups may even be 3750 * already inactive), then the scenario is tagged as 3751 * asymmetric, conservatively, without checking any of the 3752 * conditions (i-a), (i-b) or (ii). So the device is idled for bfqq. 3753 * This behavior matches also the fact that groups are created 3754 * exactly if controlling I/O is a primary concern (to 3755 * preserve bandwidth and latency guarantees). 3756 * 3757 * On the opposite end, if there are no groups with requests waiting 3758 * for completion, then only conditions (i-a) and (i-b) are actually 3759 * controlled, i.e., provided that conditions (i-a) or (i-b) holds, 3760 * idling is not performed, regardless of whether condition (ii) 3761 * holds. In other words, only if conditions (i-a) and (i-b) do not 3762 * hold, then idling is allowed, and the device tends to be prevented 3763 * from queueing many requests, possibly of several processes. Since 3764 * there are no groups with requests waiting for completion, then, to 3765 * control conditions (i-a) and (i-b) it is enough to check just 3766 * whether all the queues with requests waiting for completion also 3767 * have the same weight. 3768 * 3769 * Not checking condition (ii) evidently exposes bfqq to the 3770 * risk of getting less throughput than its fair share. 3771 * However, for queues with the same weight, a further 3772 * mechanism, preemption, mitigates or even eliminates this 3773 * problem. And it does so without consequences on overall 3774 * throughput. This mechanism and its benefits are explained 3775 * in the next three paragraphs. 3776 * 3777 * Even if a queue, say Q, is expired when it remains idle, Q 3778 * can still preempt the new in-service queue if the next 3779 * request of Q arrives soon (see the comments on 3780 * bfq_bfqq_update_budg_for_activation). If all queues and 3781 * groups have the same weight, this form of preemption, 3782 * combined with the hole-recovery heuristic described in the 3783 * comments on function bfq_bfqq_update_budg_for_activation, 3784 * are enough to preserve a correct bandwidth distribution in 3785 * the mid term, even without idling. In fact, even if not 3786 * idling allows the internal queues of the device to contain 3787 * many requests, and thus to reorder requests, we can rather 3788 * safely assume that the internal scheduler still preserves a 3789 * minimum of mid-term fairness. 3790 * 3791 * More precisely, this preemption-based, idleless approach 3792 * provides fairness in terms of IOPS, and not sectors per 3793 * second. This can be seen with a simple example. Suppose 3794 * that there are two queues with the same weight, but that 3795 * the first queue receives requests of 8 sectors, while the 3796 * second queue receives requests of 1024 sectors. In 3797 * addition, suppose that each of the two queues contains at 3798 * most one request at a time, which implies that each queue 3799 * always remains idle after it is served. Finally, after 3800 * remaining idle, each queue receives very quickly a new 3801 * request. It follows that the two queues are served 3802 * alternatively, preempting each other if needed. This 3803 * implies that, although both queues have the same weight, 3804 * the queue with large requests receives a service that is 3805 * 1024/8 times as high as the service received by the other 3806 * queue. 3807 * 3808 * The motivation for using preemption instead of idling (for 3809 * queues with the same weight) is that, by not idling, 3810 * service guarantees are preserved (completely or at least in 3811 * part) without minimally sacrificing throughput. And, if 3812 * there is no active group, then the primary expectation for 3813 * this device is probably a high throughput. 3814 * 3815 * We are now left only with explaining the two sub-conditions in the 3816 * additional compound condition that is checked below for deciding 3817 * whether the scenario is asymmetric. To explain the first 3818 * sub-condition, we need to add that the function 3819 * bfq_asymmetric_scenario checks the weights of only 3820 * non-weight-raised queues, for efficiency reasons (see comments on 3821 * bfq_weights_tree_add()). Then the fact that bfqq is weight-raised 3822 * is checked explicitly here. More precisely, the compound condition 3823 * below takes into account also the fact that, even if bfqq is being 3824 * weight-raised, the scenario is still symmetric if all queues with 3825 * requests waiting for completion happen to be 3826 * weight-raised. Actually, we should be even more precise here, and 3827 * differentiate between interactive weight raising and soft real-time 3828 * weight raising. 3829 * 3830 * The second sub-condition checked in the compound condition is 3831 * whether there is a fair amount of already in-flight I/O not 3832 * belonging to bfqq. If so, I/O dispatching is to be plugged, for the 3833 * following reason. The drive may decide to serve in-flight 3834 * non-bfqq's I/O requests before bfqq's ones, thereby delaying the 3835 * arrival of new I/O requests for bfqq (recall that bfqq is sync). If 3836 * I/O-dispatching is not plugged, then, while bfqq remains empty, a 3837 * basically uncontrolled amount of I/O from other queues may be 3838 * dispatched too, possibly causing the service of bfqq's I/O to be 3839 * delayed even longer in the drive. This problem gets more and more 3840 * serious as the speed and the queue depth of the drive grow, 3841 * because, as these two quantities grow, the probability to find no 3842 * queue busy but many requests in flight grows too. By contrast, 3843 * plugging I/O dispatching minimizes the delay induced by already 3844 * in-flight I/O, and enables bfqq to recover the bandwidth it may 3845 * lose because of this delay. 3846 * 3847 * As a side note, it is worth considering that the above 3848 * device-idling countermeasures may however fail in the following 3849 * unlucky scenario: if I/O-dispatch plugging is (correctly) disabled 3850 * in a time period during which all symmetry sub-conditions hold, and 3851 * therefore the device is allowed to enqueue many requests, but at 3852 * some later point in time some sub-condition stops to hold, then it 3853 * may become impossible to make requests be served in the desired 3854 * order until all the requests already queued in the device have been 3855 * served. The last sub-condition commented above somewhat mitigates 3856 * this problem for weight-raised queues. 3857 * 3858 * However, as an additional mitigation for this problem, we preserve 3859 * plugging for a special symmetric case that may suddenly turn into 3860 * asymmetric: the case where only bfqq is busy. In this case, not 3861 * expiring bfqq does not cause any harm to any other queues in terms 3862 * of service guarantees. In contrast, it avoids the following unlucky 3863 * sequence of events: (1) bfqq is expired, (2) a new queue with a 3864 * lower weight than bfqq becomes busy (or more queues), (3) the new 3865 * queue is served until a new request arrives for bfqq, (4) when bfqq 3866 * is finally served, there are so many requests of the new queue in 3867 * the drive that the pending requests for bfqq take a lot of time to 3868 * be served. In particular, event (2) may case even already 3869 * dispatched requests of bfqq to be delayed, inside the drive. So, to 3870 * avoid this series of events, the scenario is preventively declared 3871 * as asymmetric also if bfqq is the only busy queues 3872 */ 3873static bool idling_needed_for_service_guarantees(struct bfq_data *bfqd, 3874 struct bfq_queue *bfqq) 3875{ 3876 int tot_busy_queues = bfq_tot_busy_queues(bfqd); 3877 3878 /* No point in idling for bfqq if it won't get requests any longer */ 3879 if (unlikely(!bfqq_process_refs(bfqq))) 3880 return false; 3881 3882 return (bfqq->wr_coeff > 1 && 3883 (bfqd->wr_busy_queues < tot_busy_queues || 3884 bfqd->tot_rq_in_driver >= bfqq->dispatched + 4)) || 3885 bfq_asymmetric_scenario(bfqd, bfqq) || 3886 tot_busy_queues == 1; 3887} 3888 3889static bool __bfq_bfqq_expire(struct bfq_data *bfqd, struct bfq_queue *bfqq, 3890 enum bfqq_expiration reason) 3891{ 3892 /* 3893 * If this bfqq is shared between multiple processes, check 3894 * to make sure that those processes are still issuing I/Os 3895 * within the mean seek distance. If not, it may be time to 3896 * break the queues apart again. 3897 */ 3898 if (bfq_bfqq_coop(bfqq) && BFQQ_SEEKY(bfqq)) 3899 bfq_mark_bfqq_split_coop(bfqq); 3900 3901 /* 3902 * Consider queues with a higher finish virtual time than 3903 * bfqq. If idling_needed_for_service_guarantees(bfqq) returns 3904 * true, then bfqq's bandwidth would be violated if an 3905 * uncontrolled amount of I/O from these queues were 3906 * dispatched while bfqq is waiting for its new I/O to 3907 * arrive. This is exactly what may happen if this is a forced 3908 * expiration caused by a preemption attempt, and if bfqq is 3909 * not re-scheduled. To prevent this from happening, re-queue 3910 * bfqq if it needs I/O-dispatch plugging, even if it is 3911 * empty. By doing so, bfqq is granted to be served before the 3912 * above queues (provided that bfqq is of course eligible). 3913 */ 3914 if (RB_EMPTY_ROOT(&bfqq->sort_list) && 3915 !(reason == BFQQE_PREEMPTED && 3916 idling_needed_for_service_guarantees(bfqd, bfqq))) { 3917 if (bfqq->dispatched == 0) 3918 /* 3919 * Overloading budget_timeout field to store 3920 * the time at which the queue remains with no 3921 * backlog and no outstanding request; used by 3922 * the weight-raising mechanism. 3923 */ 3924 bfqq->budget_timeout = jiffies; 3925 3926 bfq_del_bfqq_busy(bfqq, true); 3927 } else { 3928 bfq_requeue_bfqq(bfqd, bfqq, true); 3929 /* 3930 * Resort priority tree of potential close cooperators. 3931 * See comments on bfq_pos_tree_add_move() for the unlikely(). 3932 */ 3933 if (unlikely(!bfqd->nonrot_with_queueing && 3934 !RB_EMPTY_ROOT(&bfqq->sort_list))) 3935 bfq_pos_tree_add_move(bfqd, bfqq); 3936 } 3937 3938 /* 3939 * All in-service entities must have been properly deactivated 3940 * or requeued before executing the next function, which 3941 * resets all in-service entities as no more in service. This 3942 * may cause bfqq to be freed. If this happens, the next 3943 * function returns true. 3944 */ 3945 return __bfq_bfqd_reset_in_service(bfqd); 3946} 3947 3948/** 3949 * __bfq_bfqq_recalc_budget - try to adapt the budget to the @bfqq behavior. 3950 * @bfqd: device data. 3951 * @bfqq: queue to update. 3952 * @reason: reason for expiration. 3953 * 3954 * Handle the feedback on @bfqq budget at queue expiration. 3955 * See the body for detailed comments. 3956 */ 3957static void __bfq_bfqq_recalc_budget(struct bfq_data *bfqd, 3958 struct bfq_queue *bfqq, 3959 enum bfqq_expiration reason) 3960{ 3961 struct request *next_rq; 3962 int budget, min_budget; 3963 3964 min_budget = bfq_min_budget(bfqd); 3965 3966 if (bfqq->wr_coeff == 1) 3967 budget = bfqq->max_budget; 3968 else /* 3969 * Use a constant, low budget for weight-raised queues, 3970 * to help achieve a low latency. Keep it slightly higher 3971 * than the minimum possible budget, to cause a little 3972 * bit fewer expirations. 3973 */ 3974 budget = 2 * min_budget; 3975 3976 bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last budg %d, budg left %d", 3977 bfqq->entity.budget, bfq_bfqq_budget_left(bfqq)); 3978 bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last max_budg %d, min budg %d", 3979 budget, bfq_min_budget(bfqd)); 3980 bfq_log_bfqq(bfqd, bfqq, "recalc_budg: sync %d, seeky %d", 3981 bfq_bfqq_sync(bfqq), BFQQ_SEEKY(bfqd->in_service_queue)); 3982 3983 if (bfq_bfqq_sync(bfqq) && bfqq->wr_coeff == 1) { 3984 switch (reason) { 3985 /* 3986 * Caveat: in all the following cases we trade latency 3987 * for throughput. 3988 */ 3989 case BFQQE_TOO_IDLE: 3990 /* 3991 * This is the only case where we may reduce 3992 * the budget: if there is no request of the 3993 * process still waiting for completion, then 3994 * we assume (tentatively) that the timer has 3995 * expired because the batch of requests of 3996 * the process could have been served with a 3997 * smaller budget. Hence, betting that 3998 * process will behave in the same way when it 3999 * becomes backlogged again, we reduce its 4000 * next budget. As long as we guess right, 4001 * this budget cut reduces the latency 4002 * experienced by the process. 4003 * 4004 * However, if there are still outstanding 4005 * requests, then the process may have not yet 4006 * issued its next request just because it is 4007 * still waiting for the completion of some of 4008 * the still outstanding ones. So in this 4009 * subcase we do not reduce its budget, on the 4010 * contrary we increase it to possibly boost 4011 * the throughput, as discussed in the 4012 * comments to the BUDGET_TIMEOUT case. 4013 */ 4014 if (bfqq->dispatched > 0) /* still outstanding reqs */ 4015 budget = min(budget * 2, bfqd->bfq_max_budget); 4016 else { 4017 if (budget > 5 * min_budget) 4018 budget -= 4 * min_budget; 4019 else 4020 budget = min_budget; 4021 } 4022 break; 4023 case BFQQE_BUDGET_TIMEOUT: 4024 /* 4025 * We double the budget here because it gives 4026 * the chance to boost the throughput if this 4027 * is not a seeky process (and has bumped into 4028 * this timeout because of, e.g., ZBR). 4029 */ 4030 budget = min(budget * 2, bfqd->bfq_max_budget); 4031 break; 4032 case BFQQE_BUDGET_EXHAUSTED: 4033 /* 4034 * The process still has backlog, and did not 4035 * let either the budget timeout or the disk 4036 * idling timeout expire. Hence it is not 4037 * seeky, has a short thinktime and may be 4038 * happy with a higher budget too. So 4039 * definitely increase the budget of this good 4040 * candidate to boost the disk throughput. 4041 */ 4042 budget = min(budget * 4, bfqd->bfq_max_budget); 4043 break; 4044 case BFQQE_NO_MORE_REQUESTS: 4045 /* 4046 * For queues that expire for this reason, it 4047 * is particularly important to keep the 4048 * budget close to the actual service they 4049 * need. Doing so reduces the timestamp 4050 * misalignment problem described in the 4051 * comments in the body of 4052 * __bfq_activate_entity. In fact, suppose 4053 * that a queue systematically expires for 4054 * BFQQE_NO_MORE_REQUESTS and presents a 4055 * new request in time to enjoy timestamp 4056 * back-shifting. The larger the budget of the 4057 * queue is with respect to the service the 4058 * queue actually requests in each service 4059 * slot, the more times the queue can be 4060 * reactivated with the same virtual finish 4061 * time. It follows that, even if this finish 4062 * time is pushed to the system virtual time 4063 * to reduce the consequent timestamp 4064 * misalignment, the queue unjustly enjoys for 4065 * many re-activations a lower finish time 4066 * than all newly activated queues. 4067 * 4068 * The service needed by bfqq is measured 4069 * quite precisely by bfqq->entity.service. 4070 * Since bfqq does not enjoy device idling, 4071 * bfqq->entity.service is equal to the number 4072 * of sectors that the process associated with 4073 * bfqq requested to read/write before waiting 4074 * for request completions, or blocking for 4075 * other reasons. 4076 */ 4077 budget = max_t(int, bfqq->entity.service, min_budget); 4078 break; 4079 default: 4080 return; 4081 } 4082 } else if (!bfq_bfqq_sync(bfqq)) { 4083 /* 4084 * Async queues get always the maximum possible 4085 * budget, as for them we do not care about latency 4086 * (in addition, their ability to dispatch is limited 4087 * by the charging factor). 4088 */ 4089 budget = bfqd->bfq_max_budget; 4090 } 4091 4092 bfqq->max_budget = budget; 4093 4094 if (bfqd->budgets_assigned >= bfq_stats_min_budgets && 4095 !bfqd->bfq_user_max_budget) 4096 bfqq->max_budget = min(bfqq->max_budget, bfqd->bfq_max_budget); 4097 4098 /* 4099 * If there is still backlog, then assign a new budget, making 4100 * sure that it is large enough for the next request. Since 4101 * the finish time of bfqq must be kept in sync with the 4102 * budget, be sure to call __bfq_bfqq_expire() *after* this 4103 * update. 4104 * 4105 * If there is no backlog, then no need to update the budget; 4106 * it will be updated on the arrival of a new request. 4107 */ 4108 next_rq = bfqq->next_rq; 4109 if (next_rq) 4110 bfqq->entity.budget = max_t(unsigned long, bfqq->max_budget, 4111 bfq_serv_to_charge(next_rq, bfqq)); 4112 4113 bfq_log_bfqq(bfqd, bfqq, "head sect: %u, new budget %d", 4114 next_rq ? blk_rq_sectors(next_rq) : 0, 4115 bfqq->entity.budget); 4116} 4117 4118/* 4119 * Return true if the process associated with bfqq is "slow". The slow 4120 * flag is used, in addition to the budget timeout, to reduce the 4121 * amount of service provided to seeky processes, and thus reduce 4122 * their chances to lower the throughput. More details in the comments 4123 * on the function bfq_bfqq_expire(). 4124 * 4125 * An important observation is in order: as discussed in the comments 4126 * on the function bfq_update_peak_rate(), with devices with internal 4127 * queues, it is hard if ever possible to know when and for how long 4128 * an I/O request is processed by the device (apart from the trivial 4129 * I/O pattern where a new request is dispatched only after the 4130 * previous one has been completed). This makes it hard to evaluate 4131 * the real rate at which the I/O requests of each bfq_queue are 4132 * served. In fact, for an I/O scheduler like BFQ, serving a 4133 * bfq_queue means just dispatching its requests during its service 4134 * slot (i.e., until the budget of the queue is exhausted, or the 4135 * queue remains idle, or, finally, a timeout fires). But, during the 4136 * service slot of a bfq_queue, around 100 ms at most, the device may 4137 * be even still processing requests of bfq_queues served in previous 4138 * service slots. On the opposite end, the requests of the in-service 4139 * bfq_queue may be completed after the service slot of the queue 4140 * finishes. 4141 * 4142 * Anyway, unless more sophisticated solutions are used 4143 * (where possible), the sum of the sizes of the requests dispatched 4144 * during the service slot of a bfq_queue is probably the only 4145 * approximation available for the service received by the bfq_queue 4146 * during its service slot. And this sum is the quantity used in this 4147 * function to evaluate the I/O speed of a process. 4148 */ 4149static bool bfq_bfqq_is_slow(struct bfq_data *bfqd, struct bfq_queue *bfqq, 4150 bool compensate, unsigned long *delta_ms) 4151{ 4152 ktime_t delta_ktime; 4153 u32 delta_usecs; 4154 bool slow = BFQQ_SEEKY(bfqq); /* if delta too short, use seekyness */ 4155 4156 if (!bfq_bfqq_sync(bfqq)) 4157 return false; 4158 4159 if (compensate) 4160 delta_ktime = bfqd->last_idling_start; 4161 else 4162 delta_ktime = blk_time_get(); 4163 delta_ktime = ktime_sub(delta_ktime, bfqd->last_budget_start); 4164 delta_usecs = ktime_to_us(delta_ktime); 4165 4166 /* don't use too short time intervals */ 4167 if (delta_usecs < 1000) { 4168 if (blk_queue_nonrot(bfqd->queue)) 4169 /* 4170 * give same worst-case guarantees as idling 4171 * for seeky 4172 */ 4173 *delta_ms = BFQ_MIN_TT / NSEC_PER_MSEC; 4174 else /* charge at least one seek */ 4175 *delta_ms = bfq_slice_idle / NSEC_PER_MSEC; 4176 4177 return slow; 4178 } 4179 4180 *delta_ms = delta_usecs / USEC_PER_MSEC; 4181 4182 /* 4183 * Use only long (> 20ms) intervals to filter out excessive 4184 * spikes in service rate estimation. 4185 */ 4186 if (delta_usecs > 20000) { 4187 /* 4188 * Caveat for rotational devices: processes doing I/O 4189 * in the slower disk zones tend to be slow(er) even 4190 * if not seeky. In this respect, the estimated peak 4191 * rate is likely to be an average over the disk 4192 * surface. Accordingly, to not be too harsh with 4193 * unlucky processes, a process is deemed slow only if 4194 * its rate has been lower than half of the estimated 4195 * peak rate. 4196 */ 4197 slow = bfqq->entity.service < bfqd->bfq_max_budget / 2; 4198 } 4199 4200 bfq_log_bfqq(bfqd, bfqq, "bfq_bfqq_is_slow: slow %d", slow); 4201 4202 return slow; 4203} 4204 4205/* 4206 * To be deemed as soft real-time, an application must meet two 4207 * requirements. First, the application must not require an average 4208 * bandwidth higher than the approximate bandwidth required to playback or 4209 * record a compressed high-definition video. 4210 * The next function is invoked on the completion of the last request of a 4211 * batch, to compute the next-start time instant, soft_rt_next_start, such 4212 * that, if the next request of the application does not arrive before 4213 * soft_rt_next_start, then the above requirement on the bandwidth is met. 4214 * 4215 * The second requirement is that the request pattern of the application is 4216 * isochronous, i.e., that, after issuing a request or a batch of requests, 4217 * the application stops issuing new requests until all its pending requests 4218 * have been completed. After that, the application may issue a new batch, 4219 * and so on. 4220 * For this reason the next function is invoked to compute 4221 * soft_rt_next_start only for applications that meet this requirement, 4222 * whereas soft_rt_next_start is set to infinity for applications that do 4223 * not. 4224 * 4225 * Unfortunately, even a greedy (i.e., I/O-bound) application may 4226 * happen to meet, occasionally or systematically, both the above 4227 * bandwidth and isochrony requirements. This may happen at least in 4228 * the following circumstances. First, if the CPU load is high. The 4229 * application may stop issuing requests while the CPUs are busy 4230 * serving other processes, then restart, then stop again for a while, 4231 * and so on. The other circumstances are related to the storage 4232 * device: the storage device is highly loaded or reaches a low-enough 4233 * throughput with the I/O of the application (e.g., because the I/O 4234 * is random and/or the device is slow). In all these cases, the 4235 * I/O of the application may be simply slowed down enough to meet 4236 * the bandwidth and isochrony requirements. To reduce the probability 4237 * that greedy applications are deemed as soft real-time in these 4238 * corner cases, a further rule is used in the computation of 4239 * soft_rt_next_start: the return value of this function is forced to 4240 * be higher than the maximum between the following two quantities. 4241 * 4242 * (a) Current time plus: (1) the maximum time for which the arrival 4243 * of a request is waited for when a sync queue becomes idle, 4244 * namely bfqd->bfq_slice_idle, and (2) a few extra jiffies. We 4245 * postpone for a moment the reason for adding a few extra 4246 * jiffies; we get back to it after next item (b). Lower-bounding 4247 * the return value of this function with the current time plus 4248 * bfqd->bfq_slice_idle tends to filter out greedy applications, 4249 * because the latter issue their next request as soon as possible 4250 * after the last one has been completed. In contrast, a soft 4251 * real-time application spends some time processing data, after a 4252 * batch of its requests has been completed. 4253 * 4254 * (b) Current value of bfqq->soft_rt_next_start. As pointed out 4255 * above, greedy applications may happen to meet both the 4256 * bandwidth and isochrony requirements under heavy CPU or 4257 * storage-device load. In more detail, in these scenarios, these 4258 * applications happen, only for limited time periods, to do I/O 4259 * slowly enough to meet all the requirements described so far, 4260 * including the filtering in above item (a). These slow-speed 4261 * time intervals are usually interspersed between other time 4262 * intervals during which these applications do I/O at a very high 4263 * speed. Fortunately, exactly because of the high speed of the 4264 * I/O in the high-speed intervals, the values returned by this 4265 * function happen to be so high, near the end of any such 4266 * high-speed interval, to be likely to fall *after* the end of 4267 * the low-speed time interval that follows. These high values are 4268 * stored in bfqq->soft_rt_next_start after each invocation of 4269 * this function. As a consequence, if the last value of 4270 * bfqq->soft_rt_next_start is constantly used to lower-bound the 4271 * next value that this function may return, then, from the very 4272 * beginning of a low-speed interval, bfqq->soft_rt_next_start is 4273 * likely to be constantly kept so high that any I/O request 4274 * issued during the low-speed interval is considered as arriving 4275 * to soon for the application to be deemed as soft 4276 * real-time. Then, in the high-speed interval that follows, the 4277 * application will not be deemed as soft real-time, just because 4278 * it will do I/O at a high speed. And so on. 4279 * 4280 * Getting back to the filtering in item (a), in the following two 4281 * cases this filtering might be easily passed by a greedy 4282 * application, if the reference quantity was just 4283 * bfqd->bfq_slice_idle: 4284 * 1) HZ is so low that the duration of a jiffy is comparable to or 4285 * higher than bfqd->bfq_slice_idle. This happens, e.g., on slow 4286 * devices with HZ=100. The time granularity may be so coarse 4287 * that the approximation, in jiffies, of bfqd->bfq_slice_idle 4288 * is rather lower than the exact value. 4289 * 2) jiffies, instead of increasing at a constant rate, may stop increasing 4290 * for a while, then suddenly 'jump' by several units to recover the lost 4291 * increments. This seems to happen, e.g., inside virtual machines. 4292 * To address this issue, in the filtering in (a) we do not use as a 4293 * reference time interval just bfqd->bfq_slice_idle, but 4294 * bfqd->bfq_slice_idle plus a few jiffies. In particular, we add the 4295 * minimum number of jiffies for which the filter seems to be quite 4296 * precise also in embedded systems and KVM/QEMU virtual machines. 4297 */ 4298static unsigned long bfq_bfqq_softrt_next_start(struct bfq_data *bfqd, 4299 struct bfq_queue *bfqq) 4300{ 4301 return max3(bfqq->soft_rt_next_start, 4302 bfqq->last_idle_bklogged + 4303 HZ * bfqq->service_from_backlogged / 4304 bfqd->bfq_wr_max_softrt_rate, 4305 jiffies + nsecs_to_jiffies(bfqq->bfqd->bfq_slice_idle) + 4); 4306} 4307 4308/** 4309 * bfq_bfqq_expire - expire a queue. 4310 * @bfqd: device owning the queue. 4311 * @bfqq: the queue to expire. 4312 * @compensate: if true, compensate for the time spent idling. 4313 * @reason: the reason causing the expiration. 4314 * 4315 * If the process associated with bfqq does slow I/O (e.g., because it 4316 * issues random requests), we charge bfqq with the time it has been 4317 * in service instead of the service it has received (see 4318 * bfq_bfqq_charge_time for details on how this goal is achieved). As 4319 * a consequence, bfqq will typically get higher timestamps upon 4320 * reactivation, and hence it will be rescheduled as if it had 4321 * received more service than what it has actually received. In the 4322 * end, bfqq receives less service in proportion to how slowly its 4323 * associated process consumes its budgets (and hence how seriously it 4324 * tends to lower the throughput). In addition, this time-charging 4325 * strategy guarantees time fairness among slow processes. In 4326 * contrast, if the process associated with bfqq is not slow, we 4327 * charge bfqq exactly with the service it has received. 4328 * 4329 * Charging time to the first type of queues and the exact service to 4330 * the other has the effect of using the WF2Q+ policy to schedule the 4331 * former on a timeslice basis, without violating service domain 4332 * guarantees among the latter. 4333 */ 4334void bfq_bfqq_expire(struct bfq_data *bfqd, 4335 struct bfq_queue *bfqq, 4336 bool compensate, 4337 enum bfqq_expiration reason) 4338{ 4339 bool slow; 4340 unsigned long delta = 0; 4341 struct bfq_entity *entity = &bfqq->entity; 4342 4343 /* 4344 * Check whether the process is slow (see bfq_bfqq_is_slow). 4345 */ 4346 slow = bfq_bfqq_is_slow(bfqd, bfqq, compensate, &delta); 4347 4348 /* 4349 * As above explained, charge slow (typically seeky) and 4350 * timed-out queues with the time and not the service 4351 * received, to favor sequential workloads. 4352 * 4353 * Processes doing I/O in the slower disk zones will tend to 4354 * be slow(er) even if not seeky. Therefore, since the 4355 * estimated peak rate is actually an average over the disk 4356 * surface, these processes may timeout just for bad luck. To 4357 * avoid punishing them, do not charge time to processes that 4358 * succeeded in consuming at least 2/3 of their budget. This 4359 * allows BFQ to preserve enough elasticity to still perform 4360 * bandwidth, and not time, distribution with little unlucky 4361 * or quasi-sequential processes. 4362 */ 4363 if (bfqq->wr_coeff == 1 && 4364 (slow || 4365 (reason == BFQQE_BUDGET_TIMEOUT && 4366 bfq_bfqq_budget_left(bfqq) >= entity->budget / 3))) 4367 bfq_bfqq_charge_time(bfqd, bfqq, delta); 4368 4369 if (bfqd->low_latency && bfqq->wr_coeff == 1) 4370 bfqq->last_wr_start_finish = jiffies; 4371 4372 if (bfqd->low_latency && bfqd->bfq_wr_max_softrt_rate > 0 && 4373 RB_EMPTY_ROOT(&bfqq->sort_list)) { 4374 /* 4375 * If we get here, and there are no outstanding 4376 * requests, then the request pattern is isochronous 4377 * (see the comments on the function 4378 * bfq_bfqq_softrt_next_start()). Therefore we can 4379 * compute soft_rt_next_start. 4380 * 4381 * If, instead, the queue still has outstanding 4382 * requests, then we have to wait for the completion 4383 * of all the outstanding requests to discover whether 4384 * the request pattern is actually isochronous. 4385 */ 4386 if (bfqq->dispatched == 0) 4387 bfqq->soft_rt_next_start = 4388 bfq_bfqq_softrt_next_start(bfqd, bfqq); 4389 else if (bfqq->dispatched > 0) { 4390 /* 4391 * Schedule an update of soft_rt_next_start to when 4392 * the task may be discovered to be isochronous. 4393 */ 4394 bfq_mark_bfqq_softrt_update(bfqq); 4395 } 4396 } 4397 4398 bfq_log_bfqq(bfqd, bfqq, 4399 "expire (%d, slow %d, num_disp %d, short_ttime %d)", reason, 4400 slow, bfqq->dispatched, bfq_bfqq_has_short_ttime(bfqq)); 4401 4402 /* 4403 * bfqq expired, so no total service time needs to be computed 4404 * any longer: reset state machine for measuring total service 4405 * times. 4406 */ 4407 bfqd->rqs_injected = bfqd->wait_dispatch = false; 4408 bfqd->waited_rq = NULL; 4409 4410 /* 4411 * Increase, decrease or leave budget unchanged according to 4412 * reason. 4413 */ 4414 __bfq_bfqq_recalc_budget(bfqd, bfqq, reason); 4415 if (__bfq_bfqq_expire(bfqd, bfqq, reason)) 4416 /* bfqq is gone, no more actions on it */ 4417 return; 4418 4419 /* mark bfqq as waiting a request only if a bic still points to it */ 4420 if (!bfq_bfqq_busy(bfqq) && 4421 reason != BFQQE_BUDGET_TIMEOUT && 4422 reason != BFQQE_BUDGET_EXHAUSTED) { 4423 bfq_mark_bfqq_non_blocking_wait_rq(bfqq); 4424 /* 4425 * Not setting service to 0, because, if the next rq 4426 * arrives in time, the queue will go on receiving 4427 * service with this same budget (as if it never expired) 4428 */ 4429 } else 4430 entity->service = 0; 4431 4432 /* 4433 * Reset the received-service counter for every parent entity. 4434 * Differently from what happens with bfqq->entity.service, 4435 * the resetting of this counter never needs to be postponed 4436 * for parent entities. In fact, in case bfqq may have a 4437 * chance to go on being served using the last, partially 4438 * consumed budget, bfqq->entity.service needs to be kept, 4439 * because if bfqq then actually goes on being served using 4440 * the same budget, the last value of bfqq->entity.service is 4441 * needed to properly decrement bfqq->entity.budget by the 4442 * portion already consumed. In contrast, it is not necessary 4443 * to keep entity->service for parent entities too, because 4444 * the bubble up of the new value of bfqq->entity.budget will 4445 * make sure that the budgets of parent entities are correct, 4446 * even in case bfqq and thus parent entities go on receiving 4447 * service with the same budget. 4448 */ 4449 entity = entity->parent; 4450 for_each_entity(entity) 4451 entity->service = 0; 4452} 4453 4454/* 4455 * Budget timeout is not implemented through a dedicated timer, but 4456 * just checked on request arrivals and completions, as well as on 4457 * idle timer expirations. 4458 */ 4459static bool bfq_bfqq_budget_timeout(struct bfq_queue *bfqq) 4460{ 4461 return time_is_before_eq_jiffies(bfqq->budget_timeout); 4462} 4463 4464/* 4465 * If we expire a queue that is actively waiting (i.e., with the 4466 * device idled) for the arrival of a new request, then we may incur 4467 * the timestamp misalignment problem described in the body of the 4468 * function __bfq_activate_entity. Hence we return true only if this 4469 * condition does not hold, or if the queue is slow enough to deserve 4470 * only to be kicked off for preserving a high throughput. 4471 */ 4472static bool bfq_may_expire_for_budg_timeout(struct bfq_queue *bfqq) 4473{ 4474 bfq_log_bfqq(bfqq->bfqd, bfqq, 4475 "may_budget_timeout: wait_request %d left %d timeout %d", 4476 bfq_bfqq_wait_request(bfqq), 4477 bfq_bfqq_budget_left(bfqq) >= bfqq->entity.budget / 3, 4478 bfq_bfqq_budget_timeout(bfqq)); 4479 4480 return (!bfq_bfqq_wait_request(bfqq) || 4481 bfq_bfqq_budget_left(bfqq) >= bfqq->entity.budget / 3) 4482 && 4483 bfq_bfqq_budget_timeout(bfqq); 4484} 4485 4486static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, 4487 struct bfq_queue *bfqq) 4488{ 4489 bool rot_without_queueing = 4490 !blk_queue_nonrot(bfqd->queue) && !bfqd->hw_tag, 4491 bfqq_sequential_and_IO_bound, 4492 idling_boosts_thr; 4493 4494 /* No point in idling for bfqq if it won't get requests any longer */ 4495 if (unlikely(!bfqq_process_refs(bfqq))) 4496 return false; 4497 4498 bfqq_sequential_and_IO_bound = !BFQQ_SEEKY(bfqq) && 4499 bfq_bfqq_IO_bound(bfqq) && bfq_bfqq_has_short_ttime(bfqq); 4500 4501 /* 4502 * The next variable takes into account the cases where idling 4503 * boosts the throughput. 4504 * 4505 * The value of the variable is computed considering, first, that 4506 * idling is virtually always beneficial for the throughput if: 4507 * (a) the device is not NCQ-capable and rotational, or 4508 * (b) regardless of the presence of NCQ, the device is rotational and 4509 * the request pattern for bfqq is I/O-bound and sequential, or 4510 * (c) regardless of whether it is rotational, the device is 4511 * not NCQ-capable and the request pattern for bfqq is 4512 * I/O-bound and sequential. 4513 * 4514 * Secondly, and in contrast to the above item (b), idling an 4515 * NCQ-capable flash-based device would not boost the 4516 * throughput even with sequential I/O; rather it would lower 4517 * the throughput in proportion to how fast the device 4518 * is. Accordingly, the next variable is true if any of the 4519 * above conditions (a), (b) or (c) is true, and, in 4520 * particular, happens to be false if bfqd is an NCQ-capable 4521 * flash-based device. 4522 */ 4523 idling_boosts_thr = rot_without_queueing || 4524 ((!blk_queue_nonrot(bfqd->queue) || !bfqd->hw_tag) && 4525 bfqq_sequential_and_IO_bound); 4526 4527 /* 4528 * The return value of this function is equal to that of 4529 * idling_boosts_thr, unless a special case holds. In this 4530 * special case, described below, idling may cause problems to 4531 * weight-raised queues. 4532 * 4533 * When the request pool is saturated (e.g., in the presence 4534 * of write hogs), if the processes associated with 4535 * non-weight-raised queues ask for requests at a lower rate, 4536 * then processes associated with weight-raised queues have a 4537 * higher probability to get a request from the pool 4538 * immediately (or at least soon) when they need one. Thus 4539 * they have a higher probability to actually get a fraction 4540 * of the device throughput proportional to their high 4541 * weight. This is especially true with NCQ-capable drives, 4542 * which enqueue several requests in advance, and further 4543 * reorder internally-queued requests. 4544 * 4545 * For this reason, we force to false the return value if 4546 * there are weight-raised busy queues. In this case, and if 4547 * bfqq is not weight-raised, this guarantees that the device 4548 * is not idled for bfqq (if, instead, bfqq is weight-raised, 4549 * then idling will be guaranteed by another variable, see 4550 * below). Combined with the timestamping rules of BFQ (see 4551 * [1] for details), this behavior causes bfqq, and hence any 4552 * sync non-weight-raised queue, to get a lower number of 4553 * requests served, and thus to ask for a lower number of 4554 * requests from the request pool, before the busy 4555 * weight-raised queues get served again. This often mitigates 4556 * starvation problems in the presence of heavy write 4557 * workloads and NCQ, thereby guaranteeing a higher 4558 * application and system responsiveness in these hostile 4559 * scenarios. 4560 */ 4561 return idling_boosts_thr && 4562 bfqd->wr_busy_queues == 0; 4563} 4564 4565/* 4566 * For a queue that becomes empty, device idling is allowed only if 4567 * this function returns true for that queue. As a consequence, since 4568 * device idling plays a critical role for both throughput boosting 4569 * and service guarantees, the return value of this function plays a 4570 * critical role as well. 4571 * 4572 * In a nutshell, this function returns true only if idling is 4573 * beneficial for throughput or, even if detrimental for throughput, 4574 * idling is however necessary to preserve service guarantees (low 4575 * latency, desired throughput distribution, ...). In particular, on 4576 * NCQ-capable devices, this function tries to return false, so as to 4577 * help keep the drives' internal queues full, whenever this helps the 4578 * device boost the throughput without causing any service-guarantee 4579 * issue. 4580 * 4581 * Most of the issues taken into account to get the return value of 4582 * this function are not trivial. We discuss these issues in the two 4583 * functions providing the main pieces of information needed by this 4584 * function. 4585 */ 4586static bool bfq_better_to_idle(struct bfq_queue *bfqq) 4587{ 4588 struct bfq_data *bfqd = bfqq->bfqd; 4589 bool idling_boosts_thr_with_no_issue, idling_needed_for_service_guar; 4590 4591 /* No point in idling for bfqq if it won't get requests any longer */ 4592 if (unlikely(!bfqq_process_refs(bfqq))) 4593 return false; 4594 4595 if (unlikely(bfqd->strict_guarantees)) 4596 return true; 4597 4598 /* 4599 * Idling is performed only if slice_idle > 0. In addition, we 4600 * do not idle if 4601 * (a) bfqq is async 4602 * (b) bfqq is in the idle io prio class: in this case we do 4603 * not idle because we want to minimize the bandwidth that 4604 * queues in this class can steal to higher-priority queues 4605 */ 4606 if (bfqd->bfq_slice_idle == 0 || !bfq_bfqq_sync(bfqq) || 4607 bfq_class_idle(bfqq)) 4608 return false; 4609 4610 idling_boosts_thr_with_no_issue = 4611 idling_boosts_thr_without_issues(bfqd, bfqq); 4612 4613 idling_needed_for_service_guar = 4614 idling_needed_for_service_guarantees(bfqd, bfqq); 4615 4616 /* 4617 * We have now the two components we need to compute the 4618 * return value of the function, which is true only if idling 4619 * either boosts the throughput (without issues), or is 4620 * necessary to preserve service guarantees. 4621 */ 4622 return idling_boosts_thr_with_no_issue || 4623 idling_needed_for_service_guar; 4624} 4625 4626/* 4627 * If the in-service queue is empty but the function bfq_better_to_idle 4628 * returns true, then: 4629 * 1) the queue must remain in service and cannot be expired, and 4630 * 2) the device must be idled to wait for the possible arrival of a new 4631 * request for the queue. 4632 * See the comments on the function bfq_better_to_idle for the reasons 4633 * why performing device idling is the best choice to boost the throughput 4634 * and preserve service guarantees when bfq_better_to_idle itself 4635 * returns true. 4636 */ 4637static bool bfq_bfqq_must_idle(struct bfq_queue *bfqq) 4638{ 4639 return RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_better_to_idle(bfqq); 4640} 4641 4642/* 4643 * This function chooses the queue from which to pick the next extra 4644 * I/O request to inject, if it finds a compatible queue. See the 4645 * comments on bfq_update_inject_limit() for details on the injection 4646 * mechanism, and for the definitions of the quantities mentioned 4647 * below. 4648 */ 4649static struct bfq_queue * 4650bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) 4651{ 4652 struct bfq_queue *bfqq, *in_serv_bfqq = bfqd->in_service_queue; 4653 unsigned int limit = in_serv_bfqq->inject_limit; 4654 int i; 4655 4656 /* 4657 * If 4658 * - bfqq is not weight-raised and therefore does not carry 4659 * time-critical I/O, 4660 * or 4661 * - regardless of whether bfqq is weight-raised, bfqq has 4662 * however a long think time, during which it can absorb the 4663 * effect of an appropriate number of extra I/O requests 4664 * from other queues (see bfq_update_inject_limit for 4665 * details on the computation of this number); 4666 * then injection can be performed without restrictions. 4667 */ 4668 bool in_serv_always_inject = in_serv_bfqq->wr_coeff == 1 || 4669 !bfq_bfqq_has_short_ttime(in_serv_bfqq); 4670 4671 /* 4672 * If 4673 * - the baseline total service time could not be sampled yet, 4674 * so the inject limit happens to be still 0, and 4675 * - a lot of time has elapsed since the plugging of I/O 4676 * dispatching started, so drive speed is being wasted 4677 * significantly; 4678 * then temporarily raise inject limit to one request. 4679 */ 4680 if (limit == 0 && in_serv_bfqq->last_serv_time_ns == 0 && 4681 bfq_bfqq_wait_request(in_serv_bfqq) && 4682 time_is_before_eq_jiffies(bfqd->last_idling_start_jiffies + 4683 bfqd->bfq_slice_idle) 4684 ) 4685 limit = 1; 4686 4687 if (bfqd->tot_rq_in_driver >= limit) 4688 return NULL; 4689 4690 /* 4691 * Linear search of the source queue for injection; but, with 4692 * a high probability, very few steps are needed to find a 4693 * candidate queue, i.e., a queue with enough budget left for 4694 * its next request. In fact: 4695 * - BFQ dynamically updates the budget of every queue so as 4696 * to accommodate the expected backlog of the queue; 4697 * - if a queue gets all its requests dispatched as injected 4698 * service, then the queue is removed from the active list 4699 * (and re-added only if it gets new requests, but then it 4700 * is assigned again enough budget for its new backlog). 4701 */ 4702 for (i = 0; i < bfqd->num_actuators; i++) { 4703 list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list) 4704 if (!RB_EMPTY_ROOT(&bfqq->sort_list) && 4705 (in_serv_always_inject || bfqq->wr_coeff > 1) && 4706 bfq_serv_to_charge(bfqq->next_rq, bfqq) <= 4707 bfq_bfqq_budget_left(bfqq)) { 4708 /* 4709 * Allow for only one large in-flight request 4710 * on non-rotational devices, for the 4711 * following reason. On non-rotationl drives, 4712 * large requests take much longer than 4713 * smaller requests to be served. In addition, 4714 * the drive prefers to serve large requests 4715 * w.r.t. to small ones, if it can choose. So, 4716 * having more than one large requests queued 4717 * in the drive may easily make the next first 4718 * request of the in-service queue wait for so 4719 * long to break bfqq's service guarantees. On 4720 * the bright side, large requests let the 4721 * drive reach a very high throughput, even if 4722 * there is only one in-flight large request 4723 * at a time. 4724 */ 4725 if (blk_queue_nonrot(bfqd->queue) && 4726 blk_rq_sectors(bfqq->next_rq) >= 4727 BFQQ_SECT_THR_NONROT && 4728 bfqd->tot_rq_in_driver >= 1) 4729 continue; 4730 else { 4731 bfqd->rqs_injected = true; 4732 return bfqq; 4733 } 4734 } 4735 } 4736 4737 return NULL; 4738} 4739 4740static struct bfq_queue * 4741bfq_find_active_bfqq_for_actuator(struct bfq_data *bfqd, int idx) 4742{ 4743 struct bfq_queue *bfqq; 4744 4745 if (bfqd->in_service_queue && 4746 bfqd->in_service_queue->actuator_idx == idx) 4747 return bfqd->in_service_queue; 4748 4749 list_for_each_entry(bfqq, &bfqd->active_list[idx], bfqq_list) { 4750 if (!RB_EMPTY_ROOT(&bfqq->sort_list) && 4751 bfq_serv_to_charge(bfqq->next_rq, bfqq) <= 4752 bfq_bfqq_budget_left(bfqq)) { 4753 return bfqq; 4754 } 4755 } 4756 4757 return NULL; 4758} 4759 4760/* 4761 * Perform a linear scan of each actuator, until an actuator is found 4762 * for which the following three conditions hold: the load of the 4763 * actuator is below the threshold (see comments on 4764 * actuator_load_threshold for details) and lower than that of the 4765 * next actuator (comments on this extra condition below), and there 4766 * is a queue that contains I/O for that actuator. On success, return 4767 * that queue. 4768 * 4769 * Performing a plain linear scan entails a prioritization among 4770 * actuators. The extra condition above breaks this prioritization and 4771 * tends to distribute injection uniformly across actuators. 4772 */ 4773static struct bfq_queue * 4774bfq_find_bfqq_for_underused_actuator(struct bfq_data *bfqd) 4775{ 4776 int i; 4777 4778 for (i = 0 ; i < bfqd->num_actuators; i++) { 4779 if (bfqd->rq_in_driver[i] < bfqd->actuator_load_threshold && 4780 (i == bfqd->num_actuators - 1 || 4781 bfqd->rq_in_driver[i] < bfqd->rq_in_driver[i+1])) { 4782 struct bfq_queue *bfqq = 4783 bfq_find_active_bfqq_for_actuator(bfqd, i); 4784 4785 if (bfqq) 4786 return bfqq; 4787 } 4788 } 4789 4790 return NULL; 4791} 4792 4793 4794/* 4795 * Select a queue for service. If we have a current queue in service, 4796 * check whether to continue servicing it, or retrieve and set a new one. 4797 */ 4798static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) 4799{ 4800 struct bfq_queue *bfqq, *inject_bfqq; 4801 struct request *next_rq; 4802 enum bfqq_expiration reason = BFQQE_BUDGET_TIMEOUT; 4803 4804 bfqq = bfqd->in_service_queue; 4805 if (!bfqq) 4806 goto new_queue; 4807 4808 bfq_log_bfqq(bfqd, bfqq, "select_queue: already in-service queue"); 4809 4810 /* 4811 * Do not expire bfqq for budget timeout if bfqq may be about 4812 * to enjoy device idling. The reason why, in this case, we 4813 * prevent bfqq from expiring is the same as in the comments 4814 * on the case where bfq_bfqq_must_idle() returns true, in 4815 * bfq_completed_request(). 4816 */ 4817 if (bfq_may_expire_for_budg_timeout(bfqq) && 4818 !bfq_bfqq_must_idle(bfqq)) 4819 goto expire; 4820 4821check_queue: 4822 /* 4823 * If some actuator is underutilized, but the in-service 4824 * queue does not contain I/O for that actuator, then try to 4825 * inject I/O for that actuator. 4826 */ 4827 inject_bfqq = bfq_find_bfqq_for_underused_actuator(bfqd); 4828 if (inject_bfqq && inject_bfqq != bfqq) 4829 return inject_bfqq; 4830 4831 /* 4832 * This loop is rarely executed more than once. Even when it 4833 * happens, it is much more convenient to re-execute this loop 4834 * than to return NULL and trigger a new dispatch to get a 4835 * request served. 4836 */ 4837 next_rq = bfqq->next_rq; 4838 /* 4839 * If bfqq has requests queued and it has enough budget left to 4840 * serve them, keep the queue, otherwise expire it. 4841 */ 4842 if (next_rq) { 4843 if (bfq_serv_to_charge(next_rq, bfqq) > 4844 bfq_bfqq_budget_left(bfqq)) { 4845 /* 4846 * Expire the queue for budget exhaustion, 4847 * which makes sure that the next budget is 4848 * enough to serve the next request, even if 4849 * it comes from the fifo expired path. 4850 */ 4851 reason = BFQQE_BUDGET_EXHAUSTED; 4852 goto expire; 4853 } else { 4854 /* 4855 * The idle timer may be pending because we may 4856 * not disable disk idling even when a new request 4857 * arrives. 4858 */ 4859 if (bfq_bfqq_wait_request(bfqq)) { 4860 /* 4861 * If we get here: 1) at least a new request 4862 * has arrived but we have not disabled the 4863 * timer because the request was too small, 4864 * 2) then the block layer has unplugged 4865 * the device, causing the dispatch to be 4866 * invoked. 4867 * 4868 * Since the device is unplugged, now the 4869 * requests are probably large enough to 4870 * provide a reasonable throughput. 4871 * So we disable idling. 4872 */ 4873 bfq_clear_bfqq_wait_request(bfqq); 4874 hrtimer_try_to_cancel(&bfqd->idle_slice_timer); 4875 } 4876 goto keep_queue; 4877 } 4878 } 4879 4880 /* 4881 * No requests pending. However, if the in-service queue is idling 4882 * for a new request, or has requests waiting for a completion and 4883 * may idle after their completion, then keep it anyway. 4884 * 4885 * Yet, inject service from other queues if it boosts 4886 * throughput and is possible. 4887 */ 4888 if (bfq_bfqq_wait_request(bfqq) || 4889 (bfqq->dispatched != 0 && bfq_better_to_idle(bfqq))) { 4890 unsigned int act_idx = bfqq->actuator_idx; 4891 struct bfq_queue *async_bfqq = NULL; 4892 struct bfq_queue *blocked_bfqq = 4893 !hlist_empty(&bfqq->woken_list) ? 4894 container_of(bfqq->woken_list.first, 4895 struct bfq_queue, 4896 woken_list_node) 4897 : NULL; 4898 4899 if (bfqq->bic && bfqq->bic->bfqq[0][act_idx] && 4900 bfq_bfqq_busy(bfqq->bic->bfqq[0][act_idx]) && 4901 bfqq->bic->bfqq[0][act_idx]->next_rq) 4902 async_bfqq = bfqq->bic->bfqq[0][act_idx]; 4903 /* 4904 * The next four mutually-exclusive ifs decide 4905 * whether to try injection, and choose the queue to 4906 * pick an I/O request from. 4907 * 4908 * The first if checks whether the process associated 4909 * with bfqq has also async I/O pending. If so, it 4910 * injects such I/O unconditionally. Injecting async 4911 * I/O from the same process can cause no harm to the 4912 * process. On the contrary, it can only increase 4913 * bandwidth and reduce latency for the process. 4914 * 4915 * The second if checks whether there happens to be a 4916 * non-empty waker queue for bfqq, i.e., a queue whose 4917 * I/O needs to be completed for bfqq to receive new 4918 * I/O. This happens, e.g., if bfqq is associated with 4919 * a process that does some sync. A sync generates 4920 * extra blocking I/O, which must be completed before 4921 * the process associated with bfqq can go on with its 4922 * I/O. If the I/O of the waker queue is not served, 4923 * then bfqq remains empty, and no I/O is dispatched, 4924 * until the idle timeout fires for bfqq. This is 4925 * likely to result in lower bandwidth and higher 4926 * latencies for bfqq, and in a severe loss of total 4927 * throughput. The best action to take is therefore to 4928 * serve the waker queue as soon as possible. So do it 4929 * (without relying on the third alternative below for 4930 * eventually serving waker_bfqq's I/O; see the last 4931 * paragraph for further details). This systematic 4932 * injection of I/O from the waker queue does not 4933 * cause any delay to bfqq's I/O. On the contrary, 4934 * next bfqq's I/O is brought forward dramatically, 4935 * for it is not blocked for milliseconds. 4936 * 4937 * The third if checks whether there is a queue woken 4938 * by bfqq, and currently with pending I/O. Such a 4939 * woken queue does not steal bandwidth from bfqq, 4940 * because it remains soon without I/O if bfqq is not 4941 * served. So there is virtually no risk of loss of 4942 * bandwidth for bfqq if this woken queue has I/O 4943 * dispatched while bfqq is waiting for new I/O. 4944 * 4945 * The fourth if checks whether bfqq is a queue for 4946 * which it is better to avoid injection. It is so if 4947 * bfqq delivers more throughput when served without 4948 * any further I/O from other queues in the middle, or 4949 * if the service times of bfqq's I/O requests both 4950 * count more than overall throughput, and may be 4951 * easily increased by injection (this happens if bfqq 4952 * has a short think time). If none of these 4953 * conditions holds, then a candidate queue for 4954 * injection is looked for through 4955 * bfq_choose_bfqq_for_injection(). Note that the 4956 * latter may return NULL (for example if the inject 4957 * limit for bfqq is currently 0). 4958 * 4959 * NOTE: motivation for the second alternative 4960 * 4961 * Thanks to the way the inject limit is updated in 4962 * bfq_update_has_short_ttime(), it is rather likely 4963 * that, if I/O is being plugged for bfqq and the 4964 * waker queue has pending I/O requests that are 4965 * blocking bfqq's I/O, then the fourth alternative 4966 * above lets the waker queue get served before the 4967 * I/O-plugging timeout fires. So one may deem the 4968 * second alternative superfluous. It is not, because 4969 * the fourth alternative may be way less effective in 4970 * case of a synchronization. For two main 4971 * reasons. First, throughput may be low because the 4972 * inject limit may be too low to guarantee the same 4973 * amount of injected I/O, from the waker queue or 4974 * other queues, that the second alternative 4975 * guarantees (the second alternative unconditionally 4976 * injects a pending I/O request of the waker queue 4977 * for each bfq_dispatch_request()). Second, with the 4978 * fourth alternative, the duration of the plugging, 4979 * i.e., the time before bfqq finally receives new I/O, 4980 * may not be minimized, because the waker queue may 4981 * happen to be served only after other queues. 4982 */ 4983 if (async_bfqq && 4984 icq_to_bic(async_bfqq->next_rq->elv.icq) == bfqq->bic && 4985 bfq_serv_to_charge(async_bfqq->next_rq, async_bfqq) <= 4986 bfq_bfqq_budget_left(async_bfqq)) 4987 bfqq = async_bfqq; 4988 else if (bfqq->waker_bfqq && 4989 bfq_bfqq_busy(bfqq->waker_bfqq) && 4990 bfqq->waker_bfqq->next_rq && 4991 bfq_serv_to_charge(bfqq->waker_bfqq->next_rq, 4992 bfqq->waker_bfqq) <= 4993 bfq_bfqq_budget_left(bfqq->waker_bfqq) 4994 ) 4995 bfqq = bfqq->waker_bfqq; 4996 else if (blocked_bfqq && 4997 bfq_bfqq_busy(blocked_bfqq) && 4998 blocked_bfqq->next_rq && 4999 bfq_serv_to_charge(blocked_bfqq->next_rq, 5000 blocked_bfqq) <= 5001 bfq_bfqq_budget_left(blocked_bfqq) 5002 ) 5003 bfqq = blocked_bfqq; 5004 else if (!idling_boosts_thr_without_issues(bfqd, bfqq) && 5005 (bfqq->wr_coeff == 1 || bfqd->wr_busy_queues > 1 || 5006 !bfq_bfqq_has_short_ttime(bfqq))) 5007 bfqq = bfq_choose_bfqq_for_injection(bfqd); 5008 else 5009 bfqq = NULL; 5010 5011 goto keep_queue; 5012 } 5013 5014 reason = BFQQE_NO_MORE_REQUESTS; 5015expire: 5016 bfq_bfqq_expire(bfqd, bfqq, false, reason); 5017new_queue: 5018 bfqq = bfq_set_in_service_queue(bfqd); 5019 if (bfqq) { 5020 bfq_log_bfqq(bfqd, bfqq, "select_queue: checking new queue"); 5021 goto check_queue; 5022 } 5023keep_queue: 5024 if (bfqq) 5025 bfq_log_bfqq(bfqd, bfqq, "select_queue: returned this queue"); 5026 else 5027 bfq_log(bfqd, "select_queue: no queue returned"); 5028 5029 return bfqq; 5030} 5031 5032static void bfq_update_wr_data(struct bfq_data *bfqd, struct bfq_queue *bfqq) 5033{ 5034 struct bfq_entity *entity = &bfqq->entity; 5035 5036 if (bfqq->wr_coeff > 1) { /* queue is being weight-raised */ 5037 bfq_log_bfqq(bfqd, bfqq, 5038 "raising period dur %u/%u msec, old coeff %u, w %d(%d)", 5039 jiffies_to_msecs(jiffies - bfqq->last_wr_start_finish), 5040 jiffies_to_msecs(bfqq->wr_cur_max_time), 5041 bfqq->wr_coeff, 5042 bfqq->entity.weight, bfqq->entity.orig_weight); 5043 5044 if (entity->prio_changed) 5045 bfq_log_bfqq(bfqd, bfqq, "WARN: pending prio change"); 5046 5047 /* 5048 * If the queue was activated in a burst, or too much 5049 * time has elapsed from the beginning of this 5050 * weight-raising period, then end weight raising. 5051 */ 5052 if (bfq_bfqq_in_large_burst(bfqq)) 5053 bfq_bfqq_end_wr(bfqq); 5054 else if (time_is_before_jiffies(bfqq->last_wr_start_finish + 5055 bfqq->wr_cur_max_time)) { 5056 if (bfqq->wr_cur_max_time != bfqd->bfq_wr_rt_max_time || 5057 time_is_before_jiffies(bfqq->wr_start_at_switch_to_srt + 5058 bfq_wr_duration(bfqd))) { 5059 /* 5060 * Either in interactive weight 5061 * raising, or in soft_rt weight 5062 * raising with the 5063 * interactive-weight-raising period 5064 * elapsed (so no switch back to 5065 * interactive weight raising). 5066 */ 5067 bfq_bfqq_end_wr(bfqq); 5068 } else { /* 5069 * soft_rt finishing while still in 5070 * interactive period, switch back to 5071 * interactive weight raising 5072 */ 5073 switch_back_to_interactive_wr(bfqq, bfqd); 5074 bfqq->entity.prio_changed = 1; 5075 } 5076 } 5077 if (bfqq->wr_coeff > 1 && 5078 bfqq->wr_cur_max_time != bfqd->bfq_wr_rt_max_time && 5079 bfqq->service_from_wr > max_service_from_wr) { 5080 /* see comments on max_service_from_wr */ 5081 bfq_bfqq_end_wr(bfqq); 5082 } 5083 } 5084 /* 5085 * To improve latency (for this or other queues), immediately 5086 * update weight both if it must be raised and if it must be 5087 * lowered. Since, entity may be on some active tree here, and 5088 * might have a pending change of its ioprio class, invoke 5089 * next function with the last parameter unset (see the 5090 * comments on the function). 5091 */ 5092 if ((entity->weight > entity->orig_weight) != (bfqq->wr_coeff > 1)) 5093 __bfq_entity_update_weight_prio(bfq_entity_service_tree(entity), 5094 entity, false); 5095} 5096 5097/* 5098 * Dispatch next request from bfqq. 5099 */ 5100static struct request *bfq_dispatch_rq_from_bfqq(struct bfq_data *bfqd, 5101 struct bfq_queue *bfqq) 5102{ 5103 struct request *rq = bfqq->next_rq; 5104 unsigned long service_to_charge; 5105 5106 service_to_charge = bfq_serv_to_charge(rq, bfqq); 5107 5108 bfq_bfqq_served(bfqq, service_to_charge); 5109 5110 if (bfqq == bfqd->in_service_queue && bfqd->wait_dispatch) { 5111 bfqd->wait_dispatch = false; 5112 bfqd->waited_rq = rq; 5113 } 5114 5115 bfq_dispatch_remove(bfqd->queue, rq); 5116 5117 if (bfqq != bfqd->in_service_queue) 5118 return rq; 5119 5120 /* 5121 * If weight raising has to terminate for bfqq, then next 5122 * function causes an immediate update of bfqq's weight, 5123 * without waiting for next activation. As a consequence, on 5124 * expiration, bfqq will be timestamped as if has never been 5125 * weight-raised during this service slot, even if it has 5126 * received part or even most of the service as a 5127 * weight-raised queue. This inflates bfqq's timestamps, which 5128 * is beneficial, as bfqq is then more willing to leave the 5129 * device immediately to possible other weight-raised queues. 5130 */ 5131 bfq_update_wr_data(bfqd, bfqq); 5132 5133 /* 5134 * Expire bfqq, pretending that its budget expired, if bfqq 5135 * belongs to CLASS_IDLE and other queues are waiting for 5136 * service. 5137 */ 5138 if (bfq_tot_busy_queues(bfqd) > 1 && bfq_class_idle(bfqq)) 5139 bfq_bfqq_expire(bfqd, bfqq, false, BFQQE_BUDGET_EXHAUSTED); 5140 5141 return rq; 5142} 5143 5144static bool bfq_has_work(struct blk_mq_hw_ctx *hctx) 5145{ 5146 struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; 5147 5148 /* 5149 * Avoiding lock: a race on bfqd->queued should cause at 5150 * most a call to dispatch for nothing 5151 */ 5152 return !list_empty_careful(&bfqd->dispatch) || 5153 READ_ONCE(bfqd->queued); 5154} 5155 5156static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) 5157{ 5158 struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; 5159 struct request *rq = NULL; 5160 struct bfq_queue *bfqq = NULL; 5161 5162 if (!list_empty(&bfqd->dispatch)) { 5163 rq = list_first_entry(&bfqd->dispatch, struct request, 5164 queuelist); 5165 list_del_init(&rq->queuelist); 5166 5167 bfqq = RQ_BFQQ(rq); 5168 5169 if (bfqq) { 5170 /* 5171 * Increment counters here, because this 5172 * dispatch does not follow the standard 5173 * dispatch flow (where counters are 5174 * incremented) 5175 */ 5176 bfqq->dispatched++; 5177 5178 goto inc_in_driver_start_rq; 5179 } 5180 5181 /* 5182 * We exploit the bfq_finish_requeue_request hook to 5183 * decrement tot_rq_in_driver, but 5184 * bfq_finish_requeue_request will not be invoked on 5185 * this request. So, to avoid unbalance, just start 5186 * this request, without incrementing tot_rq_in_driver. As 5187 * a negative consequence, tot_rq_in_driver is deceptively 5188 * lower than it should be while this request is in 5189 * service. This may cause bfq_schedule_dispatch to be 5190 * invoked uselessly. 5191 * 5192 * As for implementing an exact solution, the 5193 * bfq_finish_requeue_request hook, if defined, is 5194 * probably invoked also on this request. So, by 5195 * exploiting this hook, we could 1) increment 5196 * tot_rq_in_driver here, and 2) decrement it in 5197 * bfq_finish_requeue_request. Such a solution would 5198 * let the value of the counter be always accurate, 5199 * but it would entail using an extra interface 5200 * function. This cost seems higher than the benefit, 5201 * being the frequency of non-elevator-private 5202 * requests very low. 5203 */ 5204 goto start_rq; 5205 } 5206 5207 bfq_log(bfqd, "dispatch requests: %d busy queues", 5208 bfq_tot_busy_queues(bfqd)); 5209 5210 if (bfq_tot_busy_queues(bfqd) == 0) 5211 goto exit; 5212 5213 /* 5214 * Force device to serve one request at a time if 5215 * strict_guarantees is true. Forcing this service scheme is 5216 * currently the ONLY way to guarantee that the request 5217 * service order enforced by the scheduler is respected by a 5218 * queueing device. Otherwise the device is free even to make 5219 * some unlucky request wait for as long as the device 5220 * wishes. 5221 * 5222 * Of course, serving one request at a time may cause loss of 5223 * throughput. 5224 */ 5225 if (bfqd->strict_guarantees && bfqd->tot_rq_in_driver > 0) 5226 goto exit; 5227 5228 bfqq = bfq_select_queue(bfqd); 5229 if (!bfqq) 5230 goto exit; 5231 5232 rq = bfq_dispatch_rq_from_bfqq(bfqd, bfqq); 5233 5234 if (rq) { 5235inc_in_driver_start_rq: 5236 bfqd->rq_in_driver[bfqq->actuator_idx]++; 5237 bfqd->tot_rq_in_driver++; 5238start_rq: 5239 rq->rq_flags |= RQF_STARTED; 5240 } 5241exit: 5242 return rq; 5243} 5244 5245#ifdef CONFIG_BFQ_CGROUP_DEBUG 5246static void bfq_update_dispatch_stats(struct request_queue *q, 5247 struct request *rq, 5248 struct bfq_queue *in_serv_queue, 5249 bool idle_timer_disabled) 5250{ 5251 struct bfq_queue *bfqq = rq ? RQ_BFQQ(rq) : NULL; 5252 5253 if (!idle_timer_disabled && !bfqq) 5254 return; 5255 5256 /* 5257 * rq and bfqq are guaranteed to exist until this function 5258 * ends, for the following reasons. First, rq can be 5259 * dispatched to the device, and then can be completed and 5260 * freed, only after this function ends. Second, rq cannot be 5261 * merged (and thus freed because of a merge) any longer, 5262 * because it has already started. Thus rq cannot be freed 5263 * before this function ends, and, since rq has a reference to 5264 * bfqq, the same guarantee holds for bfqq too. 5265 * 5266 * In addition, the following queue lock guarantees that 5267 * bfqq_group(bfqq) exists as well. 5268 */ 5269 spin_lock_irq(&q->queue_lock); 5270 if (idle_timer_disabled) 5271 /* 5272 * Since the idle timer has been disabled, 5273 * in_serv_queue contained some request when 5274 * __bfq_dispatch_request was invoked above, which 5275 * implies that rq was picked exactly from 5276 * in_serv_queue. Thus in_serv_queue == bfqq, and is 5277 * therefore guaranteed to exist because of the above 5278 * arguments. 5279 */ 5280 bfqg_stats_update_idle_time(bfqq_group(in_serv_queue)); 5281 if (bfqq) { 5282 struct bfq_group *bfqg = bfqq_group(bfqq); 5283 5284 bfqg_stats_update_avg_queue_size(bfqg); 5285 bfqg_stats_set_start_empty_time(bfqg); 5286 bfqg_stats_update_io_remove(bfqg, rq->cmd_flags); 5287 } 5288 spin_unlock_irq(&q->queue_lock); 5289} 5290#else 5291static inline void bfq_update_dispatch_stats(struct request_queue *q, 5292 struct request *rq, 5293 struct bfq_queue *in_serv_queue, 5294 bool idle_timer_disabled) {} 5295#endif /* CONFIG_BFQ_CGROUP_DEBUG */ 5296 5297static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) 5298{ 5299 struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; 5300 struct request *rq; 5301 struct bfq_queue *in_serv_queue; 5302 bool waiting_rq, idle_timer_disabled = false; 5303 5304 spin_lock_irq(&bfqd->lock); 5305 5306 in_serv_queue = bfqd->in_service_queue; 5307 waiting_rq = in_serv_queue && bfq_bfqq_wait_request(in_serv_queue); 5308 5309 rq = __bfq_dispatch_request(hctx); 5310 if (in_serv_queue == bfqd->in_service_queue) { 5311 idle_timer_disabled = 5312 waiting_rq && !bfq_bfqq_wait_request(in_serv_queue); 5313 } 5314 5315 spin_unlock_irq(&bfqd->lock); 5316 bfq_update_dispatch_stats(hctx->queue, rq, 5317 idle_timer_disabled ? in_serv_queue : NULL, 5318 idle_timer_disabled); 5319 5320 return rq; 5321} 5322 5323/* 5324 * Task holds one reference to the queue, dropped when task exits. Each rq 5325 * in-flight on this queue also holds a reference, dropped when rq is freed. 5326 * 5327 * Scheduler lock must be held here. Recall not to use bfqq after calling 5328 * this function on it. 5329 */ 5330void bfq_put_queue(struct bfq_queue *bfqq) 5331{ 5332 struct bfq_queue *item; 5333 struct hlist_node *n; 5334 struct bfq_group *bfqg = bfqq_group(bfqq); 5335 5336 bfq_log_bfqq(bfqq->bfqd, bfqq, "put_queue: %p %d", bfqq, bfqq->ref); 5337 5338 bfqq->ref--; 5339 if (bfqq->ref) 5340 return; 5341 5342 if (!hlist_unhashed(&bfqq->burst_list_node)) { 5343 hlist_del_init(&bfqq->burst_list_node); 5344 /* 5345 * Decrement also burst size after the removal, if the 5346 * process associated with bfqq is exiting, and thus 5347 * does not contribute to the burst any longer. This 5348 * decrement helps filter out false positives of large 5349 * bursts, when some short-lived process (often due to 5350 * the execution of commands by some service) happens 5351 * to start and exit while a complex application is 5352 * starting, and thus spawning several processes that 5353 * do I/O (and that *must not* be treated as a large 5354 * burst, see comments on bfq_handle_burst). 5355 * 5356 * In particular, the decrement is performed only if: 5357 * 1) bfqq is not a merged queue, because, if it is, 5358 * then this free of bfqq is not triggered by the exit 5359 * of the process bfqq is associated with, but exactly 5360 * by the fact that bfqq has just been merged. 5361 * 2) burst_size is greater than 0, to handle 5362 * unbalanced decrements. Unbalanced decrements may 5363 * happen in te following case: bfqq is inserted into 5364 * the current burst list--without incrementing 5365 * bust_size--because of a split, but the current 5366 * burst list is not the burst list bfqq belonged to 5367 * (see comments on the case of a split in 5368 * bfq_set_request). 5369 */ 5370 if (bfqq->bic && bfqq->bfqd->burst_size > 0) 5371 bfqq->bfqd->burst_size--; 5372 } 5373 5374 /* 5375 * bfqq does not exist any longer, so it cannot be woken by 5376 * any other queue, and cannot wake any other queue. Then bfqq 5377 * must be removed from the woken list of its possible waker 5378 * queue, and all queues in the woken list of bfqq must stop 5379 * having a waker queue. Strictly speaking, these updates 5380 * should be performed when bfqq remains with no I/O source 5381 * attached to it, which happens before bfqq gets freed. In 5382 * particular, this happens when the last process associated 5383 * with bfqq exits or gets associated with a different 5384 * queue. However, both events lead to bfqq being freed soon, 5385 * and dangling references would come out only after bfqq gets 5386 * freed. So these updates are done here, as a simple and safe 5387 * way to handle all cases. 5388 */ 5389 /* remove bfqq from woken list */ 5390 if (!hlist_unhashed(&bfqq->woken_list_node)) 5391 hlist_del_init(&bfqq->woken_list_node); 5392 5393 /* reset waker for all queues in woken list */ 5394 hlist_for_each_entry_safe(item, n, &bfqq->woken_list, 5395 woken_list_node) { 5396 item->waker_bfqq = NULL; 5397 hlist_del_init(&item->woken_list_node); 5398 } 5399 5400 if (bfqq->bfqd->last_completed_rq_bfqq == bfqq) 5401 bfqq->bfqd->last_completed_rq_bfqq = NULL; 5402 5403 WARN_ON_ONCE(!list_empty(&bfqq->fifo)); 5404 WARN_ON_ONCE(!RB_EMPTY_ROOT(&bfqq->sort_list)); 5405 WARN_ON_ONCE(bfqq->dispatched); 5406 5407 kmem_cache_free(bfq_pool, bfqq); 5408 bfqg_and_blkg_put(bfqg); 5409} 5410 5411static void bfq_put_stable_ref(struct bfq_queue *bfqq) 5412{ 5413 bfqq->stable_ref--; 5414 bfq_put_queue(bfqq); 5415} 5416 5417void bfq_put_cooperator(struct bfq_queue *bfqq) 5418{ 5419 struct bfq_queue *__bfqq, *next; 5420 5421 /* 5422 * If this queue was scheduled to merge with another queue, be 5423 * sure to drop the reference taken on that queue (and others in 5424 * the merge chain). See bfq_setup_merge and bfq_merge_bfqqs. 5425 */ 5426 __bfqq = bfqq->new_bfqq; 5427 while (__bfqq) { 5428 next = __bfqq->new_bfqq; 5429 bfq_put_queue(__bfqq); 5430 __bfqq = next; 5431 } 5432} 5433 5434static void bfq_exit_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq) 5435{ 5436 if (bfqq == bfqd->in_service_queue) { 5437 __bfq_bfqq_expire(bfqd, bfqq, BFQQE_BUDGET_TIMEOUT); 5438 bfq_schedule_dispatch(bfqd); 5439 } 5440 5441 bfq_log_bfqq(bfqd, bfqq, "exit_bfqq: %p, %d", bfqq, bfqq->ref); 5442 5443 bfq_put_cooperator(bfqq); 5444 5445 bfq_release_process_ref(bfqd, bfqq); 5446} 5447 5448static void bfq_exit_icq_bfqq(struct bfq_io_cq *bic, bool is_sync, 5449 unsigned int actuator_idx) 5450{ 5451 struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, actuator_idx); 5452 struct bfq_data *bfqd; 5453 5454 if (bfqq) 5455 bfqd = bfqq->bfqd; /* NULL if scheduler already exited */ 5456 5457 if (bfqq && bfqd) { 5458 bic_set_bfqq(bic, NULL, is_sync, actuator_idx); 5459 bfq_exit_bfqq(bfqd, bfqq); 5460 } 5461} 5462 5463static void _bfq_exit_icq(struct bfq_io_cq *bic, unsigned int num_actuators) 5464{ 5465 struct bfq_iocq_bfqq_data *bfqq_data = bic->bfqq_data; 5466 unsigned int act_idx; 5467 5468 for (act_idx = 0; act_idx < num_actuators; act_idx++) { 5469 if (bfqq_data[act_idx].stable_merge_bfqq) 5470 bfq_put_stable_ref(bfqq_data[act_idx].stable_merge_bfqq); 5471 5472 bfq_exit_icq_bfqq(bic, true, act_idx); 5473 bfq_exit_icq_bfqq(bic, false, act_idx); 5474 } 5475} 5476 5477static void bfq_exit_icq(struct io_cq *icq) 5478{ 5479 struct bfq_io_cq *bic = icq_to_bic(icq); 5480 struct bfq_data *bfqd = bic_to_bfqd(bic); 5481 unsigned long flags; 5482 5483 /* 5484 * If bfqd and thus bfqd->num_actuators is not available any 5485 * longer, then cycle over all possible per-actuator bfqqs in 5486 * next loop. We rely on bic being zeroed on creation, and 5487 * therefore on its unused per-actuator fields being NULL. 5488 * 5489 * bfqd is NULL if scheduler already exited, and in that case 5490 * this is the last time these queues are accessed. 5491 */ 5492 if (bfqd) { 5493 spin_lock_irqsave(&bfqd->lock, flags); 5494 _bfq_exit_icq(bic, bfqd->num_actuators); 5495 spin_unlock_irqrestore(&bfqd->lock, flags); 5496 } else { 5497 _bfq_exit_icq(bic, BFQ_MAX_ACTUATORS); 5498 } 5499} 5500 5501/* 5502 * Update the entity prio values; note that the new values will not 5503 * be used until the next (re)activation. 5504 */ 5505static void 5506bfq_set_next_ioprio_data(struct bfq_queue *bfqq, struct bfq_io_cq *bic) 5507{ 5508 struct task_struct *tsk = current; 5509 int ioprio_class; 5510 struct bfq_data *bfqd = bfqq->bfqd; 5511 5512 if (!bfqd) 5513 return; 5514 5515 ioprio_class = IOPRIO_PRIO_CLASS(bic->ioprio); 5516 switch (ioprio_class) { 5517 default: 5518 pr_err("bdi %s: bfq: bad prio class %d\n", 5519 bdi_dev_name(bfqq->bfqd->queue->disk->bdi), 5520 ioprio_class); 5521 fallthrough; 5522 case IOPRIO_CLASS_NONE: 5523 /* 5524 * No prio set, inherit CPU scheduling settings. 5525 */ 5526 bfqq->new_ioprio = task_nice_ioprio(tsk); 5527 bfqq->new_ioprio_class = task_nice_ioclass(tsk); 5528 break; 5529 case IOPRIO_CLASS_RT: 5530 bfqq->new_ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio); 5531 bfqq->new_ioprio_class = IOPRIO_CLASS_RT; 5532 break; 5533 case IOPRIO_CLASS_BE: 5534 bfqq->new_ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio); 5535 bfqq->new_ioprio_class = IOPRIO_CLASS_BE; 5536 break; 5537 case IOPRIO_CLASS_IDLE: 5538 bfqq->new_ioprio_class = IOPRIO_CLASS_IDLE; 5539 bfqq->new_ioprio = IOPRIO_NR_LEVELS - 1; 5540 break; 5541 } 5542 5543 if (bfqq->new_ioprio >= IOPRIO_NR_LEVELS) { 5544 pr_crit("bfq_set_next_ioprio_data: new_ioprio %d\n", 5545 bfqq->new_ioprio); 5546 bfqq->new_ioprio = IOPRIO_NR_LEVELS - 1; 5547 } 5548 5549 bfqq->entity.new_weight = bfq_ioprio_to_weight(bfqq->new_ioprio); 5550 bfq_log_bfqq(bfqd, bfqq, "new_ioprio %d new_weight %d", 5551 bfqq->new_ioprio, bfqq->entity.new_weight); 5552 bfqq->entity.prio_changed = 1; 5553} 5554 5555static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, 5556 struct bio *bio, bool is_sync, 5557 struct bfq_io_cq *bic, 5558 bool respawn); 5559 5560static void bfq_check_ioprio_change(struct bfq_io_cq *bic, struct bio *bio) 5561{ 5562 struct bfq_data *bfqd = bic_to_bfqd(bic); 5563 struct bfq_queue *bfqq; 5564 int ioprio = bic->icq.ioc->ioprio; 5565 5566 /* 5567 * This condition may trigger on a newly created bic, be sure to 5568 * drop the lock before returning. 5569 */ 5570 if (unlikely(!bfqd) || likely(bic->ioprio == ioprio)) 5571 return; 5572 5573 bic->ioprio = ioprio; 5574 5575 bfqq = bic_to_bfqq(bic, false, bfq_actuator_index(bfqd, bio)); 5576 if (bfqq) { 5577 struct bfq_queue *old_bfqq = bfqq; 5578 5579 bfqq = bfq_get_queue(bfqd, bio, false, bic, true); 5580 bic_set_bfqq(bic, bfqq, false, bfq_actuator_index(bfqd, bio)); 5581 bfq_release_process_ref(bfqd, old_bfqq); 5582 } 5583 5584 bfqq = bic_to_bfqq(bic, true, bfq_actuator_index(bfqd, bio)); 5585 if (bfqq) 5586 bfq_set_next_ioprio_data(bfqq, bic); 5587} 5588 5589static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, 5590 struct bfq_io_cq *bic, pid_t pid, int is_sync, 5591 unsigned int act_idx) 5592{ 5593 u64 now_ns = blk_time_get_ns(); 5594 5595 bfqq->actuator_idx = act_idx; 5596 RB_CLEAR_NODE(&bfqq->entity.rb_node); 5597 INIT_LIST_HEAD(&bfqq->fifo); 5598 INIT_HLIST_NODE(&bfqq->burst_list_node); 5599 INIT_HLIST_NODE(&bfqq->woken_list_node); 5600 INIT_HLIST_HEAD(&bfqq->woken_list); 5601 5602 bfqq->ref = 0; 5603 bfqq->bfqd = bfqd; 5604 5605 if (bic) 5606 bfq_set_next_ioprio_data(bfqq, bic); 5607 5608 if (is_sync) { 5609 /* 5610 * No need to mark as has_short_ttime if in 5611 * idle_class, because no device idling is performed 5612 * for queues in idle class 5613 */ 5614 if (!bfq_class_idle(bfqq)) 5615 /* tentatively mark as has_short_ttime */ 5616 bfq_mark_bfqq_has_short_ttime(bfqq); 5617 bfq_mark_bfqq_sync(bfqq); 5618 bfq_mark_bfqq_just_created(bfqq); 5619 } else 5620 bfq_clear_bfqq_sync(bfqq); 5621 5622 /* set end request to minus infinity from now */ 5623 bfqq->ttime.last_end_request = now_ns + 1; 5624 5625 bfqq->creation_time = jiffies; 5626 5627 bfqq->io_start_time = now_ns; 5628 5629 bfq_mark_bfqq_IO_bound(bfqq); 5630 5631 bfqq->pid = pid; 5632 5633 /* Tentative initial value to trade off between thr and lat */ 5634 bfqq->max_budget = (2 * bfq_max_budget(bfqd)) / 3; 5635 bfqq->budget_timeout = bfq_smallest_from_now(); 5636 5637 bfqq->wr_coeff = 1; 5638 bfqq->last_wr_start_finish = jiffies; 5639 bfqq->wr_start_at_switch_to_srt = bfq_smallest_from_now(); 5640 bfqq->split_time = bfq_smallest_from_now(); 5641 5642 /* 5643 * To not forget the possibly high bandwidth consumed by a 5644 * process/queue in the recent past, 5645 * bfq_bfqq_softrt_next_start() returns a value at least equal 5646 * to the current value of bfqq->soft_rt_next_start (see 5647 * comments on bfq_bfqq_softrt_next_start). Set 5648 * soft_rt_next_start to now, to mean that bfqq has consumed 5649 * no bandwidth so far. 5650 */ 5651 bfqq->soft_rt_next_start = jiffies; 5652 5653 /* first request is almost certainly seeky */ 5654 bfqq->seek_history = 1; 5655 5656 bfqq->decrease_time_jif = jiffies; 5657} 5658 5659static struct bfq_queue **bfq_async_queue_prio(struct bfq_data *bfqd, 5660 struct bfq_group *bfqg, 5661 int ioprio_class, int ioprio, int act_idx) 5662{ 5663 switch (ioprio_class) { 5664 case IOPRIO_CLASS_RT: 5665 return &bfqg->async_bfqq[0][ioprio][act_idx]; 5666 case IOPRIO_CLASS_NONE: 5667 ioprio = IOPRIO_BE_NORM; 5668 fallthrough; 5669 case IOPRIO_CLASS_BE: 5670 return &bfqg->async_bfqq[1][ioprio][act_idx]; 5671 case IOPRIO_CLASS_IDLE: 5672 return &bfqg->async_idle_bfqq[act_idx]; 5673 default: 5674 return NULL; 5675 } 5676} 5677 5678static struct bfq_queue * 5679bfq_do_early_stable_merge(struct bfq_data *bfqd, struct bfq_queue *bfqq, 5680 struct bfq_io_cq *bic, 5681 struct bfq_queue *last_bfqq_created) 5682{ 5683 unsigned int a_idx = last_bfqq_created->actuator_idx; 5684 struct bfq_queue *new_bfqq = 5685 bfq_setup_merge(bfqq, last_bfqq_created); 5686 5687 if (!new_bfqq) 5688 return bfqq; 5689 5690 if (new_bfqq->bic) 5691 new_bfqq->bic->bfqq_data[a_idx].stably_merged = true; 5692 bic->bfqq_data[a_idx].stably_merged = true; 5693 5694 /* 5695 * Reusing merge functions. This implies that 5696 * bfqq->bic must be set too, for 5697 * bfq_merge_bfqqs to correctly save bfqq's 5698 * state before killing it. 5699 */ 5700 bfqq->bic = bic; 5701 return bfq_merge_bfqqs(bfqd, bic, bfqq); 5702} 5703 5704/* 5705 * Many throughput-sensitive workloads are made of several parallel 5706 * I/O flows, with all flows generated by the same application, or 5707 * more generically by the same task (e.g., system boot). The most 5708 * counterproductive action with these workloads is plugging I/O 5709 * dispatch when one of the bfq_queues associated with these flows 5710 * remains temporarily empty. 5711 * 5712 * To avoid this plugging, BFQ has been using a burst-handling 5713 * mechanism for years now. This mechanism has proven effective for 5714 * throughput, and not detrimental for service guarantees. The 5715 * following function pushes this mechanism a little bit further, 5716 * basing on the following two facts. 5717 * 5718 * First, all the I/O flows of a the same application or task 5719 * contribute to the execution/completion of that common application 5720 * or task. So the performance figures that matter are total 5721 * throughput of the flows and task-wide I/O latency. In particular, 5722 * these flows do not need to be protected from each other, in terms 5723 * of individual bandwidth or latency. 5724 * 5725 * Second, the above fact holds regardless of the number of flows. 5726 * 5727 * Putting these two facts together, this commits merges stably the 5728 * bfq_queues associated with these I/O flows, i.e., with the 5729 * processes that generate these IO/ flows, regardless of how many the 5730 * involved processes are. 5731 * 5732 * To decide whether a set of bfq_queues is actually associated with 5733 * the I/O flows of a common application or task, and to merge these 5734 * queues stably, this function operates as follows: given a bfq_queue, 5735 * say Q2, currently being created, and the last bfq_queue, say Q1, 5736 * created before Q2, Q2 is merged stably with Q1 if 5737 * - very little time has elapsed since when Q1 was created 5738 * - Q2 has the same ioprio as Q1 5739 * - Q2 belongs to the same group as Q1 5740 * 5741 * Merging bfq_queues also reduces scheduling overhead. A fio test 5742 * with ten random readers on /dev/nullb shows a throughput boost of 5743 * 40%, with a quadcore. Since BFQ's execution time amounts to ~50% of 5744 * the total per-request processing time, the above throughput boost 5745 * implies that BFQ's overhead is reduced by more than 50%. 5746 * 5747 * This new mechanism most certainly obsoletes the current 5748 * burst-handling heuristics. We keep those heuristics for the moment. 5749 */ 5750static struct bfq_queue *bfq_do_or_sched_stable_merge(struct bfq_data *bfqd, 5751 struct bfq_queue *bfqq, 5752 struct bfq_io_cq *bic) 5753{ 5754 struct bfq_queue **source_bfqq = bfqq->entity.parent ? 5755 &bfqq->entity.parent->last_bfqq_created : 5756 &bfqd->last_bfqq_created; 5757 5758 struct bfq_queue *last_bfqq_created = *source_bfqq; 5759 5760 /* 5761 * If last_bfqq_created has not been set yet, then init it. If 5762 * it has been set already, but too long ago, then move it 5763 * forward to bfqq. Finally, move also if bfqq belongs to a 5764 * different group than last_bfqq_created, or if bfqq has a 5765 * different ioprio, ioprio_class or actuator_idx. If none of 5766 * these conditions holds true, then try an early stable merge 5767 * or schedule a delayed stable merge. As for the condition on 5768 * actuator_idx, the reason is that, if queues associated with 5769 * different actuators are merged, then control is lost on 5770 * each actuator. Therefore some actuator may be 5771 * underutilized, and throughput may decrease. 5772 * 5773 * A delayed merge is scheduled (instead of performing an 5774 * early merge), in case bfqq might soon prove to be more 5775 * throughput-beneficial if not merged. Currently this is 5776 * possible only if bfqd is rotational with no queueing. For 5777 * such a drive, not merging bfqq is better for throughput if 5778 * bfqq happens to contain sequential I/O. So, we wait a 5779 * little bit for enough I/O to flow through bfqq. After that, 5780 * if such an I/O is sequential, then the merge is 5781 * canceled. Otherwise the merge is finally performed. 5782 */ 5783 if (!last_bfqq_created || 5784 time_before(last_bfqq_created->creation_time + 5785 msecs_to_jiffies(bfq_activation_stable_merging), 5786 bfqq->creation_time) || 5787 bfqq->entity.parent != last_bfqq_created->entity.parent || 5788 bfqq->ioprio != last_bfqq_created->ioprio || 5789 bfqq->ioprio_class != last_bfqq_created->ioprio_class || 5790 bfqq->actuator_idx != last_bfqq_created->actuator_idx) 5791 *source_bfqq = bfqq; 5792 else if (time_after_eq(last_bfqq_created->creation_time + 5793 bfqd->bfq_burst_interval, 5794 bfqq->creation_time)) { 5795 if (likely(bfqd->nonrot_with_queueing)) 5796 /* 5797 * With this type of drive, leaving 5798 * bfqq alone may provide no 5799 * throughput benefits compared with 5800 * merging bfqq. So merge bfqq now. 5801 */ 5802 bfqq = bfq_do_early_stable_merge(bfqd, bfqq, 5803 bic, 5804 last_bfqq_created); 5805 else { /* schedule tentative stable merge */ 5806 /* 5807 * get reference on last_bfqq_created, 5808 * to prevent it from being freed, 5809 * until we decide whether to merge 5810 */ 5811 last_bfqq_created->ref++; 5812 /* 5813 * need to keep track of stable refs, to 5814 * compute process refs correctly 5815 */ 5816 last_bfqq_created->stable_ref++; 5817 /* 5818 * Record the bfqq to merge to. 5819 */ 5820 bic->bfqq_data[last_bfqq_created->actuator_idx].stable_merge_bfqq = 5821 last_bfqq_created; 5822 } 5823 } 5824 5825 return bfqq; 5826} 5827 5828 5829static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd, 5830 struct bio *bio, bool is_sync, 5831 struct bfq_io_cq *bic, 5832 bool respawn) 5833{ 5834 const int ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio); 5835 const int ioprio_class = IOPRIO_PRIO_CLASS(bic->ioprio); 5836 struct bfq_queue **async_bfqq = NULL; 5837 struct bfq_queue *bfqq; 5838 struct bfq_group *bfqg; 5839 5840 bfqg = bfq_bio_bfqg(bfqd, bio); 5841 if (!is_sync) { 5842 async_bfqq = bfq_async_queue_prio(bfqd, bfqg, ioprio_class, 5843 ioprio, 5844 bfq_actuator_index(bfqd, bio)); 5845 bfqq = *async_bfqq; 5846 if (bfqq) 5847 goto out; 5848 } 5849 5850 bfqq = kmem_cache_alloc_node(bfq_pool, GFP_NOWAIT | __GFP_ZERO, 5851 bfqd->queue->node); 5852 5853 if (bfqq) { 5854 bfq_init_bfqq(bfqd, bfqq, bic, current->pid, 5855 is_sync, bfq_actuator_index(bfqd, bio)); 5856 bfq_init_entity(&bfqq->entity, bfqg); 5857 bfq_log_bfqq(bfqd, bfqq, "allocated"); 5858 } else { 5859 bfqq = &bfqd->oom_bfqq; 5860 bfq_log_bfqq(bfqd, bfqq, "using oom bfqq"); 5861 goto out; 5862 } 5863 5864 /* 5865 * Pin the queue now that it's allocated, scheduler exit will 5866 * prune it. 5867 */ 5868 if (async_bfqq) { 5869 bfqq->ref++; /* 5870 * Extra group reference, w.r.t. sync 5871 * queue. This extra reference is removed 5872 * only if bfqq->bfqg disappears, to 5873 * guarantee that this queue is not freed 5874 * until its group goes away. 5875 */ 5876 bfq_log_bfqq(bfqd, bfqq, "get_queue, bfqq not in async: %p, %d", 5877 bfqq, bfqq->ref); 5878 *async_bfqq = bfqq; 5879 } 5880 5881out: 5882 bfqq->ref++; /* get a process reference to this queue */ 5883 5884 if (bfqq != &bfqd->oom_bfqq && is_sync && !respawn) 5885 bfqq = bfq_do_or_sched_stable_merge(bfqd, bfqq, bic); 5886 return bfqq; 5887} 5888 5889static void bfq_update_io_thinktime(struct bfq_data *bfqd, 5890 struct bfq_queue *bfqq) 5891{ 5892 struct bfq_ttime *ttime = &bfqq->ttime; 5893 u64 elapsed; 5894 5895 /* 5896 * We are really interested in how long it takes for the queue to 5897 * become busy when there is no outstanding IO for this queue. So 5898 * ignore cases when the bfq queue has already IO queued. 5899 */ 5900 if (bfqq->dispatched || bfq_bfqq_busy(bfqq)) 5901 return; 5902 elapsed = blk_time_get_ns() - bfqq->ttime.last_end_request; 5903 elapsed = min_t(u64, elapsed, 2ULL * bfqd->bfq_slice_idle); 5904 5905 ttime->ttime_samples = (7*ttime->ttime_samples + 256) / 8; 5906 ttime->ttime_total = div_u64(7*ttime->ttime_total + 256*elapsed, 8); 5907 ttime->ttime_mean = div64_ul(ttime->ttime_total + 128, 5908 ttime->ttime_samples); 5909} 5910 5911static void 5912bfq_update_io_seektime(struct bfq_data *bfqd, struct bfq_queue *bfqq, 5913 struct request *rq) 5914{ 5915 bfqq->seek_history <<= 1; 5916 bfqq->seek_history |= BFQ_RQ_SEEKY(bfqd, bfqq->last_request_pos, rq); 5917 5918 if (bfqq->wr_coeff > 1 && 5919 bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time && 5920 BFQQ_TOTALLY_SEEKY(bfqq)) { 5921 if (time_is_before_jiffies(bfqq->wr_start_at_switch_to_srt + 5922 bfq_wr_duration(bfqd))) { 5923 /* 5924 * In soft_rt weight raising with the 5925 * interactive-weight-raising period 5926 * elapsed (so no switch back to 5927 * interactive weight raising). 5928 */ 5929 bfq_bfqq_end_wr(bfqq); 5930 } else { /* 5931 * stopping soft_rt weight raising 5932 * while still in interactive period, 5933 * switch back to interactive weight 5934 * raising 5935 */ 5936 switch_back_to_interactive_wr(bfqq, bfqd); 5937 bfqq->entity.prio_changed = 1; 5938 } 5939 } 5940} 5941 5942static void bfq_update_has_short_ttime(struct bfq_data *bfqd, 5943 struct bfq_queue *bfqq, 5944 struct bfq_io_cq *bic) 5945{ 5946 bool has_short_ttime = true, state_changed; 5947 5948 /* 5949 * No need to update has_short_ttime if bfqq is async or in 5950 * idle io prio class, or if bfq_slice_idle is zero, because 5951 * no device idling is performed for bfqq in this case. 5952 */ 5953 if (!bfq_bfqq_sync(bfqq) || bfq_class_idle(bfqq) || 5954 bfqd->bfq_slice_idle == 0) 5955 return; 5956 5957 /* Idle window just restored, statistics are meaningless. */ 5958 if (time_is_after_eq_jiffies(bfqq->split_time + 5959 bfqd->bfq_wr_min_idle_time)) 5960 return; 5961 5962 /* Think time is infinite if no process is linked to 5963 * bfqq. Otherwise check average think time to decide whether 5964 * to mark as has_short_ttime. To this goal, compare average 5965 * think time with half the I/O-plugging timeout. 5966 */ 5967 if (atomic_read(&bic->icq.ioc->active_ref) == 0 || 5968 (bfq_sample_valid(bfqq->ttime.ttime_samples) && 5969 bfqq->ttime.ttime_mean > bfqd->bfq_slice_idle>>1)) 5970 has_short_ttime = false; 5971 5972 state_changed = has_short_ttime != bfq_bfqq_has_short_ttime(bfqq); 5973 5974 if (has_short_ttime) 5975 bfq_mark_bfqq_has_short_ttime(bfqq); 5976 else 5977 bfq_clear_bfqq_has_short_ttime(bfqq); 5978 5979 /* 5980 * Until the base value for the total service time gets 5981 * finally computed for bfqq, the inject limit does depend on 5982 * the think-time state (short|long). In particular, the limit 5983 * is 0 or 1 if the think time is deemed, respectively, as 5984 * short or long (details in the comments in 5985 * bfq_update_inject_limit()). Accordingly, the next 5986 * instructions reset the inject limit if the think-time state 5987 * has changed and the above base value is still to be 5988 * computed. 5989 * 5990 * However, the reset is performed only if more than 100 ms 5991 * have elapsed since the last update of the inject limit, or 5992 * (inclusive) if the change is from short to long think 5993 * time. The reason for this waiting is as follows. 5994 * 5995 * bfqq may have a long think time because of a 5996 * synchronization with some other queue, i.e., because the 5997 * I/O of some other queue may need to be completed for bfqq 5998 * to receive new I/O. Details in the comments on the choice 5999 * of the queue for injection in bfq_select_queue(). 6000 * 6001 * As stressed in those comments, if such a synchronization is 6002 * actually in place, then, without injection on bfqq, the 6003 * blocking I/O cannot happen to served while bfqq is in 6004 * service. As a consequence, if bfqq is granted 6005 * I/O-dispatch-plugging, then bfqq remains empty, and no I/O 6006 * is dispatched, until the idle timeout fires. This is likely 6007 * to result in lower bandwidth and higher latencies for bfqq, 6008 * and in a severe loss of total throughput. 6009 * 6010 * On the opposite end, a non-zero inject limit may allow the 6011 * I/O that blocks bfqq to be executed soon, and therefore 6012 * bfqq to receive new I/O soon. 6013 * 6014 * But, if the blocking gets actually eliminated, then the 6015 * next think-time sample for bfqq may be very low. This in 6016 * turn may cause bfqq's think time to be deemed 6017 * short. Without the 100 ms barrier, this new state change 6018 * would cause the body of the next if to be executed 6019 * immediately. But this would set to 0 the inject 6020 * limit. Without injection, the blocking I/O would cause the 6021 * think time of bfqq to become long again, and therefore the 6022 * inject limit to be raised again, and so on. The only effect 6023 * of such a steady oscillation between the two think-time 6024 * states would be to prevent effective injection on bfqq. 6025 * 6026 * In contrast, if the inject limit is not reset during such a 6027 * long time interval as 100 ms, then the number of short 6028 * think time samples can grow significantly before the reset 6029 * is performed. As a consequence, the think time state can 6030 * become stable before the reset. Therefore there will be no 6031 * state change when the 100 ms elapse, and no reset of the 6032 * inject limit. The inject limit remains steadily equal to 1 6033 * both during and after the 100 ms. So injection can be 6034 * performed at all times, and throughput gets boosted. 6035 * 6036 * An inject limit equal to 1 is however in conflict, in 6037 * general, with the fact that the think time of bfqq is 6038 * short, because injection may be likely to delay bfqq's I/O 6039 * (as explained in the comments in 6040 * bfq_update_inject_limit()). But this does not happen in 6041 * this special case, because bfqq's low think time is due to 6042 * an effective handling of a synchronization, through 6043 * injection. In this special case, bfqq's I/O does not get 6044 * delayed by injection; on the contrary, bfqq's I/O is 6045 * brought forward, because it is not blocked for 6046 * milliseconds. 6047 * 6048 * In addition, serving the blocking I/O much sooner, and much 6049 * more frequently than once per I/O-plugging timeout, makes 6050 * it much quicker to detect a waker queue (the concept of 6051 * waker queue is defined in the comments in 6052 * bfq_add_request()). This makes it possible to start sooner 6053 * to boost throughput more effectively, by injecting the I/O 6054 * of the waker queue unconditionally on every 6055 * bfq_dispatch_request(). 6056 * 6057 * One last, important benefit of not resetting the inject 6058 * limit before 100 ms is that, during this time interval, the 6059 * base value for the total service time is likely to get 6060 * finally computed for bfqq, freeing the inject limit from 6061 * its relation with the think time. 6062 */ 6063 if (state_changed && bfqq->last_serv_time_ns == 0 && 6064 (time_is_before_eq_jiffies(bfqq->decrease_time_jif + 6065 msecs_to_jiffies(100)) || 6066 !has_short_ttime)) 6067 bfq_reset_inject_limit(bfqd, bfqq); 6068} 6069 6070/* 6071 * Called when a new fs request (rq) is added to bfqq. Check if there's 6072 * something we should do about it. 6073 */ 6074static void bfq_rq_enqueued(struct bfq_data *bfqd, struct bfq_queue *bfqq, 6075 struct request *rq) 6076{ 6077 if (rq->cmd_flags & REQ_META) 6078 bfqq->meta_pending++; 6079 6080 bfqq->last_request_pos = blk_rq_pos(rq) + blk_rq_sectors(rq); 6081 6082 if (bfqq == bfqd->in_service_queue && bfq_bfqq_wait_request(bfqq)) { 6083 bool small_req = bfqq->queued[rq_is_sync(rq)] == 1 && 6084 blk_rq_sectors(rq) < 32; 6085 bool budget_timeout = bfq_bfqq_budget_timeout(bfqq); 6086 6087 /* 6088 * There is just this request queued: if 6089 * - the request is small, and 6090 * - we are idling to boost throughput, and 6091 * - the queue is not to be expired, 6092 * then just exit. 6093 * 6094 * In this way, if the device is being idled to wait 6095 * for a new request from the in-service queue, we 6096 * avoid unplugging the device and committing the 6097 * device to serve just a small request. In contrast 6098 * we wait for the block layer to decide when to 6099 * unplug the device: hopefully, new requests will be 6100 * merged to this one quickly, then the device will be 6101 * unplugged and larger requests will be dispatched. 6102 */ 6103 if (small_req && idling_boosts_thr_without_issues(bfqd, bfqq) && 6104 !budget_timeout) 6105 return; 6106 6107 /* 6108 * A large enough request arrived, or idling is being 6109 * performed to preserve service guarantees, or 6110 * finally the queue is to be expired: in all these 6111 * cases disk idling is to be stopped, so clear 6112 * wait_request flag and reset timer. 6113 */ 6114 bfq_clear_bfqq_wait_request(bfqq); 6115 hrtimer_try_to_cancel(&bfqd->idle_slice_timer); 6116 6117 /* 6118 * The queue is not empty, because a new request just 6119 * arrived. Hence we can safely expire the queue, in 6120 * case of budget timeout, without risking that the 6121 * timestamps of the queue are not updated correctly. 6122 * See [1] for more details. 6123 */ 6124 if (budget_timeout) 6125 bfq_bfqq_expire(bfqd, bfqq, false, 6126 BFQQE_BUDGET_TIMEOUT); 6127 } 6128} 6129 6130static void bfqq_request_allocated(struct bfq_queue *bfqq) 6131{ 6132 struct bfq_entity *entity = &bfqq->entity; 6133 6134 for_each_entity(entity) 6135 entity->allocated++; 6136} 6137 6138static void bfqq_request_freed(struct bfq_queue *bfqq) 6139{ 6140 struct bfq_entity *entity = &bfqq->entity; 6141 6142 for_each_entity(entity) 6143 entity->allocated--; 6144} 6145 6146/* returns true if it causes the idle timer to be disabled */ 6147static bool __bfq_insert_request(struct bfq_data *bfqd, struct request *rq) 6148{ 6149 struct bfq_queue *bfqq = RQ_BFQQ(rq), 6150 *new_bfqq = bfq_setup_cooperator(bfqd, bfqq, rq, true, 6151 RQ_BIC(rq)); 6152 bool waiting, idle_timer_disabled = false; 6153 6154 if (new_bfqq) { 6155 struct bfq_queue *old_bfqq = bfqq; 6156 /* 6157 * Release the request's reference to the old bfqq 6158 * and make sure one is taken to the shared queue. 6159 */ 6160 bfqq_request_allocated(new_bfqq); 6161 bfqq_request_freed(bfqq); 6162 new_bfqq->ref++; 6163 /* 6164 * If the bic associated with the process 6165 * issuing this request still points to bfqq 6166 * (and thus has not been already redirected 6167 * to new_bfqq or even some other bfq_queue), 6168 * then complete the merge and redirect it to 6169 * new_bfqq. 6170 */ 6171 if (bic_to_bfqq(RQ_BIC(rq), true, 6172 bfq_actuator_index(bfqd, rq->bio)) == bfqq) { 6173 while (bfqq != new_bfqq) 6174 bfqq = bfq_merge_bfqqs(bfqd, RQ_BIC(rq), bfqq); 6175 } 6176 6177 bfq_clear_bfqq_just_created(old_bfqq); 6178 /* 6179 * rq is about to be enqueued into new_bfqq, 6180 * release rq reference on bfqq 6181 */ 6182 bfq_put_queue(old_bfqq); 6183 rq->elv.priv[1] = new_bfqq; 6184 } 6185 6186 bfq_update_io_thinktime(bfqd, bfqq); 6187 bfq_update_has_short_ttime(bfqd, bfqq, RQ_BIC(rq)); 6188 bfq_update_io_seektime(bfqd, bfqq, rq); 6189 6190 waiting = bfqq && bfq_bfqq_wait_request(bfqq); 6191 bfq_add_request(rq); 6192 idle_timer_disabled = waiting && !bfq_bfqq_wait_request(bfqq); 6193 6194 rq->fifo_time = blk_time_get_ns() + bfqd->bfq_fifo_expire[rq_is_sync(rq)]; 6195 list_add_tail(&rq->queuelist, &bfqq->fifo); 6196 6197 bfq_rq_enqueued(bfqd, bfqq, rq); 6198 6199 return idle_timer_disabled; 6200} 6201 6202#ifdef CONFIG_BFQ_CGROUP_DEBUG 6203static void bfq_update_insert_stats(struct request_queue *q, 6204 struct bfq_queue *bfqq, 6205 bool idle_timer_disabled, 6206 blk_opf_t cmd_flags) 6207{ 6208 if (!bfqq) 6209 return; 6210 6211 /* 6212 * bfqq still exists, because it can disappear only after 6213 * either it is merged with another queue, or the process it 6214 * is associated with exits. But both actions must be taken by 6215 * the same process currently executing this flow of 6216 * instructions. 6217 * 6218 * In addition, the following queue lock guarantees that 6219 * bfqq_group(bfqq) exists as well. 6220 */ 6221 spin_lock_irq(&q->queue_lock); 6222 bfqg_stats_update_io_add(bfqq_group(bfqq), bfqq, cmd_flags); 6223 if (idle_timer_disabled) 6224 bfqg_stats_update_idle_time(bfqq_group(bfqq)); 6225 spin_unlock_irq(&q->queue_lock); 6226} 6227#else 6228static inline void bfq_update_insert_stats(struct request_queue *q, 6229 struct bfq_queue *bfqq, 6230 bool idle_timer_disabled, 6231 blk_opf_t cmd_flags) {} 6232#endif /* CONFIG_BFQ_CGROUP_DEBUG */ 6233 6234static struct bfq_queue *bfq_init_rq(struct request *rq); 6235 6236static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, 6237 blk_insert_t flags) 6238{ 6239 struct request_queue *q = hctx->queue; 6240 struct bfq_data *bfqd = q->elevator->elevator_data; 6241 struct bfq_queue *bfqq; 6242 bool idle_timer_disabled = false; 6243 blk_opf_t cmd_flags; 6244 LIST_HEAD(free); 6245 6246#ifdef CONFIG_BFQ_GROUP_IOSCHED 6247 if (!cgroup_subsys_on_dfl(io_cgrp_subsys) && rq->bio) 6248 bfqg_stats_update_legacy_io(q, rq); 6249#endif 6250 spin_lock_irq(&bfqd->lock); 6251 bfqq = bfq_init_rq(rq); 6252 if (blk_mq_sched_try_insert_merge(q, rq, &free)) { 6253 spin_unlock_irq(&bfqd->lock); 6254 blk_mq_free_requests(&free); 6255 return; 6256 } 6257 6258 trace_block_rq_insert(rq); 6259 6260 if (flags & BLK_MQ_INSERT_AT_HEAD) { 6261 list_add(&rq->queuelist, &bfqd->dispatch); 6262 } else if (!bfqq) { 6263 list_add_tail(&rq->queuelist, &bfqd->dispatch); 6264 } else { 6265 idle_timer_disabled = __bfq_insert_request(bfqd, rq); 6266 /* 6267 * Update bfqq, because, if a queue merge has occurred 6268 * in __bfq_insert_request, then rq has been 6269 * redirected into a new queue. 6270 */ 6271 bfqq = RQ_BFQQ(rq); 6272 6273 if (rq_mergeable(rq)) { 6274 elv_rqhash_add(q, rq); 6275 if (!q->last_merge) 6276 q->last_merge = rq; 6277 } 6278 } 6279 6280 /* 6281 * Cache cmd_flags before releasing scheduler lock, because rq 6282 * may disappear afterwards (for example, because of a request 6283 * merge). 6284 */ 6285 cmd_flags = rq->cmd_flags; 6286 spin_unlock_irq(&bfqd->lock); 6287 6288 bfq_update_insert_stats(q, bfqq, idle_timer_disabled, 6289 cmd_flags); 6290} 6291 6292static void bfq_insert_requests(struct blk_mq_hw_ctx *hctx, 6293 struct list_head *list, 6294 blk_insert_t flags) 6295{ 6296 while (!list_empty(list)) { 6297 struct request *rq; 6298 6299 rq = list_first_entry(list, struct request, queuelist); 6300 list_del_init(&rq->queuelist); 6301 bfq_insert_request(hctx, rq, flags); 6302 } 6303} 6304 6305static void bfq_update_hw_tag(struct bfq_data *bfqd) 6306{ 6307 struct bfq_queue *bfqq = bfqd->in_service_queue; 6308 6309 bfqd->max_rq_in_driver = max_t(int, bfqd->max_rq_in_driver, 6310 bfqd->tot_rq_in_driver); 6311 6312 if (bfqd->hw_tag == 1) 6313 return; 6314 6315 /* 6316 * This sample is valid if the number of outstanding requests 6317 * is large enough to allow a queueing behavior. Note that the 6318 * sum is not exact, as it's not taking into account deactivated 6319 * requests. 6320 */ 6321 if (bfqd->tot_rq_in_driver + bfqd->queued <= BFQ_HW_QUEUE_THRESHOLD) 6322 return; 6323 6324 /* 6325 * If active queue hasn't enough requests and can idle, bfq might not 6326 * dispatch sufficient requests to hardware. Don't zero hw_tag in this 6327 * case 6328 */ 6329 if (bfqq && bfq_bfqq_has_short_ttime(bfqq) && 6330 bfqq->dispatched + bfqq->queued[0] + bfqq->queued[1] < 6331 BFQ_HW_QUEUE_THRESHOLD && 6332 bfqd->tot_rq_in_driver < BFQ_HW_QUEUE_THRESHOLD) 6333 return; 6334 6335 if (bfqd->hw_tag_samples++ < BFQ_HW_QUEUE_SAMPLES) 6336 return; 6337 6338 bfqd->hw_tag = bfqd->max_rq_in_driver > BFQ_HW_QUEUE_THRESHOLD; 6339 bfqd->max_rq_in_driver = 0; 6340 bfqd->hw_tag_samples = 0; 6341 6342 bfqd->nonrot_with_queueing = 6343 blk_queue_nonrot(bfqd->queue) && bfqd->hw_tag; 6344} 6345 6346static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) 6347{ 6348 u64 now_ns; 6349 u32 delta_us; 6350 6351 bfq_update_hw_tag(bfqd); 6352 6353 bfqd->rq_in_driver[bfqq->actuator_idx]--; 6354 bfqd->tot_rq_in_driver--; 6355 bfqq->dispatched--; 6356 6357 if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) { 6358 /* 6359 * Set budget_timeout (which we overload to store the 6360 * time at which the queue remains with no backlog and 6361 * no outstanding request; used by the weight-raising 6362 * mechanism). 6363 */ 6364 bfqq->budget_timeout = jiffies; 6365 6366 bfq_del_bfqq_in_groups_with_pending_reqs(bfqq); 6367 bfq_weights_tree_remove(bfqq); 6368 } 6369 6370 now_ns = blk_time_get_ns(); 6371 6372 bfqq->ttime.last_end_request = now_ns; 6373 6374 /* 6375 * Using us instead of ns, to get a reasonable precision in 6376 * computing rate in next check. 6377 */ 6378 delta_us = div_u64(now_ns - bfqd->last_completion, NSEC_PER_USEC); 6379 6380 /* 6381 * If the request took rather long to complete, and, according 6382 * to the maximum request size recorded, this completion latency 6383 * implies that the request was certainly served at a very low 6384 * rate (less than 1M sectors/sec), then the whole observation 6385 * interval that lasts up to this time instant cannot be a 6386 * valid time interval for computing a new peak rate. Invoke 6387 * bfq_update_rate_reset to have the following three steps 6388 * taken: 6389 * - close the observation interval at the last (previous) 6390 * request dispatch or completion 6391 * - compute rate, if possible, for that observation interval 6392 * - reset to zero samples, which will trigger a proper 6393 * re-initialization of the observation interval on next 6394 * dispatch 6395 */ 6396 if (delta_us > BFQ_MIN_TT/NSEC_PER_USEC && 6397 (bfqd->last_rq_max_size<<BFQ_RATE_SHIFT)/delta_us < 6398 1UL<<(BFQ_RATE_SHIFT - 10)) 6399 bfq_update_rate_reset(bfqd, NULL); 6400 bfqd->last_completion = now_ns; 6401 /* 6402 * Shared queues are likely to receive I/O at a high 6403 * rate. This may deceptively let them be considered as wakers 6404 * of other queues. But a false waker will unjustly steal 6405 * bandwidth to its supposedly woken queue. So considering 6406 * also shared queues in the waking mechanism may cause more 6407 * control troubles than throughput benefits. Then reset 6408 * last_completed_rq_bfqq if bfqq is a shared queue. 6409 */ 6410 if (!bfq_bfqq_coop(bfqq)) 6411 bfqd->last_completed_rq_bfqq = bfqq; 6412 else 6413 bfqd->last_completed_rq_bfqq = NULL; 6414 6415 /* 6416 * If we are waiting to discover whether the request pattern 6417 * of the task associated with the queue is actually 6418 * isochronous, and both requisites for this condition to hold 6419 * are now satisfied, then compute soft_rt_next_start (see the 6420 * comments on the function bfq_bfqq_softrt_next_start()). We 6421 * do not compute soft_rt_next_start if bfqq is in interactive 6422 * weight raising (see the comments in bfq_bfqq_expire() for 6423 * an explanation). We schedule this delayed update when bfqq 6424 * expires, if it still has in-flight requests. 6425 */ 6426 if (bfq_bfqq_softrt_update(bfqq) && bfqq->dispatched == 0 && 6427 RB_EMPTY_ROOT(&bfqq->sort_list) && 6428 bfqq->wr_coeff != bfqd->bfq_wr_coeff) 6429 bfqq->soft_rt_next_start = 6430 bfq_bfqq_softrt_next_start(bfqd, bfqq); 6431 6432 /* 6433 * If this is the in-service queue, check if it needs to be expired, 6434 * or if we want to idle in case it has no pending requests. 6435 */ 6436 if (bfqd->in_service_queue == bfqq) { 6437 if (bfq_bfqq_must_idle(bfqq)) { 6438 if (bfqq->dispatched == 0) 6439 bfq_arm_slice_timer(bfqd); 6440 /* 6441 * If we get here, we do not expire bfqq, even 6442 * if bfqq was in budget timeout or had no 6443 * more requests (as controlled in the next 6444 * conditional instructions). The reason for 6445 * not expiring bfqq is as follows. 6446 * 6447 * Here bfqq->dispatched > 0 holds, but 6448 * bfq_bfqq_must_idle() returned true. This 6449 * implies that, even if no request arrives 6450 * for bfqq before bfqq->dispatched reaches 0, 6451 * bfqq will, however, not be expired on the 6452 * completion event that causes bfqq->dispatch 6453 * to reach zero. In contrast, on this event, 6454 * bfqq will start enjoying device idling 6455 * (I/O-dispatch plugging). 6456 * 6457 * But, if we expired bfqq here, bfqq would 6458 * not have the chance to enjoy device idling 6459 * when bfqq->dispatched finally reaches 6460 * zero. This would expose bfqq to violation 6461 * of its reserved service guarantees. 6462 */ 6463 return; 6464 } else if (bfq_may_expire_for_budg_timeout(bfqq)) 6465 bfq_bfqq_expire(bfqd, bfqq, false, 6466 BFQQE_BUDGET_TIMEOUT); 6467 else if (RB_EMPTY_ROOT(&bfqq->sort_list) && 6468 (bfqq->dispatched == 0 || 6469 !bfq_better_to_idle(bfqq))) 6470 bfq_bfqq_expire(bfqd, bfqq, false, 6471 BFQQE_NO_MORE_REQUESTS); 6472 } 6473 6474 if (!bfqd->tot_rq_in_driver) 6475 bfq_schedule_dispatch(bfqd); 6476} 6477 6478/* 6479 * The processes associated with bfqq may happen to generate their 6480 * cumulative I/O at a lower rate than the rate at which the device 6481 * could serve the same I/O. This is rather probable, e.g., if only 6482 * one process is associated with bfqq and the device is an SSD. It 6483 * results in bfqq becoming often empty while in service. In this 6484 * respect, if BFQ is allowed to switch to another queue when bfqq 6485 * remains empty, then the device goes on being fed with I/O requests, 6486 * and the throughput is not affected. In contrast, if BFQ is not 6487 * allowed to switch to another queue---because bfqq is sync and 6488 * I/O-dispatch needs to be plugged while bfqq is temporarily 6489 * empty---then, during the service of bfqq, there will be frequent 6490 * "service holes", i.e., time intervals during which bfqq gets empty 6491 * and the device can only consume the I/O already queued in its 6492 * hardware queues. During service holes, the device may even get to 6493 * remaining idle. In the end, during the service of bfqq, the device 6494 * is driven at a lower speed than the one it can reach with the kind 6495 * of I/O flowing through bfqq. 6496 * 6497 * To counter this loss of throughput, BFQ implements a "request 6498 * injection mechanism", which tries to fill the above service holes 6499 * with I/O requests taken from other queues. The hard part in this 6500 * mechanism is finding the right amount of I/O to inject, so as to 6501 * both boost throughput and not break bfqq's bandwidth and latency 6502 * guarantees. In this respect, the mechanism maintains a per-queue 6503 * inject limit, computed as below. While bfqq is empty, the injection 6504 * mechanism dispatches extra I/O requests only until the total number 6505 * of I/O requests in flight---i.e., already dispatched but not yet 6506 * completed---remains lower than this limit. 6507 * 6508 * A first definition comes in handy to introduce the algorithm by 6509 * which the inject limit is computed. We define as first request for 6510 * bfqq, an I/O request for bfqq that arrives while bfqq is in 6511 * service, and causes bfqq to switch from empty to non-empty. The 6512 * algorithm updates the limit as a function of the effect of 6513 * injection on the service times of only the first requests of 6514 * bfqq. The reason for this restriction is that these are the 6515 * requests whose service time is affected most, because they are the 6516 * first to arrive after injection possibly occurred. 6517 * 6518 * To evaluate the effect of injection, the algorithm measures the 6519 * "total service time" of first requests. We define as total service 6520 * time of an I/O request, the time that elapses since when the 6521 * request is enqueued into bfqq, to when it is completed. This 6522 * quantity allows the whole effect of injection to be measured. It is 6523 * easy to see why. Suppose that some requests of other queues are 6524 * actually injected while bfqq is empty, and that a new request R 6525 * then arrives for bfqq. If the device does start to serve all or 6526 * part of the injected requests during the service hole, then, 6527 * because of this extra service, it may delay the next invocation of 6528 * the dispatch hook of BFQ. Then, even after R gets eventually 6529 * dispatched, the device may delay the actual service of R if it is 6530 * still busy serving the extra requests, or if it decides to serve, 6531 * before R, some extra request still present in its queues. As a 6532 * conclusion, the cumulative extra delay caused by injection can be 6533 * easily evaluated by just comparing the total service time of first 6534 * requests with and without injection. 6535 * 6536 * The limit-update algorithm works as follows. On the arrival of a 6537 * first request of bfqq, the algorithm measures the total time of the 6538 * request only if one of the three cases below holds, and, for each 6539 * case, it updates the limit as described below: 6540 * 6541 * (1) If there is no in-flight request. This gives a baseline for the 6542 * total service time of the requests of bfqq. If the baseline has 6543 * not been computed yet, then, after computing it, the limit is 6544 * set to 1, to start boosting throughput, and to prepare the 6545 * ground for the next case. If the baseline has already been 6546 * computed, then it is updated, in case it results to be lower 6547 * than the previous value. 6548 * 6549 * (2) If the limit is higher than 0 and there are in-flight 6550 * requests. By comparing the total service time in this case with 6551 * the above baseline, it is possible to know at which extent the 6552 * current value of the limit is inflating the total service 6553 * time. If the inflation is below a certain threshold, then bfqq 6554 * is assumed to be suffering from no perceivable loss of its 6555 * service guarantees, and the limit is even tentatively 6556 * increased. If the inflation is above the threshold, then the 6557 * limit is decreased. Due to the lack of any hysteresis, this 6558 * logic makes the limit oscillate even in steady workload 6559 * conditions. Yet we opted for it, because it is fast in reaching 6560 * the best value for the limit, as a function of the current I/O 6561 * workload. To reduce oscillations, this step is disabled for a 6562 * short time interval after the limit happens to be decreased. 6563 * 6564 * (3) Periodically, after resetting the limit, to make sure that the 6565 * limit eventually drops in case the workload changes. This is 6566 * needed because, after the limit has gone safely up for a 6567 * certain workload, it is impossible to guess whether the 6568 * baseline total service time may have changed, without measuring 6569 * it again without injection. A more effective version of this 6570 * step might be to just sample the baseline, by interrupting 6571 * injection only once, and then to reset/lower the limit only if 6572 * the total service time with the current limit does happen to be 6573 * too large. 6574 * 6575 * More details on each step are provided in the comments on the 6576 * pieces of code that implement these steps: the branch handling the 6577 * transition from empty to non empty in bfq_add_request(), the branch 6578 * handling injection in bfq_select_queue(), and the function 6579 * bfq_choose_bfqq_for_injection(). These comments also explain some 6580 * exceptions, made by the injection mechanism in some special cases. 6581 */ 6582static void bfq_update_inject_limit(struct bfq_data *bfqd, 6583 struct bfq_queue *bfqq) 6584{ 6585 u64 tot_time_ns = blk_time_get_ns() - bfqd->last_empty_occupied_ns; 6586 unsigned int old_limit = bfqq->inject_limit; 6587 6588 if (bfqq->last_serv_time_ns > 0 && bfqd->rqs_injected) { 6589 u64 threshold = (bfqq->last_serv_time_ns * 3)>>1; 6590 6591 if (tot_time_ns >= threshold && old_limit > 0) { 6592 bfqq->inject_limit--; 6593 bfqq->decrease_time_jif = jiffies; 6594 } else if (tot_time_ns < threshold && 6595 old_limit <= bfqd->max_rq_in_driver) 6596 bfqq->inject_limit++; 6597 } 6598 6599 /* 6600 * Either we still have to compute the base value for the 6601 * total service time, and there seem to be the right 6602 * conditions to do it, or we can lower the last base value 6603 * computed. 6604 * 6605 * NOTE: (bfqd->tot_rq_in_driver == 1) means that there is no I/O 6606 * request in flight, because this function is in the code 6607 * path that handles the completion of a request of bfqq, and, 6608 * in particular, this function is executed before 6609 * bfqd->tot_rq_in_driver is decremented in such a code path. 6610 */ 6611 if ((bfqq->last_serv_time_ns == 0 && bfqd->tot_rq_in_driver == 1) || 6612 tot_time_ns < bfqq->last_serv_time_ns) { 6613 if (bfqq->last_serv_time_ns == 0) { 6614 /* 6615 * Now we certainly have a base value: make sure we 6616 * start trying injection. 6617 */ 6618 bfqq->inject_limit = max_t(unsigned int, 1, old_limit); 6619 } 6620 bfqq->last_serv_time_ns = tot_time_ns; 6621 } else if (!bfqd->rqs_injected && bfqd->tot_rq_in_driver == 1) 6622 /* 6623 * No I/O injected and no request still in service in 6624 * the drive: these are the exact conditions for 6625 * computing the base value of the total service time 6626 * for bfqq. So let's update this value, because it is 6627 * rather variable. For example, it varies if the size 6628 * or the spatial locality of the I/O requests in bfqq 6629 * change. 6630 */ 6631 bfqq->last_serv_time_ns = tot_time_ns; 6632 6633 6634 /* update complete, not waiting for any request completion any longer */ 6635 bfqd->waited_rq = NULL; 6636 bfqd->rqs_injected = false; 6637} 6638 6639/* 6640 * Handle either a requeue or a finish for rq. The things to do are 6641 * the same in both cases: all references to rq are to be dropped. In 6642 * particular, rq is considered completed from the point of view of 6643 * the scheduler. 6644 */ 6645static void bfq_finish_requeue_request(struct request *rq) 6646{ 6647 struct bfq_queue *bfqq = RQ_BFQQ(rq); 6648 struct bfq_data *bfqd; 6649 unsigned long flags; 6650 6651 /* 6652 * rq either is not associated with any icq, or is an already 6653 * requeued request that has not (yet) been re-inserted into 6654 * a bfq_queue. 6655 */ 6656 if (!rq->elv.icq || !bfqq) 6657 return; 6658 6659 bfqd = bfqq->bfqd; 6660 6661 if (rq->rq_flags & RQF_STARTED) 6662 bfqg_stats_update_completion(bfqq_group(bfqq), 6663 rq->start_time_ns, 6664 rq->io_start_time_ns, 6665 rq->cmd_flags); 6666 6667 spin_lock_irqsave(&bfqd->lock, flags); 6668 if (likely(rq->rq_flags & RQF_STARTED)) { 6669 if (rq == bfqd->waited_rq) 6670 bfq_update_inject_limit(bfqd, bfqq); 6671 6672 bfq_completed_request(bfqq, bfqd); 6673 } 6674 bfqq_request_freed(bfqq); 6675 bfq_put_queue(bfqq); 6676 RQ_BIC(rq)->requests--; 6677 spin_unlock_irqrestore(&bfqd->lock, flags); 6678 6679 /* 6680 * Reset private fields. In case of a requeue, this allows 6681 * this function to correctly do nothing if it is spuriously 6682 * invoked again on this same request (see the check at the 6683 * beginning of the function). Probably, a better general 6684 * design would be to prevent blk-mq from invoking the requeue 6685 * or finish hooks of an elevator, for a request that is not 6686 * referred by that elevator. 6687 * 6688 * Resetting the following fields would break the 6689 * request-insertion logic if rq is re-inserted into a bfq 6690 * internal queue, without a re-preparation. Here we assume 6691 * that re-insertions of requeued requests, without 6692 * re-preparation, can happen only for pass_through or at_head 6693 * requests (which are not re-inserted into bfq internal 6694 * queues). 6695 */ 6696 rq->elv.priv[0] = NULL; 6697 rq->elv.priv[1] = NULL; 6698} 6699 6700static void bfq_finish_request(struct request *rq) 6701{ 6702 bfq_finish_requeue_request(rq); 6703 6704 if (rq->elv.icq) { 6705 put_io_context(rq->elv.icq->ioc); 6706 rq->elv.icq = NULL; 6707 } 6708} 6709 6710/* 6711 * Removes the association between the current task and bfqq, assuming 6712 * that bic points to the bfq iocontext of the task. 6713 * Returns NULL if a new bfqq should be allocated, or the old bfqq if this 6714 * was the last process referring to that bfqq. 6715 */ 6716static struct bfq_queue * 6717bfq_split_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq) 6718{ 6719 bfq_log_bfqq(bfqq->bfqd, bfqq, "splitting queue"); 6720 6721 if (bfqq_process_refs(bfqq) == 1 && !bfqq->new_bfqq) { 6722 bfqq->pid = current->pid; 6723 bfq_clear_bfqq_coop(bfqq); 6724 bfq_clear_bfqq_split_coop(bfqq); 6725 return bfqq; 6726 } 6727 6728 bic_set_bfqq(bic, NULL, true, bfqq->actuator_idx); 6729 6730 bfq_put_cooperator(bfqq); 6731 6732 bfq_release_process_ref(bfqq->bfqd, bfqq); 6733 return NULL; 6734} 6735 6736static struct bfq_queue * 6737__bfq_get_bfqq_handle_split(struct bfq_data *bfqd, struct bfq_io_cq *bic, 6738 struct bio *bio, bool split, bool is_sync, 6739 bool *new_queue) 6740{ 6741 unsigned int act_idx = bfq_actuator_index(bfqd, bio); 6742 struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync, act_idx); 6743 struct bfq_iocq_bfqq_data *bfqq_data = &bic->bfqq_data[act_idx]; 6744 6745 if (likely(bfqq && bfqq != &bfqd->oom_bfqq)) 6746 return bfqq; 6747 6748 if (new_queue) 6749 *new_queue = true; 6750 6751 if (bfqq) 6752 bfq_put_queue(bfqq); 6753 bfqq = bfq_get_queue(bfqd, bio, is_sync, bic, split); 6754 6755 bic_set_bfqq(bic, bfqq, is_sync, act_idx); 6756 if (split && is_sync) { 6757 if ((bfqq_data->was_in_burst_list && bfqd->large_burst) || 6758 bfqq_data->saved_in_large_burst) 6759 bfq_mark_bfqq_in_large_burst(bfqq); 6760 else { 6761 bfq_clear_bfqq_in_large_burst(bfqq); 6762 if (bfqq_data->was_in_burst_list) 6763 /* 6764 * If bfqq was in the current 6765 * burst list before being 6766 * merged, then we have to add 6767 * it back. And we do not need 6768 * to increase burst_size, as 6769 * we did not decrement 6770 * burst_size when we removed 6771 * bfqq from the burst list as 6772 * a consequence of a merge 6773 * (see comments in 6774 * bfq_put_queue). In this 6775 * respect, it would be rather 6776 * costly to know whether the 6777 * current burst list is still 6778 * the same burst list from 6779 * which bfqq was removed on 6780 * the merge. To avoid this 6781 * cost, if bfqq was in a 6782 * burst list, then we add 6783 * bfqq to the current burst 6784 * list without any further 6785 * check. This can cause 6786 * inappropriate insertions, 6787 * but rarely enough to not 6788 * harm the detection of large 6789 * bursts significantly. 6790 */ 6791 hlist_add_head(&bfqq->burst_list_node, 6792 &bfqd->burst_list); 6793 } 6794 bfqq->split_time = jiffies; 6795 } 6796 6797 return bfqq; 6798} 6799 6800/* 6801 * Only reset private fields. The actual request preparation will be 6802 * performed by bfq_init_rq, when rq is either inserted or merged. See 6803 * comments on bfq_init_rq for the reason behind this delayed 6804 * preparation. 6805 */ 6806static void bfq_prepare_request(struct request *rq) 6807{ 6808 rq->elv.icq = ioc_find_get_icq(rq->q); 6809 6810 /* 6811 * Regardless of whether we have an icq attached, we have to 6812 * clear the scheduler pointers, as they might point to 6813 * previously allocated bic/bfqq structs. 6814 */ 6815 rq->elv.priv[0] = rq->elv.priv[1] = NULL; 6816} 6817 6818static struct bfq_queue *bfq_waker_bfqq(struct bfq_queue *bfqq) 6819{ 6820 struct bfq_queue *new_bfqq = bfqq->new_bfqq; 6821 struct bfq_queue *waker_bfqq = bfqq->waker_bfqq; 6822 6823 if (!waker_bfqq) 6824 return NULL; 6825 6826 while (new_bfqq) { 6827 if (new_bfqq == waker_bfqq) { 6828 /* 6829 * If waker_bfqq is in the merge chain, and current 6830 * is the only process, waker_bfqq can be freed. 6831 */ 6832 if (bfqq_process_refs(waker_bfqq) == 1) 6833 return NULL; 6834 6835 return waker_bfqq; 6836 } 6837 6838 new_bfqq = new_bfqq->new_bfqq; 6839 } 6840 6841 /* 6842 * If waker_bfqq is not in the merge chain, and it's procress reference 6843 * is 0, waker_bfqq can be freed. 6844 */ 6845 if (bfqq_process_refs(waker_bfqq) == 0) 6846 return NULL; 6847 6848 return waker_bfqq; 6849} 6850 6851static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd, 6852 struct bfq_io_cq *bic, 6853 struct bio *bio, 6854 unsigned int idx, 6855 bool is_sync) 6856{ 6857 struct bfq_queue *waker_bfqq; 6858 struct bfq_queue *bfqq; 6859 bool new_queue = false; 6860 6861 bfqq = __bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync, 6862 &new_queue); 6863 if (unlikely(new_queue)) 6864 return bfqq; 6865 6866 /* If the queue was seeky for too long, break it apart. */ 6867 if (!bfq_bfqq_coop(bfqq) || !bfq_bfqq_split_coop(bfqq) || 6868 bic->bfqq_data[idx].stably_merged) 6869 return bfqq; 6870 6871 waker_bfqq = bfq_waker_bfqq(bfqq); 6872 6873 /* Update bic before losing reference to bfqq */ 6874 if (bfq_bfqq_in_large_burst(bfqq)) 6875 bic->bfqq_data[idx].saved_in_large_burst = true; 6876 6877 bfqq = bfq_split_bfqq(bic, bfqq); 6878 if (bfqq) { 6879 bfq_bfqq_resume_state(bfqq, bfqd, bic, true); 6880 return bfqq; 6881 } 6882 6883 bfqq = __bfq_get_bfqq_handle_split(bfqd, bic, bio, true, is_sync, NULL); 6884 if (unlikely(bfqq == &bfqd->oom_bfqq)) 6885 return bfqq; 6886 6887 bfq_bfqq_resume_state(bfqq, bfqd, bic, false); 6888 bfqq->waker_bfqq = waker_bfqq; 6889 bfqq->tentative_waker_bfqq = NULL; 6890 6891 /* 6892 * If the waker queue disappears, then new_bfqq->waker_bfqq must be 6893 * reset. So insert new_bfqq into the 6894 * woken_list of the waker. See 6895 * bfq_check_waker for details. 6896 */ 6897 if (waker_bfqq) 6898 hlist_add_head(&bfqq->woken_list_node, 6899 &bfqq->waker_bfqq->woken_list); 6900 6901 return bfqq; 6902} 6903 6904/* 6905 * If needed, init rq, allocate bfq data structures associated with 6906 * rq, and increment reference counters in the destination bfq_queue 6907 * for rq. Return the destination bfq_queue for rq, or NULL is rq is 6908 * not associated with any bfq_queue. 6909 * 6910 * This function is invoked by the functions that perform rq insertion 6911 * or merging. One may have expected the above preparation operations 6912 * to be performed in bfq_prepare_request, and not delayed to when rq 6913 * is inserted or merged. The rationale behind this delayed 6914 * preparation is that, after the prepare_request hook is invoked for 6915 * rq, rq may still be transformed into a request with no icq, i.e., a 6916 * request not associated with any queue. No bfq hook is invoked to 6917 * signal this transformation. As a consequence, should these 6918 * preparation operations be performed when the prepare_request hook 6919 * is invoked, and should rq be transformed one moment later, bfq 6920 * would end up in an inconsistent state, because it would have 6921 * incremented some queue counters for an rq destined to 6922 * transformation, without any chance to correctly lower these 6923 * counters back. In contrast, no transformation can still happen for 6924 * rq after rq has been inserted or merged. So, it is safe to execute 6925 * these preparation operations when rq is finally inserted or merged. 6926 */ 6927static struct bfq_queue *bfq_init_rq(struct request *rq) 6928{ 6929 struct request_queue *q = rq->q; 6930 struct bio *bio = rq->bio; 6931 struct bfq_data *bfqd = q->elevator->elevator_data; 6932 struct bfq_io_cq *bic; 6933 const int is_sync = rq_is_sync(rq); 6934 struct bfq_queue *bfqq; 6935 unsigned int a_idx = bfq_actuator_index(bfqd, bio); 6936 6937 if (unlikely(!rq->elv.icq)) 6938 return NULL; 6939 6940 /* 6941 * Assuming that RQ_BFQQ(rq) is set only if everything is set 6942 * for this rq. This holds true, because this function is 6943 * invoked only for insertion or merging, and, after such 6944 * events, a request cannot be manipulated any longer before 6945 * being removed from bfq. 6946 */ 6947 if (RQ_BFQQ(rq)) 6948 return RQ_BFQQ(rq); 6949 6950 bic = icq_to_bic(rq->elv.icq); 6951 bfq_check_ioprio_change(bic, bio); 6952 bfq_bic_update_cgroup(bic, bio); 6953 bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, a_idx, is_sync); 6954 6955 bfqq_request_allocated(bfqq); 6956 bfqq->ref++; 6957 bic->requests++; 6958 bfq_log_bfqq(bfqd, bfqq, "get_request %p: bfqq %p, %d", 6959 rq, bfqq, bfqq->ref); 6960 6961 rq->elv.priv[0] = bic; 6962 rq->elv.priv[1] = bfqq; 6963 6964 /* 6965 * If a bfq_queue has only one process reference, it is owned 6966 * by only this bic: we can then set bfqq->bic = bic. in 6967 * addition, if the queue has also just been split, we have to 6968 * resume its state. 6969 */ 6970 if (likely(bfqq != &bfqd->oom_bfqq) && !bfqq->new_bfqq && 6971 bfqq_process_refs(bfqq) == 1) 6972 bfqq->bic = bic; 6973 6974 /* 6975 * Consider bfqq as possibly belonging to a burst of newly 6976 * created queues only if: 6977 * 1) A burst is actually happening (bfqd->burst_size > 0) 6978 * or 6979 * 2) There is no other active queue. In fact, if, in 6980 * contrast, there are active queues not belonging to the 6981 * possible burst bfqq may belong to, then there is no gain 6982 * in considering bfqq as belonging to a burst, and 6983 * therefore in not weight-raising bfqq. See comments on 6984 * bfq_handle_burst(). 6985 * 6986 * This filtering also helps eliminating false positives, 6987 * occurring when bfqq does not belong to an actual large 6988 * burst, but some background task (e.g., a service) happens 6989 * to trigger the creation of new queues very close to when 6990 * bfqq and its possible companion queues are created. See 6991 * comments on bfq_handle_burst() for further details also on 6992 * this issue. 6993 */ 6994 if (unlikely(bfq_bfqq_just_created(bfqq) && 6995 (bfqd->burst_size > 0 || 6996 bfq_tot_busy_queues(bfqd) == 0))) 6997 bfq_handle_burst(bfqd, bfqq); 6998 6999 return bfqq; 7000} 7001 7002static void 7003bfq_idle_slice_timer_body(struct bfq_data *bfqd, struct bfq_queue *bfqq) 7004{ 7005 enum bfqq_expiration reason; 7006 unsigned long flags; 7007 7008 spin_lock_irqsave(&bfqd->lock, flags); 7009 7010 /* 7011 * Considering that bfqq may be in race, we should firstly check 7012 * whether bfqq is in service before doing something on it. If 7013 * the bfqq in race is not in service, it has already been expired 7014 * through __bfq_bfqq_expire func and its wait_request flags has 7015 * been cleared in __bfq_bfqd_reset_in_service func. 7016 */ 7017 if (bfqq != bfqd->in_service_queue) { 7018 spin_unlock_irqrestore(&bfqd->lock, flags); 7019 return; 7020 } 7021 7022 bfq_clear_bfqq_wait_request(bfqq); 7023 7024 if (bfq_bfqq_budget_timeout(bfqq)) 7025 /* 7026 * Also here the queue can be safely expired 7027 * for budget timeout without wasting 7028 * guarantees 7029 */ 7030 reason = BFQQE_BUDGET_TIMEOUT; 7031 else if (bfqq->queued[0] == 0 && bfqq->queued[1] == 0) 7032 /* 7033 * The queue may not be empty upon timer expiration, 7034 * because we may not disable the timer when the 7035 * first request of the in-service queue arrives 7036 * during disk idling. 7037 */ 7038 reason = BFQQE_TOO_IDLE; 7039 else 7040 goto schedule_dispatch; 7041 7042 bfq_bfqq_expire(bfqd, bfqq, true, reason); 7043 7044schedule_dispatch: 7045 bfq_schedule_dispatch(bfqd); 7046 spin_unlock_irqrestore(&bfqd->lock, flags); 7047} 7048 7049/* 7050 * Handler of the expiration of the timer running if the in-service queue 7051 * is idling inside its time slice. 7052 */ 7053static enum hrtimer_restart bfq_idle_slice_timer(struct hrtimer *timer) 7054{ 7055 struct bfq_data *bfqd = container_of(timer, struct bfq_data, 7056 idle_slice_timer); 7057 struct bfq_queue *bfqq = bfqd->in_service_queue; 7058 7059 /* 7060 * Theoretical race here: the in-service queue can be NULL or 7061 * different from the queue that was idling if a new request 7062 * arrives for the current queue and there is a full dispatch 7063 * cycle that changes the in-service queue. This can hardly 7064 * happen, but in the worst case we just expire a queue too 7065 * early. 7066 */ 7067 if (bfqq) 7068 bfq_idle_slice_timer_body(bfqd, bfqq); 7069 7070 return HRTIMER_NORESTART; 7071} 7072 7073static void __bfq_put_async_bfqq(struct bfq_data *bfqd, 7074 struct bfq_queue **bfqq_ptr) 7075{ 7076 struct bfq_queue *bfqq = *bfqq_ptr; 7077 7078 bfq_log(bfqd, "put_async_bfqq: %p", bfqq); 7079 if (bfqq) { 7080 bfq_bfqq_move(bfqd, bfqq, bfqd->root_group); 7081 7082 bfq_log_bfqq(bfqd, bfqq, "put_async_bfqq: putting %p, %d", 7083 bfqq, bfqq->ref); 7084 bfq_put_queue(bfqq); 7085 *bfqq_ptr = NULL; 7086 } 7087} 7088 7089/* 7090 * Release all the bfqg references to its async queues. If we are 7091 * deallocating the group these queues may still contain requests, so 7092 * we reparent them to the root cgroup (i.e., the only one that will 7093 * exist for sure until all the requests on a device are gone). 7094 */ 7095void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg) 7096{ 7097 int i, j, k; 7098 7099 for (k = 0; k < bfqd->num_actuators; k++) { 7100 for (i = 0; i < 2; i++) 7101 for (j = 0; j < IOPRIO_NR_LEVELS; j++) 7102 __bfq_put_async_bfqq(bfqd, &bfqg->async_bfqq[i][j][k]); 7103 7104 __bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq[k]); 7105 } 7106} 7107 7108/* 7109 * See the comments on bfq_limit_depth for the purpose of 7110 * the depths set in the function. Return minimum shallow depth we'll use. 7111 */ 7112static void bfq_depth_updated(struct request_queue *q) 7113{ 7114 struct bfq_data *bfqd = q->elevator->elevator_data; 7115 unsigned int nr_requests = q->nr_requests; 7116 7117 /* 7118 * In-word depths if no bfq_queue is being weight-raised: 7119 * leaving 25% of tags only for sync reads. 7120 * 7121 * In next formulas, right-shift the value 7122 * (1U<<bt->sb.shift), instead of computing directly 7123 * (1U<<(bt->sb.shift - something)), to be robust against 7124 * any possible value of bt->sb.shift, without having to 7125 * limit 'something'. 7126 */ 7127 /* no more than 50% of tags for async I/O */ 7128 bfqd->async_depths[0][0] = max(nr_requests >> 1, 1U); 7129 /* 7130 * no more than 75% of tags for sync writes (25% extra tags 7131 * w.r.t. async I/O, to prevent async I/O from starving sync 7132 * writes) 7133 */ 7134 bfqd->async_depths[0][1] = max((nr_requests * 3) >> 2, 1U); 7135 7136 /* 7137 * In-word depths in case some bfq_queue is being weight- 7138 * raised: leaving ~63% of tags for sync reads. This is the 7139 * highest percentage for which, in our tests, application 7140 * start-up times didn't suffer from any regression due to tag 7141 * shortage. 7142 */ 7143 /* no more than ~18% of tags for async I/O */ 7144 bfqd->async_depths[1][0] = max((nr_requests * 3) >> 4, 1U); 7145 /* no more than ~37% of tags for sync writes (~20% extra tags) */ 7146 bfqd->async_depths[1][1] = max((nr_requests * 6) >> 4, 1U); 7147 7148 blk_mq_set_min_shallow_depth(q, 1); 7149} 7150 7151static void bfq_exit_queue(struct elevator_queue *e) 7152{ 7153 struct bfq_data *bfqd = e->elevator_data; 7154 struct bfq_queue *bfqq, *n; 7155 unsigned int actuator; 7156 7157 hrtimer_cancel(&bfqd->idle_slice_timer); 7158 7159 spin_lock_irq(&bfqd->lock); 7160 list_for_each_entry_safe(bfqq, n, &bfqd->idle_list, bfqq_list) 7161 bfq_deactivate_bfqq(bfqd, bfqq, false, false); 7162 spin_unlock_irq(&bfqd->lock); 7163 7164 for (actuator = 0; actuator < bfqd->num_actuators; actuator++) 7165 WARN_ON_ONCE(bfqd->rq_in_driver[actuator]); 7166 WARN_ON_ONCE(bfqd->tot_rq_in_driver); 7167 7168 hrtimer_cancel(&bfqd->idle_slice_timer); 7169 7170 /* release oom-queue reference to root group */ 7171 bfqg_and_blkg_put(bfqd->root_group); 7172 7173#ifdef CONFIG_BFQ_GROUP_IOSCHED 7174 blkcg_deactivate_policy(bfqd->queue->disk, &blkcg_policy_bfq); 7175#else 7176 spin_lock_irq(&bfqd->lock); 7177 bfq_put_async_queues(bfqd, bfqd->root_group); 7178 kfree(bfqd->root_group); 7179 spin_unlock_irq(&bfqd->lock); 7180#endif 7181 7182 blk_stat_disable_accounting(bfqd->queue); 7183 blk_queue_flag_clear(QUEUE_FLAG_DISABLE_WBT_DEF, bfqd->queue); 7184 wbt_enable_default(bfqd->queue->disk); 7185 7186 kfree(bfqd); 7187} 7188 7189static void bfq_init_root_group(struct bfq_group *root_group, 7190 struct bfq_data *bfqd) 7191{ 7192 int i; 7193 7194#ifdef CONFIG_BFQ_GROUP_IOSCHED 7195 root_group->entity.parent = NULL; 7196 root_group->my_entity = NULL; 7197 root_group->bfqd = bfqd; 7198#endif 7199 root_group->rq_pos_tree = RB_ROOT; 7200 for (i = 0; i < BFQ_IOPRIO_CLASSES; i++) 7201 root_group->sched_data.service_tree[i] = BFQ_SERVICE_TREE_INIT; 7202 root_group->sched_data.bfq_class_idle_last_service = jiffies; 7203} 7204 7205static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq) 7206{ 7207 struct bfq_data *bfqd; 7208 unsigned int i; 7209 struct blk_independent_access_ranges *ia_ranges = q->disk->ia_ranges; 7210 7211 bfqd = kzalloc_node(sizeof(*bfqd), GFP_KERNEL, q->node); 7212 if (!bfqd) 7213 return -ENOMEM; 7214 7215 eq->elevator_data = bfqd; 7216 7217 spin_lock_irq(&q->queue_lock); 7218 q->elevator = eq; 7219 spin_unlock_irq(&q->queue_lock); 7220 7221 /* 7222 * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues. 7223 * Grab a permanent reference to it, so that the normal code flow 7224 * will not attempt to free it. 7225 * Set zero as actuator index: we will pretend that 7226 * all I/O requests are for the same actuator. 7227 */ 7228 bfq_init_bfqq(bfqd, &bfqd->oom_bfqq, NULL, 1, 0, 0); 7229 bfqd->oom_bfqq.ref++; 7230 bfqd->oom_bfqq.new_ioprio = BFQ_DEFAULT_QUEUE_IOPRIO; 7231 bfqd->oom_bfqq.new_ioprio_class = IOPRIO_CLASS_BE; 7232 bfqd->oom_bfqq.entity.new_weight = 7233 bfq_ioprio_to_weight(bfqd->oom_bfqq.new_ioprio); 7234 7235 /* oom_bfqq does not participate to bursts */ 7236 bfq_clear_bfqq_just_created(&bfqd->oom_bfqq); 7237 7238 /* 7239 * Trigger weight initialization, according to ioprio, at the 7240 * oom_bfqq's first activation. The oom_bfqq's ioprio and ioprio 7241 * class won't be changed any more. 7242 */ 7243 bfqd->oom_bfqq.entity.prio_changed = 1; 7244 7245 bfqd->queue = q; 7246 7247 bfqd->num_actuators = 1; 7248 /* 7249 * If the disk supports multiple actuators, copy independent 7250 * access ranges from the request queue structure. 7251 */ 7252 spin_lock_irq(&q->queue_lock); 7253 if (ia_ranges) { 7254 /* 7255 * Check if the disk ia_ranges size exceeds the current bfq 7256 * actuator limit. 7257 */ 7258 if (ia_ranges->nr_ia_ranges > BFQ_MAX_ACTUATORS) { 7259 pr_crit("nr_ia_ranges higher than act limit: iars=%d, max=%d.\n", 7260 ia_ranges->nr_ia_ranges, BFQ_MAX_ACTUATORS); 7261 pr_crit("Falling back to single actuator mode.\n"); 7262 } else { 7263 bfqd->num_actuators = ia_ranges->nr_ia_ranges; 7264 7265 for (i = 0; i < bfqd->num_actuators; i++) { 7266 bfqd->sector[i] = ia_ranges->ia_range[i].sector; 7267 bfqd->nr_sectors[i] = 7268 ia_ranges->ia_range[i].nr_sectors; 7269 } 7270 } 7271 } 7272 7273 /* Otherwise use single-actuator dev info */ 7274 if (bfqd->num_actuators == 1) { 7275 bfqd->sector[0] = 0; 7276 bfqd->nr_sectors[0] = get_capacity(q->disk); 7277 } 7278 spin_unlock_irq(&q->queue_lock); 7279 7280 INIT_LIST_HEAD(&bfqd->dispatch); 7281 7282 hrtimer_setup(&bfqd->idle_slice_timer, bfq_idle_slice_timer, CLOCK_MONOTONIC, 7283 HRTIMER_MODE_REL); 7284 7285 bfqd->queue_weights_tree = RB_ROOT_CACHED; 7286#ifdef CONFIG_BFQ_GROUP_IOSCHED 7287 bfqd->num_groups_with_pending_reqs = 0; 7288#endif 7289 7290 INIT_LIST_HEAD(&bfqd->active_list[0]); 7291 INIT_LIST_HEAD(&bfqd->active_list[1]); 7292 INIT_LIST_HEAD(&bfqd->idle_list); 7293 INIT_HLIST_HEAD(&bfqd->burst_list); 7294 7295 bfqd->hw_tag = -1; 7296 bfqd->nonrot_with_queueing = blk_queue_nonrot(bfqd->queue); 7297 7298 bfqd->bfq_max_budget = bfq_default_max_budget; 7299 7300 bfqd->bfq_fifo_expire[0] = bfq_fifo_expire[0]; 7301 bfqd->bfq_fifo_expire[1] = bfq_fifo_expire[1]; 7302 bfqd->bfq_back_max = bfq_back_max; 7303 bfqd->bfq_back_penalty = bfq_back_penalty; 7304 bfqd->bfq_slice_idle = bfq_slice_idle; 7305 bfqd->bfq_timeout = bfq_timeout; 7306 7307 bfqd->bfq_large_burst_thresh = 8; 7308 bfqd->bfq_burst_interval = msecs_to_jiffies(180); 7309 7310 bfqd->low_latency = true; 7311 7312 /* 7313 * Trade-off between responsiveness and fairness. 7314 */ 7315 bfqd->bfq_wr_coeff = 30; 7316 bfqd->bfq_wr_rt_max_time = msecs_to_jiffies(300); 7317 bfqd->bfq_wr_min_idle_time = msecs_to_jiffies(2000); 7318 bfqd->bfq_wr_min_inter_arr_async = msecs_to_jiffies(500); 7319 bfqd->bfq_wr_max_softrt_rate = 7000; /* 7320 * Approximate rate required 7321 * to playback or record a 7322 * high-definition compressed 7323 * video. 7324 */ 7325 bfqd->wr_busy_queues = 0; 7326 7327 /* 7328 * Begin by assuming, optimistically, that the device peak 7329 * rate is equal to 2/3 of the highest reference rate. 7330 */ 7331 bfqd->rate_dur_prod = ref_rate[blk_queue_nonrot(bfqd->queue)] * 7332 ref_wr_duration[blk_queue_nonrot(bfqd->queue)]; 7333 bfqd->peak_rate = ref_rate[blk_queue_nonrot(bfqd->queue)] * 2 / 3; 7334 7335 /* see comments on the definition of next field inside bfq_data */ 7336 bfqd->actuator_load_threshold = 4; 7337 7338 spin_lock_init(&bfqd->lock); 7339 7340 /* 7341 * The invocation of the next bfq_create_group_hierarchy 7342 * function is the head of a chain of function calls 7343 * (bfq_create_group_hierarchy->blkcg_activate_policy-> 7344 * blk_mq_freeze_queue) that may lead to the invocation of the 7345 * has_work hook function. For this reason, 7346 * bfq_create_group_hierarchy is invoked only after all 7347 * scheduler data has been initialized, apart from the fields 7348 * that can be initialized only after invoking 7349 * bfq_create_group_hierarchy. This, in particular, enables 7350 * has_work to correctly return false. Of course, to avoid 7351 * other inconsistencies, the blk-mq stack must then refrain 7352 * from invoking further scheduler hooks before this init 7353 * function is finished. 7354 */ 7355 bfqd->root_group = bfq_create_group_hierarchy(bfqd, q->node); 7356 if (!bfqd->root_group) 7357 goto out_free; 7358 bfq_init_root_group(bfqd->root_group, bfqd); 7359 bfq_init_entity(&bfqd->oom_bfqq.entity, bfqd->root_group); 7360 bfq_depth_updated(q); 7361 7362 /* We dispatch from request queue wide instead of hw queue */ 7363 blk_queue_flag_set(QUEUE_FLAG_SQ_SCHED, q); 7364 7365 blk_queue_flag_set(QUEUE_FLAG_DISABLE_WBT_DEF, q); 7366 wbt_disable_default(q->disk); 7367 blk_stat_enable_accounting(q); 7368 7369 return 0; 7370 7371out_free: 7372 kfree(bfqd); 7373 return -ENOMEM; 7374} 7375 7376static void bfq_slab_kill(void) 7377{ 7378 kmem_cache_destroy(bfq_pool); 7379} 7380 7381static int __init bfq_slab_setup(void) 7382{ 7383 bfq_pool = KMEM_CACHE(bfq_queue, 0); 7384 if (!bfq_pool) 7385 return -ENOMEM; 7386 return 0; 7387} 7388 7389static ssize_t bfq_var_show(unsigned int var, char *page) 7390{ 7391 return sprintf(page, "%u\n", var); 7392} 7393 7394static int bfq_var_store(unsigned long *var, const char *page) 7395{ 7396 unsigned long new_val; 7397 int ret = kstrtoul(page, 10, &new_val); 7398 7399 if (ret) 7400 return ret; 7401 *var = new_val; 7402 return 0; 7403} 7404 7405#define SHOW_FUNCTION(__FUNC, __VAR, __CONV) \ 7406static ssize_t __FUNC(struct elevator_queue *e, char *page) \ 7407{ \ 7408 struct bfq_data *bfqd = e->elevator_data; \ 7409 u64 __data = __VAR; \ 7410 if (__CONV == 1) \ 7411 __data = jiffies_to_msecs(__data); \ 7412 else if (__CONV == 2) \ 7413 __data = div_u64(__data, NSEC_PER_MSEC); \ 7414 return bfq_var_show(__data, (page)); \ 7415} 7416SHOW_FUNCTION(bfq_fifo_expire_sync_show, bfqd->bfq_fifo_expire[1], 2); 7417SHOW_FUNCTION(bfq_fifo_expire_async_show, bfqd->bfq_fifo_expire[0], 2); 7418SHOW_FUNCTION(bfq_back_seek_max_show, bfqd->bfq_back_max, 0); 7419SHOW_FUNCTION(bfq_back_seek_penalty_show, bfqd->bfq_back_penalty, 0); 7420SHOW_FUNCTION(bfq_slice_idle_show, bfqd->bfq_slice_idle, 2); 7421SHOW_FUNCTION(bfq_max_budget_show, bfqd->bfq_user_max_budget, 0); 7422SHOW_FUNCTION(bfq_timeout_sync_show, bfqd->bfq_timeout, 1); 7423SHOW_FUNCTION(bfq_strict_guarantees_show, bfqd->strict_guarantees, 0); 7424SHOW_FUNCTION(bfq_low_latency_show, bfqd->low_latency, 0); 7425#undef SHOW_FUNCTION 7426 7427#define USEC_SHOW_FUNCTION(__FUNC, __VAR) \ 7428static ssize_t __FUNC(struct elevator_queue *e, char *page) \ 7429{ \ 7430 struct bfq_data *bfqd = e->elevator_data; \ 7431 u64 __data = __VAR; \ 7432 __data = div_u64(__data, NSEC_PER_USEC); \ 7433 return bfq_var_show(__data, (page)); \ 7434} 7435USEC_SHOW_FUNCTION(bfq_slice_idle_us_show, bfqd->bfq_slice_idle); 7436#undef USEC_SHOW_FUNCTION 7437 7438#define STORE_FUNCTION(__FUNC, __PTR, MIN, MAX, __CONV) \ 7439static ssize_t \ 7440__FUNC(struct elevator_queue *e, const char *page, size_t count) \ 7441{ \ 7442 struct bfq_data *bfqd = e->elevator_data; \ 7443 unsigned long __data, __min = (MIN), __max = (MAX); \ 7444 int ret; \ 7445 \ 7446 ret = bfq_var_store(&__data, (page)); \ 7447 if (ret) \ 7448 return ret; \ 7449 if (__data < __min) \ 7450 __data = __min; \ 7451 else if (__data > __max) \ 7452 __data = __max; \ 7453 if (__CONV == 1) \ 7454 *(__PTR) = msecs_to_jiffies(__data); \ 7455 else if (__CONV == 2) \ 7456 *(__PTR) = (u64)__data * NSEC_PER_MSEC; \ 7457 else \ 7458 *(__PTR) = __data; \ 7459 return count; \ 7460} 7461STORE_FUNCTION(bfq_fifo_expire_sync_store, &bfqd->bfq_fifo_expire[1], 1, 7462 INT_MAX, 2); 7463STORE_FUNCTION(bfq_fifo_expire_async_store, &bfqd->bfq_fifo_expire[0], 1, 7464 INT_MAX, 2); 7465STORE_FUNCTION(bfq_back_seek_max_store, &bfqd->bfq_back_max, 0, INT_MAX, 0); 7466STORE_FUNCTION(bfq_back_seek_penalty_store, &bfqd->bfq_back_penalty, 1, 7467 INT_MAX, 0); 7468STORE_FUNCTION(bfq_slice_idle_store, &bfqd->bfq_slice_idle, 0, INT_MAX, 2); 7469#undef STORE_FUNCTION 7470 7471#define USEC_STORE_FUNCTION(__FUNC, __PTR, MIN, MAX) \ 7472static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count)\ 7473{ \ 7474 struct bfq_data *bfqd = e->elevator_data; \ 7475 unsigned long __data, __min = (MIN), __max = (MAX); \ 7476 int ret; \ 7477 \ 7478 ret = bfq_var_store(&__data, (page)); \ 7479 if (ret) \ 7480 return ret; \ 7481 if (__data < __min) \ 7482 __data = __min; \ 7483 else if (__data > __max) \ 7484 __data = __max; \ 7485 *(__PTR) = (u64)__data * NSEC_PER_USEC; \ 7486 return count; \ 7487} 7488USEC_STORE_FUNCTION(bfq_slice_idle_us_store, &bfqd->bfq_slice_idle, 0, 7489 UINT_MAX); 7490#undef USEC_STORE_FUNCTION 7491 7492static ssize_t bfq_max_budget_store(struct elevator_queue *e, 7493 const char *page, size_t count) 7494{ 7495 struct bfq_data *bfqd = e->elevator_data; 7496 unsigned long __data; 7497 int ret; 7498 7499 ret = bfq_var_store(&__data, (page)); 7500 if (ret) 7501 return ret; 7502 7503 if (__data == 0) 7504 bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd); 7505 else { 7506 if (__data > INT_MAX) 7507 __data = INT_MAX; 7508 bfqd->bfq_max_budget = __data; 7509 } 7510 7511 bfqd->bfq_user_max_budget = __data; 7512 7513 return count; 7514} 7515 7516/* 7517 * Leaving this name to preserve name compatibility with cfq 7518 * parameters, but this timeout is used for both sync and async. 7519 */ 7520static ssize_t bfq_timeout_sync_store(struct elevator_queue *e, 7521 const char *page, size_t count) 7522{ 7523 struct bfq_data *bfqd = e->elevator_data; 7524 unsigned long __data; 7525 int ret; 7526 7527 ret = bfq_var_store(&__data, (page)); 7528 if (ret) 7529 return ret; 7530 7531 if (__data < 1) 7532 __data = 1; 7533 else if (__data > INT_MAX) 7534 __data = INT_MAX; 7535 7536 bfqd->bfq_timeout = msecs_to_jiffies(__data); 7537 if (bfqd->bfq_user_max_budget == 0) 7538 bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd); 7539 7540 return count; 7541} 7542 7543static ssize_t bfq_strict_guarantees_store(struct elevator_queue *e, 7544 const char *page, size_t count) 7545{ 7546 struct bfq_data *bfqd = e->elevator_data; 7547 unsigned long __data; 7548 int ret; 7549 7550 ret = bfq_var_store(&__data, (page)); 7551 if (ret) 7552 return ret; 7553 7554 if (__data > 1) 7555 __data = 1; 7556 if (!bfqd->strict_guarantees && __data == 1 7557 && bfqd->bfq_slice_idle < 8 * NSEC_PER_MSEC) 7558 bfqd->bfq_slice_idle = 8 * NSEC_PER_MSEC; 7559 7560 bfqd->strict_guarantees = __data; 7561 7562 return count; 7563} 7564 7565static ssize_t bfq_low_latency_store(struct elevator_queue *e, 7566 const char *page, size_t count) 7567{ 7568 struct bfq_data *bfqd = e->elevator_data; 7569 unsigned long __data; 7570 int ret; 7571 7572 ret = bfq_var_store(&__data, (page)); 7573 if (ret) 7574 return ret; 7575 7576 if (__data > 1) 7577 __data = 1; 7578 if (__data == 0 && bfqd->low_latency != 0) 7579 bfq_end_wr(bfqd); 7580 bfqd->low_latency = __data; 7581 7582 return count; 7583} 7584 7585#define BFQ_ATTR(name) \ 7586 __ATTR(name, 0644, bfq_##name##_show, bfq_##name##_store) 7587 7588static const struct elv_fs_entry bfq_attrs[] = { 7589 BFQ_ATTR(fifo_expire_sync), 7590 BFQ_ATTR(fifo_expire_async), 7591 BFQ_ATTR(back_seek_max), 7592 BFQ_ATTR(back_seek_penalty), 7593 BFQ_ATTR(slice_idle), 7594 BFQ_ATTR(slice_idle_us), 7595 BFQ_ATTR(max_budget), 7596 BFQ_ATTR(timeout_sync), 7597 BFQ_ATTR(strict_guarantees), 7598 BFQ_ATTR(low_latency), 7599 __ATTR_NULL 7600}; 7601 7602static struct elevator_type iosched_bfq_mq = { 7603 .ops = { 7604 .limit_depth = bfq_limit_depth, 7605 .prepare_request = bfq_prepare_request, 7606 .requeue_request = bfq_finish_requeue_request, 7607 .finish_request = bfq_finish_request, 7608 .exit_icq = bfq_exit_icq, 7609 .insert_requests = bfq_insert_requests, 7610 .dispatch_request = bfq_dispatch_request, 7611 .next_request = elv_rb_latter_request, 7612 .former_request = elv_rb_former_request, 7613 .allow_merge = bfq_allow_bio_merge, 7614 .bio_merge = bfq_bio_merge, 7615 .request_merge = bfq_request_merge, 7616 .requests_merged = bfq_requests_merged, 7617 .request_merged = bfq_request_merged, 7618 .has_work = bfq_has_work, 7619 .depth_updated = bfq_depth_updated, 7620 .init_sched = bfq_init_queue, 7621 .exit_sched = bfq_exit_queue, 7622 }, 7623 7624 .icq_size = sizeof(struct bfq_io_cq), 7625 .icq_align = __alignof__(struct bfq_io_cq), 7626 .elevator_attrs = bfq_attrs, 7627 .elevator_name = "bfq", 7628 .elevator_owner = THIS_MODULE, 7629}; 7630MODULE_ALIAS("bfq-iosched"); 7631 7632static int __init bfq_init(void) 7633{ 7634 int ret; 7635 7636#ifdef CONFIG_BFQ_GROUP_IOSCHED 7637 ret = blkcg_policy_register(&blkcg_policy_bfq); 7638 if (ret) 7639 return ret; 7640#endif 7641 7642 ret = -ENOMEM; 7643 if (bfq_slab_setup()) 7644 goto err_pol_unreg; 7645 7646 /* 7647 * Times to load large popular applications for the typical 7648 * systems installed on the reference devices (see the 7649 * comments before the definition of the next 7650 * array). Actually, we use slightly lower values, as the 7651 * estimated peak rate tends to be smaller than the actual 7652 * peak rate. The reason for this last fact is that estimates 7653 * are computed over much shorter time intervals than the long 7654 * intervals typically used for benchmarking. Why? First, to 7655 * adapt more quickly to variations. Second, because an I/O 7656 * scheduler cannot rely on a peak-rate-evaluation workload to 7657 * be run for a long time. 7658 */ 7659 ref_wr_duration[0] = msecs_to_jiffies(7000); /* actually 8 sec */ 7660 ref_wr_duration[1] = msecs_to_jiffies(2500); /* actually 3 sec */ 7661 7662 ret = elv_register(&iosched_bfq_mq); 7663 if (ret) 7664 goto slab_kill; 7665 7666 return 0; 7667 7668slab_kill: 7669 bfq_slab_kill(); 7670err_pol_unreg: 7671#ifdef CONFIG_BFQ_GROUP_IOSCHED 7672 blkcg_policy_unregister(&blkcg_policy_bfq); 7673#endif 7674 return ret; 7675} 7676 7677static void __exit bfq_exit(void) 7678{ 7679 elv_unregister(&iosched_bfq_mq); 7680#ifdef CONFIG_BFQ_GROUP_IOSCHED 7681 blkcg_policy_unregister(&blkcg_policy_bfq); 7682#endif 7683 bfq_slab_kill(); 7684} 7685 7686module_init(bfq_init); 7687module_exit(bfq_exit); 7688 7689MODULE_AUTHOR("Paolo Valente"); 7690MODULE_LICENSE("GPL"); 7691MODULE_DESCRIPTION("MQ Budget Fair Queueing I/O Scheduler");
~/
Read-only file preview.