diff options
| author | Ming Lei <ming.lei@redhat.com> | 2021-01-22 10:33:12 +0800 |
|---|---|---|
| committer | Martin K. Petersen <martin.petersen@oracle.com> | 2021-03-04 17:36:59 -0500 |
| commit | 2a5a24aa83382a88c43d18a901fab66e6ffe1199 (patch) | |
| tree | 85d9a1aa8d76bb25fe9fdc56f9a1b74f575ce485 /block/blk-mq.h | |
| parent | d022d18c045fc2ccf92d0f14cf80f98eb0a8e119 (diff) | |
scsi: blk-mq: Return budget token from .get_budget callback
SCSI uses a global atomic variable to track queue depth for each
LUN/request queue.
This doesn't scale well when there are lots of CPU cores and the disk is
very fast. It has been observed that IOPS is affected a lot by tracking
queue depth via sdev->device_busy in the I/O path.
Return budget token from .get_budget callback. The budget token can be
passed to driver so that we can replace the atomic variable with
sbitmap_queue and alleviate the scaling problems that way.
Link: https://lore.kernel.org/r/20210122023317.687987-9-ming.lei@redhat.com
Cc: Omar Sandoval <osandov@fb.com>
Cc: Kashyap Desai <kashyap.desai@broadcom.com>
Cc: Sumanesh Samanta <sumanesh.samanta@broadcom.com>
Cc: Ewan D. Milne <emilne@redhat.com>
Tested-by: Sumanesh Samanta <sumanesh.samanta@broadcom.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Diffstat (limited to 'block/blk-mq.h')
| -rw-r--r-- | block/blk-mq.h | 25 |
1 files changed, 21 insertions, 4 deletions
diff --git a/block/blk-mq.h b/block/blk-mq.h index 3616453ca28c..9ce64bc4a6c8 100644 --- a/block/blk-mq.h +++ b/block/blk-mq.h @@ -187,17 +187,34 @@ unsigned int blk_mq_in_flight(struct request_queue *q, void blk_mq_in_flight_rw(struct request_queue *q, struct block_device *part, unsigned int inflight[2]); -static inline void blk_mq_put_dispatch_budget(struct request_queue *q) +static inline void blk_mq_put_dispatch_budget(struct request_queue *q, + int budget_token) { if (q->mq_ops->put_budget) - q->mq_ops->put_budget(q); + q->mq_ops->put_budget(q, budget_token); } -static inline bool blk_mq_get_dispatch_budget(struct request_queue *q) +static inline int blk_mq_get_dispatch_budget(struct request_queue *q) { if (q->mq_ops->get_budget) return q->mq_ops->get_budget(q); - return true; + return 0; +} + +static inline void blk_mq_set_rq_budget_token(struct request *rq, int token) +{ + if (token < 0) + return; + + if (rq->q->mq_ops->set_rq_budget_token) + rq->q->mq_ops->set_rq_budget_token(rq, token); +} + +static inline int blk_mq_get_rq_budget_token(struct request *rq) +{ + if (rq->q->mq_ops->get_rq_budget_token) + return rq->q->mq_ops->get_rq_budget_token(rq); + return -1; } static inline void __blk_mq_inc_active_requests(struct blk_mq_hw_ctx *hctx) |