diff options
| author | Pavel Begunkov <asml.silence@gmail.com> | 2024-01-17 00:57:27 +0000 |
|---|---|---|
| committer | Jens Axboe <axboe@kernel.dk> | 2024-01-17 09:45:24 -0700 |
| commit | d381099f980b5f6c3c7e150baf13b0aaefc66c29 (patch) | |
| tree | c53607ec46b10a246c94519271b04f557fb919bf /io_uring | |
| parent | dc12d1799ce710fd90abbe0ced71e7e1ae0894fc (diff) | |
io_uring: clean up local tw add-wait sync
Kill a smp_mb__after_atomic() right before wake_up, it's useless, and
add a comment explaining implicit barriers from cmpxchg and
synchronsation around ->cq_wait_nr with the waiter.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/3007f3c2d53c72b61de56919ef56b53158b8276f.1705438669.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'io_uring')
| -rw-r--r-- | io_uring/io_uring.c | 10 |
1 files changed, 8 insertions, 2 deletions
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d40c767a6216..3ab7e6a46149 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1332,6 +1332,14 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) } while (!try_cmpxchg(&ctx->work_llist.first, &first, &req->io_task_work.node)); + /* + * cmpxchg implies a full barrier, which pairs with the barrier + * in set_current_state() on the io_cqring_wait() side. It's used + * to ensure that either we see updated ->cq_wait_nr, or waiters + * going to sleep will observe the work added to the list, which + * is similar to the wait/wawke task state sync. + */ + if (!first) { if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); @@ -1346,8 +1354,6 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) /* either not enough or the previous add has already woken it up */ if (nr_wait > nr_tw || nr_tw_prev >= nr_wait) return; - /* pairs with set_current_state() in io_cqring_wait() */ - smp_mb__after_atomic(); wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); } |