Fix memory consumption related to SMB AIO reads
Description
Problem/Justification
None
Impact
None
is duplicated by
Activity
Show:
Bug Clerk November 2, 2024 at 3:54 PM
This issue has now been closed. Comments made after this point may not be viewed by the TrueNAS Teams. Please open a new issue if you have found a problem or need to re-engage with the TrueNAS Engineering Teams.
Andrew Walker November 2, 2024 at 11:54 AMEdited
^^^ master
^^^ stable/electriceel
^^^ 24.10.0.1
^^^ 24.04.2.4
Complete
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Andrew Walker
Andrew WalkerReporter
Andrew Walker
Andrew WalkerImpact
High
Affects versions
Priority
More fields
Time tracking
More fields
Time trackingKatalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Katalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Created November 1, 2024 at 12:59 PM
Updated February 27, 2025 at 9:06 PM
Resolved November 2, 2024 at 3:54 PM
Add backpressure mechanism for AIO reads
In some edge cases where storage backend can't keep up with client
reads, io_uring / samba will allow queueing up reads beyond what
is reasonable leading to OOM errors in some extreme edge cases.
This commit keeps a counter of outstanding read requests per-TCON
and switches to pread(2) once the queue depth limit is reached.
This effectively applies backpressure to client and prevents excessive
memory consumption. A total count of synchronous reads performed is
logged when TCON is disconnected if log level for the VFS module is
sufficiently high (3 or higher).
Fix memory leak related to AIO memory pool for some failure scenarios
for AIO reads.
The memory pool for AIO buffers is allocated under a memory context
that persists for the duration of the SMB session. When chunks of
pool are assigned out for a particular AIO request, a separate
small allocation (io_link) is performed with a pointer to the data
buffer and a talloc destructor set such that when the io_link is
freed the buffer is returned to the pool.
Initially these were performed as two separate steps (assigning out
buffer from pool and setting up io_link) which allowed for situations
to arise in which the request could error out before the io_link was
set resulting in the buffer never being returned to the memory pool.
This commit revises the workflow and API for the memory pool such
that the chunk is always assigned with an associated io_link allocated
under the memory context of the initial SMB2 read request. Once the
request is successful and we're getting ready to respond to client
then we reparent the io_link to the context of the response. This
ensures that the buffer is always returned when a limited-life
talloc chunk is freed.