Why OpenList Quark WebDAV Uploads Use Excessive Memory: Causes and Optimization

When using rclone to upload local files to Quark Cloud Drive through OpenList’s WebDAV interface, OpenList’s memory usage may increase significantly, even triggering a container OOM. Investigation found that the key factors are the full-file cache used before uploads by the Quark driver and the file page cache included in container statistics.

This analysis focuses on OpenList v4.2.7, the standard Quark driver, Docker, and cgroup v2. The implementation should be rechecked for other drivers and versions.

Why Quark uploads require caching

The OpenList WebDAV entry point wraps the HTTP request body into a file stream and then passes it to the target storage driver. This step itself does not inherently read the entire file into memory.

However, the standard Quark driver first obtains the file’s MD5 and SHA1. The WebDAV upload path does not provide these two hashes to the file object, so the driver calls:

stream.CacheFullAndWriter(&up, io.MultiWriter(writers...))

This call caches the entire file while calculating the hashes. The driver then submits pre-upload and hash requests to Quark; if instant upload is not possible, it reads the data from the cache and uploads it in chunks.

Hash computation can use very little memory; what actually consumes space is the file content retained for subsequent readback. Even if Quark supports chunked uploads, that does not mean the preceding full-file cache is absent.

HybridCache in v4.2.7 has both memory and file backends. When memory use is allowed, it attempts to expand the cache and switches to files after allocation becomes constrained; some of this memory is managed through anonymous mmap. Therefore, all cache usage cannot be constrained merely by adjusting Go GC parameters.

rclone’s --transfers controls the number of files transferred simultaneously. Two files being uploaded concurrently means that two full-file caches may exist at the same time. Reducing rclone’s own --buffer-size does not directly limit this portion of the cache in the OpenList process.

Why a temporary directory still results in memory usage

For example:

"temp_dir": "data/temp"

This only specifies where temporary files are stored; it does not force every cache to be written to disk. To determine the actual location, the relative path must also be considered together with OpenList’s working directory and Docker mounts.

Additionally, the cache expansion check in v4.2.7 uses system available-memory statistics and does not directly read the container’s memory.max and memory.current in that check. As a result, even when the host still has free memory, the container may hit its own memory limit first.

Switching to disk temporary files

First, pause uploads and back up the configuration. Set the following at the top level of config.json:

"min_free_memory": -1

This is a field fragment, not a complete configuration file. When editing it, preserve the other content and a valid JSON structure.

The initialization code in v4.2.7 interprets a negative value as disabling HybridCache’s memory cache. 0 causes the threshold to be calculated automatically and does not mean disabled. This adjustment changes only the corresponding caching mechanism, not the entire process’s hard memory limit.

Ensure that temp_dir is located on an actual disk and has enough space. If the temporary directory is located on tmpfs, the so-called file cache will still consume memory and will not produce the expected result. Caching complete files also means that enough disk space must be reserved for concurrent files.

Confirm the Docker working directory and mounts:

container='openlist'
docker inspect -f '工作目录={{.Config.WorkingDir}}
{{range .Mounts}}{{println .Source "->" .Destination}}{{end}}' "$container"

In a deployment where the working directory has been confirmed as /opt/openlist and the temporary directory as data/temp, you can check:

docker exec openlist df -h /opt/openlist/data/temp

Replace these with the actual paths for other deployments. After completing the configuration, restart the container:

docker restart openlist
docker logs --tail 100 openlist 2>&1 |
    grep -E 'disable memory cache|min free memory|auto memory limit'

Seeing the following log indicates that the configuration has taken effect:

disable memory cache

At this point, even if auto memory limit: 4MB is still printed, it does not mean that the cache has been re-enabled. If the configuration does not take effect, check whether the file actually in use was edited and whether an environment variable is overriding it; the variable prefix should be based on the actual startup configuration.

How to determine whether memory usage has actually improved

Do not look only at memory.current. It includes anonymous memory, the file page cache, and other chargeable memory; after switching to disk files, Linux will still use RAM to cache file data.

Run the following commands on the Docker host; they apply to cgroup v2. Modify the container name as appropriate:

(
    container='openlist'
    pid=$(docker inspect -f '{{.State.Pid}}' "$container") || exit 1
    [ "${pid:-0}" -gt 0 ] || { echo '容器未运行'; exit 1; }
    cg=$(awk -F: '$1 == "0" {print $3}' "/proc/$pid/cgroup")
    [ -n "$cg" ] || { echo '未找到 cgroup v2 路径'; exit 1; }
    dir="/sys/fs/cgroup$cg"

    awk '
        $1 ~ /^(anon|file|shmem|file_dirty|file_writeback|kernel|sock)$/ {
            printf "%-16s %.1f MiB\n", $1, $2/1048576
        }
    ' "$dir/memory.stat"
    cat "$dir/memory.events"
    docker stats --no-stream "$container"
)

The measured results after one optimization were:

MetricUsage
anon26.8 MiB
file131.4 MiB
sock5.5 MiB
shmem0.0 MiB
file_dirty131.3 MiB
file_writeback0.0 MiB
Memory displayed by Docker36.2 MiB

This shows that anonymous memory was low at that moment, while most of the usage came from the file page cache, with the majority not yet written back to disk. file_dirty, shmem, and similar entries are subcategories and must not be added to file again.

file_writeback=0 only means that no data was being written back at the instant of sampling; it does not mean that the files were not using disk. Dirty pages must be written back before they can be reclaimed, so not all file cache should be considered immediately reclaimable.

On Linux, Docker CLI deducts some of the file cache, so the memory it displays may be significantly lower than memory.current. The two serve different purposes and should be evaluated together with memory.stat.

These results support the effectiveness of the cache policy adjustment, but a single sample cannot prove that there is no leak throughout the entire runtime. If abnormal behavior is suspected, compare anon, total usage, and OOM events across multiple runs of the same task, rather than drawing conclusions solely because memory did not immediately decrease after the upload ended.

Setting container limits according to the workload

First, complete a representative high-load synchronization and record the total memory peak. If the kernel provides memory.peak, you can read it; otherwise, sampling memory.current every second can only obtain an approximate peak during the observation period and may miss brief spikes.

After this adjustment, a sampled peak of approximately 478 MiB was observed, so 768 MiB can be used as a starting point for a trial run:

docker update --memory 768m openlist

This limit still requires validation under the full workload and is not a recommended fixed value for every environment. If Docker reports a conflict with the existing swap configuration, first inspect the memory-swap setting and decide on a strategy.

When using Compose, merge the limit into the existing service:

services:
  openlist:
    # 保留原有镜像、挂载和网络设置
    mem_limit: 768m

Do not set the limit too low based on the tens of MiB displayed by Docker, because file cache also counts toward the cgroup memory limit. After setting it, observe whether oom and oom_kill in memory.events increase, and whether uploads fail or become noticeably slower. If the workload is relatively heavy, you can first reduce rclone Transfers to 1 for comparison.

References