webdav: serve archives via WebDAV / HTTP, fixes #9942#9944
Open
ThomasWaldmann wants to merge 17 commits into
Open
webdav: serve archives via WebDAV / HTTP, fixes #9942#9944ThomasWaldmann wants to merge 17 commits into
ThomasWaldmann wants to merge 17 commits into
Conversation
New "borg webdav" command: it starts a read-only HTTP server on localhost, so archive contents can be browsed and files downloaded with a web browser. This first step implements plain HTTP only (GET/HEAD, HTML directory listings, file downloads). The WebDAV methods (OPTIONS/PROPFIND), which enable native mounting by Windows Explorer, macOS Finder and the Linux desktop file managers, are planned to be added on top of this later. Implementation notes: - only uses the python standard library, no new dependencies. - the directory tree of an archive is built in memory when it is first accessed, so a big archive only costs time/memory if it is browsed. - all repository access is serialized with a lock, because borgstore connections are not thread-safe. - a LockRefresher keeps the repository lock of an idle server alive, so it is not killed as stale, see borgbackup#9872. - missing chunks abort the download connection, so damaged files are never served as silently corrupted content. - symlinks and special files are listed, but not downloadable. Note: downloads via HTTP do not preserve POSIX metadata (owner, group, mode, timestamps, xattrs, ACLs) - use borg extract or borg export-tar for full-fidelity restores. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #9944 +/- ##
==========================================
+ Coverage 85.49% 85.58% +0.09%
==========================================
Files 93 95 +2
Lines 16122 16762 +640
Branches 2466 2572 +106
==========================================
+ Hits 13783 14346 +563
- Misses 1632 1680 +48
- Partials 707 736 +29 ☔ View full report in Codecov by Harness. |
…ackup#9942 The served archives can now be mounted as a read-only network file system using the WebDAV client built into most operating systems and file managers: - Windows Explorer: "Map network drive" -> http://localhost:8000/ - macOS Finder: "Go > Connect to Server" -> http://localhost:8000 - GNOME Files / KDE Dolphin: dav://localhost:8000/ - Linux kernel mount: mount -t davfs (davfs2 package) Implemented (still python stdlib only): - OPTIONS advertising class 1 DAV support. - PROPFIND with Depth 0/1 (infinity is refused, as RFC 4918 permits), supporting allprop / propname / named-prop request bodies, answering with 207 multistatus XML. Request body size is limited. - live properties: resourcetype, displayname, getcontentlength, getcontenttype, getlastmodified, creationdate, getetag, and empty supportedlock/lockdiscovery (no locking - read-only server). - Range requests (single range): only the chunks overlapping the requested range are fetched and decrypted (chunk sizes are known in advance), so the many small ranged reads DAV file systems do for big files stay efficient. 206/416 handling, If-Range. - conditional requests: ETag (archive contents are immutable, so mtime+size is a solid validator), If-None-Match -> 304. - all writing/locking methods (PUT, DELETE, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, ...) are rejected with 405 + Allow, and clients see a read-only file system. - symbolic links and special files are not exposed via WebDAV (the protocol has no concept of them); they remain visible in the web browser listings. Verified against real clients: rclone (recursive PROPFIND listing and reads) and the macOS WebDAV file system (mount_webdav): mount, ls, byte- identical reads of multi-chunk files, write attempts correctly rejected with EROFS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Evaluation of the CodeQL py/http-response-splitting alerts: - Content-Disposition (real issue, fixed): the fallback file name in the header was built from the archived file name, which can contain any byte except NUL and "/" - including CR/LF. A file named e.g. "x\r\nEvil-Header: ..." planted in the backed up data would have injected headers into the download response. Now every non-printable character is replaced before building the fallback name (the RFC 8187 encoded name was already safe, quote() percent-encodes CR/LF). - Location (not exploitable, hardened anyway): the redirect echoed the raw request line path, which can never contain raw CR/LF (they would terminate the request line). The Location value is now rebuilt from the parsed path segments via percent-encoding, so it provably only contains URL-safe ASCII and no client input is echoed at all. Both send_header lines carry codeql[] suppression comments with the justification nearby, as CodeQL's taint tracking does not recognize these sanitizers. Add a test that plants file and directory names containing CR/LF in an archive and asserts that no header can be injected via download and redirect responses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ThomasWaldmann
changed the title
ThomasWaldmann
changed the title
Each segment of the path shown in the heading (e.g. "archive/data/docs/") is now a link to the corresponding directory. The h1 links inherit the heading color, so the page looks unchanged. The browser tab title stays plain text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A malicious PROPFIND body could declare internal XML entities that
expand exponentially ("billion laughs"). Reject any DTD before parsing,
using expat's StartDoctypeDeclHandler: it fires at the start of the
doctype declaration, before any entity is declared or expanded, so no
custom entities can exist and no expansion is possible. Using expat's
callback rather than a substring check on the raw body makes this
independent of the document encoding (a DTD in e.g. UTF-16 does not
contain the ascii bytes "<!DOCTYPE"). The request body is also size
limited to 1 MiB.
Add tests with UTF-8 and UTF-16 encoded entity expansion payloads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A mounted WebDAV file system reads a big file as many small, sequential range requests, which tend to hit the same content chunk over and over. Without caching, each request re-fetched and re-decrypted the chunk. Add a shared LRU cache of decrypted chunks (keyed by chunk id, so it is valid across archives and requests), the same approach borg mount uses. Its size is set by BORG_MOUNT_DATA_CACHE_ENTRIES (default: number of CPUs), just like for borg mount. The cache is guarded by repo_lock (the server is multi-threaded and LRUCache is not thread-safe), while writing to the client still happens outside the lock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tests created files with names that are illegal on Windows file systems (< and > in the "funny name", CR/LF and : in the header injection test), so the file creation raised OSError in the test setup and all webdav tests failed on the windows CI. Guard the creation of and the assertions about such files with is_win32, and skip the header injection test on Windows (it fundamentally needs a CR/LF file name, which Windows cannot create). The escaping and header injection code is platform independent and stays fully covered on the Linux and macOS CI. The other webdav tests (download, ranges, PROPFIND, conditional GET, data cache, ...) now run on Windows, too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The CI "security" job runs bandit, which flags the xml.etree import (B405) and the ET.fromstring() call (B314) as parsing untrusted XML. The parse is guarded: reject_dtd() rejects any DTD before parsing, so no custom entities can be declared and no entity expansion is possible, and the request body is size-limited. Mark both lines with # nosec (with a justification), the same mechanism borg already uses elsewhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a -f/--foreground flag; without it, the command daemonizes and runs in the background, matching borg mount. Bind the socket in the foreground first (so bind errors like "port in use" are reported before forking, and the listening fd survives the fork), then migrate the repository lock to the forked process. The LockRefresher and server threads are started after the fork, since threads do not survive fork(). The daemon stops on SIGTERM (the usual way to stop a daemon) or SIGINT, shutting down cleanly and releasing the repository lock. Daemonizing needs os.fork(), which does not exist on Windows; there the command stays in the foreground (borg webdav is meant to be usable on Windows, unlike borg mount). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FUNNY_NAME test file contains "<>" (and CR/LF for the injection test),
which are illegal in Windows file names, so it is skipped on Windows - which
left the Windows CI job exercising no non-ASCII file name at all. Add
UNICODE_NAME ("grüße.txt"), which is legal on every platform, and assert its
percent-encoding in the browser listing link, the download URL round-trip and
the PROPFIND href, unconditionally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Parse the PROPFIND request body with expat directly instead of xml.etree.ElementTree.fromstring(). This keeps the same, encoding-proof DTD rejection (expat's StartDoctypeDeclHandler fires before any entity can be declared, so entity expansion / "billion laughs" is impossible), but removes the xml.etree parsing sink entirely - the CodeQL py/xml-bomb query only recognises defusedxml as safe and cannot see the DTD rejection, so it kept flagging the fromstring() call even though the code was not vulnerable. xml.etree is still imported, but now only to *build* the response XML (which is not an XML-bomb sink), so its bandit B405 nosec stays and the B314 nosec on fromstring is gone. No new dependency. Behaviour is unchanged: valid allprop/propname/prop bodies parse identically and the existing UTF-8/UTF-16 bomb and garbage-body tests still get a 400. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a per-directory tar download: append ?tar (or ?tar=1) to a directory URL, or click the download icon shown next to the heading in the browser listings. Unlike a plain file download, the tar preserves POSIX metadata (owner, group, mode, sub-second timestamps, symlinks, special files, xattrs, ACLs), so it is the metadata-lossless way to restore a whole directory tree over the server. The tar is rooted at the requested directory (paths above it are stripped); ?tar on the archive root exports the whole archive. The tar size is not known in advance (PAX header sizes vary), so it is streamed with chunked transfer encoding. Repository access (item iteration and chunk fetching) is serialized under repo_lock, but each chunk is written to the client outside the lock: a slow client cannot block other requests, and the LockRefresher can keep the repository lock alive during a long download (holding the lock for the whole stream would starve it and let the lock go stale). A missing chunk leaves the chunked stream unterminated and closes the connection, so the client detects the truncation instead of receiving a corrupt archive. To avoid duplicating the borg-item -> tar mapping, item_to_tarinfo() and item_to_paxheaders() are factored out of TarMixIn._export_tar to module level in tar_cmds.py (behaviour-preserving; item_to_tarinfo now returns needs_content and the caller builds the content stream). webdav writes the tar blocks itself (TarInfo.tobuf for the header, chunk data, 512-byte padding, end-of-archive marker) so it can control the per-chunk locking. Verified end-to-end with the system tar tool: modes, symlink targets and a multi-chunk file are preserved byte-identically. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…position The tar download's file name is derived from the client-supplied URL path and flows into the Content-Disposition header, so CodeQL flags it as py/http-response-splitting - like the plain-download and redirect headers before it. The value is already sanitized: _content_disposition() strips non-printables (killing CR/LF) for the fallback name and percent-encodes the RFC 8187 name. Put the header on a single line so the # codeql[py/http-response-splitting] marker sits on the flagged line (it was on the wrapped closing-paren line before), matching the existing suppressions, and extend the header-injection test to a ?tar download of a directory whose name contains CR/LF. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the breadcrumb path, only the parent segments need to be links (for navigating up); the last segment is the directory being viewed, so show it as plain text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
Browser
Finder (macOS)
|
Raise patch coverage by exercising code paths the existing tests missed: - test_webdav_command_serves_and_stops runs the actual `borg webdav` command in the foreground, checks it serves a request, stops it with SIGTERM and confirms the repository lock was released (a following borg command works). This covers do_webdav(), which nothing exercised before. - test_webdav_special_files (POSIX) backs up a fifo and checks it is listed but not downloadable, refused with 403 on GET, hidden from WebDAV, and included in a ?tar download as a fifo entry. - extend test_webdav_errors with a GET on a symlink (403) and an oversized PROPFIND body (413, rejected via Content-Length without reading the body). - extend the tar test with a ?tar download at the archive root (whole archive, no path prefix stripped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CodeQL keeps flagging the Content-Disposition and Location headers as py/http-response-splitting. The values are already newline-free (the Content-Disposition fallback replaces non-printables, the RFC 8187 name and the Location are percent-encoded), but that is not visible to the analyser, and borg's CodeQL runs as advanced setup which does not honour inline "# codeql[...]" suppression comments (the marker sat right on the flagged line). Route every header value that derives from a client-supplied path through a small strip_crlf() helper right at the sink. It is a no-op by construction, but makes the CR/LF safety explicit as an actual sanitizer instead of relying on a suppression comment. Drop the now-misleading "# codeql[...]" markers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "Client notes and known issues" section to the webdav docs covering the common client-side quirks of a read-only WebDAV server: the Windows Explorer ~47 MiB download limit / WebClient service / Basic-auth-over-HTTP restriction, Finder writing (and harmlessly failing to write) .DS_Store/AppleDouble/.Trash, davfs2 needing use_locks 0, and the protocol-level notes (Depth: infinity refused, trailing-slash redirect, symlinks/special files and metadata only via the ?tar download). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
borg webdavcommand that starts a read-only WebDAV / HTTP server on localhost, so archive contents can be browsed with a web browser or mounted as a read-only network file system with the WebDAV client built into most operating systems. Addresses #9942.Usage
By default the command daemonizes; pass
--foreground/-fto keep it in the foreground (then Ctrl-C stops it). A daemonized server is stopped by sending it a signal (e.g.kill, which sends SIGTERM); either way the repository lock is released on shutdown.Then either open
http://localhost:8123/in a browser (styled green-on-black like borgbackup.org, with the borg logo; the heading is a breadcrumb - every path segment links to its directory), or mount it:http://localhost:8123/(ornet use Z: http://localhost:8123/)http://localhost:8123dav://localhost:8123/mount -t davfs(davfs2 package)Why this approach
Ecosystem prior art: kopia (
kopia mount --webdav), rustic (rustic webdav, a default feature) and vykar (itsmountcommand is a WebDAV server) all converged on WebDAV as the answer to platform pain around mounting; restic has an open PR for it. No new dependencies - everything is python stdlib.What is implemented
HTTP (for browsers): HTML directory listings with breadcrumb navigation (archives at the top level, selected via the archive filter options; lazily built per-archive directory trees below), file downloads, Range requests, conditional requests (ETag / If-None-Match / If-Range).
WebDAV class 1 (for mounting): OPTIONS (
DAV: 1), PROPFIND with Depth 0/1 (infinity refused per RFC 4918) answering 207 multistatus XML with the usual live properties; all writing/locking methods rejected with 405, so clients see a read-only file system. Range requests only fetch/decrypt the chunks overlapping the requested range (chunk sizes are known in advance), so the many small ranged reads DAV file systems do stay efficient.Directory download as a tar archive: append
?tarto any directory URL (the browser listings show a download icon next to the heading). Unlike a plain file download, the tar preserves POSIX metadata (owner, group, mode, sub-second timestamps, symlinks, special files, xattrs, ACLs), so it is the metadata-lossless way to restore a whole directory tree over the server. It is a PAX tarball, streamed uncompressed with chunked transfer encoding (the size is not known in advance). Repository access is serialized under the repo lock, but each chunk is written to the client outside the lock - so a slow client cannot block other requests and the lock refresher can keep the repository lock alive during a long download. A missing chunk leaves the chunked stream unterminated and closes the connection, so the client detects the truncation instead of receiving a corrupt archive. The borg-item -> tar mapping is shared withborg export-tar(factored out to module level, no duplication).Daemonize: like
borg mount, the command daemonizes by default (--foregroundto stay in the foreground). The listening socket is bound in the foreground first (so bind errors are reported before forking), then the repository lock is migrated to the forked process; SIGTERM/SIGINT shut the server down cleanly and release the lock. Daemonizing needsos.fork(), so on Windows the command stays in the foreground.Chunk cache: decrypted+decompressed content chunks are kept in an LRU cache (shared across requests, keyed by chunk id), sized by
BORG_MOUNT_DATA_CACHE_ENTRIESjust likeborg mount. This avoids re-decrypting the same chunk for the many small sequential range reads a mounted file system does. It complements the repository's own pack cache, which caches raw (still encrypted/compressed) pack bytes but not the per-chunk decrypt+decompress result.Security (thanks to CodeQL for the review):
--bindoption yet - that should arrive together with auth/TLS).py/http-response-splittingfindings; a header-injection test covers it.xml.etree): expat's doctype callback rejects any DTD before an entity can be declared - encoding-independent, so it rules out XML entity expansion attacks ("billion laughs"). Parsing withoutxml.etreealso removes thepy/xml-bombsink entirely, so that alert is genuinely fixed (nothing to dismiss).Not in scope (follow-ups): auth + TLS +
--bind(needed before serving on anything but localhost); batching cold chunk reads on remote repos.Caveats
?tardirectory download above, orborg extract/borg export-tar, for full-fidelity restores.?tardownloads.FileSizeLimitInBytesregistry value); browsers or other clients are not affected.Testing
13 tests start the server in-process: browsing + escaping of hostile file names, breadcrumbs, single/multi-chunk downloads, non-ASCII file names (percent-encoding exercised on every platform), hardlinks, PROPFIND structure and edge cases (incl. UTF-8 and UTF-16 XML entity expansion payloads), ranges (incl. chunk-boundary crossing), conditional GET, the data cache, the
?tardirectory download (structure, content, preserved metadata, path stripping, HEAD), header-injection attempts, 404/405/403 behavior. Tests that need file names illegal on Windows (<>and CR/LF) are guarded/skipped there; the escaping code stays fully covered on Linux and macOS.Verified against real clients: rclone (recursive PROPFIND listing, reads), the macOS WebDAV file system (
mount_webdav: mount, ls, byte-identical reads of multi-chunk files, write attempts correctly rejected with EROFS), and the systemtartool for?tardownloads (modes, symlink targets and a multi-chunk file preserved byte-identically).🤖 Generated with Claude Code