How to Use Wget with Chunked Transfer Encoding on Ubuntu

This article provides a concise guide on using the wget command-line tool within the Ubuntu operating system to download files from servers utilizing chunked transfer encoding. You will learn how wget manages this transmission method automatically and discover specific command flags that ensure download stability when dealing with streamed or dynamically generated content.

Understanding Chunked Transfer Encoding

Chunked transfer encoding is a mechanism used by web servers to send content over HTTP without knowing the total file size beforehand. This is common for dynamic content or large streams. Fortunately, the wget utility supports HTTP/1.1 standards natively, meaning it handles chunked transfer encoding automatically without requiring special configuration flags to enable the feature.

Basic Download Command

To download a file from a server using chunked encoding, you simply use the standard wget syntax. Open your Ubuntu terminal and enter the following command:

wget https://example.com/path/to/file

Wget will detect the Transfer-Encoding: chunked header in the HTTP response and assemble the chunks into a complete file on your disk automatically.

Ensuring Stability for Chunked Downloads

While wget handles the encoding automatically, connections involving chunked transfers can sometimes be unstable due to server timeouts or keep-alive issues. If your download hangs or fails prematurely, use the following flags to improve reliability.

Disable Keep-Alive

Some servers struggle to maintain persistent connections during chunked transfers. Disabling keep-alive forces wget to close and reopen connections if necessary, which can prevent hanging:

wget --no-keep-alive https://example.com/path/to/file

Adjust Timeout Settings

Chunked transfers may have pauses between data chunks that wget might interpret as a timeout. Increasing the timeout threshold ensures wget waits long enough for the next chunk to arrive:

wget --timeout=60 --dns-timeout=60 https://example.com/path/to/file

Resume Interrupted Downloads

If a chunked download is interrupted, you can resume it using the -c flag. Wget will attempt to reconcile the existing partial file with the server’s current state:

wget -c https://example.com/path/to/file

Verifying the Download

Once the process completes, verify that the file was assembled correctly. You can check the file type using the file command to ensure it is not corrupted:

file downloaded_filename

By using these standard commands and stability flags, you can effectively manage downloads involving chunked transfer encoding on Ubuntu without needing complex scripts or additional software.