how do I download a large file (via HTTP) in .NET

I need to download a LARGE file (2GB) over HTTP in a C# console app. Problem is, after about 1.2GB, the app runs out of memory.

Here’s the code I’m using:

WebClient request = new WebClient();
request.Credentials = new NetworkCredential(username, password);
byte[] fileData = request.DownloadData(baseURL + fName);

As you can see… I’m reading the file directly into memory. I’m pretty sure I could solve this if I were to read the data back from HTTP in chunks and write it to a file on disk.

Does anyone know how I could do this?

How to download a binary file via Net::HTTP::Get?

I am trying to download a binary file via HTTP using the following Ruby script. #!/usr/bin/env ruby require ‘net/http’ require ‘uri’ def http_download(resource, filename, debug = false) uri = URI.pars

How do I download a file over HTTP using Ruby?

How do I download a file over HTTP using Ruby?

How do I download a binary file over HTTP?

How do I download and save a binary file over HTTP using Ruby? The URL is http://somedomain.net/flv/sample/sample.flv. I am on the Windows platform and I would prefer not to run any external program.

How to download a certain byte of the file via http

How to download first 125 bytes and 125 last byte file via HTTP protocol ?

How do i download a file with CA-VO

I’d like to download a file using HTTP. How do I do it?

How do I download a CSV file from a web server to a local machine using HTTP? [duplicate]

This question already has an answer here: How to download a file from a website in C# 6 answers How do I download a CSV file from a web server to a local machine using HTTP?

How do I download an msi file via an asp.net button?

So, I’ve created my wonderful winforms app that I want to unleash upon the world, and now I am trying to create a simple website to host some basic information and link to the setup file (msi installe

How do I download a file over HTTP using Python?

I have a small utility that I use to download an MP3 from a website on a schedule and then builds/updates a podcast XML file which I’ve obviously added to iTunes. The text processing that creates/upda

How do I download a file via default Android Downloader?

How can I download files using Android downloader? (The downloader that WebBrowser is using that too). I tried something like this : Intent i = new Intent(Intent.ACTION_VIEW , Uri.parse(MyUrl)); sta

How can I download a single file from multiple locations via HTTP?

I need to download a big file quickly, but all sources I can find have throttled bandwidth. Each of them seem to support HTTP 1.1 Byte Serving (Range Requests), since I can pause and resume the downlo

Answers

You need to get the response stream and then read in blocks, writing each block to a file to allow memory to be reused.

As you have written it, the whole response, all 2GB, needs to be in memory. Even on a 64bit system that will hit the 2GB limit for a single .NET object.


Update: easier option. Get WebClient to do the work for you: with its DownloadFile method which will put the data directly into a file.

The WebClient class is the one for simplified scenarios. Once you get past simple scenarios (and you have), you’ll have to fall back a bit and use WebRequest.

With WebRequest, you’ll have access to the response stream, and you’ll be able to loop over it, reading a bit and writing a bit, until you’re done.


Example:

public void MyDownloadFile(Uri url, string outputFilePath)
{
    const int BUFFER_SIZE = 16 * 1024;
    using (var outputFileStream = File.Create(outputFilePath, BUFFER_SIZE))
    {
        var req = WebRequest.Create(url);
        using (var response = req.GetResponse())
        {
            using (var responseStream = response.GetResponseStream())
            {
                var buffer = new byte[BUFFER_SIZE];
                int bytesRead;
                do
                {
                    bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE);
                    outputFileStream.Write(buffer, 0, bytesRead);
                } while (bytesRead > 0);
            }
        }
    }
}

Note that if WebClient.DownloadFile works, then I’d call it the best solution. I wrote the above before the “DownloadFile” answer was posted. I also wrote it way too early in the morning, so a grain of salt (and testing) may be required.

If you use WebClient.DownloadFile you could save it directly into a file.

i would use something like this

WebClient.OpenRead returns a Stream, just use Read to loop over the contents, so the data is not buffered in memory but can be written in blocks to a file.