There's a memory leak problem in Curl::Easy.download

[liudangyi]

I use the following code.

photos.each do |photo|
  filename = "../thumbs/#{photo[:id]}.jpeg"
  Curl::Easy.download(photo[:thumb_url], filename) unless File.exist? filename
end

And it quickly (about 50M/s) eats up my memory.

[daedric]

This path would probably fix it but since the current master segfault on my laptop I will not create a pull request

diff --git a/ext/curb_multi.c b/ext/curb_multi.c
index db9a35a..281ff52 100644
--- a/ext/curb_multi.c
+++ b/ext/curb_multi.c
@@ -68,7 +68,7 @@ rb_hash_clear_i(VALUE key, VALUE value, VALUE dummy) {

 static void curl_multi_free(ruby_curl_multi *rbcm) {

-  if (rbcm && !rbcm->requests == Qnil && rb_type(rbcm->requests) == T_HASH && RHASH_SIZE(rbcm->requests) > 0) {
+  if (rbcm && !NIL_P(rbcm->requests) && rb_type(rbcm->requests) == T_HASH && RHASH_SIZE(rbcm->requests) > 0) {

     rb_hash_foreach( rbcm->requests, (int (*)())curl_multi_flush_easy, (VALUE)rbcm );

[taf2]

Thanks for finding this definitely looks like a bug... and your patch makes good sense. looking into why it is now causing a segfault in your example

[taf2]

Do you have a sample test case I can run to repeat the crash?

[taf2]

I believe this is fixed now in 0.9.0 please confirm.

[HolyWalley]

As I can see it's not fixed

curb (0.9.11)

puts "MEMORY USAGE: %d MB" % (`ps -o rss= -p #{Process.pid}`.to_i / 1024)
=> MEMORY USAGE: 107 MB

Curl::Easy.download(direct_link_to_large_file)

puts "MEMORY USAGE: %d MB" % (`ps -o rss= -p #{Process.pid}`.to_i / 1024)
=> MEMORY USAGE: 139 MB

in my case file size was 32MB which is eq to difference in memory, so, I suppose it's been loaded into memory somewhere.

I experimented with file on our corporate s3, so, can't share direct link here.