Define USE_CURL_MULTI for git-core

[svoop@…]

The git-core port configures http.h with "#undef USE_CURL_MULTI" which makes it impossible to use "git push" over HTTP(S). This was introduced about a year ago, see ticket #12008 for details.

I've overridden this and compiled git-core with "#define USE_CURL_MULTI". Config and compile worked fine and "git push" over HTTPS is possible without error. (Tested on a x86 MacBook Pro with OS X 10.5.x.)

Could you change the default of USE_CURL_MULTI in case there are no other blockers? Thanks!

[jmr@…]

Cc maintainer.

[raimue@…]

There is patch-http.h.diff which explicitely removes the USE_CURL_MULTI option. I don't know if there is/was a valid reason to do so?

[thierry.moisan@…]

Removing patch-http.h.diff makes git-core works as expected. I tested it.

[bryan@…]

ticket 12008 did not introduce the patch: 12008 was simply a refactoring of previous patches to match the macports policy. The patches used to be arranged by function, now they are arranged by file.

USE_CURL_MULTI used to be horribly broken, causing severe data corruption.

Curl v7.16.0 was supposed to make USE_CURL_MULTI functional, but that caused severe data corruption, but on a much less frequent basis. Reproducing the bug required a very large checkout on a powerpc mac. By very large, I mean much larger than linux-kernel: I used an embedded linux distribution (which included 3 different linux kernel sources, 3 different libc's and hundreds of apps) for my testing. Intel macs appeared to work fine.

Curl v7.16.2 may have fixed the problem, but the decision was made not to trust it since nobody had the days of effort required to do an exhaustive test. At that time, disabling multi only slowed down checkouts.

We've had 8 releases of Curl since 7.16.0, most of which include some sort of patch to multi. So it's probably working fine now. Unfortunately, testing to make sure takes hours. Does somebody have the time to do so?

[bryan@…]

Has anybody used large repositories with CURL_MULTI turned on over the last few months? Has anybody used PPC?

[maccheck@…]

Cc Me!

[maccheck@…]

I removed patch-http.h.diff in my Portfile update for git 1.6.0.3 (see #16955). It seems that git itself now has a thorough check for the different curl versions.

[bryan@…]

I double checked the git version check, this is the same version check that has always been there. It assumes that 7.16.0 is good, which isn't (see above).

I might have some time tomorrow to do some testing. My iBook fan has stopped squealing, so I again have access to a PowerPC mac. I no longer have access to the repository causing the problems. (Local http pulls never exhibited the problems). Is anybody aware of a very large repository?

[maccheck@…]

Maybe the new Android open source repo? This is based on git. I cannot help as I sold my last PPC Mac 2 month ago... :-(

[maccheck@…]

Replying to bryan@…:

I double checked the git version check, this is the same version check that has always been there. It assumes that 7.16.0 is good, which isn't (see above). I might have some time tomorrow to do some testing. My iBook fan has stopped squealing, so I again have access to a PowerPC mac. I no longer have access to the repository causing the problems. (Local http pulls never exhibited the problems). Is anybody aware of a very large repository?

Bryan, did you have the time to test a large repo with your PPC machine? Another large repo should be the X.org project at git://anongit.freedesktop.org/git/cairo. I had no issues to clone this with my Intel machine.

BTW: stock MacPorts curl ist now @7.19.0_0. Maybe this issue doesn't exist anymore here?

[bryan@…]

I finally got all of the Android prereqs installed, and downloaded the Android source. They use a large number of small - medium size repositories, and use the git: protocol to access the repositories. I am going to try and see if I could change the repository and try again.

I tried the http protocol on the cairo repository, but I got a message saying that they're down for maintenance. Hopefully that's a real message and it comes up later. But it's only a 25meg repository. The repository that was causing problems in the past was on the order of 500 megabytes.

It's quite likely the problem doesn't exist anymore. But it is possible it hasn't been fixed, and the problem was fairly serious. Give me until Tuesday to do some more testing. If I can't find enough time to do some solid testing by then, maybe we should just let CURL_MULTI through -- what are the chances anyone's going to work on a 500Meg repository over HTTP on a PowerPC? Much less than somebody wanting to push to an HTTP repository.

[maccheck@…]

Sorry, I didn't look close enough and hadn't realized that these repos used solely the git protocol. So we need a HTTP mirror of the Linux kernel? That should be big enough, shouldn't it!? ;-)

If I find a link I'll post it here...

[bryan@…]

The tree I used last time was the source for complete embedded linux distribution. It included 3 linux kernels, 3 different libc's and the source to approximately 200 different applications. So it was an order of magnitude bigger than just the linux kernel. Although, the kernel seems to have grown substantially in the meantime.

Sometimes I could pull the whole thing without an error - I got an error about 50% of the time, so it was the right size for testing. I was able to successfully pull a linux kernel continuously in a loop overnight -- smaller repositories never triggered the error.

I'm still looking for big repositories. Here's a good place to start: http://git.or.cz/gitwiki/GitHosting. None of them seem to list sizes. The biggest I've found so far is kdebase, which is slightly bigger than the kernel. Let me know if you find something bigger.

The biggest .git repository in the android tree is the linux kernel.

[bryan@…]

It appears that the bug is still active. I'm going to follow up upstream.

[maccheck@…]

Bryan, if you can confirm that this curl problem still exists, please have a look at the newly submitted changes in #16955. I undid the removal of the http.h patch there. I should work again for you (hopefully). At least we could release version 1.6.0.3 and investigate this issue here simultaneously.

[bryan@…]

Never mind. It's not USE_CURL_MULTI. It appears that HTTP in git is just broken, period.

[maccheck@…]

So my final summary would be that it's okay to remove the http.h patch from the git-core Portfile. I will update #16955 accordingly.