Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Problem with msvcrt60 vs. msvcr71 vs. strdup/free

Reply
Thread Tools

Problem with msvcrt60 vs. msvcr71 vs. strdup/free

 
 
Gerhard Haering
Guest
Posts: n/a
 
      12-21-2004
Hello,

I used to build Python extension modules with mingw. Now, Python has
switched to the MSVCR71 runtime with version 2.4, and I thought mingw
has support for this. But I get problems with symbols being referenced
from the wrong DLLs.

You can see the problem by compiling this:

##################
#include <string.h>

int main()
{
char* s;
int i;


for (i = 0; i < 10; i++) {
s = strdup("foo");
free(s);
}

return 0;
}
##################

with gcc x.c -lmsvcr71

Then if you run a.exe it crashes.

If you use depends.exe on it, you see that it resolves strdup() via
msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
the one malloc, but free() a different free() from the other DLL,
which is undoubtedly the reason for the crash.

Is there any way I can force mingw to not link in msvcr for things
like strdup?

I think I found a very strange and ugly workaround: if I use _strdup()
instead of strdup(), it is resolved using the msvcr71 DLL.

I hope it is (soon) possible to use only the msvcr71 runtime from the
mingw compiler.

-- Gerhard
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFByCmWdIO4ozGCH14RAu0nAJwKYhMLulwl52UltcFk00 sy55yEpQCfaTsS
Ba8Q5W8g23lRe/VM/K/kBCU=
=L8+w
-----END PGP SIGNATURE-----

 
Reply With Quote
 
 
 
 
Thomas Heller
Guest
Posts: n/a
 
      12-21-2004
Gerhard Haering <(E-Mail Removed)> writes:

> Hello,
>
> I used to build Python extension modules with mingw. Now, Python has
> switched to the MSVCR71 runtime with version 2.4, and I thought mingw
> has support for this. But I get problems with symbols being referenced
> from the wrong DLLs.
>
> You can see the problem by compiling this:
>
> ##################
> #include <string.h>
>
> int main()
> {
> char* s;
> int i;
>
>
> for (i = 0; i < 10; i++) {
> s = strdup("foo");
> free(s);
> }
>
> return 0;
> }
> ##################
>
> with gcc x.c -lmsvcr71
>
> Then if you run a.exe it crashes.
>
> If you use depends.exe on it, you see that it resolves strdup() via
> msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
> the one malloc, but free() a different free() from the other DLL,
> which is undoubtedly the reason for the crash.
>
> Is there any way I can force mingw to not link in msvcr for things
> like strdup?


Only guesswork, but replacing this section in the
lib\gcc-lib\mingw32\3.2.3\specs file (I can only guess wht this file does)

*libgcc:
%{mthreads:-lmingwthrd} -lmingw32 -lgcc -lmoldname -lmingwex -lmsvcrt

with this one:

*libgcc:
%{mthreads:-lmingwthrd} -lmingw32 -lgcc -lmoldname -lmingwex -lmsvcr71

seemed to do the trick.

Thomas
 
Reply With Quote
 
 
 
 
abkhd@earth.co.jp
Guest
Posts: n/a
 
      12-22-2004
From: "" <(E-Mail Removed)>
Newsgroups: comp.lang.python
Subject: Re: Problem with msvcrt60 vs. msvcr71 vs. strdup/free
Date: Wed, 22 Dec 2004 11:07:02 -0800

Gerhard Haering wrote:
> Hello,
>
> I used to build Python extension modules with mingw. Now, Python has
> switched to the MSVCR71 runtime with version 2.4, and I thought mingw
> has support for this. But I get problems with symbols being

referenced
> from the wrong DLLs.
>
> You can see the problem by compiling this:
>
> ##################
> #include <string.h>
>
> int main()
> {
> char* s;
> int i;
>
>
> for (i = 0; i < 10; i++) {
> s = strdup("foo");
> free(s);
> }
>
> return 0;
> }
> ##################
>
> with gcc x.c -lmsvcr71
>
> Then if you run a.exe it crashes.
>
> If you use depends.exe on it, you see that it resolves strdup() via
> msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
> the one malloc, but free() a different free() from the other DLL,
> which is undoubtedly the reason for the crash.
>
> Is there any way I can force mingw to not link in msvcr for things
> like strdup?



Hello


Believe it or not I do not get a crash here.

Using objdump on a.exe I get:

[CUT]

00005014 00005084 00000000 00000000 000052b8 00005100

DLL Name: msvcrt.dll
vma: Hint/Ord Member-Name Bound-To
51b4 81 _strdup

00005028 00005090 00000000 00000000 000052f8 0000510c

DLL Name: msvcr71.dll
vma: Hint/Ord Member-Name Bound-To
51c0 59 __getmainargs
51d0 82 __p__environ
51e0 105 __set_app_type
51f4 154 _cexit
5200 211 _fileno
520c 222 _fmode
5218 225 _fpreset
5224 273 _iob
522c 390 _onexit
5238 435 _setmode
5244 599 atexit
5250 634 free
5258 715 signal

0000503c 000050cc 00000000 00000000 00005314 00005148

DLL Name: msvcrt.dll
vma: Hint/Ord Member-Name Bound-To
5264 510 abort
526c 537 fflush
5278 546 fprintf
5284 603 malloc

00005050 00000000 00000000 00000000 00000000 00000000


[CUT]


Having said that, let me add that I know where you are coming from. And
even though your a.exe did not crash with me now, I know that other
dlls and programs, in my case, did.

And let it be known that unfortunately the measures suggested by the
good people here do not always work. Changing the entry in the specs
file works most of the time, but not on mmap.pyd (of the official
Python source distribution) for example, where a call to some _lseek of
some sort (I forgot the exact call) is still linked from msvcrt.dll not
from msvcr71. And yes the test_mmap crashes if I compile it that way.

Also changing the specs file seems to work only when the linking is
simple. But if the linking gets complex (as in linking many objects
files to build, say, pythonXY.dll) then even the specs change will not
help. To illustrate, and in trying to get the pyMinGW built Python to
link with msvcr71 that way, one ends up with a pythonXY.dll that
references both runtime libraries: msvcrt, and msvcr71 with strange
consequences. This particular version of Python crashes on site import
hallucinating about (and I paraphrase!) "Null result from PY...XXX
without Error."

On MSDN, it is stated: "If your DLLs pass CRT resources across the
msvcrt.dll and msvcr71.dll boundary, you will encounter issues with
mismatched CRTs and need to recompile your project with Visual C++
..NET." [1]

I have a gut feeling that "DLLs" here does not mean inter-dll passing
of resources, but intra-dll passing of resources. In other words that
if the self-same DLL uses both run-times then you are in trouble; but I
don't think that this relates to the situation where one DLL is using
msvcrt and another DLL is using msvcr71. Please bear with me.

In this light, I truly don't know why would we try, or recommend
others, to link with libmsvcr71.a, if some mix-up is still possible. I
have yet to see a crash occur when Python24.dll or an extension of the
same uses the old way of linking with libmsvcrt.a. That way we are
assured that no references are made to msvcr71 when msvcrt is the one
used when linking.

I could be mistaken, but I think people are confusing the behavior of
the MS compilers with MinGW's. The old MS compiler (V. 6) uses
msvcrt.lib by default, and even when the newer MSVS uses the same name
for the its default runtime, that msvcrt link implicitly means the
newer msvcr71 [2]. This behind the scenes behavior is what seems to
have people mixed up. You see if one has older projects compiled using
V. 6, then linking these object files to other object files compiled
with the newer MSVS could effectively mean that the resulting DLL will
have references to both run-times, something by the way is also
mentioned in the MSDN resources [3].

But as far as I know this is not the case with regards to MinGW. MinGW
explicitly uses msvcrt, and so old and newer projects should in theory
have no problems if they stick to msvcrt, as opposed to going the MS
compilers' path-- shooting ourselves in the foot-- and risking linking
to two run-times at the same time.

I have tried changing the specs, and changing the location of msvcr71
and the results are never 100% re-assuring. And so I have resorted to
linking with the good old msvcrt and I have not had a crash due to this
to date. Of course this is my humble experience and I could be mistaken
big time.

For what it is worth, however, my extension DLLs are clean in that they
reference msvcrt only and not msvcr71, and hence all the malloc, and
free and what have you are never shared with any other runtime. My
point being that lacking empirical evidence to the contrary, I think
this is how making extensions to Python 2.4 with MinGW should be: using
the default runtime libmsvcrt.a. It would be fantastic if the good
people at MinGW, or others, can get that hard-wiring issue of the
msvcrt resolved. But until that happens I think it is safer to extend
Python 2.4 the old way. It seems to work fine.



Regards,
Khalid






[1] C-RunTime Libraries from MSDN:
http://tinyurl.com/q20a

[2] Ibid.

[3] Ibid.

 
Reply With Quote
 
=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
Guest
Posts: n/a
 
      12-22-2004
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Believe it or not I do not get a crash here.


I can believe this. strdup does malloc, i.e. gets some
memory from the heap, and free puts it back into the heap.
Since this is a different CRT, it puts it back into
a different heap. This is a leak, but it should not cause
a crash.

> I have a gut feeling that "DLLs" here does not mean inter-dll passing
> of resources, but intra-dll passing of resources. In other words that
> if the self-same DLL uses both run-times then you are in trouble; but I
> don't think that this relates to the situation where one DLL is using
> msvcrt and another DLL is using msvcr71. Please bear with me.


I think you are wrong. What DLL invokes the CRT function does not matter
- that the CRT functions come from different DLLs matters. Each CRT
is loaded exactly once into an address space, regardless of how many
DLLs refer to it. However, different CRTs have different sets of global
variables, and this is what hurts.

E.g. malloc use a pointer to the free list in the heap, and so does
free. Each CRT has its own heap, and each has its own pointer to the
free list. Then, doing malloc in one CRT and free in the other does
not cause crashes (perhaps unless you use the debug CRT, which tries
to determine whether it owns the pointer being freed). The CRT will
put the memory into its own free list, and allocate it again if its
own malloc asks for it. However, if you repeatedly do malloc() in one
CRT and free() in another, you get a memory leak, since malloc() will
not find the memory that has been freed.

> In this light, I truly don't know why would we try, or recommend
> others, to link with libmsvcr71.a, if some mix-up is still possible. I
> have yet to see a crash occur when Python24.dll or an extension of the
> same uses the old way of linking with libmsvcrt.a. That way we are
> assured that no references are made to msvcr71 when msvcrt is the one
> used when linking.


This is very easy to create. Just do fopen() in an extension linked
with msvcrt.dll, then do PyRun_AnyFile. This will crash, because fread()
on a file opened in a different CRT will always crash.

> I could be mistaken, but I think people are confusing the behavior of
> the MS compilers with MinGW's. The old MS compiler (V. 6) uses
> msvcrt.lib by default, and even when the newer MSVS uses the same name
> for the its default runtime, that msvcrt link implicitly means the
> newer msvcr71 [2].


You are mistaken. The name of the import library is, and always was,
msvcrt.lib. However, different versions of msvcrt.lib refer to different
DLLs: msvcrt40.dll, msvcr70.dll, and msvcr71.dll.

> This behind the scenes behavior is what seems to
> have people mixed up. You see if one has older projects compiled using
> V. 6, then linking these object files to other object files compiled
> with the newer MSVS could effectively mean that the resulting DLL will
> have references to both run-times, something by the way is also
> mentioned in the MSDN resources [3].


I think you are misreading [3]: It explicitly says

"If you have a .lib or .obj file that needs to link to msvcrt.lib, then
you should *not* have to recompile it to work with the new msvcrt.lib in
Visual C++ .NET." (emphasis mine)

The only case where you can link a single DLL with different CRTs using
MS tools is when you combine different versions of the CRTs, e.g. debug
and non-debug, or DLL and non-DLL.

> But as far as I know this is not the case with regards to MinGW. MinGW
> explicitly uses msvcrt, and so old and newer projects should in theory
> have no problems if they stick to msvcrt, as opposed to going the MS
> compilers' path-- shooting ourselves in the foot-- and risking linking
> to two run-times at the same time.


Python extensions should never ever link with msvcrt.dll. As your
reference says:

'The msvcrt.dll is now a "known DLL," meaning that it is a system
component owned and built by Windows. It is intended for future use only
by system-level components. An application should use and redistribute
msvcr71.dll'

Now, a Python extension is released from the need to distribute
msvcr71.dll, since Python already does that. It should still link
with that DLL.

> I have tried changing the specs, and changing the location of msvcr71
> and the results are never 100% re-assuring. And so I have resorted to
> linking with the good old msvcrt and I have not had a crash due to this
> to date. Of course this is my humble experience and I could be mistaken
> big time.


You are.

> For what it is worth, however, my extension DLLs are clean in that they
> reference msvcrt only and not msvcr71, and hence all the malloc, and
> free and what have you are never shared with any other runtime.


How do you know? What if your extension returns a pointer malloc'ed by
msvcrt.dll, and returns it to python24.dll, and then python24.dll
frees the pointer, using free() from msvcr71.dll?

> My
> point being that lacking empirical evidence to the contrary, I think
> this is how making extensions to Python 2.4 with MinGW should be: using
> the default runtime libmsvcrt.a. It would be fantastic if the good
> people at MinGW, or others, can get that hard-wiring issue of the
> msvcrt resolved. But until that happens I think it is safer to extend
> Python 2.4 the old way. It seems to work fine.


Indeed, it *will* work most of the time, as you rarely pass resources
across CRTs in a typical Python extension. However, it is very hard
to tell from the source code of the extension if such resource passing
could ever happen (e.g. what about setlocale(), atexit(), malloc(),
....), so if you need something better than "seems to work", you really
need to make sure that you understand all the issues, and have a build
process that resolves them all.

Regards,
Martin
 
Reply With Quote
 
abkhd@earth.co.jp
Guest
Posts: n/a
 
      12-23-2004
Martin v. L÷wis wrote:
> Indeed, it *will* work most of the time, as you rarely pass resources
> across CRTs in a typical Python extension. However, it is very hard
> to tell from the source code of the extension if such resource

passing
> could ever happen (e.g. what about setlocale(), atexit(), malloc(),
> ...), so if you need something better than "seems to work", you

really
> need to make sure that you understand all the issues, and have a

build
> process that resolves them all.
>
> Regards,
> Martin





You know if I had not tried the example you gave, I'd probably still be
arguing: "but it works." Oh well. Not anymore.

For those who are interested, here is a patch to demo.c of the
\python\dist\src\Demo\embed\demo.c demonstrating what Martin has argued
about above. With such evidence at hand I guess it is safe to say that
the MinGW die-hards will have to improvise.

But seriously, and since I don't have the official Python 2.4
distribution yet, and since the pyMinGW compiled Python I use relies on
msvcrt, as opposed to msvcr71, I needed to reverse the roles here to
see the problem first hand: making the "extension" demo use msvcr71 to
see how it reacts with the home-made python24.dll which uses msvcrt.
And sure enough: crash.

Having said that, this seems to me not to be an issue for those who
have or can build Python and/or the extensions they need from sources.
Indeed recompiling the same "demo" to use msvcrt (to use the same
run-time library that my python24.dll uses) instead of msvcr71 yields
the expected correct results without crashes.

And so I suspect this approach (compiling sources yourself) can shield
people from this ugly mess, if they can build Python 2.4 and the
extensions and have them (core and extension) use one and the same
run-time library. But indeed this might be an issue for those who [1]
use MinGW as their first choice and [2] cannot or won't bother to build
their Python and/or extensions from sources, and hence have to rely on
binary distributions.


Regards
Khalid



--- demo.c Sat Nov 17 08:30:20 2001
+++ demo2.c Thu Dec 23 02:02:40 2004
@@ -34,6 +34,20 @@
/* Note that you can call any public function of the Python
interpreter here, e.g. call_object(). */

+ FILE *in;
+ int sts;
+
+ if ((in = fopen(argv[1], "rt")) == NULL) {
+ fprintf(stderr, "Cannot open program file.\n");
+ return 1;
+ }
+
+ printf("Working on %s\n", argv[1]);
+ sts = PyRun_AnyFile(in, argv[1]) != 0;
+ printf("sts: %d\n", sts);
+ fclose(in);
+
+
/* Some more application specific code */
printf("\nGoodbye, cruel world\n");

 
Reply With Quote
 
Scott David Daniels
Guest
Posts: n/a
 
      12-23-2004
(E-Mail Removed) wrote:
> And so I suspect this approach (compiling sources yourself) can shield
> people from this ugly mess, if they can build Python 2.4 and the
> extensions and have them (core and extension) use one and the same
> run-time library. But indeed this might be an issue for those who [1]
> use MinGW as their first choice and [2] cannot or won't bother to build
> their Python and/or extensions from sources, and hence have to rely on
> binary distributions.


I encourage anyone who gets further into solving the "How do I use
MinGW to build Python2.4 (and later) standard-distribution compatible
modules (.pyd s)?" question to share any clues they have. The MS
free compiler is useful to many, but not all of us.

I expect that MinGW is the most frequent choice for making Python
modules in the "free compiler outside of MS" universe. Let's make
that one work well, and (once we've done that) we can pass the lessons
on for other compiler is desired.

--Scott David Daniels
(E-Mail Removed)

p.s. I am able (and willing) to use the MS free compiler, but I have
sympathy for those who aren't. I'd like to see it possible to use
_lots_ of different compilers, but we are at the beginning of the
problem, not the end. If Intel's compiler is as much faster than
MS's as it used to be, I may eventually wind up using theirs anyway.
 
Reply With Quote
 
=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
Guest
Posts: n/a
 
      12-23-2004
Scott David Daniels wrote:
> I encourage anyone who gets further into solving the "How do I use
> MinGW to build Python2.4 (and later) standard-distribution compatible
> modules (.pyd s)?" question to share any clues they have. The MS
> free compiler is useful to many, but not all of us.


I think the simplest answer is "It works out of the box". Just do

python setup.py build --compiler=mingw32

If it fails, report what the failure is (the first failure should
be lack of libpython24.a, for which you can find build instructions
in Google).

Regards,
Martin
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
msvcr71.dll Konte Python 2 02-28-2007 02:49 PM
msvcr71.dll Konte Python 0 02-28-2007 09:40 AM
A solution to the MSVCRT vs MSVCR71 problem? sturlamolden Python 7 01-21-2007 06:55 PM
Missing MSVCR71.dll Hoop Python 2 08-08-2006 05:36 PM
Windows error message msvcr71.dll =?Utf-8?B?YmlsbA==?= Microsoft Certification 1 05-09-2004 09:55 AM



Advertisments