@rspfile #17

New Issue

claunia · 2026-01-29T22:23:17Z

claunia commented

2026-01-29 22:23:17 +00:00

Originally created by @SilvanScherrer on GitHub (Jan 28, 2020).

Originally assigned to: @dmik on GitHub.

When building something with
gcc -Zomf -Zexe -Zlinker "DISABLE 1121" test.c
all works w/o issues.
But when we use a rspfile like the following:
gcc -Zomf -Zexe -Zlinker "DISABLE 1121" @object1.rsp
we get an error
Error! E30033: file ld6JgbBQ.: lie(10): directive error near 'DISABLE\'

Originally created by @SilvanScherrer on GitHub (Jan 28, 2020). Originally assigned to: @dmik on GitHub. When building something with `` gcc -Zomf -Zexe -Zlinker "DISABLE 1121" test.c `` all works w/o issues. But when we use a rspfile like the following: `` gcc -Zomf -Zexe -Zlinker "DISABLE 1121" @object1.rsp `` we get an error `` Error! E30033: file ld6JgbBQ.: lie(10): directive error near 'DISABLE\' ``

claunia closed this issue

2026-01-29 22:23:18 +00:00

claunia commented

2026-01-29 22:23:19 +00:00

@SilvanScherrer commented on GitHub (Jan 28, 2020):

with gcc4 all works as it should. we just verified that.

@SilvanScherrer commented on GitHub (Jan 28, 2020): with gcc4 all works as it should. we just verified that.

claunia commented

2026-01-29 22:23:20 +00:00

@komh commented on GitHub (Jan 28, 2020):

Split it. That is,

-Zlinker DISABLE -Zlinker 1121

See https://github.com/komh/kdllar/issues/3.

@komh commented on GitHub (Jan 28, 2020): Split it. That is, ```-Zlinker DISABLE -Zlinker 1121``` See https://github.com/komh/kdllar/issues/3.

claunia commented

2026-01-29 22:23:21 +00:00

@dmik commented on GitHub (Feb 6, 2020):

Nothing has been changed in GCC itself in this regard. Tthe gcc.exe frontend itself and all other relative executables uses expandargv from libiberty to process response files. This is the documentation for it:

@deftypefn Extension void expandargv (int *@var{argcp}, char ***@var{argvp})

The @var{argcp} and @code{argvp} arguments are pointers to the usual
@code{argc} and @code{argv} arguments to @code{main}.  This function
looks for arguments that begin with the character @samp{@@}.  Any such
arguments are interpreted as ``response files''.  The contents of the
response file are interpreted as additional command line options.  In
particular, the file is separated into whitespace-separated strings;
each such string is taken as a command-line option.  The new options
are inserted in place of the option naming the response file, and
@code{*argcp} and @code{*argvp} will be updated.  If the value of
@code{*argvp} is modified by this function, then the new value has
been dynamically allocated and can be deallocated by the caller with
@code{freeargv}.  However, most callers will simply call
@code{expandargv} near the beginning of @code{main} and allow the
operating system to free the memory when the program exits.

@end deftypefn

It clearly states that arguments in the response file are space-separated. EMX (LIBC), on the contrary, uses line endings as argument separators. Of course, this means that the libiberty approach can't handle options with spaces the way LIBC does them.

Libiberty hasn't changed in this regard between what's used in GCC4 and GCC9. This means that the only way this can work in GCC4 is that somehow EMX code is used (via a direct call to _response at the beginning of main) to handle response files instead of libiberty. What I don't understand is that from what I see in the GCC4 code, _response is called after expandargv so it should be the same.. But given there is a huge mess of everything in there (e.g. some macros used in EMX contain calls to _response), there must be an explanation to that. I just don't want to spend time digging it. I will try to fix itn in GCC9 by using _response directly there as well.

@dmik commented on GitHub (Feb 6, 2020): Nothing has been changed in GCC itself in this regard. Tthe `gcc.exe` frontend itself and all other relative executables uses `expandargv` from `libiberty` to process response files. This is the [documentation](https://github.com/bitwiseworks/gcc-os2/blob/fee94d14baa6b5e1a16b6a75a33340215a34f047/libiberty/argv.c#L346) for it: ``` @deftypefn Extension void expandargv (int *@var{argcp}, char ***@var{argvp}) The @var{argcp} and @code{argvp} arguments are pointers to the usual @code{argc} and @code{argv} arguments to @code{main}. This function looks for arguments that begin with the character @samp{@@}. Any such arguments are interpreted as ``response files''. The contents of the response file are interpreted as additional command line options. In particular, the file is separated into whitespace-separated strings; each such string is taken as a command-line option. The new options are inserted in place of the option naming the response file, and @code{*argcp} and @code{*argvp} will be updated. If the value of @code{*argvp} is modified by this function, then the new value has been dynamically allocated and can be deallocated by the caller with @code{freeargv}. However, most callers will simply call @code{expandargv} near the beginning of @code{main} and allow the operating system to free the memory when the program exits. @end deftypefn ``` It clearly states that arguments in the response file are space-separated. EMX (LIBC), on the contrary, uses line endings as argument separators. Of course, this means that the `libiberty` approach can't handle options with spaces the way LIBC does them. Libiberty hasn't changed in this regard between what's used in GCC4 and GCC9. This means that the only way this can work in GCC4 is that somehow EMX code is used (via a direct call to `_response` at the beginning of `main`) to handle response files instead of libiberty. What I don't understand is that from what I see in the GCC4 code, `_response` is called *after* `expandargv` so it should be the same.. But given there is a huge mess of everything in there (e.g. some macros used in EMX contain calls to `_response`), there must be an explanation to that. I just don't want to spend time digging it. I will try to fix itn in GCC9 by using `_response` directly there as well.

claunia commented

2026-01-29 22:23:22 +00:00

@dmik commented on GitHub (Feb 6, 2020):

What I don't like here is that _response is not compatible with libiberty (used everywhere in gcc and everywhere else in the Unix world) per se. So it's a potential source of future problems due to incompatibility. But in this particular case I think that keeping GCC for OS/2 compatible to its own older OS/2 builds is more important.

@dmik commented on GitHub (Feb 6, 2020): What I don't like here is that `_response` is not compatible with `libiberty` (used everywhere in gcc and everywhere else in the Unix world) per se. So it's a potential source of future problems due to incompatibility. But in this particular case I think that keeping GCC for OS/2 compatible to its own older OS/2 builds is more important.

claunia commented

2026-01-29 22:23:23 +00:00

@dmik commented on GitHub (Feb 6, 2020):

OTOH, I see that libiberty allows preserving white space with either double or single quotes (as well as with backslashes but that's disabled on OS/2 because they are valid path separators). This means that having -Zlinker "DISABLE 1121" in a response file or on the command line should work even with libiberty. Somehow OS/2 bits in gcc's main break this. This needs to be found out.

Note that things get screwed up only when @file is present on the gcc command line. This means, that this works:

gcc -Zomf hello.c -Zlinker "DISABLE 1121"

while this doesn't:

gcc -Zomf hello.c -Zlinker "DISABLE 1121" @hello.rsp

even if hello.rsp itself doesn't contain -Zlinker "DISABLE 1121" or such.

@dmik commented on GitHub (Feb 6, 2020): OTOH, I see that `libiberty` allows preserving white space with either double or single quotes (as well as with backslashes but that's disabled on OS/2 because they are valid path separators). This means that having `-Zlinker "DISABLE 1121"` in a response file or on the command line should work even with `libiberty`. Somehow OS/2 bits in gcc's `main` break this. This needs to be found out. Note that things get screwed up only when `@file` is present on the gcc command line. This means, that this works: ``` gcc -Zomf hello.c -Zlinker "DISABLE 1121" ``` while this doesn't: ``` gcc -Zomf hello.c -Zlinker "DISABLE 1121" @hello.rsp ``` even if `hello.rsp` itself doesn't contain `-Zlinker "DISABLE 1121"` or such.

claunia commented

2026-01-29 22:23:23 +00:00

@dmik commented on GitHub (Feb 6, 2020):

I think I know what it is. It's quite complicated.

GCC uses libiberty not only to parse response files via expandargv but also to create them via writeargv.
GCC uses response files to pass some options to tools it runs (including the linker) if and only if a response file was used in its own invocation (as I guess, assuming that if it's needed for gcc, it is needed for tools as well). This is done for input files and for output files (%i and %o in GCC SPEC).
Linker options (including -Zlinker) somehow appear as part of the %o option substitution.
The difference between GCC 4 and GCC 9 is that the former only uses response files for %o options if HAVE_GNU_LD is defined. In GCC 9, they removed a check for HAVE_GNU_LD.
OS/2 builds have HAVE_GNU_LD defined (as configure somehow passes this test) but GCC 4 builds explicitly disabled HAVE_GNU_LD in gcc.c — in order to specifically stop from using response files for linker options, I suppose.
Since GCC 9 removed HAVE_GNU_LD usage from %o processing, we got response files for linker options with this ticket's problem. Note that its OS/2 override was also removed from gcc.c (as not used anywwhere).

An obvious fix is to bring HAVE_GNU_LD back into gcc.c. I really wonder why GCC devs removed. But in order to know that, I have to make a full clone of the GCC repo which is enormously big. Sigh...

From what I see, HAVE_GNU_LD is used in other GCC sources specifically to guard places dealing with response files. From what I get their logic is as follows: If the linker is not ld from binutils, it will most likely know nothing about libiberty response files so better not use them. This is a fair assumption and it worked really well in case of emxomfld which tries to mimic ld but doesn't use libiberty response files. In fact, it uses LIBC own response files (_response/-Zargs-resp) which, as pointed out above, is not compatible with libiberty ones.

Another fix to that problem would be to use libiberty instead of LIBC for response files in all EMX tools (emxomfld, emxomf, emxexp and alike) but this may break some makefiles so I'm not sure if we should do that. I will create a ticket though.

As a matter of fact, GCC for OS/2 itself has been always using libiberty response files so making it use LIBC ones may break some makefiles too. This is really so weird that gcc and the rest of EMX disagree here. Was certainly NOT a wise decision back there to create such a terrible mix.

Another problem is that the GCC version of libiberty disables using of backslash as an escape character when reading response files — for obvious reasons of it being a valid path separator under OS/2. However, it doesn't do so when writing response files. I can actually make it use quotes instead but this will not fix the problem as LIBC response file processing doesn't give any special treating to quotes.

So complicated.

@dmik commented on GitHub (Feb 6, 2020): I think I know what it is. It's quite complicated. 1. GCC uses libiberty not only to parse response files via `expandargv` but also to create them via `writeargv`. 2. GCC uses response files to pass some options to tools it runs (including the linker) if and only if a response file was used in its own invocation (as I guess, assuming that if it's needed for gcc, it is needed for tools as well). This is done for input files and for output files (%i and %o in GCC SPEC). 3. Linker options (including -Zlinker) somehow appear as part of the %o option substitution. 4. The difference between GCC 4 and GCC 9 is that the former only uses response files for %o options if HAVE_GNU_LD is defined. In GCC 9, they removed a check for HAVE_GNU_LD. 5. OS/2 builds have HAVE_GNU_LD defined (as configure somehow passes this test) but GCC 4 builds explicitly disabled HAVE_GNU_LD in `gcc.c` — in order to specifically stop from using response files for linker options, I suppose. 6. Since GCC 9 removed HAVE_GNU_LD usage from %o processing, we got response files for linker options with this ticket's problem. Note that its OS/2 override was also removed from `gcc.c` (as not used anywwhere). An obvious fix is to bring HAVE_GNU_LD back into `gcc.c`. I really wonder why GCC devs removed. But in order to know that, I have to make a full clone of the GCC repo which is enormously big. Sigh... From what I see, HAVE_GNU_LD is used in other GCC sources specifically to guard places dealing with response files. From what I get their logic is as follows: If the linker is not `ld` from [binutils](https://github.com/bitwiseworks/binutils-os2), it will most likely know nothing about libiberty response files so better not use them. This is a fair assumption and it worked really well in case of `emxomfld` which tries to mimic `ld` but doesn't use libiberty response files. In fact, it uses LIBC own response files (`_response`/`-Zargs-resp`) which, as pointed out above, is not compatible with libiberty ones. Another fix to that problem would be to use libiberty instead of LIBC for response files in all EMX tools (`emxomfld`, `emxomf`, `emxexp` and alike) but this may break some makefiles so I'm not sure if we should do that. I will create a ticket though. As a matter of fact, GCC for OS/2 itself has been always using libiberty response files so making it use LIBC ones may break some makefiles too. This is really so weird that gcc and the rest of EMX disagree here. Was certainly **NOT** a wise decision back there to create such a terrible mix. Another problem is that the GCC version of libiberty disables using of backslash as an escape character when reading response files — for obvious reasons of it being a valid path separator under OS/2. However, it doesn't do so when writing response files. I can actually make it use quotes instead but this will not fix the problem as LIBC response file processing doesn't give any special treating to quotes. So complicated.

claunia commented

2026-01-29 22:23:25 +00:00

@dryeo commented on GitHub (Feb 6, 2020):

At some point, response files had to be added to calling emxomfld IIRC
(might have been emxomfar) due to Mozilla builds breaking as using a
pipe was running into a 64kb limit. IIRC, it was Yuri or Paul who added
the support.

@dryeo commented on GitHub (Feb 6, 2020): At some point, response files had to be added to calling emxomfld IIRC (might have been emxomfar) due to Mozilla builds breaking as using a pipe was running into a 64kb limit. IIRC, it was Yuri or Paul who added the support.

claunia commented

2026-01-29 22:23:25 +00:00

@dryeo commented on GitHub (Feb 6, 2020):

The related bug, http://trac.netlabs.org/libc/ticket/216 though it
doesn't show Yuri's work.

@dryeo commented on GitHub (Feb 6, 2020): The related bug, http://trac.netlabs.org/libc/ticket/216 though it doesn't show Yuri's work.

claunia commented

2026-01-29 22:23:27 +00:00

@dmik commented on GitHub (Feb 6, 2020):

Thanks for the feedback, Dave. Although I really doubt that using response files in GCC when running subprograms was added because of this ticket (or because of the OS/2 limitation). Most likely the GCC folks needed it on Windows (mingw) or such. Anyway, this sheds some light on why they could do that.

But the disagreement between EMX tools and GCC wrt response files I mention above has nothing to do with this particular problem I suppose. I believe EMX tools have been using LIBC response files since the very beginning (and GCC has been using libiberty since its very beginning too).

However, I now think that changing EMX tools to use libiberty instead is pretty safe. I doubt there are any makefiles that manually create response files for those tools. And GCC will handle it nicely anyway. I will think on that more.

Just for the record, HAVE_GNU_LD was removed by this original commit e7208389c8 (original gitweb). No explanation in the commit itself on why it was done so. Perhaps, the commit author simply didn't know why it originally appeared there.

@dmik commented on GitHub (Feb 6, 2020): Thanks for the feedback, Dave. Although I really doubt that using response files in GCC when running subprograms was added because of this ticket (or because of the OS/2 limitation). Most likely the GCC folks needed it on Windows (mingw) or such. Anyway, this sheds some light on why they could do that. But the disagreement between EMX tools and GCC wrt response files I mention above has nothing to do with this particular problem I suppose. I believe EMX tools have been using LIBC response files since the very beginning (and GCC has been using libiberty since its very beginning too). However, I now think that changing EMX tools to use libiberty instead is pretty safe. I doubt there are any makefiles that manually create response files for those tools. And GCC will handle it nicely anyway. I will think on that more. Just for the record, HAVE_GNU_LD was removed by this original commit https://github.com/gcc-mirror/gcc/commit/e7208389c8381feaf8c7c60d975b06c446978006 (original [gitweb](https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=e7208389c8381feaf8c7c60d975b06c446978006)). No explanation in the commit itself on why it was done so. Perhaps, the commit author simply didn't know why it originally appeared there.

claunia commented

2026-01-29 22:23:27 +00:00

@dmik commented on GitHub (Jun 18, 2020):

I'm reopening this as I've found the source of the problem. It's the following libiberty code:

7d9b429416/libiberty/pex-djgpp.c (L218)

In short: GCC starts driver processes (including emxomfld.exe) with the help of libiberty's pex_run_in_environment. And on EMX its worker detects if the size of arguments is going to be larger than 30K (DosExecPgm has a limit of 32K), and if so, creates a response file on its own.

However, GCC doesn't know about that and decides to create its own response file under some circumstances (see the comments above). As a result, emxomfld ends up with two response files: one created by libiberty and passed to it as its only argument (it has a name $TEMP/grspXXXX.tmp btw) and the other one created by GCC ($TEMP/xxxxxxxx) which is put, among all other arguments, into the libiberty's response file (i.e. a nested response file). Nested response files are not supported by LIBC (and by GCC/libiberty too IIRC). Hence, after calling LIBC's _response to resolve any possible response files, emxomfld ends up with @$/TMP/xxxxxxxx among its arguments (note the leading @!) which it treats as an input file. And, of course, _abspath fails on that file with "File not found" because it's in fact a non-existing file.

The solution here is to disable response file creation by GCC completely on EMX since this is done as needed by libiberty when starting child driver processes.

@dmik commented on GitHub (Jun 18, 2020): I'm reopening this as I've found the source of the problem. It's the following libiberty code: https://github.com/bitwiseworks/gcc-os2/blob/7d9b429416ea417d73c6a5b1d472a09bea0681ee/libiberty/pex-djgpp.c#L218 In short: GCC starts driver processes (including `emxomfld.exe`) with the help of libiberty's `pex_run_in_environment`. And on EMX its worker detects if the size of arguments is going to be larger than 30K (DosExecPgm has a limit of 32K), and if so, creates a response file on its own. However, GCC doesn't know about that and decides to create its own response file under some circumstances (see the comments above). As a result, `emxomfld` ends up with two response files: one created by libiberty and passed to it as its only argument (it has a name `$TEMP/grspXXXX.tmp` btw) and the other one created by GCC (`$TEMP/xxxxxxxx`) which is put, among all other arguments, into the libiberty's response file (i.e. a nested response file). Nested response files are not supported by LIBC (and by GCC/libiberty too IIRC). Hence, after calling LIBC's `_response` to resolve any possible response files, `emxomfld` ends up with `@$/TMP/xxxxxxxx` among its arguments (note the leading `@`!) which it treats as an input file. And, of course, `_abspath` fails on that file with "File not found" because it's in fact a non-existing file. The solution here is to disable response file creation by GCC completely on EMX since this is done as needed by libiberty when starting child driver processes.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bitwiseworks/gcc-os2#17