diff options
| -rw-r--r-- | AUTHORS | 2 | ||||
| -rw-r--r-- | ChangeLog | 18 | ||||
| -rw-r--r-- | NEWS | 12 | ||||
| -rw-r--r-- | doc/make.texi | 177 | ||||
| -rw-r--r-- | job.c | 67 | ||||
| -rw-r--r-- | job.h | 8 | ||||
| -rw-r--r-- | main.c | 6 | ||||
| -rw-r--r-- | make.1 | 13 | ||||
| -rw-r--r-- | makeint.h | 6 | ||||
| -rw-r--r-- | tests/ChangeLog | 5 | ||||
| -rw-r--r-- | tests/scripts/features/output-sync | 111 |
11 files changed, 320 insertions, 105 deletions
@@ -37,7 +37,7 @@ GNU make porting efforts: Port to MS-DOS (DJGPP), OS/2, and MS-Windows (native/MinGW) by: DJ Delorie <dj@delorie.com> Rob Tulloh <rob_tulloh@tivoli.com> - Eli Zaretskii <eliz@is.elta.co.il> + Eli Zaretskii <eliz@gnu.org> Jonathan Grant <jg@jguk.org> Andreas Beuning <andreas.buening@nexgo.de> Earnie Boyd <earnie@uses.sf.net> @@ -1,3 +1,21 @@ +2013-04-28 Paul Smith <psmith@gnu.org> + + Implement a "per-job" output synchronization option. + + * main.c (decode_output_sync_flags): Recognize the new option. + * makeint.h (OUTPUT_SYNC_JOB): Add new values for "job" + * job.c (assign_child_tempfiles): In per-job mode, truncate the + temp file for re-use by the next job. + (sync_output): Don't close the temp files as we may still use them. + (free_child): Close the temp files here as we definitely don't + need them. + (new_job): In per-job output mode, sync_output() after each job. + * job.h (struct child): Avoid ifdefs. + * make.1: Add new options to the man page. + * doc/make.texi (Parallel Output): Break documentation on input + and output into separate sections for readability. Document the + new "job" and "none" modes. + 2013-04-27 Paul Smith <psmith@gnu.org> * job.c (construct_command_argv_internal): Fix oneshell support @@ -1,6 +1,6 @@ GNU make NEWS -*-indented-text-*- History of user-visible changes. - 16 April 2013 + 27 April 2013 See the end of this file for copyrights and conditions. @@ -31,13 +31,17 @@ http://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=101&set * New command line option: --output-sync (-O) enables grouping of output by target or by recursive make. This is useful during parallel builds to avoid - mixing output from different jobs together giving hard-to-understand results. - Original implementation by David Boyce <dsb@boyski.com>. Patch was reworked - by Frank Heckenbach <f.heckenbach@fh-soft.de>. + mixing output from different jobs together giving hard-to-understand + results. Original implementation by David Boyce <dsb@boyski.com>. + Reworked and enhanced by Frank Heckenbach <f.heckenbach@fh-soft.de>. + Windows support by Eli Zaretskii <eliz@gnu.org>. * New feature: The "job server" capability is now supported on Windows. Implementation contributed by Troy Runkel <Troy.Runkel@mathworks.com> +* New feature: The .ONESHELL capability is now supported on Windows. Support + added by Eli Zaretskii <eliz@gnu.org>. + * New feature: "!=" shell assignment operator as an alternative to the $(shell ...) function. Implemented for compatibility with BSD makefiles. WARNING: Backward-incompatibility! diff --git a/doc/make.texi b/doc/make.texi index ce8e07b..64be3b3 100644 --- a/doc/make.texi +++ b/doc/make.texi @@ -219,6 +219,11 @@ Recipe Execution * Choosing the Shell:: How @code{make} chooses the shell used to run recipes. +Parallel Execution + +* Parallel Output:: Handling output during parallel execution +* Parallel Input:: Handling input during parallel execution + Recursive Use of @code{make} * MAKE Variable:: The special effects of using @samp{$(MAKE)}. @@ -4057,48 +4062,16 @@ If there is nothing looking like an integer after the @samp{-j} option, there is no limit on the number of job slots. The default number of job slots is one, which means serial execution (one thing at a time). -When running several recipes simultaneously the output from each -recipe appears as soon as it is generated, with the result that -messages from different recipes may be interspersed. To avoid this -you can use the @samp{--output-sync} (@samp{-O}) option; if this -option is provided then the output from each recipe will be saved -until the recipe is complete, then written all at once. This ensures -that output from different recipes is not mixed together. - -Another problem is that two processes cannot both take input from the -same device; so to make sure that only one recipe tries to take input -from the terminal at once, @code{make} will invalidate the standard -input streams of all but one running recipe. This means that -attempting to read from standard input will usually be a fatal error (a -@samp{Broken pipe} signal) for most child processes if there are -several. -@cindex broken pipe -@cindex standard input - -It is unpredictable which recipe will have a valid standard input stream -(which will come from the terminal, or wherever you redirect the standard -input of @code{make}). The first recipe run will always get it first, and -the first recipe started after that one finishes will get it next, and so -on. - -We will change how this aspect of @code{make} works if we find a better -alternative. In the mean time, you should not rely on any recipe using -standard input at all if you are using the parallel execution feature; but -if you are not using this feature, then standard input works normally in -all recipes. - -Finally, handling recursive @code{make} invocations raises issues. For -more information on this, see -@ref{Options/Recursion, ,Communicating Options to a Sub-@code{make}}. +Handling recursive @code{make} invocations raises issues for parallel +execution. For more information on this, see @ref{Options/Recursion, +,Communicating Options to a Sub-@code{make}}. If a recipe fails (is killed by a signal or exits with a nonzero -status), and errors are not ignored for that recipe -(@pxref{Errors, ,Errors in Recipes}), -the remaining recipe lines to remake the same target will not be run. -If a recipe fails and the @samp{-k} or @samp{--keep-going} -option was not given -(@pxref{Options Summary, ,Summary of Options}), -@code{make} aborts execution. If make +status), and errors are not ignored for that recipe (@pxref{Errors, +,Errors in Recipes}), the remaining recipe lines to remake the same +target will not be run. If a recipe fails and the @samp{-k} or +@samp{--keep-going} option was not given (@pxref{Options Summary, +,Summary of Options}), @code{make} aborts execution. If make terminates for any reason (including a signal) with child processes running, it waits for them to finish before actually exiting.@refill @@ -4131,6 +4104,105 @@ average goes below that limit, or until all the other jobs finish. By default, there is no load limit. +@menu +* Parallel Output:: Handling output during parallel execution +* Parallel Input:: Handling input during parallel execution +@end menu + +@node Parallel Output, Parallel Input, Parallel, Parallel +@subsection Output During Parallel Execution +@cindex output during parallel execution +@cindex parallel execution, output during + +When running several recipes in parallel the output from each +recipe appears as soon as it is generated, with the result that +messages from different recipes may be interspersed, sometimes even +appearing on the same line. This can make reading the output very +difficult. + +@cindex @code{--output-sync} +@cindex @code{-O} +To avoid this you can use the @samp{--output-sync} (@samp{-O}) option. +This option instructs @code{make} to save the output from the commands +it invokes and print it all once the commands are completed. +Additionally, if there are multiple recursive @code{make} invocations +running in parallel, they will communicate so that only one of them is +generating output at a time. + +There are four levels of granularity when synchronizing output, +specified by giving an argument to the option (e.g., @samp{-Ojob} or +@samp{--output-sync=make}). + +@table @code +@item none +The is the default: all output is sent directly as it is generated and +no synchronization is performed. + +@item job +Output from each individual line of the recipe is grouped and printed +as soon as that line is complete. If a recipe consists of multiple +lines, they may be interspersed with lines from other recipes. + +@item target +Output from the entire recipe for each target is grouped and printed +once the target is complete. This is the default if the +@code{--output-sync} or @code{-O} option is given with no argument. + +@item make +Output from each recursive invocation of @code{make} is grouped and +printed once the recursive invocation is complete. + +@end table + +Regardless of the mode chosen, the total build time will be the same. +The only difference is in how the output appears. + +The @samp{make} mode provides the most comprehensive grouping, +allowing output from all targets built by a given makefile to appear +together. However, there will be long interludes during the build +where no output appears while a recursive @code{make} is running, +followed by a burst of output. This mode is best for builds being +performed in the background, where the output will be examined later. + +The @samp{job} mode is mainly useful for front-ends that may be +watching the output of @code{make} and looking for certain generated +output to determine when recipes are started and completed. + +You should be aware that some programs may act differently when they +determine they're writing output to a terminal versus a file +(typically described as ``interactive'' vs. ``non-interactive'' +modes). If your makefile invokes a program like this then using the +output synchronization options will cause the program to believe it's +running in ``non-interactive'' mode even when it's writing to the +terminal. Of course, invoking @code{make} with output redirected to a +file will elicit the same behavior. + +@node Parallel Input, , Parallel Output, Parallel +@subsection Input During Parallel Execution +@cindex input during parallel execution +@cindex parallel execution, input during +@cindex standard input + +Two processes cannot both take input from the same device at the same +time. To make sure that only one recipe tries to take input from the +terminal at once, @code{make} will invalidate the standard input +streams of all but one running recipe. If another recipe attempts to +read from standard input it will usually incur a fatal error (a +@samp{Broken pipe} signal). +@cindex broken pipe + +It is unpredictable which recipe will have a valid standard input stream +(which will come from the terminal, or wherever you redirect the standard +input of @code{make}). The first recipe run will always get it first, and +the first recipe started after that one finishes will get it next, and so +on. + +We will change how this aspect of @code{make} works if we find a better +alternative. In the mean time, you should not rely on any recipe using +standard input at all if you are using the parallel execution feature; but +if you are not using this feature, then standard input works normally in +all recipes. + @node Errors, Interrupts, Parallel, Recipes @section Errors in Recipes @cindex errors (in recipes) @@ -8628,16 +8700,19 @@ uninterrupted sequence. This option is only useful when using the (@pxref{Parallel, ,Parallel Execution}) Without this option output will be displayed as it is generated by the recipes.@refill -With no type or the type @samp{target}, output from each individual -target is grouped together. With the type @samp{make}, the output -from an entire recursive make is grouped together. The latter -achieves better grouping of output from related jobs, but causes -longer delay since messages do not appear until the entire recursive -make has completed (this does not increase the total build time, -though). In general @samp{target} mode is useful when watching the -output while make runs, and @samp{make} mode is useful when running a -complex parallel build in the background and checking its output -afterwards. +With no type or the type @samp{target}, output from the entire recipe +of each target is grouped together. With the type @samp{job}, output +from each job in the recipe is grouped together. With the type +@samp{make}, the output from an entire recursive make is grouped +together. The latter achieves better grouping of output from related +jobs, but causes longer delay since messages do not appear until the +entire recursive make has completed (this does not increase the total +build time, though). In general @samp{target} mode is useful when +watching the output while make runs, and @samp{make} mode is useful +when running a complex parallel build in the background and checking +its output afterwards. The @samp{job} mode may be helpful for tools +which watch the output to determine when recipes have started or +stopped. @item -q @cindex @code{-q} @@ -10886,7 +10961,7 @@ will load the dynamic object @file{../mk_funcs.so}. After the object is loaded, @code{make} will invoke the function @code{init_mk_func}. Regardless of how many times an object file appears in a @code{load} -directive, it will only be loaded (and it's setup function will only +directive, it will only be loaded (and its setup function will only be invoked) once. @vindex .LOADED @@ -557,24 +557,41 @@ child_handler (int sig UNUSED) static int assign_child_tempfiles (struct child *c, int combined) { - /* If we already have a temp file assigned we're done. */ - if (c->outfd != -1 && c->errfd != -1) - return 1; - - if (STREAM_OK (stdout)) + /* If we don't have a temp file, get one. */ + if (c->outfd < 0 && c->errfd < 0) { - c->outfd = open_tmpfd (); - CLOSE_ON_EXEC (c->outfd); + if (STREAM_OK (stdout)) + { + c->outfd = open_tmpfd (); + CLOSE_ON_EXEC (c->outfd); + } + + if (STREAM_OK (stderr)) + { + if (c->outfd >= 0 && combined) + c->errfd = c->outfd; + else + { + c->errfd = open_tmpfd (); + CLOSE_ON_EXEC (c->errfd); + } + } + + return 1; } - if (STREAM_OK (stderr)) + /* We already have a temp file. On per-job output, truncate and reset. */ + if (output_sync == OUTPUT_SYNC_JOB) { - if (c->outfd >= 0 && combined) - c->errfd = c->outfd; - else + if (c->outfd >= 0) { - c->errfd = open_tmpfd (); - CLOSE_ON_EXEC (c->errfd); + lseek(c->outfd, 0, SEEK_SET); + ftruncate(c->outfd, 0); + } + if (c->errfd >= 0 && c->errfd != c->outfd) + { + lseek(c->errfd, 0, SEEK_SET); + ftruncate(c->outfd, 0); } } @@ -690,14 +707,8 @@ sync_output (struct child *c) /* Exit the critical section. */ if (sem) - release_semaphore (sem); + release_semaphore (sem); } - - if (c->outfd >= 0) - close (c->outfd); - if (c->errfd >= 0) - close (c->errfd); - c->outfd = c->errfd = -1; } #endif /* OUTPUT_SYNC */ @@ -995,6 +1006,11 @@ reap_children (int block, int err) c->sh_batch_file = NULL; } +#ifdef OUTPUT_SYNC + if (output_sync == OUTPUT_SYNC_JOB) + sync_output (c); +#endif + /* If this child had the good stdin, say it is now free. */ if (c->good_stdin) good_stdin_used = 0; @@ -1073,7 +1089,7 @@ reap_children (int block, int err) #ifdef OUTPUT_SYNC /* Synchronize parallel output if requested */ - if (output_sync) + if (output_sync > OUTPUT_SYNC_JOB) sync_output (c); #endif /* OUTPUT_SYNC */ @@ -1131,6 +1147,11 @@ reap_children (int block, int err) static void free_child (struct child *child) { + if (child->outfd >= 0) + close (child->outfd); + if (child->errfd >= 0 && child->errfd != child->outfd) + close (child->errfd); + if (!jobserver_tokens) fatal (NILF, "INTERNAL: Freeing child %p (%s) but no tokens left!\n", child, child->file->name); @@ -1613,7 +1634,7 @@ start_job_command (struct child *child) /* If it still looks like we can synchronize, create a temp file to hold stdout (and one for stderr if separate). */ if (output_sync == OUTPUT_SYNC_MAKE - || (output_sync == OUTPUT_SYNC_TARGET && !(flags & COMMANDS_RECURSE))) + || (output_sync && !(flags & COMMANDS_RECURSE))) { if (!assign_child_tempfiles (child, combined_output)) output_sync = 0; @@ -2035,9 +2056,7 @@ new_job (struct file *file) c->file = file; c->command_lines = lines; c->sh_batch_file = NULL; -#ifdef OUTPUT_SYNC c->outfd = c->errfd = -1; -#endif /* Cache dontcare flag because file->dontcare can be changed once we return. Check dontcare inheritance mechanism for details. */ @@ -99,16 +99,14 @@ struct child #endif unsigned int command_line; /* Index into command_lines. */ - pid_t pid; /* Child process's ID number. */ + int outfd; /* File descriptor for saving stdout */ + int errfd; /* File descriptor for saving stderr */ + pid_t pid; /* Child process's ID number. */ unsigned int remote:1; /* Nonzero if executing remotely. */ unsigned int noerror:1; /* Nonzero if commands contained a '-'. */ unsigned int good_stdin:1; /* Nonzero if this child has a good stdin. */ unsigned int deleted:1; /* Nonzero if targets have been deleted. */ unsigned int dontcare:1; /* Saved dontcare flag. */ -#ifdef OUTPUT_SYNC - int outfd; /* File descriptor for saving stdout */ - int errfd; /* File descriptor for saving stderr */ -#endif }; extern struct child *children; @@ -695,7 +695,11 @@ decode_output_sync_flags (void) { const char *p = *pp; - if (streq (p, "target")) + if (streq (p, "none")) + output_sync = OUTPUT_SYNC_NONE; + else if (streq (p, "job")) + output_sync = OUTPUT_SYNC_JOB; + else if (streq (p, "target")) output_sync = OUTPUT_SYNC_TARGET; else if (streq (p, "make")) output_sync = OUTPUT_SYNC_MAKE; @@ -227,11 +227,20 @@ other jobs. If .I type is not specified or is .B target -output is grouped together on a per-target basis. If +the output from the entire recipe for each target is grouped together. If +.I type +is +.B job +the output from each job within a recipe is grouped together. +If .I type is .B make -output from an entire recursive make is grouped together. +output from an entire recursive make is grouped together. If +.I type +is +.B none +output synchronization is disabled. .TP 0.5i \fB\-p\fR, \fB\-\-print\-data\-base\fR Print the data base (rules and variable values) that results from @@ -538,8 +538,10 @@ int strncasecmp (const char *s1, const char *s2, int n); # define OUTPUT_SYNC #endif -#define OUTPUT_SYNC_TARGET 1 -#define OUTPUT_SYNC_MAKE 2 +#define OUTPUT_SYNC_NONE 0 +#define OUTPUT_SYNC_JOB 1 +#define OUTPUT_SYNC_TARGET 2 +#define OUTPUT_SYNC_MAKE 3 extern const gmk_floc *reading_file; extern const gmk_floc **expanding_var; diff --git a/tests/ChangeLog b/tests/ChangeLog index 0502aba..642ba85 100644 --- a/tests/ChangeLog +++ b/tests/ChangeLog @@ -1,3 +1,8 @@ +2013-04-28 Paul Smith <psmith@gnu.org> + + * scripts/features/output-sync (output_sync_set): Add tests for + the per-job syntax mode. + 2013-04-15 Paul Smith <psmith@gnu.org> * scripts/features/output-sync (output_sync_set): New arg syntax. diff --git a/tests/scripts/features/output-sync b/tests/scripts/features/output-sync index dce2ac4..babc08f 100644 --- a/tests/scripts/features/output-sync +++ b/tests/scripts/features/output-sync @@ -15,6 +15,18 @@ else { $sleep_command = "sleep"; } +# The following subdirectories with Makefiles are used in several +# of the following tests. The model is: +# foo/Makefile - has a "foo" target that waits for the bar target +# bar/Makefile - has a "bar" target that runs immediately +# - has a "baz" target that waits for the foo target +# +# So, you start the two sub-makes in parallel and first the "bar" target is +# built, followed by "foo", followed by "baz". The trick is that first each +# target prints a "start" statement, then waits (if appropriate), then prints +# an end statement. Thus we can tell if the -O flag is working, since +# otherwise these statements would be mixed together. + @syncfiles = (); sub output_sync_clean { @@ -36,19 +48,21 @@ sub output_sync_set { return "date > ../mksync.$_[0]"; } -@syncfiles = qw(mksync.foo mksync.bar); +@syncfiles = qw(mksync.foo mksync.foo_start mksync.bar mksync.bar_start); -# The following subdirectories with Makefiles are used in several -# of the following tests. output_sync_clean(); mkdir('foo', 0777); mkdir('bar', 0777); $set_foo = output_sync_set('foo'); $set_bar = output_sync_set('bar'); +$set_foo_start = output_sync_set('foo_start'); +$set_bar_start = output_sync_set('bar_start'); $wait_foo = output_sync_wait('foo'); $wait_bar = output_sync_wait('bar'); +$wait_foo_start = output_sync_set('foo_start'); +$wait_bar_start = output_sync_set('bar_start'); open(MAKEFILE,"> foo/Makefile"); print MAKEFILE <<EOF; @@ -56,9 +70,25 @@ all: foo foo: foo-base ; \@$set_foo -foo-base: ; \@echo foo: start; $wait_bar ; echo foo: end - -foo-fail: ; \@$wait_bar ; false +foo-base: +\t\@echo foo: start +\t\@$wait_bar +\t\@echo foo: end + +foo-job: foo-job-base ; \@$set_foo + +foo-job-base: +\t\@$wait_bar_start +\t\@echo foo: start +\t\@$set_foo_start +\t\@$wait_bar +\t\@echo foo: end + +foo-fail: +\t\@echo foo-fail: start +\t\@$wait_bar +\t\@echo foo-fail: end +\t\@false EOF close(MAKEFILE); @@ -66,15 +96,28 @@ open(MAKEFILE,"> bar/Makefile"); print MAKEFILE <<EOF; all: bar baz -bar: ; \@echo bar: start; echo bar: end; $set_bar +bar: bar-base ; \@$set_bar +bar-base: +\t\@echo bar: start +\t\@echo bar: end -baz: baz-base +bar-job: bar-job-base ; \@$set_bar + +bar-job-base: +\t\@echo bar: start +\t\@$set_bar_start +\t\@$wait_foo_start +\t\@echo bar: end -baz-base: ; \@echo baz: start; $wait_foo; echo baz: end +baz: baz-base +baz-base: +\t\@echo baz: start +\t\@$wait_foo +\t\@echo baz: end EOF close(MAKEFILE); -# Test coarse synchronization. +# Test per-make synchronization. unlink(@syncfiles); run_make_test(qq! all: make-foo make-bar @@ -102,7 +145,7 @@ baz: end #MAKE#[1]: Leaving directory '#PWD#/bar' #MAKE#[1]: Leaving directory '#PWD#/bar'\n", 0, 6); -# Test fine synchronization. +# Test per-target synchronization. # Note we have to sleep again here after starting the foo makefile before # starting the bar makefile, otherwise the "entering/leaving" messages for the # submakes might be ordered differently than we expect. @@ -154,16 +197,54 @@ bar: end #MAKE#[1]: Leaving directory '#PWD#/bar' #MAKE#[1]: Leaving directory '#PWD#/bar' #MAKE#[1]: Entering directory '#PWD#/foo' -Makefile:7: recipe for target 'foo-fail' failed -#MAKE#[1]: Leaving directory '#PWD#/foo' -#MAKE#[1]: Entering directory '#PWD#/foo' -#MAKE#[1]: *** [foo-fail] Error 1 +Makefile:20: recipe for target 'foo-fail' failed +make[1]: Leaving directory '/home/psmith/src/make/make/tests/foo' +make[1]: Entering directory '/home/psmith/src/make/make/tests/foo' +make[1]: *** [foo-fail] Error 1 +make[1]: Leaving directory '/home/psmith/src/make/make/tests/foo' +make[1]: Entering directory '/home/psmith/src/make/make/tests/foo' +foo-fail: start +foo-fail: end #MAKE#[1]: Leaving directory '#PWD#/foo' #MAKE#[1]: Leaving directory '#PWD#/foo' #MAKEFILE#:4: recipe for target 'make-foo-fail' failed #MAKE#: *** [make-foo-fail] Error 2\n", 512); +# Test the per-job synchronization. +# For this we'll have bar-job: +# print start, invoke bar-start, wait for foo-start, print end, print-bar-end +# And foo-job: +# wait for bar-start, print foo-start, wait for bar-end, print end + +unlink(@syncfiles); +run_make_test(qq! +all: make-foo make-bar + +make-foo: ; \$(MAKE) -C foo foo-job + +make-bar: ; $sleep_command 1 ; \$(MAKE) -C bar bar-job!, + '-j --output-sync=job', +"#MAKEPATH# -C foo foo-job +$sleep_command 1 ; #MAKEPATH# -C bar bar-job +#MAKE#[1]: Entering directory '#PWD#/foo' +#MAKE#[1]: Entering directory '#PWD#/foo' +foo: start +#MAKE#[1]: Leaving directory '#PWD#/foo' +#MAKE#[1]: Entering directory '#PWD#/bar' +#MAKE#[1]: Entering directory '#PWD#/bar' +bar: start +#MAKE#[1]: Leaving directory '#PWD#/bar' +#MAKE#[1]: Entering directory '#PWD#/bar' +bar: end +#MAKE#[1]: Leaving directory '#PWD#/bar' +#MAKE#[1]: Leaving directory '#PWD#/bar' +#MAKE#[1]: Entering directory '#PWD#/foo' +foo: end +#MAKE#[1]: Leaving directory '#PWD#/foo' +#MAKE#[1]: Leaving directory '#PWD#/foo'\n", 0, 6); + + # Remove temporary directories and contents. output_sync_clean(); |