Add support for per-job output sync.

A new flag to the -O/--output-sync, "job", selects a per-job (that is, per
line of a recipe) output synchronization.  To support this move the close of
the temp file out of the sync_output() function and don't do it until we free
the child, since we may call sync_output() multiple times in a given recipe.
When we set up for a new temp file, if we're in per-job mode we truncate the
file and seek to the beginning to re-use it for every job.

11 files changed, 320 insertions, 105 deletions

diff --git a/AUTHORS b/AUTHORS
index 84a7126..67bdd40 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -37,7 +37,7 @@ GNU make porting efforts:
   Port to MS-DOS (DJGPP), OS/2, and MS-Windows (native/MinGW) by:
       DJ Delorie <dj@delorie.com>
       Rob Tulloh <rob_tulloh@tivoli.com>
-      Eli Zaretskii <eliz@is.elta.co.il>
+      Eli Zaretskii <eliz@gnu.org>
       Jonathan Grant <jg@jguk.org>
       Andreas Beuning <andreas.buening@nexgo.de>
       Earnie Boyd <earnie@uses.sf.net>
diff --git a/ChangeLog b/ChangeLog
index d0dabb1..91802ae 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,21 @@
+2013-04-28  Paul Smith  <psmith@gnu.org>
+
+	Implement a "per-job" output synchronization option.
+
+	* main.c (decode_output_sync_flags): Recognize the new option.
+	* makeint.h (OUTPUT_SYNC_JOB): Add new values for "job"
+	* job.c (assign_child_tempfiles): In per-job mode, truncate the
+	temp file for re-use by the next job.
+	(sync_output): Don't close the temp files as we may still use them.
+	(free_child): Close the temp files here as we definitely don't
+	need them.
+	(new_job): In per-job output mode, sync_output() after each job.
+	* job.h (struct child): Avoid ifdefs.
+	* make.1: Add new options to the man page.
+	* doc/make.texi (Parallel Output): Break documentation on input
+	and output into separate sections for readability.  Document the
+	new "job" and "none" modes.
+
 2013-04-27  Paul Smith  <psmith@gnu.org>
 
 	* job.c (construct_command_argv_internal): Fix oneshell support
diff --git a/NEWS b/NEWS
index 8442311..5a0c20a 100644
--- a/NEWS
+++ b/NEWS
@@ -1,6 +1,6 @@
 GNU make NEWS                                               -*-indented-text-*-
   History of user-visible changes.
-  16 April 2013
+  27 April 2013
 
 See the end of this file for copyrights and conditions.
 
@@ -31,13 +31,17 @@ http://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=101&set
 
 * New command line option: --output-sync (-O) enables grouping of output by
   target or by recursive make.  This is useful during parallel builds to avoid
-  mixing output from different jobs together giving hard-to-understand results.
-  Original implementation by David Boyce <dsb@boyski.com>.  Patch was reworked
-  by Frank Heckenbach <f.heckenbach@fh-soft.de>.
+  mixing output from different jobs together giving hard-to-understand
+  results.  Original implementation by David Boyce <dsb@boyski.com>.
+  Reworked and enhanced by Frank Heckenbach <f.heckenbach@fh-soft.de>.
+  Windows support by Eli Zaretskii <eliz@gnu.org>.
 
 * New feature: The "job server" capability is now supported on Windows.
   Implementation contributed by Troy Runkel <Troy.Runkel@mathworks.com>
 
+* New feature: The .ONESHELL capability is now supported on Windows.  Support
+  added by Eli Zaretskii <eliz@gnu.org>.
+
 * New feature: "!=" shell assignment operator as an alternative to the
   $(shell ...) function.  Implemented for compatibility with BSD makefiles.
   WARNING: Backward-incompatibility!
diff --git a/doc/make.texi b/doc/make.texi
index ce8e07b..64be3b3 100644
--- a/doc/make.texi
+++ b/doc/make.texi
@@ -219,6 +219,11 @@ Recipe Execution
 * Choosing the Shell::          How @code{make} chooses the shell used
                                   to run recipes.
 
+Parallel Execution
+
+* Parallel Output::             Handling output during parallel execution
+* Parallel Input::              Handling input during parallel execution
+
 Recursive Use of @code{make}
 
 * MAKE Variable::               The special effects of using @samp{$(MAKE)}.
@@ -4057,48 +4062,16 @@ If there is nothing looking like an integer after the @samp{-j} option,
 there is no limit on the number of job slots.  The default number of job
 slots is one, which means serial execution (one thing at a time).
 
-When running several recipes simultaneously the output from each
-recipe appears as soon as it is generated, with the result that
-messages from different recipes may be interspersed.  To avoid this
-you can use the @samp{--output-sync} (@samp{-O}) option; if this
-option is provided then the output from each recipe will be saved
-until the recipe is complete, then written all at once.  This ensures
-that output from different recipes is not mixed together.
-
-Another problem is that two processes cannot both take input from the
-same device; so to make sure that only one recipe tries to take input
-from the terminal at once, @code{make} will invalidate the standard
-input streams of all but one running recipe.  This means that
-attempting to read from standard input will usually be a fatal error (a
-@samp{Broken pipe} signal) for most child processes if there are
-several.
-@cindex broken pipe
-@cindex standard input
-
-It is unpredictable which recipe will have a valid standard input stream
-(which will come from the terminal, or wherever you redirect the standard
-input of @code{make}).  The first recipe run will always get it first, and
-the first recipe started after that one finishes will get it next, and so
-on.
-
-We will change how this aspect of @code{make} works if we find a better
-alternative.  In the mean time, you should not rely on any recipe using
-standard input at all if you are using the parallel execution feature; but
-if you are not using this feature, then standard input works normally in
-all recipes.
-
-Finally, handling recursive @code{make} invocations raises issues.  For
-more information on this, see
-@ref{Options/Recursion, ,Communicating Options to a Sub-@code{make}}.
+Handling recursive @code{make} invocations raises issues for parallel
+execution.  For more information on this, see @ref{Options/Recursion,
+,Communicating Options to a Sub-@code{make}}.
 
 If a recipe fails (is killed by a signal or exits with a nonzero
-status), and errors are not ignored for that recipe
-(@pxref{Errors, ,Errors in Recipes}),
-the remaining recipe lines to remake the same target will not be run.
-If a recipe fails and the @samp{-k} or @samp{--keep-going}
-option was not given
-(@pxref{Options Summary, ,Summary of Options}),
-@code{make} aborts execution.  If make
+status), and errors are not ignored for that recipe (@pxref{Errors,
+,Errors in Recipes}), the remaining recipe lines to remake the same
+target will not be run.  If a recipe fails and the @samp{-k} or
+@samp{--keep-going} option was not given (@pxref{Options Summary,
+,Summary of Options}), @code{make} aborts execution.  If make
 terminates for any reason (including a signal) with child processes
 running, it waits for them to finish before actually exiting.@refill
 
@@ -4131,6 +4104,105 @@ average goes below that limit, or until all the other jobs finish.
 
 By default, there is no load limit.
 
+@menu
+* Parallel Output::             Handling output during parallel execution
+* Parallel Input::              Handling input during parallel execution
+@end menu
+
+@node Parallel Output, Parallel Input, Parallel, Parallel
+@subsection Output During Parallel Execution
+@cindex output during parallel execution
+@cindex parallel execution, output during
+
+When running several recipes in parallel the output from each
+recipe appears as soon as it is generated, with the result that
+messages from different recipes may be interspersed, sometimes even
+appearing on the same line.  This can make reading the output very
+difficult.
+
+@cindex @code{--output-sync}
+@cindex @code{-O}
+To avoid this you can use the @samp{--output-sync} (@samp{-O}) option.
+This option instructs @code{make} to save the output from the commands
+it invokes and print it all once the commands are completed.
+Additionally, if there are multiple recursive @code{make} invocations
+running in parallel, they will communicate so that only one of them is
+generating output at a time.
+
+There are four levels of granularity when synchronizing output,
+specified by giving an argument to the option (e.g.,  @samp{-Ojob} or
+@samp{--output-sync=make}).
+
+@table @code
+@item none
+The is the default: all output is sent directly as it is generated and
+no synchronization is performed.
+
+@item job
+Output from each individual line of the recipe is grouped and printed
+as soon as that line is complete.  If a recipe consists of multiple
+lines, they may be interspersed with lines from other recipes.
+
+@item target
+Output from the entire recipe for each target is grouped and printed
+once the target is complete.  This is the default if the
+@code{--output-sync} or @code{-O} option is given with no argument.
+
+@item make
+Output from each recursive invocation of @code{make} is grouped and
+printed once the recursive invocation is complete.
+
+@end table
+
+Regardless of the mode chosen, the total build time will be the same.
+The only difference is in how the output appears.
+
+The @samp{make} mode provides the most comprehensive grouping,
+allowing output from all targets built by a given makefile to appear
+together.  However, there will be long interludes during the build
+where no output appears while a recursive @code{make} is running,
+followed by a burst of output.  This mode is best for builds being
+performed in the background, where the output will be examined later.
+
+The @samp{job} mode is mainly useful for front-ends that may be
+watching the output of @code{make} and looking for certain generated
+output to determine when recipes are started and completed.
+
+You should be aware that some programs may act differently when they
+determine they're writing output to a terminal versus a file
+(typically described as ``interactive'' vs. ``non-interactive''
+modes).  If your makefile invokes a program like this then using the
+output synchronization options will cause the program to believe it's
+running in ``non-interactive'' mode even when it's writing to the
+terminal.  Of course, invoking @code{make} with output redirected to a
+file will elicit the same behavior.
+
+@node Parallel Input,  , Parallel Output, Parallel
+@subsection Input During Parallel Execution
+@cindex input during parallel execution
+@cindex parallel execution, input during
+@cindex standard input
+
+Two processes cannot both take input from the same device at the same
+time.  To make sure that only one recipe tries to take input from the
+terminal at once, @code{make} will invalidate the standard input
+streams of all but one running recipe.  If another recipe attempts to
+read from standard input it will usually incur a fatal error (a
+@samp{Broken pipe} signal).
+@cindex broken pipe
+
+It is unpredictable which recipe will have a valid standard input stream
+(which will come from the terminal, or wherever you redirect the standard
+input of @code{make}).  The first recipe run will always get it first, and
+the first recipe started after that one finishes will get it next, and so
+on.
+
+We will change how this aspect of @code{make} works if we find a better
+alternative.  In the mean time, you should not rely on any recipe using
+standard input at all if you are using the parallel execution feature; but
+if you are not using this feature, then standard input works normally in
+all recipes.
+
 @node Errors, Interrupts, Parallel, Recipes
 @section Errors in Recipes
 @cindex errors (in recipes)
@@ -8628,16 +8700,19 @@ uninterrupted sequence.  This option is only useful when using the
 (@pxref{Parallel, ,Parallel Execution})  Without this option output
 will be displayed as it is generated by the recipes.@refill
 
-With no type or the type @samp{target}, output from each individual
-target is grouped together.  With the type @samp{make}, the output
-from an entire recursive make is grouped together.  The latter
-achieves better grouping of output from related jobs, but causes
-longer delay since messages do not appear until the entire recursive
-make has completed (this does not increase the total build time,
-though).  In general @samp{target} mode is useful when watching the
-output while make runs, and @samp{make} mode is useful when running a
-complex parallel build in the background and checking its output
-afterwards.
+With no type or the type @samp{target}, output from the entire recipe
+of each target is grouped together.  With the type @samp{job}, output
+from each job in the recipe is grouped together.  With the type
+@samp{make}, the output from an entire recursive make is grouped
+together.  The latter achieves better grouping of output from related
+jobs, but causes longer delay since messages do not appear until the
+entire recursive make has completed (this does not increase the total
+build time, though).  In general @samp{target} mode is useful when
+watching the output while make runs, and @samp{make} mode is useful
+when running a complex parallel build in the background and checking
+its output afterwards.  The @samp{job} mode may be helpful for tools
+which watch the output to determine when recipes have started or
+stopped.
 
 @item -q
 @cindex @code{-q}
@@ -10886,7 +10961,7 @@ will load the dynamic object @file{../mk_funcs.so}.  After the object
 is loaded, @code{make} will invoke the function @code{init_mk_func}.
 
 Regardless of how many times an object file appears in a @code{load}
-directive, it will only be loaded (and it's setup function will only
+directive, it will only be loaded (and its setup function will only
 be invoked) once.
 
 @vindex .LOADED
diff --git a/job.c b/job.c
index addeca2..941cf21 100644
--- a/job.c
+++ b/job.c
@@ -557,24 +557,41 @@ child_handler (int sig UNUSED)
 static int
 assign_child_tempfiles (struct child *c, int combined)
 {
-  /* If we already have a temp file assigned we're done.  */
-  if (c->outfd != -1 && c->errfd != -1)
-    return 1;
-
-  if (STREAM_OK (stdout))
+  /* If we don't have a temp file, get one.  */
+  if (c->outfd < 0 && c->errfd < 0)
     {
-      c->outfd = open_tmpfd ();
-      CLOSE_ON_EXEC (c->outfd);
+      if (STREAM_OK (stdout))
+        {
+          c->outfd = open_tmpfd ();
+          CLOSE_ON_EXEC (c->outfd);
+        }
+
+      if (STREAM_OK (stderr))
+        {
+          if (c->outfd >= 0 && combined)
+            c->errfd = c->outfd;
+          else
+            {
+              c->errfd = open_tmpfd ();
+              CLOSE_ON_EXEC (c->errfd);
+            }
+        }
+
+      return 1;
     }
 
-  if (STREAM_OK (stderr))
+  /* We already have a temp file.  On per-job output, truncate and reset.  */
+  if (output_sync == OUTPUT_SYNC_JOB)
     {
-      if (c->outfd >= 0 && combined)
-        c->errfd = c->outfd;
-      else
+      if (c->outfd >= 0)
         {
-          c->errfd = open_tmpfd ();
-          CLOSE_ON_EXEC (c->errfd);
+          lseek(c->outfd, 0, SEEK_SET);
+          ftruncate(c->outfd, 0);
+        }
+      if (c->errfd >= 0 && c->errfd != c->outfd)
+        {
+          lseek(c->errfd, 0, SEEK_SET);
+          ftruncate(c->outfd, 0);
         }
     }
 
@@ -690,14 +707,8 @@ sync_output (struct child *c)
 
       /* Exit the critical section.  */
       if (sem)
-	release_semaphore (sem);
+        release_semaphore (sem);
     }
-
-  if (c->outfd >= 0)
-    close (c->outfd);
-  if (c->errfd >= 0)
-    close (c->errfd);
-  c->outfd = c->errfd = -1;
 }
 #endif /* OUTPUT_SYNC */
 
@@ -995,6 +1006,11 @@ reap_children (int block, int err)
         c->sh_batch_file = NULL;
       }
 
+#ifdef OUTPUT_SYNC
+      if (output_sync == OUTPUT_SYNC_JOB)
+        sync_output (c);
+#endif
+
       /* If this child had the good stdin, say it is now free.  */
       if (c->good_stdin)
         good_stdin_used = 0;
@@ -1073,7 +1089,7 @@ reap_children (int block, int err)
 
 #ifdef OUTPUT_SYNC
       /* Synchronize parallel output if requested */
-      if (output_sync)
+      if (output_sync > OUTPUT_SYNC_JOB)
         sync_output (c);
 #endif /* OUTPUT_SYNC */
 
@@ -1131,6 +1147,11 @@ reap_children (int block, int err)
 static void
 free_child (struct child *child)
 {
+  if (child->outfd >= 0)
+    close (child->outfd);
+  if (child->errfd >= 0 && child->errfd != child->outfd)
+    close (child->errfd);
+
   if (!jobserver_tokens)
     fatal (NILF, "INTERNAL: Freeing child %p (%s) but no tokens left!\n",
            child, child->file->name);
@@ -1613,7 +1634,7 @@ start_job_command (struct child *child)
           /* If it still looks like we can synchronize, create a temp
               file to hold stdout (and one for stderr if separate). */
           if (output_sync == OUTPUT_SYNC_MAKE
-              || (output_sync == OUTPUT_SYNC_TARGET && !(flags & COMMANDS_RECURSE)))
+              || (output_sync && !(flags & COMMANDS_RECURSE)))
             {
               if (!assign_child_tempfiles (child, combined_output))
                 output_sync = 0;
@@ -2035,9 +2056,7 @@ new_job (struct file *file)
   c->file = file;
   c->command_lines = lines;
   c->sh_batch_file = NULL;
-#ifdef OUTPUT_SYNC
   c->outfd = c->errfd = -1;
-#endif
 
   /* Cache dontcare flag because file->dontcare can be changed once we
      return. Check dontcare inheritance mechanism for details.  */
diff --git a/job.h b/job.h
index df74270..f15b54c 100644
--- a/job.h
+++ b/job.h
@@ -99,16 +99,14 @@ struct child
 #endif
 
     unsigned int command_line;	/* Index into command_lines.  */
-    pid_t pid;			/* Child process's ID number.  */
+    int          outfd;		/* File descriptor for saving stdout */
+    int          errfd;		/* File descriptor for saving stderr */
+    pid_t        pid;		/* Child process's ID number.  */
     unsigned int remote:1;	/* Nonzero if executing remotely.  */
     unsigned int noerror:1;	/* Nonzero if commands contained a '-'.  */
     unsigned int good_stdin:1;	/* Nonzero if this child has a good stdin.  */
     unsigned int deleted:1;	/* Nonzero if targets have been deleted.  */
     unsigned int dontcare:1;    /* Saved dontcare flag.  */
-#ifdef OUTPUT_SYNC
-    int outfd;			/* File descriptor for saving stdout */
-    int errfd;			/* File descriptor for saving stderr */
-#endif
   };
 
 extern struct child *children;
diff --git a/main.c b/main.c
index 0037587..be3d190 100644
--- a/main.c
+++ b/main.c
@@ -695,7 +695,11 @@ decode_output_sync_flags (void)
     {
       const char *p = *pp;
 
-      if (streq (p, "target"))
+      if (streq (p, "none"))
+        output_sync = OUTPUT_SYNC_NONE;
+      else if (streq (p, "job"))
+        output_sync = OUTPUT_SYNC_JOB;
+      else if (streq (p, "target"))
         output_sync = OUTPUT_SYNC_TARGET;
       else if (streq (p, "make"))
         output_sync = OUTPUT_SYNC_MAKE;
diff --git a/make.1 b/make.1
index 6eeaaf8..326bf86 100644
--- a/make.1
+++ b/make.1
@@ -227,11 +227,20 @@ other jobs.  If
 .I type
 is not specified or is
 .B target
-output is grouped together on a per-target basis.  If
+the output from the entire recipe for each target is grouped together.  If
+.I type
+is
+.B job
+the output from each job within a recipe is grouped together.
+If
 .I type
 is
 .B make
-output from an entire recursive make is grouped together.
+output from an entire recursive make is grouped together.  If
+.I type
+is
+.B none
+output synchronization is disabled.
 .TP 0.5i
 \fB\-p\fR, \fB\-\-print\-data\-base\fR
 Print the data base (rules and variable values) that results from
diff --git a/makeint.h b/makeint.h
index d1b00c6..643b611 100644
--- a/makeint.h
+++ b/makeint.h
@@ -538,8 +538,10 @@ int strncasecmp (const char *s1, const char *s2, int n);
 # define OUTPUT_SYNC
 #endif
 
-#define OUTPUT_SYNC_TARGET 1
-#define OUTPUT_SYNC_MAKE 2
+#define OUTPUT_SYNC_NONE   0
+#define OUTPUT_SYNC_JOB    1
+#define OUTPUT_SYNC_TARGET 2
+#define OUTPUT_SYNC_MAKE   3
 
 extern const gmk_floc *reading_file;
 extern const gmk_floc **expanding_var;
diff --git a/tests/ChangeLog b/tests/ChangeLog
index 0502aba..642ba85 100644
--- a/tests/ChangeLog
+++ b/tests/ChangeLog
@@ -1,3 +1,8 @@
+2013-04-28  Paul Smith  <psmith@gnu.org>
+
+	* scripts/features/output-sync (output_sync_set): Add tests for
+	the per-job syntax mode.
+
 2013-04-15  Paul Smith  <psmith@gnu.org>
 
 	* scripts/features/output-sync (output_sync_set): New arg syntax.
diff --git a/tests/scripts/features/output-sync b/tests/scripts/features/output-sync
index dce2ac4..babc08f 100644
--- a/tests/scripts/features/output-sync
+++ b/tests/scripts/features/output-sync
@@ -15,6 +15,18 @@ else {
   $sleep_command = "sleep";
 }
 
+# The following subdirectories with Makefiles are used in several
+# of the following tests.  The model is:
+#   foo/Makefile - has a "foo" target that waits for the bar target
+#   bar/Makefile - has a "bar" target that runs immediately
+#                - has a "baz" target that waits for the foo target
+#
+# So, you start the two sub-makes in parallel and first the "bar" target is
+# built, followed by "foo", followed by "baz".  The trick is that first each
+# target prints a "start" statement, then waits (if appropriate), then prints
+# an end statement.  Thus we can tell if the -O flag is working, since
+# otherwise these statements would be mixed together.
+
 @syncfiles = ();
 
 sub output_sync_clean {
@@ -36,19 +48,21 @@ sub output_sync_set {
     return "date > ../mksync.$_[0]";
 }
 
-@syncfiles = qw(mksync.foo mksync.bar);
+@syncfiles = qw(mksync.foo mksync.foo_start mksync.bar mksync.bar_start);
 
-# The following subdirectories with Makefiles are used in several
-# of the following tests.
 output_sync_clean();
 mkdir('foo', 0777);
 mkdir('bar', 0777);
 
 $set_foo = output_sync_set('foo');
 $set_bar = output_sync_set('bar');
+$set_foo_start = output_sync_set('foo_start');
+$set_bar_start = output_sync_set('bar_start');
 
 $wait_foo = output_sync_wait('foo');
 $wait_bar = output_sync_wait('bar');
+$wait_foo_start = output_sync_set('foo_start');
+$wait_bar_start = output_sync_set('bar_start');
 
 open(MAKEFILE,"> foo/Makefile");
 print MAKEFILE <<EOF;
@@ -56,9 +70,25 @@ all: foo
 
 foo: foo-base ; \@$set_foo
 
-foo-base: ; \@echo foo: start; $wait_bar ; echo foo: end
-
-foo-fail: ; \@$wait_bar ; false
+foo-base:
+\t\@echo foo: start
+\t\@$wait_bar
+\t\@echo foo: end
+
+foo-job: foo-job-base ; \@$set_foo
+
+foo-job-base:
+\t\@$wait_bar_start
+\t\@echo foo: start
+\t\@$set_foo_start
+\t\@$wait_bar
+\t\@echo foo: end
+
+foo-fail:
+\t\@echo foo-fail: start
+\t\@$wait_bar
+\t\@echo foo-fail: end
+\t\@false
 EOF
 close(MAKEFILE);
 
@@ -66,15 +96,28 @@ open(MAKEFILE,"> bar/Makefile");
 print MAKEFILE <<EOF;
 all: bar baz
 
-bar: ; \@echo bar: start; echo bar: end; $set_bar
+bar: bar-base ; \@$set_bar
+bar-base:
+\t\@echo bar: start
+\t\@echo bar: end
 
-baz: baz-base
+bar-job: bar-job-base ; \@$set_bar
+
+bar-job-base:
+\t\@echo bar: start
+\t\@$set_bar_start
+\t\@$wait_foo_start
+\t\@echo bar: end
 
-baz-base: ; \@echo baz: start; $wait_foo; echo baz: end
+baz: baz-base
+baz-base:
+\t\@echo baz: start
+\t\@$wait_foo
+\t\@echo baz: end
 EOF
 close(MAKEFILE);
 
-# Test coarse synchronization.
+# Test per-make synchronization.
 unlink(@syncfiles);
 run_make_test(qq!
 all: make-foo make-bar
@@ -102,7 +145,7 @@ baz: end
 #MAKE#[1]: Leaving directory '#PWD#/bar'
 #MAKE#[1]: Leaving directory '#PWD#/bar'\n", 0, 6);
 
-# Test fine synchronization.
+# Test per-target synchronization.
 # Note we have to sleep again here after starting the foo makefile before
 # starting the bar makefile, otherwise the "entering/leaving" messages for the
 # submakes might be ordered differently than we expect.
@@ -154,16 +197,54 @@ bar: end
 #MAKE#[1]: Leaving directory '#PWD#/bar'
 #MAKE#[1]: Leaving directory '#PWD#/bar'
 #MAKE#[1]: Entering directory '#PWD#/foo'
-Makefile:7: recipe for target 'foo-fail' failed
-#MAKE#[1]: Leaving directory '#PWD#/foo'
-#MAKE#[1]: Entering directory '#PWD#/foo'
-#MAKE#[1]: *** [foo-fail] Error 1
+Makefile:20: recipe for target 'foo-fail' failed
+make[1]: Leaving directory '/home/psmith/src/make/make/tests/foo'
+make[1]: Entering directory '/home/psmith/src/make/make/tests/foo'
+make[1]: *** [foo-fail] Error 1
+make[1]: Leaving directory '/home/psmith/src/make/make/tests/foo'
+make[1]: Entering directory '/home/psmith/src/make/make/tests/foo'
+foo-fail: start
+foo-fail: end
 #MAKE#[1]: Leaving directory '#PWD#/foo'
 #MAKE#[1]: Leaving directory '#PWD#/foo'
 #MAKEFILE#:4: recipe for target 'make-foo-fail' failed
 #MAKE#: *** [make-foo-fail] Error 2\n",
 512);
 
+# Test the per-job synchronization.
+# For this we'll have bar-job:
+#   print start, invoke bar-start, wait for foo-start, print end, print-bar-end
+# And foo-job:
+#   wait for bar-start, print foo-start, wait for bar-end, print end
+
+unlink(@syncfiles);
+run_make_test(qq!
+all: make-foo make-bar
+
+make-foo: ; \$(MAKE) -C foo foo-job
+
+make-bar: ; $sleep_command 1 ; \$(MAKE) -C bar bar-job!,
+              '-j --output-sync=job',
+"#MAKEPATH# -C foo foo-job
+$sleep_command 1 ; #MAKEPATH# -C bar bar-job
+#MAKE#[1]: Entering directory '#PWD#/foo'
+#MAKE#[1]: Entering directory '#PWD#/foo'
+foo: start
+#MAKE#[1]: Leaving directory '#PWD#/foo'
+#MAKE#[1]: Entering directory '#PWD#/bar'
+#MAKE#[1]: Entering directory '#PWD#/bar'
+bar: start
+#MAKE#[1]: Leaving directory '#PWD#/bar'
+#MAKE#[1]: Entering directory '#PWD#/bar'
+bar: end
+#MAKE#[1]: Leaving directory '#PWD#/bar'
+#MAKE#[1]: Leaving directory '#PWD#/bar'
+#MAKE#[1]: Entering directory '#PWD#/foo'
+foo: end
+#MAKE#[1]: Leaving directory '#PWD#/foo'
+#MAKE#[1]: Leaving directory '#PWD#/foo'\n", 0, 6);
+
+
 # Remove temporary directories and contents.
 output_sync_clean();