Remake
 All Classes Files Functions Variables Typedefs Enumerations Enumerator Macros Pages
remake.cpp
Go to the documentation of this file.
1 /**
2 @mainpage Remake, a build system that bridges the gap between make and redo.
3 
4 As with <b>make</b>, <b>remake</b> uses a centralized rule file, which is
5 named <b>Remakefile</b>. It contains rules with a <em>make</em>-like
6 syntax:
7 
8 @verbatim
9 target1 target2 ... : dependency1 dependency2 ...
10  shell script
11  that builds
12  the targets
13 @endverbatim
14 
15 A target is known to be up-to-date if all its dependencies are. If it
16 has no known dependencies yet the file already exits, it is assumed to
17 be up-to-date. Obsolete targets are rebuilt thanks to the shell script
18 provided by the rule.
19 
20 As with <b>redo</b>, <b>remake</b> supports dynamic dependencies in
21 addition to these static dependencies. Whenever a script executes
22 <tt>remake dependency4 dependency5 ...</tt>, these dependencies are
23 rebuilt if they are obsolete. (So <b>remake</b> acts like
24 <b>redo-ifchange</b>.) Moreover, these dependencies are stored in file
25 <b>.remake</b> so that they are remembered in subsequent runs. Note that
26 dynamic dependencies from previous runs are only used to decide whether a
27 target is obsolete; they are not automatically rebuilt when they are
28 obsolete yet a target depends on them. They will only be rebuilt once the
29 dynamic call to <b>remake</b> is executed.
30 
31 In other words, the following two rules have almost the same behavior.
32 
33 @verbatim
34 target1 target2 ... : dependency1 dependency2 ...
35  shell script
36 
37 target1 target2 ... :
38  remake dependency1 dependency2 ...
39  shell script
40 @endverbatim
41 
42 (There is a difference if the targets already exist, have never been
43 built before, and the dependencies are either younger or obsolete, since
44 the targets will not be rebuilt in the second case.)
45 
46 The above usage of dynamic dependencies is hardly useful. Their strength
47 lies in the fact that they can be computed on the fly:
48 
49 @verbatim
50 %.o : %.c
51  gcc -MMD -MF $1.d -o $1 -c ${1%.o}.c
52  remake -r < $1.d
53  rm $1.d
54 
55 %.cmo : %.ml
56  ocamldep ${1%.cmo}.ml | remake -r $1
57  ocamlc -c ${1%.cmo}.ml
58 
59 after.xml: before.xml rules.xsl
60  xsltproc --load-trace -o after.xml rules.xsl before.xml 2> deps
61  remake $(sed -n -e "\\,//,! s,^.*URL=\"\\([^\"]*\\).*\$,\\1,p" deps)
62  rm deps
63 @endverbatim
64 
65 Note that the first rule fails if any of the header files included by
66 a C source file has to be automatically generated. In that case, one
67 should perform a first call to <b>remake</b> them before calling the
68 compiler. (Dependencies from several calls to <b>remake</b> are
69 cumulative, so they will all be remembered the next time.)
70 
71 \section sec-usage Usage
72 
73 Usage: <tt>remake <i>options</i> <i>targets</i></tt>
74 
75 Options:
76 
77 - <tt>-d</tt>: Echo script commands.
78 - <tt>-j[N]</tt>, <tt>--jobs=[N]</tt>: Allow N jobs at once; infinite jobs
79  with no argument.
80 - <tt>-k</tt>, <tt>--keep-going</tt>: Keep going when some targets cannot be made.
81 - <tt>-r</tt>: Look up targets from the dependencies on standard input.
82 - <tt>-s</tt>, <tt>--silent</tt>, <tt>--quiet</tt>: Do not echo targets.
83 
84 \section sec-syntax Syntax
85 
86 Lines starting with a space character or a tabulation are assumed to be rule
87 scripts. They are only allowed after a rule header.
88 
89 Lines starting with <tt>#</tt> are considered to be comments and are ignored.
90 They do interrupt rule scripts though.
91 
92 Any other line is either a rule header or a variable definition. If such a
93 line ends with a backslash, the following line break is ignored and the line
94 extends to the next one.
95 
96 Rule headers are a nonempty list of names, followed by a colon, followed by
97 another list of names, possibly empty. Variable definitions are a single
98 name followed by equal followed by a list of names, possibly empty. Basically,
99 the syntax of a rule is as follows:
100 
101 @verbatim
102 targets : prerequisites
103  shell script
104 @endverbatim
105 
106 List of names are space-separated sequences of names. If a name contains a
107 space character, it should be put into double quotes. Names can not be any
108 of the following special characters <tt>:$(),="</tt>. Again, quotation
109 should be used. Quotation marks can be escaped by a backslash inside
110 quoted names.
111 
112 \subsection sec-variables Variables
113 
114 Variables can be used to factor lists of targets or dependencies. They are
115 expanded as they are encountered during <b>Remakefile</b> parsing.
116 
117 @verbatim
118 VAR1 = c d
119 VAR2 = a $(VAR1) b
120 $(VAR2) e :
121 @endverbatim
122 
123 Variables can be used inside rule scripts; they are available as non-exported
124 shell variables there.
125 
126 \subsection sec-functions Built-in functions
127 
128 <b>remake</b> also supports a few built-in functions inspired from <b>make</b>.
129 
130 - <tt>$(addprefix <i>prefix</i>, <i>list</i>)</tt> returns the list obtained
131  by prepending its first argument to each element of its second argument.
132 - <tt>$(addsuffix <i>suffix</i>, <i>list</i>)</tt> returns the list obtained
133  by appending its first argument to each element of its second argument.
134 
135 Note that functions are ignored inside scripts.
136 
137 \section sec-semantics Semantics
138 
139 \subsection src-obsolete When are targets obsolete?
140 
141 A target is obsolete:
142 
143 - if there is no file corresponding to the target, or to one of its siblings
144  in a multi-target rule,
145 - if any of its dynamic dependencies from a previous run or any of its static
146  prerequisites is obsolete,
147 - if the latest file corresponding to its siblings or itself is older than any
148  of its dynamic dependencies or static prerequisites.
149 
150 In all the other cases, it is assumed to be up-to-date (and so are all its
151 siblings). Note that the last rule above says "latest" and not "earliest". While
152 it might cause some obsolete targets to go unnoticed in corner cases, it allows
153 for the following kind of rules:
154 
155 @verbatim
156 config.h stamp-config_h: config.h.in config.status
157  ./config.status config.h
158  touch stamp-config_h
159 @endverbatim
160 
161 A <tt>config.status</tt> file generally does not update header files (here
162 <tt>config.h</tt>) if they would not change. As a consequence, if not for the
163 <tt>stamp-config_h</tt> file above, a header would always be considered obsolete
164 once one of its prerequisites is modified. Note that touching <tt>config.h</tt>
165 rather than <tt>stamp-config_h</tt> would defeat the point of not updating it
166 in the first place, since the program files would need to be rebuilt.
167 
168 Once all the static prerequisites of a target have been rebuilt, <b>remake</b>
169 checks if the target still needs to be built. If it was obsolete only because
170 its dependencies needed to be rebuilt and none of them changed, the target is
171 assumed to be up-to-date.
172 
173 \subsection sec-rules How are targets (re)built?
174 
175 There are two kinds of rules. If any of the targets or prerequisites contains
176 a <tt>%</tt> character, the rule is said to be <em>generic</em>. All the
177 targets of the rule shall then contain a single <tt>%</tt> character. All the
178 other rules are said to be <em>specific</em>.
179 
180 A rule is said to <em>match</em> a given target:
181 
182 - if it is specific and the target appears inside its target list,
183 - if it is generic and there is a way to replace the <tt>%</tt> character
184  from one of its targets so that it matches the given target.
185 
186 When <b>remake</b> tries to build a given target, it looks for a specific rule
187 that matches it. If there is one and its script is nonempty, it uses it to
188 rebuild the target.
189 
190 Otherwise, it looks for a generic rule that match the target. If there are
191 several matching rules, it chooses the one with the shortest pattern (and if
192 there are several ones, the earliest one). <b>remake</b> then looks for
193 specific rules that match each target of the generic rule. All the
194 prerequisites of these specific rules are added to those of the generic rule.
195 The script of the generic rule is used to build the target.
196 
197 Example:
198 
199 @verbatim
200 t%1 t2%: p1 p%2
201  commands building t%1 and t2%
202 
203 t2z: p4
204  commands building t2z
205 
206 ty1: p3
207 
208 # t2x is built by the first rule (which also builds tx1) and its prerequisites are p1, px2
209 # t2y is built by the first rule (which also builds ty1) and its prerequisites are p1, py2, p3
210 # t2z is built by the second rule and its prerequisite is p4
211 @endverbatim
212 
213 The set of rules from <b>Remakefile</b> is ill-formed:
214 
215 - if any specific rule matching a target of the generic rule has a nonempty script,
216 - if any target of the generic rule is matched by a generic rule with a shorter pattern.
217 
218 \section sec-compilation Compilation
219 
220 - On Linux, MacOSX, and BSD: <tt>g++ -o remake remake.cpp</tt>
221 - On Windows: <tt>g++ -o remake.exe remake.cpp -lws2_32</tt>
222 
223 Installing <b>remake</b> is needed only if <b>Remakefile</b> does not
224 specify the path to the executable for its recursive calls. Thanks to its
225 single source file, <b>remake</b> can be shipped inside other packages and
226 built at configuration time.
227 
228 \section sec-differences Differences with other build systems
229 
230 Differences with <b>make</b>:
231 
232 - Dynamic dependencies are supported.
233 - For rules with multiple targets, the shell script is executed only once
234  and is assumed to build all the targets. There is no need for
235  convoluted rules that are robust enough for parallel builds. For generic
236  rules, this is similar to the behavior of pattern rules from <b>gmake</b>.
237 - As with <b>redo</b>, only one shell is run when executing a script,
238  rather than one per script line. Note that the shells are run with
239  option <tt>-e</tt>, thus causing them to exit as soon as an error is
240  encountered.
241 - The dependencies of generic rules (known as implicit rules in make lingo)
242  are not used to decide between several of them. <b>remake</b> does not
243  select one for which it could satisfy the dependencies.
244 - Variables and built-in functions are expanded as they are encountered
245  during <b>Remakefile</b> parsing.
246 
247 Differences with <b>redo</b>:
248 
249 - As with <b>make</b>, it is possible to write the following kind of rules
250  in <b>remake</b>.
251 @verbatim
252 Remakefile: Remakefile.in ./config.status
253  ./config.status Remakefile
254 @endverbatim
255 - If a target is already built the first time <b>remake</b> runs, it still
256  uses the static prerequisites of rules mentioning it to check whether it
257  needs to be rebuilt. It does not assume it to be up-to-date. As with
258  <b>redo</b> though, if its obsolete status would be due to a dynamic
259  dependency, it will go unnoticed; it should be removed beforehand.
260 - <b>remake</b> has almost no features: no checksum-based dependencies, no
261  compatibility with token servers, etc.
262 
263 Differences with both <b>make</b> and <b>redo</b>:
264 
265 - Multiple targets are supported.
266 - When executing shell scripts, positional variables <tt>$1</tt>,
267  <tt>$2</tt>, etc, point to the target names of the rule obtained after
268  substituting <tt>%</tt>. No other variables are defined.
269 
270 \section sec-limitations Limitations
271 
272 - When the user or a script calls <b>remake</b>, the current working
273  directory should be the one containing <b>Remakefile</b> (and thus
274  <b>.remake</b> too).
275 - Some cases of ill-formed rules are not caught by <b>remake</b> and can
276  thus lead to unpredictable behaviors.
277 
278 \section sec-links Links
279 
280 @see http://cr.yp.to/redo.html for the philosophy of <b>redo</b> and
281 https://github.com/apenwarr/redo for an implementation and some comprehensive documentation.
282 
283 \section sec-licensing Licensing
284 
285 @author Guillaume Melquiond
286 @version 0.4
287 @date 2012-2013
288 @copyright
289 This program is free software: you can redistribute it and/or modify
290 it under the terms of the GNU General Public License as published by
291 the Free Software Foundation, either version 3 of the License, or
292 (at your option) any later version.
293 \n
294 This program is distributed in the hope that it will be useful,
295 but WITHOUT ANY WARRANTY; without even the implied warranty of
296 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
297 GNU General Public License for more details.
298 */
299 
300 #ifdef _WIN32
301 #define WIN32_LEAN_AND_MEAN
302 #define WINDOWS
303 #endif
304 
305 #include <fstream>
306 #include <iostream>
307 #include <list>
308 #include <map>
309 #include <set>
310 #include <sstream>
311 #include <string>
312 #include <vector>
313 #include <cassert>
314 #include <cstdlib>
315 #include <ctime>
316 #include <errno.h>
317 #include <fcntl.h>
318 #include <signal.h>
319 #include <unistd.h>
320 #include <sys/stat.h>
321 #include <sys/types.h>
322 
323 #ifdef __APPLE__
324 #define MACOSX
325 #endif
326 
327 #ifdef __linux__
328 #define LINUX
329 #endif
330 
331 #ifdef WINDOWS
332 #include <windows.h>
333 #include <winbase.h>
334 #include <winsock2.h>
335 #define pid_t HANDLE
336 typedef SOCKET socket_t;
337 #else
338 #include <sys/socket.h>
339 #include <sys/un.h>
340 #include <sys/wait.h>
341 typedef int socket_t;
342 enum { INVALID_SOCKET = -1 };
343 #endif
344 
345 #if defined(WINDOWS) || defined(MACOSX)
346 enum { MSG_NOSIGNAL = 0 };
347 #endif
348 
349 typedef std::list<std::string> string_list;
350 
351 typedef std::set<std::string> string_set;
352 
353 /**
354  * Reference-counted shared object.
355  * @note The default constructor delays the creation of the object until it
356  * is first dereferenced.
357  */
358 template<class T>
359 struct ref_ptr
360 {
361  struct content
362  {
363  size_t cnt;
364  T val;
365  content(): cnt(1) {}
366  content(T const &t): cnt(1), val(t) {}
367  };
368  mutable content *ptr;
369  ref_ptr(): ptr(NULL) {}
370  ref_ptr(T const &t): ptr(new content(t)) {}
371  ref_ptr(ref_ptr const &p): ptr(p.ptr) { if (ptr) ++ptr->cnt; }
372  ~ref_ptr() { if (ptr && --ptr->cnt == 0) delete ptr; }
374  {
375  if (ptr == p.ptr) return *this;
376  if (ptr && --ptr->cnt == 0) delete ptr;
377  ptr = p.ptr;
378  if (ptr) ++ptr->cnt;
379  return *this;
380  }
381  T &operator*() const
382  {
383  if (!ptr) ptr = new content;
384  return ptr->val;
385  }
386  T *operator->() const { return &**this; }
387 };
388 
390 {
393 };
394 
395 typedef std::map<std::string, ref_ptr<dependency_t> > dependency_map;
396 
397 typedef std::map<std::string, string_list> variable_map;
398 
399 /**
400  * Build status of a target.
401  */
403 {
404  Uptodate, ///< Target is up-to-date.
405  Todo, ///< Target is missing or obsolete.
406  Recheck, ///< Target has an obsolete dependency.
407  Running, ///< Target is being rebuilt.
408  Remade, ///< Target was successfully rebuilt.
409  Failed ///< Build failed for target.
410 };
411 
412 /**
413  * Build status of a target.
414  */
415 struct status_t
416 {
417  status_e status; ///< Actual status.
418  time_t last; ///< Last-modified date.
419 };
420 
421 typedef std::map<std::string, status_t> status_map;
422 
423 /**
424  * A rule loaded from Remakefile.
425  */
426 struct rule_t
427 {
428  string_list targets; ///< Files produced by this rule.
429  string_list deps; ///< Files used for an implicit call to remake at the start of the script.
430  std::string script; ///< Shell script for building the targets.
431 };
432 
433 typedef std::list<rule_t> rule_list;
434 
435 typedef std::map<std::string, ref_ptr<rule_t> > rule_map;
436 
437 typedef std::map<int, string_list> job_targets_map;
438 
439 typedef std::map<pid_t, int> pid_job_map;
440 
441 /**
442  * Client waiting for a request complete.
443  *
444  * There are two kinds of clients:
445  * - real clients, which are instances of remake created by built scripts,
446  * - pseudo clients, which are created by the server to build specific targets.
447  *
448  * Among pseudo clients, there are two categories:
449  * - original clients, which are created for the targets passed on the
450  * command line by the user or for the initial regeneration of the rule file,
451  * - dependency clients, which are created to handle rules that have
452  * explicit dependencies and thus to emulate a call to remake.
453  */
454 struct client_t
455 {
456  socket_t socket; ///< Socket used to reply to the client (invalid for pseudo clients).
457  int job_id; ///< Job for which the built script called remake and spawned the client (negative for original clients).
458  bool failed; ///< Whether some targets failed in mode -k.
459  string_list pending; ///< Targets not yet started.
460  string_set running; ///< Targets being built.
461  rule_t *delayed; ///< Rule that implicitly created a dependency client, and which script has to be started on request completion.
462  client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(NULL) {}
463 };
464 
465 typedef std::list<client_t> client_list;
466 
467 /**
468  * Map from variable names to their content.
469  */
471 
472 /**
473  * Precomputed variable assignments for shell usage.
474  */
475 static std::string variable_block;
476 
477 /**
478  * Map from targets to their known dependencies.
479  */
481 
482 /**
483  * Map from targets to their build status.
484  */
486 
487 /**
488  * Set of generic rules loaded from Remakefile.
489  */
491 
492 /**
493  * Map from targets to specific rules loaded from Remakefile.
494  */
496 
497 /**
498  * Map from jobs to targets being built.
499  */
501 
502 /**
503  * Map from jobs to shell pids.
504  */
506 
507 /**
508  * List of clients waiting for a request to complete.
509  * New clients are put to front, so that the build process is depth-first.
510  */
512 
513 /**
514  * Maximum number of parallel jobs (non-positive if unbounded).
515  * Can be modified by the -j option.
516  */
517 static int max_active_jobs = 1;
518 
519 /**
520  * Whether to keep building targets in case of failure.
521  * Can be modified by the -k option.
522  */
523 static bool keep_going = false;
524 
525 /**
526  * Number of jobs currently running:
527  * - it increases when a process is created in #run_script,
528  * - it decreases when a completion message is received in #server_loop.
529  *
530  * @note There might be some jobs running while #clients is empty.
531  * Indeed, if a client requested two targets to be rebuilt, if they
532  * are running concurrently, if one of them fails, the client will
533  * get a failure notice and might terminate before the other target
534  * finishes.
535  */
536 static int running_jobs = 0;
537 
538 /**
539  * Number of jobs currently waiting for a build request to finish:
540  * - it increases when a build request is received in #accept_client
541  * (since the client is presumably waiting for the reply),
542  * - it decreases when a reply is sent in #complete_request.
543  */
544 static int waiting_jobs = 0;
545 
546 /**
547  * Global counter used to produce increasing job numbers.
548  * @see job_targets
549  */
550 static int job_counter = 0;
551 
552 /**
553  * Socket on which the server listens for client request.
554  */
556 
557 /**
558  * Whether the request of an original client failed.
559  */
560 static bool build_failure;
561 
562 /**
563  * Name of the server socket in the file system.
564  */
565 static char *socket_name;
566 
567 /**
568  * Name of the first target of the first specific rule, used for default run.
569  */
570 static std::string first_target;
571 
572 /**
573  * Whether a short message should be displayed for each target.
574  */
575 static bool show_targets = true;
576 
577 /**
578  * Whether script commands are echoed.
579  */
580 static bool echo_scripts = false;
581 
582 static time_t now = time(NULL);
583 
584 static std::string working_dir;
585 
586 #ifndef WINDOWS
587 static volatile sig_atomic_t got_SIGCHLD = 0;
588 
589 static void child_sig_handler(int)
590 {
591  got_SIGCHLD = 1;
592 }
593 #endif
594 
595 struct log
596 {
597  bool active, open;
598  int depth;
599  log(): active(false), open(false), depth(0)
600  {
601  }
602  std::ostream &operator()()
603  {
604  if (open) std::cerr << std::endl;
605  assert(depth >= 0);
606  std::cerr << std::string(depth * 2, ' ');
607  open = false;
608  return std::cerr;
609  }
610  std::ostream &operator()(bool o)
611  {
612  if (o && open) std::cerr << std::endl;
613  if (!o) --depth;
614  assert(depth >= 0);
615  if (o || !open) std::cerr << std::string(depth * 2, ' ');
616  if (o) ++depth;
617  open = o;
618  return std::cerr;
619  }
620 };
621 
623 
625 {
628  {
629  }
631  {
632  if (debug.active && still_open) debug(false) << "done\n";
633  }
634 };
635 
636 #define DEBUG if (debug.active) debug()
637 #define DEBUG_open log_auto_close auto_close; if (debug.active) debug(true)
638 #define DEBUG_close if ((auto_close.still_open = false), debug.active) debug(false)
639 
640 /**
641  * Return the original string if it does not contain any special characters,
642  * a quoted and escaped string otherwise.
643  */
644 static std::string escape_string(std::string const &s)
645 {
646  char const *quoted_char = ",: '";
647  char const *escaped_char = "\"\\$!";
648  bool need_quotes = false;
649  size_t len = s.length(), nb = len;
650  for (size_t i = 0; i < len; ++i)
651  {
652  if (strchr(quoted_char, s[i])) need_quotes = true;
653  if (strchr(escaped_char, s[i])) ++nb;
654  }
655  if (nb != len) need_quotes = true;
656  if (!need_quotes) return s;
657  std::string t(nb + 2, '\\');
658  t[0] = '"';
659  for (size_t i = 0, j = 1; i < len; ++i, ++j)
660  {
661  if (strchr(escaped_char, s[i])) ++j;
662  t[j] = s[i];
663  }
664  t[nb + 1] = '"';
665  return t;
666 }
667 
668 /**
669  * Initialize #working_dir.
670  */
672 {
673  char buf[1024];
674  char *res = getcwd(buf, sizeof(buf));
675  if (!res)
676  {
677  perror("Failed to get working directory");
678  exit(EXIT_FAILURE);
679  }
680  working_dir = buf;
681 }
682 
683 /**
684  * Normalize an absolute path with respect to the working directory.
685  * Paths outside the working subtree are left unchanged.
686  */
687 static std::string normalize_abs(std::string const &s)
688 {
689  size_t l = working_dir.length();
690  if (s.compare(0, l, working_dir)) return s;
691  size_t ll = s.length();
692  if (ll == l) return ".";
693  if (s[l] != '/')
694  {
695  size_t pos = s.rfind('/', l);
696  assert(pos != std::string::npos);
697  return s.substr(pos + 1);
698  }
699  if (ll == l + 1) return ".";
700  return s.substr(l + 1);
701 }
702 
703 /**
704  * Normalize a target name.
705  */
706 static std::string normalize(std::string const &s)
707 {
708 #ifdef WINDOWS
709  char const *delim = "/\\";
710 #else
711  char delim = '/';
712 #endif
713  size_t prev = 0, len = s.length();
714  size_t pos = s.find_first_of(delim);
715  if (pos == std::string::npos) return s;
716  bool absolute = pos == 0;
717  string_list l;
718  for (;;)
719  {
720  if (pos != prev)
721  {
722  std::string n = s.substr(prev, pos - prev);
723  if (n == "..")
724  {
725  if (!l.empty()) l.pop_back();
726  else if (!absolute)
727  return normalize(working_dir + '/' + s);
728  }
729  else if (n != ".")
730  l.push_back(n);
731  }
732  ++pos;
733  if (pos >= len) break;
734  prev = pos;
735  pos = s.find_first_of(delim, prev);
736  if (pos == std::string::npos) pos = len;
737  }
738  string_list::const_iterator i = l.begin(), i_end = l.end();
739  if (i == i_end) return absolute ? "/" : ".";
740  std::string n;
741  if (absolute) n.push_back('/');
742  n.append(*i);
743  for (++i; i != i_end; ++i)
744  {
745  n.push_back('/');
746  n.append(*i);
747  }
748  if (absolute) return normalize_abs(n);
749  return n;
750 }
751 
752 /**
753  * Normalize the content of a list of targets.
754  */
756 {
757  for (string_list::iterator i = l.begin(),
758  i_end = l.end(); i != i_end; ++i)
759  {
760  *i = normalize(*i);
761  }
762 }
763 
764 /**
765  * Skip spaces.
766  */
767 static void skip_spaces(std::istream &in)
768 {
769  char c;
770  while (strchr(" \t", (c = in.get()))) {}
771  if (in.good()) in.putback(c);
772 }
773 
774 /**
775  * Skip end of line.
776  */
777 static void skip_eol(std::istream &in)
778 {
779  char c;
780  while (strchr("\r\n", (c = in.get()))) {}
781  if (in.good()) in.putback(c);
782 }
783 
785 
786 /**
787  * Skip spaces and return the kind of the next token.
788  */
789 static token_e next_token(std::istream &in)
790 {
791  while (true)
792  {
793  skip_spaces(in);
794  char c = in.peek();
795  if (!in.good()) return Eof;
796  switch (c)
797  {
798  case ':': return Colon;
799  case ',': return Comma;
800  case '=': return Equal;
801  case '$': return Dollar;
802  case ')': return Rightpar;
803  case '\r':
804  case '\n':
805  return Eol;
806  case '\\':
807  in.ignore(1);
808  c = in.peek();
809  if (c != '\r' && c != '\n')
810  {
811  in.putback('\\');
812  return Word;
813  }
814  skip_eol(in);
815  break;
816  default:
817  return Word;
818  }
819  }
820 }
821 
822 /**
823  * Read a (possibly quoted) word.
824  */
825 static std::string read_word(std::istream &in)
826 {
827  int c = in.get();
828  std::string res;
829  if (!in.good()) return res;
830  char const *separators = " \t\r\n:$(),=\"";
831  bool quoted = c == '"';
832  if (!quoted)
833  {
834  if (strchr(separators, c))
835  {
836  in.putback(c);
837  return res;
838  }
839  res += c;
840  }
841  while (true)
842  {
843  c = in.get();
844  if (!in.good()) return res;
845  if (quoted)
846  {
847  if (c == '\\')
848  res += in.get();
849  else if (c == '"')
850  return res;
851  else
852  res += c;
853  }
854  else
855  {
856  if (strchr(separators, c))
857  {
858  in.putback(c);
859  return res;
860  }
861  res += c;
862  }
863  }
864 }
865 
866 static string_list read_words(std::istream &in);
867 
868 /**
869  * Execute a built-in function @a name and append its result to @a dest.
870  */
871 static void execute_function(std::istream &in, std::string const &name, string_list &dest)
872 {
873  if (false)
874  {
875  error:
876  std::cerr << "Failed to load rules: syntax error" << std::endl;
877  exit(EXIT_FAILURE);
878  }
879  skip_spaces(in);
880  string_list fix = read_words(in);
881  if (next_token(in) != Comma) goto error;
882  in.ignore(1);
883  string_list names = read_words(in);
884  if (next_token(in) != Rightpar) goto error;
885  in.ignore(1);
886  size_t fixl = fix.size();
887  if (name == "addprefix")
888  {
889  for (string_list::const_iterator i = names.begin(),
890  i_end = names.end(); i != i_end; ++i)
891  {
892  if (!fixl)
893  {
894  dest.push_back(*i);
895  continue;
896  }
897  string_list::const_iterator k = fix.begin();
898  for (size_t j = 1; j != fixl; ++j)
899  {
900  dest.push_back(*k++);
901  }
902  dest.push_back(*k++ + *i);
903  }
904  }
905  else if (name == "addsuffix")
906  {
907  for (string_list::const_iterator i = names.begin(),
908  i_end = names.end(); i != i_end; ++i)
909  {
910  if (!fixl)
911  {
912  dest.push_back(*i);
913  continue;
914  }
915  string_list::const_iterator k = fix.begin();
916  dest.push_back(*i + *k++);
917  for (size_t j = 1; j != fixl; ++j)
918  {
919  dest.push_back(*k++);
920  }
921  }
922  }
923  else goto error;
924 }
925 
926 /**
927  * Read a list of words, possibly executing functions.
928  */
929 static string_list read_words(std::istream &in)
930 {
931  if (false)
932  {
933  error:
934  std::cerr << "Failed to load rules: syntax error" << std::endl;
935  exit(EXIT_FAILURE);
936  }
937  string_list res;
938  while (true)
939  {
940  switch (next_token(in))
941  {
942  case Word:
943  res.push_back(read_word(in));
944  break;
945  case Dollar:
946  {
947  in.ignore(1);
948  if (in.get() != '(') goto error;
949  std::string name = read_word(in);
950  if (name.empty()) goto error;
951  token_e tok = next_token(in);
952  if (tok == Rightpar)
953  {
954  in.ignore(1);
955  variable_map::const_iterator i = variables.find(name);
956  if (i != variables.end())
957  res.insert(res.end(), i->second.begin(), i->second.end());
958  }
959  else execute_function(in, name, res);
960  }
961  default:
962  return res;
963  }
964  }
965 }
966 
967 /**
968  * Load dependencies from @a in.
969  */
970 static void load_dependencies(std::istream &in)
971 {
972  while (!in.eof())
973  {
974  string_list targets = read_words(in);
975  if (targets.empty()) return;
976  DEBUG << "reading dependencies of target " << targets.front() << std::endl;
977  if (in.get() != ':')
978  {
979  std::cerr << "Failed to load database" << std::endl;
980  exit(EXIT_FAILURE);
981  }
983  dep->targets = targets;
984  string_list d = read_words(in);
985  dep->deps.insert(d.begin(), d.end());
986  for (string_list::const_iterator i = targets.begin(),
987  i_end = targets.end(); i != i_end; ++i)
988  {
989  dependencies[*i] = dep;
990  }
991  skip_eol(in);
992  }
993 }
994 
995 /**
996  * Load known dependencies from file <tt>.remake</tt>.
997  */
998 static void load_dependencies()
999 {
1000  DEBUG_open << "Loading database... ";
1001  std::ifstream in(".remake");
1002  if (!in.good())
1003  {
1004  DEBUG_close << "not found\n";
1005  return;
1006  }
1007  load_dependencies(in);
1008 }
1009 
1010 /**
1011  * Read a rule starting with target @a first, if nonempty.
1012  * Store into #generic_rules or #specific_rules depending on its genericity.
1013  */
1014 static void load_rule(std::istream &in, std::string const &first)
1015 {
1016  DEBUG_open << "Reading rule for target " << first << "... ";
1017  if (false)
1018  {
1019  error:
1020  DEBUG_close << "failed\n";
1021  std::cerr << "Failed to load rules: syntax error" << std::endl;
1022  exit(EXIT_FAILURE);
1023  }
1024  rule_t rule;
1025 
1026  // Read targets and check genericity.
1027  string_list targets = read_words(in);
1028  if (!first.empty()) targets.push_front(first);
1029  else if (targets.empty()) goto error;
1030  else DEBUG << "actual target: " << targets.front() << std::endl;
1031  bool generic = false;
1032  normalize_list(targets);
1033  for (string_list::const_iterator i = targets.begin(),
1034  i_end = targets.end(); i != i_end; ++i)
1035  {
1036  if (i->empty()) goto error;
1037  if ((i->find('%') != std::string::npos) != generic)
1038  {
1039  if (i == targets.begin()) generic = true;
1040  else goto error;
1041  }
1042  }
1043  std::swap(rule.targets, targets);
1044  skip_spaces(in);
1045  if (in.get() != ':') goto error;
1046 
1047  // Read dependencies.
1048  rule.deps = read_words(in);
1049  normalize_list(rule.deps);
1050  skip_spaces(in);
1051  char c = in.get();
1052  if (c != '\r' && c != '\n') goto error;
1053  skip_eol(in);
1054 
1055  // Read script.
1056  std::ostringstream buf;
1057  while (true)
1058  {
1059  char c = in.get();
1060  if (!in.good()) break;
1061  if (c == '\t' || c == ' ')
1062  {
1063  in.get(*buf.rdbuf());
1064  if (in.fail() && !in.eof()) in.clear();
1065  }
1066  else if (c == '\r' || c == '\n')
1067  buf << c;
1068  else
1069  {
1070  in.putback(c);
1071  break;
1072  }
1073  }
1074  rule.script = buf.str();
1075 
1076  // Add generic rules to the correct set.
1077  if (generic)
1078  {
1079  generic_rules.push_back(rule);
1080  return;
1081  }
1082 
1083  // Rules with a nonempty script lump all their targets in the same
1084  // dependency set, while other rules behave as if they had been
1085  // replicated for each of their targets.
1086  if (!rule.script.empty())
1087  {
1089  dep->targets = rule.targets;
1090  dep->deps.insert(rule.deps.begin(), rule.deps.end());
1091  for (string_list::const_iterator i = rule.targets.begin(),
1092  i_end = rule.targets.end(); i != i_end; ++i)
1093  {
1095  dep->deps.insert(d->deps.begin(), d->deps.end());
1096  d = dep;
1097  }
1098  }
1099  else
1100  {
1101  for (string_list::const_iterator i = rule.targets.begin(),
1102  i_end = rule.targets.end(); i != i_end; ++i)
1103  {
1105  if (dep->targets.empty()) dep->targets.push_back(*i);
1106  dep->deps.insert(rule.deps.begin(), rule.deps.end());
1107  }
1108  }
1109 
1110  if (first_target.empty())
1111  first_target = rule.targets.front();
1112 
1113  ref_ptr<rule_t> r(rule);
1114  for (string_list::const_iterator i = rule.targets.begin(),
1115  i_end = rule.targets.end(); i != i_end; ++i)
1116  {
1117  std::pair<rule_map::iterator,bool> j =
1118  specific_rules.insert(std::make_pair(*i, r));
1119  if (j.second) continue;
1120  std::cerr << "Failed to load rules: " << *i
1121  << " cannot be the target of several rules" << std::endl;
1122  exit(EXIT_FAILURE);
1123  }
1124 }
1125 
1126 /**
1127  * Save all the dependencies in file <tt>.remake</tt>.
1128  */
1129 static void save_dependencies()
1130 {
1131  DEBUG_open << "Saving database... ";
1132  std::ofstream db(".remake");
1133  while (!dependencies.empty())
1134  {
1135  ref_ptr<dependency_t> dep = dependencies.begin()->second;
1136  for (string_list::const_iterator i = dep->targets.begin(),
1137  i_end = dep->targets.end(); i != i_end; ++i)
1138  {
1139  db << escape_string(*i) << ' ';
1140  dependencies.erase(*i);
1141  }
1142  db << ':';
1143  for (string_set::const_iterator i = dep->deps.begin(),
1144  i_end = dep->deps.end(); i != i_end; ++i)
1145  {
1146  db << ' ' << escape_string(*i);
1147  }
1148  db << std::endl;
1149  }
1150 }
1151 
1152 /**
1153  * Load rules.
1154  * If some rules have dependencies and non-generic targets, add these
1155  * dependencies to the targets.
1156  */
1157 static void load_rules()
1158 {
1159  DEBUG_open << "Loading rules... ";
1160  if (false)
1161  {
1162  error:
1163  std::cerr << "Failed to load rules: syntax error" << std::endl;
1164  exit(EXIT_FAILURE);
1165  }
1166  std::ifstream in("Remakefile");
1167  if (!in.good())
1168  {
1169  std::cerr << "Failed to load rules: no Remakefile found" << std::endl;
1170  exit(EXIT_FAILURE);
1171  }
1172  skip_eol(in);
1173 
1174  // Read rules
1175  while (in.good())
1176  {
1177  char c = in.peek();
1178  if (c == '#')
1179  {
1180  while (in.get() != '\n') {}
1181  skip_eol(in);
1182  continue;
1183  }
1184  if (c == ' ' || c == '\t') goto error;
1185  token_e tok = next_token(in);
1186  if (tok == Word)
1187  {
1188  std::string name = read_word(in);
1189  if (name.empty()) goto error;
1190  if (next_token(in) == Equal)
1191  {
1192  in.ignore(1);
1193  DEBUG << "Assignment to variable " << name << std::endl;
1194  variables[name] = read_words(in);
1195  skip_eol(in);
1196  }
1197  else load_rule(in, name);
1198  }
1199  else if (tok == Dollar)
1200  load_rule(in, std::string());
1201  else goto error;
1202  }
1203 
1204  // Generate script for variable assignment
1205  std::ostringstream buf;
1206  for (variable_map::const_iterator i = variables.begin(),
1207  i_end = variables.end(); i != i_end; ++i)
1208  {
1209  std::ostringstream var;
1210  bool first = true;
1211  for (string_list::const_iterator j = i->second.begin(),
1212  j_end = i->second.end(); j != j_end; ++j)
1213  {
1214  if (first) first = false;
1215  else var << ' ';
1216  var << *j;
1217  }
1218  buf << i->first << '=' << escape_string(var.str()) << std::endl;
1219  }
1220  variable_block = buf.str();
1221 }
1222 
1223 /**
1224  * Substitute a pattern into a list of strings.
1225  */
1226 static void substitute_pattern(std::string const &pat, string_list const &src, string_list &dst)
1227 {
1228  for (string_list::const_iterator i = src.begin(),
1229  i_end = src.end(); i != i_end; ++i)
1230  {
1231  size_t pos = i->find('%');
1232  if (pos == std::string::npos)dst.push_back(*i);
1233  else dst.push_back(i->substr(0, pos) + pat + i->substr(pos + 1));
1234  }
1235 }
1236 
1237 /**
1238  * Find a generic rule matching @a target:
1239  * - the one leading to shorter matches has priority,
1240  * - among equivalent rules, the earliest one has priority.
1241  */
1242 static rule_t find_generic_rule(std::string const &target)
1243 {
1244  size_t tlen = target.length(), plen = tlen + 1;
1245  rule_t rule;
1246  for (rule_list::const_iterator i = generic_rules.begin(),
1247  i_end = generic_rules.end(); i != i_end; ++i)
1248  {
1249  for (string_list::const_iterator j = i->targets.begin(),
1250  j_end = i->targets.end(); j != j_end; ++j)
1251  {
1252  size_t len = j->length();
1253  if (tlen < len) continue;
1254  if (plen <= tlen - (len - 1)) continue;
1255  size_t pos = j->find('%');
1256  if (pos == std::string::npos) continue;
1257  size_t len2 = len - (pos + 1);
1258  if (j->compare(0, pos, target, 0, pos) ||
1259  j->compare(pos + 1, len2, target, tlen - len2, len2))
1260  continue;
1261  plen = tlen - (len - 1);
1262  std::string pat = target.substr(pos, plen);
1263  rule = rule_t();
1264  rule.script = i->script;
1265  substitute_pattern(pat, i->targets, rule.targets);
1266  substitute_pattern(pat, i->deps, rule.deps);
1267  break;
1268  }
1269  }
1270  return rule;
1271 }
1272 
1273 /**
1274  * Find a specific rule matching @a target. Return a generic one otherwise.
1275  * If there is both a specific rule with an empty script and a generic rule, the
1276  * generic one is returned after adding the dependencies of the specific one.
1277  */
1278 static rule_t find_rule(std::string const &target)
1279 {
1280  rule_map::const_iterator i = specific_rules.find(target),
1281  i_end = specific_rules.end();
1282  // If there is a specific rule with a script, return it.
1283  if (i != i_end && !i->second->script.empty()) return *i->second;
1284  rule_t grule = find_generic_rule(target);
1285  // If there is no generic rule, return the specific rule (no script), if any.
1286  if (grule.targets.empty())
1287  {
1288  if (i != i_end) return *i->second;
1289  return grule;
1290  }
1291  // Optimize the lookup when there is only one target (alread looked up).
1292  if (grule.targets.size() == 1)
1293  {
1294  if (i != i_end)
1295  grule.deps.insert(grule.deps.end(),
1296  i->second->deps.begin(), i->second->deps.end());
1297  return grule;
1298  }
1299  // Add the dependencies of the specific rules of every target to the
1300  // generic rule. If any of those rules has a nonempty script, error out.
1301  for (string_list::const_iterator j = grule.targets.begin(),
1302  j_end = grule.targets.end(); j != j_end; ++j)
1303  {
1304  i = specific_rules.find(*j);
1305  if (i == i_end) continue;
1306  if (!i->second->script.empty()) return rule_t();
1307  grule.deps.insert(grule.deps.end(),
1308  i->second->deps.begin(), i->second->deps.end());
1309  }
1310  return grule;
1311 }
1312 
1313 /**
1314  * Compute and memoize the status of @a target:
1315  * - if the file does not exist, the target is obsolete,
1316  * - if any dependency is obsolete or younger than the file, it is obsolete,
1317  * - otherwise it is up-to-date.
1318  *
1319  * @note With multiple targets, they all share the same status. (If one is
1320  * obsolete, they all are.) For the second rule above, the latest target
1321  * is chosen, not the oldest!
1322  */
1323 static status_t const &get_status(std::string const &target)
1324 {
1325  std::pair<status_map::iterator,bool> i =
1326  status.insert(std::make_pair(target, status_t()));
1327  status_t &ts = i.first->second;
1328  if (!i.second) return ts;
1329  DEBUG_open << "Checking status of " << target << "... ";
1330  dependency_map::const_iterator j = dependencies.find(target);
1331  if (j == dependencies.end())
1332  {
1333  struct stat s;
1334  if (stat(target.c_str(), &s) != 0)
1335  {
1336  DEBUG_close << "missing\n";
1337  ts.status = Todo;
1338  ts.last = 0;
1339  return ts;
1340  }
1341  DEBUG_close << "up-to-date\n";
1342  ts.status = Uptodate;
1343  ts.last = s.st_mtime;
1344  return ts;
1345  }
1346  dependency_t const &dep = *j->second;
1347  status_e st = Uptodate;
1348  time_t latest = 0;
1349  for (string_list::const_iterator k = dep.targets.begin(),
1350  k_end = dep.targets.end(); k != k_end; ++k)
1351  {
1352  struct stat s;
1353  if (stat(k->c_str(), &s) != 0)
1354  {
1355  if (st == Uptodate) DEBUG_close << *k << " missing\n";
1356  s.st_mtime = 0;
1357  st = Todo;
1358  }
1359  status[*k].last = s.st_mtime;
1360  if (s.st_mtime > latest) latest = s.st_mtime;
1361  }
1362  if (st == Todo) goto update;
1363  for (string_set::const_iterator k = dep.deps.begin(),
1364  k_end = dep.deps.end(); k != k_end; ++k)
1365  {
1366  status_t const &ts_ = get_status(*k);
1367  if (latest < ts_.last)
1368  {
1369  DEBUG_close << "older than " << *k << std::endl;
1370  st = Todo;
1371  goto update;
1372  }
1373  if (ts_.status == Uptodate) continue;
1374  if (st == Uptodate)
1375  DEBUG << "obsolete dependency " << *k << std::endl;
1376  st = Recheck;
1377  }
1378  if (st == Uptodate) DEBUG_close << "all siblings up-to-date\n";
1379  update:
1380  for (string_list::const_iterator k = dep.targets.begin(),
1381  k_end = dep.targets.end(); k != k_end; ++k)
1382  {
1383  status[*k].status = st;
1384  }
1385  return ts;
1386 }
1387 
1388 /**
1389  * Change the status of @a target to #Remade or #Uptodate depending on whether
1390  * its modification time changed.
1391  */
1392 static void update_status(std::string const &target)
1393 {
1394  DEBUG_open << "Rechecking status of " << target << "... ";
1395  status_map::iterator i = status.find(target);
1396  assert (i != status.end());
1397  status_t &ts = i->second;
1398  ts.status = Remade;
1399  if (ts.last >= now)
1400  {
1401  DEBUG_close << "possibly remade\n";
1402  return;
1403  }
1404  struct stat s;
1405  if (stat(target.c_str(), &s) != 0)
1406  {
1407  DEBUG_close << "missing\n";
1408  ts.last = 0;
1409  }
1410  else if (s.st_mtime != ts.last)
1411  {
1412  DEBUG_close << "remade\n";
1413  ts.last = s.st_mtime;
1414  }
1415  else
1416  {
1417  DEBUG_close << "unchanged\n";
1418  ts.status = Uptodate;
1419  }
1420 }
1421 
1422 /**
1423  * Check if all the prerequisites of @a target ended being up-to-date.
1424  */
1425 static bool still_need_rebuild(std::string const &target)
1426 {
1427  DEBUG_open << "Rechecking obsoleteness of " << target << "... ";
1428  status_map::const_iterator i = status.find(target);
1429  assert (i != status.end());
1430  if (i->second.status != Recheck) return true;
1431  dependency_map::const_iterator j = dependencies.find(target);
1432  assert(j != dependencies.end());
1433  dependency_t const &dep = *j->second;
1434  for (string_set::const_iterator k = dep.deps.begin(),
1435  k_end = dep.deps.end(); k != k_end; ++k)
1436  {
1437  if (status[*k].status != Uptodate) return true;
1438  }
1439  for (string_list::const_iterator k = dep.targets.begin(),
1440  k_end = dep.targets.end(); k != k_end; ++k)
1441  {
1442  status[*k].status = Uptodate;
1443  }
1444  DEBUG_close << "no longer obsolete\n";
1445  return false;
1446 }
1447 
1448 /**
1449  * Handle job completion.
1450  */
1451 static void complete_job(int job_id, bool success)
1452 {
1453  DEBUG_open << "Completing job " << job_id << "... ";
1454  job_targets_map::iterator i = job_targets.find(job_id);
1455  assert(i != job_targets.end());
1456  string_list const &targets = i->second;
1457  if (success)
1458  {
1459  for (string_list::const_iterator j = targets.begin(),
1460  j_end = targets.end(); j != j_end; ++j)
1461  {
1462  update_status(*j);
1463  }
1464  }
1465  else
1466  {
1467  DEBUG_close << "failed\n";
1468  std::cerr << "Failed to build";
1469  for (string_list::const_iterator j = targets.begin(),
1470  j_end = targets.end(); j != j_end; ++j)
1471  {
1472  status[*j].status = Failed;
1473  std::cerr << ' ' << *j;
1474  remove(j->c_str());
1475  }
1476  std::cerr << std::endl;
1477  }
1478  job_targets.erase(i);
1479 }
1480 
1481 /**
1482  * Execute the script from @a rule.
1483  */
1484 static bool run_script(int job_id, rule_t const &rule)
1485 {
1486  if (show_targets)
1487  {
1488  std::cout << "Building";
1489  for (string_list::const_iterator i = rule.targets.begin(),
1490  i_end = rule.targets.end(); i != i_end; ++i)
1491  {
1492  std::cout << ' ' << *i;
1493  }
1494  std::cout << std::endl;
1495  }
1496 
1498  dep->targets = rule.targets;
1499  dep->deps.insert(rule.deps.begin(), rule.deps.end());
1500  for (string_list::const_iterator i = rule.targets.begin(),
1501  i_end = rule.targets.end(); i != i_end; ++i)
1502  {
1503  dependencies[*i] = dep;
1504  }
1505 
1506  DEBUG_open << "Starting script for job " << job_id << "... ";
1507 #ifdef WINDOWS
1508  HANDLE pfd[2];
1509  if (false)
1510  {
1511  error2:
1512  CloseHandle(pfd[0]);
1513  CloseHandle(pfd[1]);
1514  error:
1515  DEBUG_close << "failed\n";
1516  complete_job(job_id, false);
1517  return false;
1518  }
1519  if (!CreatePipe(&pfd[0], &pfd[1], NULL, 0))
1520  goto error;
1521  if (!SetHandleInformation(pfd[0], HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT))
1522  goto error2;
1523  STARTUPINFO si;
1524  ZeroMemory(&si, sizeof(STARTUPINFO));
1525  si.cb = sizeof(STARTUPINFO);
1526  si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
1527  si.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
1528  si.hStdInput = pfd[0];
1529  si.dwFlags |= STARTF_USESTDHANDLES;
1530  PROCESS_INFORMATION pi;
1531  ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
1532  std::ostringstream buf;
1533  buf << job_id;
1534  if (!SetEnvironmentVariable("REMAKE_JOB_ID", buf.str().c_str()))
1535  goto error2;
1536  std::ostringstream argv;
1537  argv << "SH.EXE -e -s";
1538  if (echo_scripts) argv << " -v";
1539  for (string_list::const_iterator i = rule.targets.begin(),
1540  i_end = rule.targets.end(); i != i_end; ++i)
1541  {
1542  argv << " \"" << escape_string(*i) << '"';
1543  }
1544  if (!CreateProcess(NULL, (char *)argv.str().c_str(), NULL, NULL,
1545  true, 0, NULL, NULL, &si, &pi))
1546  {
1547  goto error2;
1548  }
1549  CloseHandle(pi.hThread);
1550  std::string script = variable_block + rule.script;
1551  DWORD len = script.length(), wlen;
1552  if (!WriteFile(pfd[1], script.c_str(), len, &wlen, NULL) || wlen < len)
1553  std::cerr << "Unexpected failure while sending script to shell" << std::endl;
1554  CloseHandle(pfd[0]);
1555  CloseHandle(pfd[1]);
1556  ++running_jobs;
1557  job_pids[pi.hProcess] = job_id;
1558  return true;
1559 #else
1560  int pfd[2];
1561  if (false)
1562  {
1563  error2:
1564  close(pfd[0]);
1565  close(pfd[1]);
1566  error:
1567  DEBUG_close << "failed\n";
1568  complete_job(job_id, false);
1569  return false;
1570  }
1571  if (pipe(pfd) == -1)
1572  goto error;
1573  if (pid_t pid = fork())
1574  {
1575  if (pid == -1) goto error2;
1576  std::string script = variable_block + rule.script;
1577  ssize_t len = script.length();
1578  if (write(pfd[1], script.c_str(), len) < len)
1579  std::cerr << "Unexpected failure while sending script to shell" << std::endl;
1580  close(pfd[0]);
1581  close(pfd[1]);
1582  ++running_jobs;
1583  job_pids[pid] = job_id;
1584  return true;
1585  }
1586  // Child process starts here.
1587  std::ostringstream buf;
1588  buf << job_id;
1589  if (setenv("REMAKE_JOB_ID", buf.str().c_str(), 1))
1590  _exit(EXIT_FAILURE);
1591  int num = echo_scripts ? 4 : 3;
1592  char const **argv = new char const *[num + rule.targets.size() + 1];
1593  argv[0] = "sh";
1594  argv[1] = "-e";
1595  argv[2] = "-s";
1596  if (echo_scripts) argv[3] = "-v";
1597  for (string_list::const_iterator i = rule.targets.begin(),
1598  i_end = rule.targets.end(); i != i_end; ++i, ++num)
1599  {
1600  argv[num] = i->c_str();
1601  }
1602  argv[num] = NULL;
1603  if (pfd[0] != 0)
1604  {
1605  dup2(pfd[0], 0);
1606  close(pfd[0]);
1607  }
1608  close(pfd[1]);
1609  execv("/bin/sh", (char **)argv);
1610  _exit(EXIT_FAILURE);
1611 #endif
1612 }
1613 
1614 /**
1615  * Create a job for @a target according to the loaded rules.
1616  * Mark all the targets from the rule as running and reset their dependencies.
1617  * If the rule has dependencies, create a new client to build them just
1618  * before @a current, and change @a current so that it points to it.
1619  */
1620 static bool start(std::string const &target, client_list::iterator &current)
1621 {
1622  DEBUG_open << "Starting job " << job_counter << " for " << target << "... ";
1623  rule_t rule = find_rule(target);
1624  if (rule.targets.empty())
1625  {
1626  status[target].status = Failed;
1627  DEBUG_close << "failed\n";
1628  std::cerr << "No rule for building " << target << std::endl;
1629  return false;
1630  }
1631  for (string_list::const_iterator i = rule.targets.begin(),
1632  i_end = rule.targets.end(); i != i_end; ++i)
1633  {
1634  status[*i].status = Running;
1635  }
1636  int job_id = job_counter++;
1637  job_targets[job_id] = rule.targets;
1638  if (!rule.deps.empty())
1639  {
1640  current = clients.insert(current, client_t());
1641  current->job_id = job_id;
1642  current->pending = rule.deps;
1643  current->delayed = new rule_t(rule);
1644  return true;
1645  }
1646  return run_script(job_id, rule);
1647 }
1648 
1649 /**
1650  * Send a reply to a client then remove it.
1651  * If the client was a dependency client, start the actual script.
1652  */
1653 static void complete_request(client_t &client, bool success)
1654 {
1655  DEBUG_open << "Completing request from client of job " << client.job_id << "... ";
1656  if (client.delayed)
1657  {
1658  assert(client.socket == INVALID_SOCKET);
1659  if (success)
1660  {
1661  if (still_need_rebuild(client.delayed->targets.front()))
1662  run_script(client.job_id, *client.delayed);
1663  else complete_job(client.job_id, true);
1664  }
1665  else complete_job(client.job_id, false);
1666  delete client.delayed;
1667  }
1668  else if (client.socket != INVALID_SOCKET)
1669  {
1670  char res = success ? 1 : 0;
1671  send(client.socket, &res, 1, 0);
1672  #ifdef WINDOWS
1673  closesocket(client.socket);
1674  #else
1675  close(client.socket);
1676  #endif
1677  --waiting_jobs;
1678  }
1679 
1680  if (client.job_id < 0 && !success) build_failure = true;
1681 }
1682 
1683 /**
1684  * Return whether there are slots for starting new jobs.
1685  */
1686 static bool has_free_slots()
1687 {
1688  if (max_active_jobs <= 0) return true;
1690 }
1691 
1692 /**
1693  * Update clients as long as there are free slots:
1694  * - check for running targets that have finished,
1695  * - start as many pending targets as allowed,
1696  * - complete the request if there are neither running nor pending targets
1697  * left or if any of them failed.
1698  */
1699 static void update_clients()
1700 {
1701  DEBUG_open << "Updating clients... ";
1702  for (client_list::iterator i = clients.begin(), i_next = i,
1703  i_end = clients.end(); i != i_end && has_free_slots(); i = i_next)
1704  {
1705  ++i_next;
1706  DEBUG_open << "Handling client from job " << i->job_id << "... ";
1707  if (false)
1708  {
1709  failed:
1710  complete_request(*i, false);
1711  clients.erase(i);
1712  DEBUG_close << "failed\n";
1713  continue;
1714  }
1715 
1716  // Remove running targets that have finished.
1717  for (string_set::iterator j = i->running.begin(), j_next = j,
1718  j_end = i->running.end(); j != j_end; j = j_next)
1719  {
1720  ++j_next;
1721  status_map::const_iterator k = status.find(*j);
1722  assert(k != status.end());
1723  switch (k->second.status)
1724  {
1725  case Running:
1726  break;
1727  case Failed:
1728  if (!keep_going) goto failed;
1729  i->failed = true;
1730  // no break
1731  case Uptodate:
1732  case Remade:
1733  i->running.erase(j);
1734  break;
1735  case Recheck:
1736  case Todo:
1737  assert(false);
1738  }
1739  }
1740 
1741  // Start pending targets.
1742  while (!i->pending.empty())
1743  {
1744  std::string target = i->pending.front();
1745  i->pending.pop_front();
1746  switch (get_status(target).status)
1747  {
1748  case Running:
1749  i->running.insert(target);
1750  break;
1751  case Failed:
1752  pending_failed:
1753  if (!keep_going) goto failed;
1754  i->failed = true;
1755  // no break
1756  case Uptodate:
1757  case Remade:
1758  break;
1759  case Recheck:
1760  case Todo:
1761  client_list::iterator j = i;
1762  if (!start(target, i)) goto pending_failed;
1763  j->running.insert(target);
1764  if (!has_free_slots()) return;
1765  // Job start might insert a dependency client.
1766  i_next = i;
1767  ++i_next;
1768  break;
1769  }
1770  }
1771 
1772  // Try to complete request.
1773  // (This might start a new job if it was a dependency client.)
1774  if (i->running.empty())
1775  {
1776  if (i->failed) goto failed;
1777  complete_request(*i, true);
1778  clients.erase(i);
1779  DEBUG_close << "finished\n";
1780  }
1781  }
1782 }
1783 
1784 /**
1785  * Create a named unix socket that listens for build requests. Also set
1786  * the REMAKE_SOCKET environment variable that will be inherited by all
1787  * the job scripts.
1788  */
1789 static void create_server()
1790 {
1791  if (false)
1792  {
1793  error:
1794  perror("Failed to create server");
1795 #ifndef WINDOWS
1796  error2:
1797 #endif
1798  exit(EXIT_FAILURE);
1799  }
1800  DEBUG_open << "Creating server... ";
1801 
1802 #ifdef WINDOWS
1803  // Prepare a windows socket.
1804  struct sockaddr_in socket_addr;
1805  socket_addr.sin_family = AF_INET;
1806  socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
1807  socket_addr.sin_port = 0;
1808 
1809  // Create and listen to the socket.
1810  socket_fd = socket(AF_INET, SOCK_STREAM, 0);
1811  if (socket_fd < 0) goto error;
1812  if (!SetHandleInformation((HANDLE)socket_fd, HANDLE_FLAG_INHERIT, 0))
1813  goto error;
1814  if (bind(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
1815  goto error;
1816  int len = sizeof(sockaddr_in);
1817  if (getsockname(socket_fd, (struct sockaddr *)&socket_addr, &len))
1818  goto error;
1819  std::ostringstream buf;
1820  buf << socket_addr.sin_port;
1821  if (!SetEnvironmentVariable("REMAKE_SOCKET", buf.str().c_str()))
1822  goto error;
1823  if (listen(socket_fd, 1000)) goto error;
1824 #else
1825  // Set a handler for SIGCHLD then block the signal (unblocked during select).
1826  sigset_t sigmask;
1827  sigemptyset(&sigmask);
1828  sigaddset(&sigmask, SIGCHLD);
1829  if (sigprocmask(SIG_BLOCK, &sigmask, NULL) == -1) goto error;
1830  struct sigaction sa;
1831  sa.sa_flags = 0;
1832  sa.sa_handler = &child_sig_handler;
1833  sigemptyset(&sa.sa_mask);
1834  if (sigaction(SIGCHLD, &sa, NULL) == -1) goto error;
1835 
1836  // Prepare a named unix socket in temporary directory.
1837  socket_name = tempnam(NULL, "rmk-");
1838  if (!socket_name) goto error2;
1839  struct sockaddr_un socket_addr;
1840  size_t len = strlen(socket_name);
1841  if (len >= sizeof(socket_addr.sun_path) - 1) goto error2;
1842  socket_addr.sun_family = AF_UNIX;
1843  strcpy(socket_addr.sun_path, socket_name);
1844  len += sizeof(socket_addr.sun_family);
1845  if (setenv("REMAKE_SOCKET", socket_name, 1)) goto error;
1846 
1847  // Create and listen to the socket.
1848 #ifdef LINUX
1849  socket_fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
1850  if (socket_fd < 0) goto error;
1851 #else
1852  socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
1853  if (socket_fd < 0) goto error;
1854  if (fcntl(socket_fd, F_SETFD, FD_CLOEXEC) < 0) goto error;
1855 #endif
1856  if (bind(socket_fd, (struct sockaddr *)&socket_addr, len))
1857  goto error;
1858  if (listen(socket_fd, 1000)) goto error;
1859 #endif
1860 }
1861 
1862 /**
1863  * Accept a connection from a client, get the job it spawned from,
1864  * get the targets, and mark them as dependencies of the job targets.
1865  */
1867 {
1868  DEBUG_open << "Handling client request... ";
1869 
1870  // Accept connection.
1871 #ifdef WINDOWS
1872  socket_t fd = accept(socket_fd, NULL, NULL);
1873  if (fd == INVALID_SOCKET) return;
1874  if (!SetHandleInformation((HANDLE)fd, HANDLE_FLAG_INHERIT, 0))
1875  {
1876  error2:
1877  std::cerr << "Unexpected failure while setting connection with client" << std::endl;
1878  closesocket(fd);
1879  return;
1880  }
1881  // WSAEventSelect puts sockets into nonblocking mode, so disable it here.
1882  u_long nbio = 0;
1883  if (ioctlsocket(fd, FIONBIO, &nbio)) goto error2;
1884 #elif defined(LINUX)
1885  int fd = accept4(socket_fd, NULL, NULL, SOCK_CLOEXEC);
1886  if (fd < 0) return;
1887 #else
1888  int fd = accept(socket_fd, NULL, NULL);
1889  if (fd < 0) return;
1890  if (fcntl(fd, F_SETFD, FD_CLOEXEC) < 0) return;
1891 #endif
1892  clients.push_front(client_t());
1893  client_list::iterator proc = clients.begin();
1894 
1895  if (false)
1896  {
1897  error:
1898  DEBUG_close << "failed\n";
1899  std::cerr << "Received an ill-formed client message" << std::endl;
1900  #ifdef WINDOWS
1901  closesocket(fd);
1902  #else
1903  close(fd);
1904  #endif
1905  clients.erase(proc);
1906  return;
1907  }
1908 
1909  // Receive message. Stop when encountering two nuls in a row.
1910  std::vector<char> buf;
1911  size_t len = 0;
1912  while (len < sizeof(int) + 2 || buf[len - 1] || buf[len - 2])
1913  {
1914  buf.resize(len + 1024);
1915  ssize_t l = recv(fd, &buf[0] + len, 1024, 0);
1916  if (l <= 0) goto error;
1917  len += l;
1918  }
1919 
1920  // Parse job that spawned the client.
1921  int job_id;
1922  memcpy(&job_id, &buf[0], sizeof(int));
1923  proc->socket = fd;
1924  proc->job_id = job_id;
1925  job_targets_map::const_iterator i = job_targets.find(job_id);
1926  if (i == job_targets.end()) goto error;
1927  DEBUG << "receiving request from job " << job_id << std::endl;
1928 
1929  // Parse the targets and mark them as dependencies from the job targets.
1930  dependency_t &dep = *dependencies[job_targets[job_id].front()];
1931  char const *p = &buf[0] + sizeof(int);
1932  while (true)
1933  {
1934  len = strlen(p);
1935  if (len == 0)
1936  {
1937  ++waiting_jobs;
1938  return;
1939  }
1940  std::string target(p, p + len);
1941  DEBUG << "adding dependency " << target << " to job\n";
1942  proc->pending.push_back(target);
1943  dep.deps.insert(target);
1944  p += len + 1;
1945  }
1946 }
1947 
1948 /**
1949  * Loop until all the jobs have finished.
1950  */
1952 {
1953  while (true)
1954  {
1955  update_clients();
1956  if (running_jobs == 0)
1957  {
1958  assert(clients.empty());
1959  break;
1960  }
1961  DEBUG_open << "Handling events... ";
1962  #ifdef WINDOWS
1963  size_t len = job_pids.size() + 1;
1964  HANDLE h[len];
1965  int num = 0;
1966  for (pid_job_map::const_iterator i = job_pids.begin(),
1967  i_end = job_pids.end(); i != i_end; ++i, ++num)
1968  {
1969  h[num] = i->first;
1970  }
1971  WSAEVENT aev = WSACreateEvent();
1972  h[num] = aev;
1973  WSAEventSelect(socket_fd, aev, FD_ACCEPT);
1974  DWORD w = WaitForMultipleObjects(len, h, false, INFINITE);
1975  WSAEventSelect(socket_fd, aev, 0);
1976  WSACloseEvent(aev);
1977  if (w < WAIT_OBJECT_0 || WAIT_OBJECT_0 + len <= w)
1978  continue;
1979  if (w == WAIT_OBJECT_0 + len - 1)
1980  {
1981  accept_client();
1982  continue;
1983  }
1984  pid_t pid = h[w - WAIT_OBJECT_0];
1985  DWORD s = 0;
1986  bool res = GetExitCodeProcess(pid, &s) && s == 0;
1987  CloseHandle(pid);
1988  pid_job_map::iterator i = job_pids.find(pid);
1989  assert(i != job_pids.end());
1990  int job_id = i->second;
1991  job_pids.erase(i);
1992  --running_jobs;
1993  complete_job(job_id, res);
1994  #else
1995  sigset_t emptymask;
1996  sigemptyset(&emptymask);
1997  fd_set fdset;
1998  FD_ZERO(&fdset);
1999  FD_SET(socket_fd, &fdset);
2000  int ret = pselect(socket_fd + 1, &fdset, NULL, NULL, NULL, &emptymask);
2001  if (ret > 0 /* && FD_ISSET(socket_fd, &fdset)*/) accept_client();
2002  if (!got_SIGCHLD) continue;
2003  got_SIGCHLD = 0;
2004  pid_t pid;
2005  int status;
2006  while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
2007  {
2008  bool res = WIFEXITED(status) && WEXITSTATUS(status) == 0;
2009  pid_job_map::iterator i = job_pids.find(pid);
2010  assert(i != job_pids.end());
2011  int job_id = i->second;
2012  job_pids.erase(i);
2013  --running_jobs;
2014  complete_job(job_id, res);
2015  }
2016  #endif
2017  }
2018 }
2019 
2020 /**
2021  * Load dependencies and rules, listen to client requests, and loop until
2022  * all the requests have completed.
2023  * If Remakefile is obsolete, perform a first run with it only, then reload
2024  * the rules, and perform a second with the original clients.
2025  */
2026 void server_mode(string_list const &targets)
2027 {
2029  load_rules();
2030  create_server();
2031  if (get_status("Remakefile").status != Uptodate)
2032  {
2033  clients.push_back(client_t());
2034  clients.back().pending.push_back("Remakefile");
2035  server_loop();
2036  if (build_failure) goto early_exit;
2037  variables.clear();
2038  specific_rules.clear();
2039  generic_rules.clear();
2040  first_target.clear();
2041  load_rules();
2042  }
2043  clients.push_back(client_t());
2044  if (!targets.empty()) clients.back().pending = targets;
2045  else if (!first_target.empty())
2046  clients.back().pending.push_back(first_target);
2047  server_loop();
2048  early_exit:
2049  close(socket_fd);
2050  remove(socket_name);
2052  exit(build_failure ? EXIT_FAILURE : EXIT_SUCCESS);
2053 }
2054 
2055 /**
2056  * Connect to the server @a socket_name, send a build request for @a targets,
2057  * and exit with the status returned by the server.
2058  */
2059 void client_mode(char *socket_name, string_list const &targets)
2060 {
2061  if (false)
2062  {
2063  error:
2064  perror("Failed to send targets to server");
2065  exit(EXIT_FAILURE);
2066  }
2067  if (targets.empty()) exit(EXIT_SUCCESS);
2068  DEBUG_open << "Connecting to server... ";
2069 
2070  // Connect to server.
2071 #ifdef WINDOWS
2072  struct sockaddr_in socket_addr;
2073  socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2074  if (socket_fd < 0) goto error;
2075  socket_addr.sin_family = AF_INET;
2076  socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2077  socket_addr.sin_port = atoi(socket_name);
2078  if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2079  goto error;
2080 #else
2081  struct sockaddr_un socket_addr;
2082  size_t len = strlen(socket_name);
2083  if (len >= sizeof(socket_addr.sun_path) - 1) exit(EXIT_FAILURE);
2084  socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2085  if (socket_fd < 0) goto error;
2086  socket_addr.sun_family = AF_UNIX;
2087  strcpy(socket_addr.sun_path, socket_name);
2088  if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(socket_addr.sun_family) + len))
2089  goto error;
2090 #ifdef MACOSX
2091  int set_option = 1;
2092  if (setsockopt(socket_fd, SOL_SOCKET, SO_NOSIGPIPE, &set_option, sizeof(set_option)))
2093  goto error;
2094 #endif
2095 #endif
2096 
2097  // Send current job id.
2098  char *id = getenv("REMAKE_JOB_ID");
2099  int job_id = id ? atoi(id) : -1;
2100  if (send(socket_fd, (char *)&job_id, sizeof(job_id), MSG_NOSIGNAL) != sizeof(job_id))
2101  goto error;
2102 
2103  // Send tagets.
2104  for (string_list::const_iterator i = targets.begin(),
2105  i_end = targets.end(); i != i_end; ++i)
2106  {
2107  DEBUG_open << "Sending " << *i << "... ";
2108  ssize_t len = i->length() + 1;
2109  if (send(socket_fd, i->c_str(), len, MSG_NOSIGNAL) != len)
2110  goto error;
2111  }
2112 
2113  // Send terminating nul and wait for reply.
2114  char result = 0;
2115  if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2116  if (recv(socket_fd, &result, 1, 0) != 1) exit(EXIT_FAILURE);
2117  exit(result ? EXIT_SUCCESS : EXIT_FAILURE);
2118 }
2119 
2120 /**
2121  * Display usage and exit with @a exit_status.
2122  */
2123 void usage(int exit_status)
2124 {
2125  std::cerr << "Usage: remake [options] [target] ...\n"
2126  "Options\n"
2127  " -d Echo script commands.\n"
2128  " -d -d Print lots of debugging information.\n"
2129  " -h, --help Print this message and exit.\n"
2130  " -j[N], --jobs=[N] Allow N jobs at once; infinite jobs with no arg.\n"
2131  " -k Keep going when some targets cannot be made.\n"
2132  " -r Look up targets from the dependencies on standard input.\n"
2133  " -s, --silent, --quiet Do not echo targets.\n";
2134  exit(exit_status);
2135 }
2136 
2137 /**
2138  * This program behaves in two different ways.
2139  *
2140  * - If the environment contains the REMAKE_SOCKET variable, the client
2141  * connects to this socket and sends to the server its build targets.
2142  * It exits once it receives the server reply.
2143  *
2144  * - Otherwise, it creates a server that waits for build requests. It
2145  * also creates a pseudo-client that requests the targets passed on the
2146  * command line.
2147  */
2148 int main(int argc, char *argv[])
2149 {
2150  init_working_dir();
2151 
2152  string_list targets;
2153  bool indirect_targets = false;
2154 
2155  // Parse command-line arguments.
2156  for (int i = 1; i < argc; ++i)
2157  {
2158  std::string arg = argv[i];
2159  if (arg.empty()) usage(EXIT_FAILURE);
2160  if (arg == "-h" || arg == "--help") usage(EXIT_SUCCESS);
2161  if (arg == "-d")
2162  if (echo_scripts) debug.active = true;
2163  else echo_scripts = true;
2164  else if (arg == "-k" || arg =="--keep-going")
2165  keep_going = true;
2166  else if (arg == "-s" || arg == "--silent" || arg == "--quiet")
2167  show_targets = false;
2168  else if (arg == "-r")
2169  indirect_targets = true;
2170  else if (arg.compare(0, 2, "-j") == 0)
2171  max_active_jobs = atoi(arg.c_str() + 2);
2172  else if (arg.compare(0, 7, "--jobs=") == 0)
2173  max_active_jobs = atoi(arg.c_str() + 7);
2174  else
2175  {
2176  if (arg[0] == '-') usage(1);
2177  targets.push_back(normalize(arg));
2178  DEBUG << "New target: " << arg << '\n';
2179  }
2180  }
2181 
2182  if (indirect_targets)
2183  {
2184  load_dependencies(std::cin);
2185  string_list l;
2186  targets.swap(l);
2187  if (l.empty() && !dependencies.empty())
2188  {
2189  l.push_back(dependencies.begin()->second->targets.front());
2190  }
2191  for (string_list::const_iterator i = l.begin(),
2192  i_end = l.end(); i != i_end; ++i)
2193  {
2194  dependency_map::const_iterator j = dependencies.find(*i);
2195  if (j == dependencies.end()) continue;
2196  dependency_t const &dep = *j->second;
2197  for (string_set::const_iterator k = dep.deps.begin(),
2198  k_end = dep.deps.end(); k != k_end; ++k)
2199  {
2200  targets.push_back(normalize(*k));
2201  }
2202  }
2203  dependencies.clear();
2204  }
2205 
2206 #ifdef WINDOWS
2207  WSADATA wsaData;
2208  if (WSAStartup(MAKEWORD(2,2), &wsaData))
2209  {
2210  std::cerr << "Unexpected failure while initializing Windows Socket" << std::endl;
2211  return 1;
2212  }
2213 #endif
2214 
2215  // Run as client if REMAKE_SOCKET is present in the environment.
2216  if (char *sn = getenv("REMAKE_SOCKET")) client_mode(sn, targets);
2217 
2218  // Otherwise run as server.
2219  server_mode(targets);
2220 }