Merge commit 'flex-2.5.39^^'

This merges the same version of flex as we're using in prebuilts/misc
currently (but without the tiny branch that switched the version to
2.5.39 that weren't in the upstream master branch).

Also adds the METADATA, LICENSE, NOTICE, and MODULE_LICENSE_* files as
required.

Test: No Android.* files
Change-Id: If69488a1d78d5c7d4c1cc8955d55788bd5a639f2
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..1120ef6
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,42 @@
+*.tar.bz2
+*.tar.gz
+.bootstrap
+.deps
+ABOUT-NLS
+ChangeLog
+INSTALL
+Makefile
+Makefile.in
+aclocal.m4
+autom4te.cache
+compile
+conf.in
+config.cache
+config.guess
+config.h
+config.log
+config.status*
+config.sub
+configure
+depcomp
+flex
+flex-*
+install-sh
+missing
+mkinstalldirs
+parse.c
+parse.h
+scan.c
+skel.c
+stamp-*
+config.rpath
+ylwrap
+*.o
+.libs
+*.lo
+*.la
+libtool
+ltmain.sh
+*~
+*.orig
+*.rej
diff --git a/.indent.pro b/.indent.pro
new file mode 100644
index 0000000..3f8232f
--- /dev/null
+++ b/.indent.pro
@@ -0,0 +1,15 @@
+--blank-lines-after-declarations
+--blank-lines-after-procedures
+-br  /* open braces on same line */
+-nce /* start else on new line */
+-nbc /* vars on same line */
+-di8 /* line up var decl at col 8 */
+-brs /* struct brace on same line */
+-i8  /* indent 4  */
+-lp  /* line up parens */
+-ts8 /* tab stop */
+-bbo /* break before && || */
+-hnl /* honor newlines */
+--space-special-semicolon
+--line-length75
+--dont-break-procedure-type
diff --git a/AUTHORS b/AUTHORS
new file mode 100644
index 0000000..93b3528
--- /dev/null
+++ b/AUTHORS
@@ -0,0 +1,16 @@
+
+In 2001, Will Estes took over as maintainer of flex.
+
+John Millaway is a co-author of the current version of flex. He has
+contributed a large number of new features, fixed a large number of
+outstanding bugs and has made significant contributions to the flex
+documentation.
+
+Aaron Stone has contributed several bug fixes to the flex codebase.
+
+Vern Paxson wrote flex with the help of many ideas and much
+inspiration from Van Jacobson.  Original version by Jef Poskanzer.
+
+The fast table representation is a partial implementation of a design
+done by Van Jacobson.  The implementation was done by Kevin Gong and
+Vern Paxson.
diff --git a/BUGS b/BUGS
new file mode 100644
index 0000000..f456b69
--- /dev/null
+++ b/BUGS
@@ -0,0 +1,107 @@
+This is a list of bugs still in the queue at lex.sf.net at the time we closed
+out the project and moved it to flex.sf.net.
+
+-------------------------------------------------------------
+
+Some strict compilers warn about a few internal flex variables signedness. They
+are bogus warnings and can be ignored, but people send in reports nonethless.
+
+-------------------------------------------------------------
+
+The initializer of the yy_transition array in the
+generated scanner
+contains fewer entries than the declared size of the array.
+
+Examples include yy_transition[6504] with 6250 entries,
+yy_transition[13215] with 12961 entries. This looks
+like it
+is always 254 fewer entries than the declared size.
+
+This bug is present in flex 2.5.4a as well. It appears to be harmless.
+
+-------------------------------------------------------------
+The examples in the chapter "Generating C++ Scanners"
+contain suspicious code. Attached is a patch that
+corrects this, and after these
+modifications this example compiles and works.
+
+-------------------------------------------------------------
+
+C++ scanners derived from the yyFlexLexer base class
+will not compile with flex-2.5.31 because the
+<FlexLexer.h> automatically gets included into the
+scanner twice. Because yyFlexLexer is now defined by
+default even if no prefix is specified, including
+FlexLexer.h twice causes the class yyFlexLexer to be
+declared twice. In flex-2.5.4 because yyFlexLexer was
+not defined by flex in the scanner code, including
+FlexLexer.h more than once only declared yyFlexLexer
+once. I appreciate that this is because of the M4
+additions to flex, but I can not find a way to stop
+flex defining yyFlexLexer if it is not needed.
+
+Here is an example of a class that will not compile:
+
+derived_lexer.h:
+
+#ifndef __derived_lexer__
+#define __derived_lexer__
+
+#include <FlexLexer.h>
+
+class derived_lexer : public yyFlexLexer
+{
+public:
+derived_lexer(std::istream* arg_yyin = 0) :
+yyFlexLexer(arg_yyin){}
+int yylex();
+int x;
+};
+#endif
+
+derived_lexer.l:
+
+%{
+#include "derived_lexer.h"
+%}
+
+%option yyclass="derived_lexer"
+
+%%
+[0-9]+ {
+x = atoi(yytext);
+}
+%%
+
+main.cpp:
+
+#include "derived_lexer.h"
+#include <fstream>
+
+int main()
+{
+std::ifstream input;
+input.open("input");
+derived_lexer lexer(&input);
+lexer.yylex();
+}
+
+-------------------------------------------------------------
+
+Hi, the anomally is that if I generate a c++ parser it will not make 
+yy_scan_buffer and friends.
+this is happenning on flex version 2.5.4.
+Is this the intent, if so how do I make a c++ parser read from my buffer?
+
+P.S. in c++ it will only generate:
+#ifndef YY_NO_SCAN_BUFFER
+#endif
+
+
+#ifndef YY_NO_SCAN_STRING
+#endif
+
+
+#ifndef YY_NO_SCAN_BYTES
+#endif
+-------------------------------------------------------------
diff --git a/COPYING b/COPYING
new file mode 100644
index 0000000..684b011
--- /dev/null
+++ b/COPYING
@@ -0,0 +1,42 @@
+Flex carries the copyright used for BSD software, slightly modified
+because it originated at the Lawrence Berkeley (not Livermore!) Laboratory,
+which operates under a contract with the Department of Energy:
+
+Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007 The Flex Project.
+
+Copyright (c) 1990, 1997 The Regents of the University of California.
+All rights reserved.
+
+This code is derived from software contributed to Berkeley by
+Vern Paxson.
+
+The United States Government has rights in this work pursuant
+to contract no. DE-AC03-76SF00098 between the United States
+Department of Energy and the University of California.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+Neither the name of the University nor the names of its contributors
+may be used to endorse or promote products derived from this software
+without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.
+
+This basically says "do whatever you please with this software except
+remove this notice or take advantage of the University's (or the flex
+authors') name".
+
+Note that the "flex.skl" scanner skeleton carries no copyright notice.
+You are free to do whatever you please with scanners generated using flex;
+for them, you are not even bound by the above copyright.
diff --git a/FlexLexer.h b/FlexLexer.h
new file mode 100644
index 0000000..bad4ce0
--- /dev/null
+++ b/FlexLexer.h
@@ -0,0 +1,206 @@
+// -*-C++-*-
+// FlexLexer.h -- define interfaces for lexical analyzer classes generated
+// by flex
+
+// Copyright (c) 1993 The Regents of the University of California.
+// All rights reserved.
+//
+// This code is derived from software contributed to Berkeley by
+// Kent Williams and Tom Epperly.
+//
+//  Redistribution and use in source and binary forms, with or without
+//  modification, are permitted provided that the following conditions
+//  are met:
+
+//  1. Redistributions of source code must retain the above copyright
+//  notice, this list of conditions and the following disclaimer.
+//  2. Redistributions in binary form must reproduce the above copyright
+//  notice, this list of conditions and the following disclaimer in the
+//  documentation and/or other materials provided with the distribution.
+
+//  Neither the name of the University nor the names of its contributors
+//  may be used to endorse or promote products derived from this software
+//  without specific prior written permission.
+
+//  THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+//  IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+//  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+//  PURPOSE.
+
+// This file defines FlexLexer, an abstract class which specifies the
+// external interface provided to flex C++ lexer objects, and yyFlexLexer,
+// which defines a particular lexer class.
+//
+// If you want to create multiple lexer classes, you use the -P flag
+// to rename each yyFlexLexer to some other xxFlexLexer.  You then
+// include <FlexLexer.h> in your other sources once per lexer class:
+//
+//	#undef yyFlexLexer
+//	#define yyFlexLexer xxFlexLexer
+//	#include <FlexLexer.h>
+//
+//	#undef yyFlexLexer
+//	#define yyFlexLexer zzFlexLexer
+//	#include <FlexLexer.h>
+//	...
+
+#ifndef __FLEX_LEXER_H
+// Never included before - need to define base class.
+#define __FLEX_LEXER_H
+
+#include <iostream>
+#  ifndef FLEX_STD
+#    define FLEX_STD std::
+#  endif
+
+extern "C++" {
+
+struct yy_buffer_state;
+typedef int yy_state_type;
+
+class FlexLexer {
+public:
+	virtual ~FlexLexer()	{ }
+
+	const char* YYText() const	{ return yytext; }
+	int YYLeng()	const	{ return yyleng; }
+
+	virtual void
+		yy_switch_to_buffer( struct yy_buffer_state* new_buffer ) = 0;
+	virtual struct yy_buffer_state*
+		yy_create_buffer( FLEX_STD istream* s, int size ) = 0;
+	virtual void yy_delete_buffer( struct yy_buffer_state* b ) = 0;
+	virtual void yyrestart( FLEX_STD istream* s ) = 0;
+
+	virtual int yylex() = 0;
+
+	// Call yylex with new input/output sources.
+	int yylex( FLEX_STD istream* new_in, FLEX_STD ostream* new_out = 0 )
+		{
+		switch_streams( new_in, new_out );
+		return yylex();
+		}
+
+	// Switch to new input/output streams.  A nil stream pointer
+	// indicates "keep the current one".
+	virtual void switch_streams( FLEX_STD istream* new_in = 0,
+					FLEX_STD ostream* new_out = 0 ) = 0;
+
+	int lineno() const		{ return yylineno; }
+
+	int debug() const		{ return yy_flex_debug; }
+	void set_debug( int flag )	{ yy_flex_debug = flag; }
+
+protected:
+	char* yytext;
+	int yyleng;
+	int yylineno;		// only maintained if you use %option yylineno
+	int yy_flex_debug;	// only has effect with -d or "%option debug"
+};
+
+}
+#endif // FLEXLEXER_H
+
+#if defined(yyFlexLexer) || ! defined(yyFlexLexerOnce)
+// Either this is the first time through (yyFlexLexerOnce not defined),
+// or this is a repeated include to define a different flavor of
+// yyFlexLexer, as discussed in the flex manual.
+#define yyFlexLexerOnce
+
+extern "C++" {
+
+class yyFlexLexer : public FlexLexer {
+public:
+	// arg_yyin and arg_yyout default to the cin and cout, but we
+	// only make that assignment when initializing in yylex().
+	yyFlexLexer( FLEX_STD istream* arg_yyin = 0, FLEX_STD ostream* arg_yyout = 0 );
+
+	virtual ~yyFlexLexer();
+
+	void yy_switch_to_buffer( struct yy_buffer_state* new_buffer );
+	struct yy_buffer_state* yy_create_buffer( FLEX_STD istream* s, int size );
+	void yy_delete_buffer( struct yy_buffer_state* b );
+	void yyrestart( FLEX_STD istream* s );
+
+	void yypush_buffer_state( struct yy_buffer_state* new_buffer );
+	void yypop_buffer_state();
+
+	virtual int yylex();
+	virtual void switch_streams( FLEX_STD istream* new_in, FLEX_STD ostream* new_out = 0 );
+	virtual int yywrap();
+
+protected:
+	virtual int LexerInput( char* buf, int max_size );
+	virtual void LexerOutput( const char* buf, int size );
+	virtual void LexerError( const char* msg );
+
+	void yyunput( int c, char* buf_ptr );
+	int yyinput();
+
+	void yy_load_buffer_state();
+	void yy_init_buffer( struct yy_buffer_state* b, FLEX_STD istream* s );
+	void yy_flush_buffer( struct yy_buffer_state* b );
+
+	int yy_start_stack_ptr;
+	int yy_start_stack_depth;
+	int* yy_start_stack;
+
+	void yy_push_state( int new_state );
+	void yy_pop_state();
+	int yy_top_state();
+
+	yy_state_type yy_get_previous_state();
+	yy_state_type yy_try_NUL_trans( yy_state_type current_state );
+	int yy_get_next_buffer();
+
+	FLEX_STD istream* yyin;	// input source for default LexerInput
+	FLEX_STD ostream* yyout;	// output sink for default LexerOutput
+
+	// yy_hold_char holds the character lost when yytext is formed.
+	char yy_hold_char;
+
+	// Number of characters read into yy_ch_buf.
+	int yy_n_chars;
+
+	// Points to current character in buffer.
+	char* yy_c_buf_p;
+
+	int yy_init;		// whether we need to initialize
+	int yy_start;		// start state number
+
+	// Flag which is used to allow yywrap()'s to do buffer switches
+	// instead of setting up a fresh yyin.  A bit of a hack ...
+	int yy_did_buffer_switch_on_eof;
+
+
+	size_t yy_buffer_stack_top; /**< index of top of stack. */
+	size_t yy_buffer_stack_max; /**< capacity of stack. */
+	struct yy_buffer_state ** yy_buffer_stack; /**< Stack as an array. */
+	void yyensure_buffer_stack(void);
+
+	// The following are not always needed, but may be depending
+	// on use of certain flex features (like REJECT or yymore()).
+
+	yy_state_type yy_last_accepting_state;
+	char* yy_last_accepting_cpos;
+
+	yy_state_type* yy_state_buf;
+	yy_state_type* yy_state_ptr;
+
+	char* yy_full_match;
+	int* yy_full_state;
+	int yy_full_lp;
+
+	int yy_lp;
+	int yy_looking_for_trail_begin;
+
+	int yy_more_flag;
+	int yy_more_len;
+	int yy_more_offset;
+	int yy_prev_more_offset;
+};
+
+}
+
+#endif // yyFlexLexer || ! yyFlexLexerOnce
+
diff --git a/LICENSE b/LICENSE
new file mode 120000
index 0000000..d24842f
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1 @@
+COPYING
\ No newline at end of file
diff --git a/METADATA b/METADATA
new file mode 100644
index 0000000..c6e6456
--- /dev/null
+++ b/METADATA
@@ -0,0 +1,17 @@
+name: "flex"
+description:
+    "The Fast Lexical Analyzer - scanner generator for lexing in C and C++"
+
+third_party {
+  url {
+    type: HOMEPAGE
+    value: "http://flex.sourceforge.net/"
+  }
+  url {
+    type: GIT
+    value: "https://github.com/westes/flex"
+  }
+  version: "flex-2.5.39"
+  last_upgrade_date { year: 2018 month: 5 day: 22 }
+  license_type: NOTICE
+}
diff --git a/MODULE_LICENSE_BSD_LIKE b/MODULE_LICENSE_BSD_LIKE
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/MODULE_LICENSE_BSD_LIKE
diff --git a/Makefile.am b/Makefile.am
new file mode 100644
index 0000000..fc51921
--- /dev/null
+++ b/Makefile.am
@@ -0,0 +1,214 @@
+# This file is part of flex.
+
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+
+# Neither the name of the University nor the names of its contributors
+# may be used to endorse or promote products derived from this software
+# without specific prior written permission.
+
+# THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE.
+
+# Notes on building:
+
+# Possible values for DEFS:
+#
+# By default, flex generates 8-bit scanners when using table compression,
+# and 7-bit scanners when using uncompressed tables (-f or -F options).
+# For flex to always generate 8-bit scanners, add "-DDEFAULT_CSIZE=256"
+# to DEFS.
+#
+# For Vax/VMS, add "-DVMS" to DEFS.
+#
+# For MS-DOS, add "-DMS_DOS" to DEFS.  See the directory MISC/MSDOS for
+# additional info.
+
+AM_YFLAGS = -d
+ACLOCAL_AMFLAGS = -I m4
+m4 = @M4@
+indent = @INDENT@
+
+bin_PROGRAMS = flex
+lib_LTLIBRARIES = \
+	libfl.la \
+	libfl_pic.la
+
+flex_SOURCES = \
+	ccl.c \
+	dfa.c \
+	ecs.c \
+	scanflags.c \
+	gen.c \
+	main.c \
+	misc.c \
+	nfa.c \
+	parse.y \
+	scan.l \
+	skel.c \
+	sym.c \
+	tblcmp.c \
+	yylex.c \
+	options.c \
+	scanopt.c \
+	buf.c \
+	tables.c \
+	tables_shared.c \
+	filter.c \
+	regex.c
+
+
+LDADD = lib/libcompat.la
+
+libfl_la_SOURCES = \
+	libmain.c \
+	libyywrap.c
+
+libfl_la_LDFLAGS = -no-undefined -version-info @SHARED_VERSION_INFO@
+
+libfl_pic_la_SOURCES = \
+	libmain.c \
+	libyywrap.c
+
+libfl_pic_la_LDFLAGS = -no-undefined -version-info @SHARED_VERSION_INFO@
+
+noinst_HEADERS = \
+	flexdef.h \
+	flexint.h \
+	version.h \
+	options.h \
+	scanopt.h \
+	tables.h \
+	tables_shared.h
+
+include_HEADERS = \
+	FlexLexer.h
+
+dist_doc_DATA = \
+	AUTHORS \
+	COPYING \
+	NEWS \
+	ONEWS \
+	README \
+	TODO
+
+EXTRA_DIST = \
+	.indent.pro \
+	ABOUT-NLS \
+	INSTALL \
+	autogen.sh \
+	flex.skl \
+	mkskel.sh \
+	config.rpath \
+	gettext.h
+
+BUILT_SOURCES = \
+	skel.c
+
+SUBDIRS = \
+	lib \
+	. \
+	doc \
+	examples \
+	po \
+	tests
+
+localedir = $(datadir)/locale
+AM_CPPFLAGS = -DLOCALEDIR=\"$(localedir)\" -I$(top_srcdir)/intl
+LIBS = @LIBINTL@ @LIBS@
+
+skel.c: flex.skl mkskel.sh flexint.h tables_shared.h
+	sed 's/m4_/m4postproc_/g; s/m4preproc_/m4_/g' $(srcdir)/flex.skl | $(m4) -P -DFLEX_MAJOR_VERSION=`echo $(VERSION)|cut -f 1 -d .` -DFLEX_MINOR_VERSION=`echo $(VERSION)|cut -f 2 -d .` -DFLEX_SUBMINOR_VERSION=`echo $(VERSION)|cut -f 3 -d .` | sed 's/m4postproc_/m4_/g' | $(SHELL) $(srcdir)/mkskel.sh  >skel.c
+
+# Explicitly describe dependencies.
+# You can recreate this with `gcc -I. -MM *.c'
+buf.o: buf.c flexdef.h flexint.h
+ccl.o: ccl.c flexdef.h flexint.h
+dfa.o: dfa.c flexdef.h flexint.h tables.h tables_shared.h
+ecs.o: ecs.c flexdef.h flexint.h
+scanflags.o: scanflags.c flexdef.h flexint.h
+gen.o: gen.c flexdef.h flexint.h tables.h tables_shared.h
+libmain.o: libmain.c
+libyywrap.o: libyywrap.c
+main.o: main.c flexdef.h flexint.h version.h options.h scanopt.h \
+ tables.h tables_shared.h
+misc.o: misc.c flexdef.h flexint.h tables.h tables_shared.h
+nfa.o: nfa.c flexdef.h flexint.h
+options.o: options.c options.h scanopt.h flexdef.h flexint.h
+parse.o: parse.c flexdef.h flexint.h tables.h tables_shared.h
+scan.o: scan.c flexdef.h flexint.h parse.h
+scanopt.o: scanopt.c flexdef.h flexint.h scanopt.h
+skel.o: skel.c flexdef.h flexint.h
+sym.o: sym.c flexdef.h flexint.h
+tables.o: tables.c flexdef.h flexint.h tables.h tables_shared.h
+tables_shared.o: tables_shared.c flexdef.h flexint.h tables.h \
+ tables_shared.h
+tblcmp.o: tblcmp.c flexdef.h flexint.h
+yylex.o: yylex.c flexdef.h flexint.h parse.h
+filter.o: filter.c flexdef.h flexint.h
+
+# Create the ChangeLog, but only if we're inside a git working directory
+
+ChangeLog: $(srcdir)/tools/git2cl
+	if [ -d $(srcdir)/.git ] ; then \
+		$(srcdir)/tools/git2cl > $@ \
+	; fi
+
+# Run GNU indent on sources. Don't run this unless all the sources compile cleanly.
+#
+# Whole idea:
+#   1. Check for .indent.pro, otherwise indent will use unknown
+#      settings, or worse, the GNU defaults.)
+#   2. Check that this is GNU indent.
+#   3. Make sure to process only the NON-generated .c and .h files.
+#   4. Run indent twice per file. The first time is a test.
+#      Otherwise, indent overwrites your file even if it fails!
+indentfiles = \
+	buf.c \
+	ccl.c \
+	dfa.c \
+	ecs.c \
+	scanflags.c \
+	filter.c \
+	flexdef.h \
+	gen.c \
+	libmain.c \
+	libyywrap.c \
+	main.c \
+	misc.c \
+	nfa.c \
+	options.c \
+	options.h \
+	regex.c \
+	scanopt.c \
+	scanopt.h \
+	sym.c \
+	tables.c \
+	tables.h \
+	tables_shared.c \
+	tables_shared.h \
+	tblcmp.c
+
+indent:
+	if [ -f .indent.pro ] ; then \
+	for f in $(indentfiles);\
+	do\
+		echo indenting $$f ;\
+		$(indent) < $$f >/dev/null && indent $$f || echo $$f FAILED to indent ;\
+	done \
+	fi
+
+install-exec-hook:
+	cd $(DESTDIR)$(bindir) && \
+		$(LN_S) -f flex$(EXEEXT) flex++$(EXEEXT)
+
+.PHONY: ChangeLog tags indent
diff --git a/NEWS b/NEWS
new file mode 100644
index 0000000..530c355
--- /dev/null
+++ b/NEWS
@@ -0,0 +1,528 @@
+This is the file NEWS for the flex package. It records user -visible
+changes between releases of flex.
+
+See the file COPYING for copying conditions.
+
+* version 2.5.38
+
+** internationalization
+
+*** add sr translation from the translation project
+
+*** update da, es, ko, nl, pt_BR, ro, ru, sv, tr, vi, zh_CN translations from the translation project
+
+*** rename zh_tw to its proper zh_TW name
+
+* version 2.5.37 released 2012-08-03
+
+** Import flex into git. See
+   git://flex.git.sourceforge.net/gitroot/flex/flex.
+
+** Fix make install target to not fail when the flex++ program is
+   already installed
+
+** New translations from the translation project: de, fi, pl, vi
+
+* version 2.5.36 released 2012-07-20
+
+** various portability fixes that quiet compiler warnings on 64-bit
+   hosts
+
+** various manual fixes, including correcting the name of a %option and
+   updating some simple examples to use ANSI C syntax
+
+** various bug fixes that prevent certain error conditions from
+   persisting when they should not persist
+
+** improvements to the test suite so it behaves better when linking
+   compiled files
+
+** new translations from the translation project: ca, da, es, fi, fr,
+   ga, ko, pt_br, ro, ru, sv, tr, zh_cn
+
+** the flex distribution is now built with automake 1.10.1 and automake
+   2.61
+
+* version 2.5.35 released 2008-02-26
+
+** fixed bug that prevented flex from accepting certain comments in the
+  scanner file (resolves bugs #1849809 and #1849805)
+
+** fix bug that prevented headers for all functions from being generated
+  (resolves bug #1628314)
+
+** change yy_size_t to be size_t (resolves bug #1849812)
+
+** new de, nl, pl, pt_br, vi translations from the translation project
+
+* version 2.5.34 released 2007-12-12
+
+** introduce yylex_init_extra; see the manual for details
+
+** introduce %option extra-type="your_type *" (resolves bug #1744505)
+
+** The flex program now parses multiple short concatenated options (resolves bug
+  #1619820). Thanks to Petr Machata of Red Hat on this issue.
+
+** better checking after yyalloc/yyrealloc (resolves bug #1595967)
+
+** flex now provides for a libfl_pic.a compiled with position
+   independent code. Particularly useful when including a flex scanner
+   in a shared library and with more recent versions of gcc. Thanks to the Debian project for the idea.
+
+** SourceForge feature request #1658379: Expose YY_BUF_SIZE in the
+	header file.
+
+** flex better escapes filenames with special characters in them
+   (resolves bug #1623600)
+
+** a memory leak was plugged(resolves bug #1601111)
+
+** pattern language expanded; see the manual for details on the below
+   highlights
+
+*** pattern options added to specify patterns as case-insensitive or
+    case-sensitive
+
+*** pattern options to specify whether the "." character should match
+    the newline character
+
+*** pattern options added to allow ignoring of whitespace in patterns
+
+*** POSIX character classes may be negated in patterns
+
+*** patterns may now use set difference, union operators
+
+** the manual now contains an appendix listing various common patterns
+   which may be useful when writing scanners
+
+** some memory leaks were removed from the C++ scanner (but the C++
+  scanner is still experimental and may change radically without
+  notice)
+
+** c++ scanners can now use yywrap
+
+** added new unit test for c++ and yywrap
+
+** portability fixes to some unit tests
+
+** flex man page and flex manual in pdf now distributed in the flex
+distribution
+
+** new ca, vi, ga, nl translations from the translation project
+
+** flex no longer comes with an rpm spec file
+
+** flex development now happens with automake 1.9.6
+
+* version 2.5.33 released 2006-2-20
+
+** all flex resources are now to be found from the website at
+   http://flex.sourceforge.net/
+
+** there was no release 2.5.32 published
+
+** numerous bug and security fixes
+
+** new nl, vi, sv, ro, po, ga, ca, fr, tr translations from the translation project
+
+** upgrade to use gettext 0.12 (this now makes the "pdf" and "ps"
+   targets in the build system able to be run successfully)
+
+* version 2.5.31 released 2003-4-1
+
+** remove --enable-maintainer-mode configure option; none of the
+   Makefiles were using it and it can be unduely confusing
+
+* version 2.5.30 released 2003-4-1
+
+** yylineno is per-buffer in reentrant scanners
+
+** added %top directive for placing code at the top of the generated
+   scanner; see manual for details
+
+** flex now uses m4 to generate scanners; while this means that
+   scanners are more readable, it means that flex requires m4 to be
+   installed; see manual for details
+
+* version 2.5.29 released 2003-3-5
+
+** Automatic stack management for multiple input buffers in C and C++ scanners
+
+** moved the flex documentation to a new doc/ subdirectory
+
+** cleanups to the yy namespace
+
+* version 2.5.28 released 2003-2-12
+
+** flex is now hosted at sourceforge
+
+** Fixed trailing slash bug in YY_INPUT macro def
+
+** Flex now warns if always-interactive is specified with fast or full
+
+* version 2.5.27 released 2003-1-21
+
+** flex now works with recent bison versions
+
+** new pt_br translation from the translation project
+
+* version 2.5.26 released 2003-1-14
+
+** Fixed table deserialization bug on big-endian archs. Patch sent from Bryce Nichols <bryce@bnichols.org>
+
+** yyleng has proper declarations now; this caused flex to generate
+   unusable scanners for some programs
+
+** the flex distribution now includes a spec file suitable for use
+   with rpm
+
+** some more c++ fixes
+
+** new es translation from the translation project
+
+** slight tweeks to the flex_int*_t types
+
+** flex now warns about pattern ranges that might be ambiguous when
+   generating a case-insensitive scanner
+
+
+* version 2.5.25 released 2002-12-2
+
+** flex now uses flex_int*_t types. For C99 systems, they are just the
+   int*_t types; for non-C99 systems, we just make some typedefs
+
+** new pt_br translation from the translation project
+
+* version 2.5.24 released 2002-11-25
+
+* more portability fixes
+
+** the manual continues to be updated and edited, but it's still got a
+   ways to go
+
+** it is possible to have multiple c++ scanners in the same program again
+
+** new turkish translation from the translation project
+
+* version 2.5.23 released 2002-10-21
+
+** more portability fixes
+
+** the manual includes a title page and a table-of-contents when printed
+
+** the test suite can be run with "make check" from the top-level
+   directory
+
+** configure now accepts the --enable-maintainer-mode option
+
+** gettext functionality is now only available externally
+
+** the constant FLEX_BETA is defined if flex is a beta release
+
+** the script create-test was not included in the distribution and it
+   should have been
+
+* version 2.5.22 released 2002-10-10
+
+** more portability fixes around how we get ahold of the integral
+   types; there is a constant FLEX_NEED_INTEGRAL_TYPE_DEFINITIONS
+   which you should define if you don't have the <inttypes.h> header
+   file (after you complain to your C vendor for not providing a
+   reasonable C environment)
+
+** more test suite cleanups; in particular, the test suite should run
+   correctly when build from a different directory
+
+** upgraded automake to 1.7 and consequently autoconf to 2.54; this
+   means, among other things, that there is some support for
+formatting the manual in postscript and pdf in the distributed
+   Makefile.in (and therefore in the Makefile built by configure)
+
+** the flex.1 manpage is generated by help2man; (this has been true
+   for quite a while but was not listed here)
+
+** flex now includes three defined constants to indicate which version
+  of flex generated a scanner (YY_FLEX_{MAJOR,MINOR,SUBMINOR}_VERSION)
+
+** flex tries its best to output only the relevant portions of the
+   skeleton when generating a scanner, thus avoiding as much
+   conditional compilation as possible
+
+* version 2.5.21 released 2002-9-17
+
+** one of the tests in the test suite broke the dist target
+
+* version 2.5.20 released 2002-9-16
+
+** A flex scanner has the ability to save the DFA tables to a file,
+   and load them at runtime when needed; see the manual for details
+
+** Added %option bison-bridge (--bison-bridge)
+
+** Removed %option reentrant-bison/--reentrant-bison/-Rb
+
+** yylineno is present in all scanners; Modified nasty performance
+   penalty warning with yylineno in documentation
+
+** test-table-opts is now run last in the test suite because it's so fat
+
+** flex can, to some extent, diagnose where internal problems occur
+
+** new translations from the translation project: fr, ca, de, ru, sv
+
+**Flex generates C99 defs now; see YY_TRADITIONAL_FUNC_DEFS in the
+  manual if that's not a good thing for you
+
+* version 2.5.19 released 2002-9-5
+
+** prevent segfault on input lines which are longer than the allocated
+   space (problem report from Manoj Srivastava
+   <srivasta@golden-gryphon.com>)
+
+** Changed option 'header' to 'header-file'
+
+* version 2.5.18 released 2002-9-4
+
+** portability fixes for integer constants and in the way the test
+   suite reports its results
+
+** the test for bison was reporting bison missing when it was, in
+   fact, found
+
+** if we don't find GNU indent, we're more careful when we're not
+   finding it
+
+* version 2.5.17 released 2002-8-29
+
+** more portability fixes
+
+** updated config.sub and config.guess
+
+** flex is indented by GNU indent (this was done earlier but not
+   explicitly documented)
+
+* version 2.5.16 released 2002-8-28
+
+** c++ scanners compile again
+
+** there is now an indent target in the top-level Makefile; configure
+   checks for GNU indent which is required for proper operation of the
+   indent target
+
+** some more portability fixes were made
+
+** %options and invocation sections of manual merged
+
+** a c++ test was added to the test suite
+
+** we're trying to clean up more files in the test suite's make clean
+   targets
+
+* version 2.5.15 released 2002-8-21
+
+** reject-state buffer is now dynamically allocated and REJECT buffer
+   variables are reentrant-safe
+
+** manual now discusses memory usage
+
+** skeleton now processed by m4 before mkskel.sh; (this only matters
+   if you want to change the skeleton or if you're doing flex development)
+
+** zh_cn translation added from translation project
+
+** a bug that caused a segfault has now been fixed
+
+** the test suite now respects the usual CFLAGS, etc. variables
+
+** removed some warnings which some tests trigggered with the -s option
+
+** the flex-generated header file now tries to be smarter about
+   conditionally including start conditions
+
+** tables code omitted from generated scanner when not used
+
+* version 2.5.14 released 2002-8-15
+
+** the tests using the reentrant c scanner as c++ were reworked
+   slightly to be sure that the c++ was enforced
+
+** de translation now included in the distribution
+
+** various portability fixes regarding nls support, c++ include
+   headers, etc.
+
+* version 2.5.13 released 2002-8-15
+
+** the header file output with %option header is now much smaller
+
+** Fixed type mismatch in printf in scanner skeleton
+
+** yylex_init now reports errors
+
+* version 2.5.12 released 2002-8-8
+
+** updated gettext support to 0.11.5
+
+** new fr translation from the translation project
+
+** bison is no longer needed to build flex; If you are building flex
+   from a release (i.e., not from a cvs snapshot), then you don't need
+   to have a pre-built lex around either (unless you modify scan.l, of
+   course); (This has been true for some time, but was not mentioned
+   here.)
+
+* version 2.5.11 released 2002-7-31
+
+** Fixed bug where yyless did not consider yylineno
+
+** the yylineno performance hit is now gone
+
+** fixed some typos in the manual and we now include texinfo.tex in
+   the distribution
+
+** traditional prototypes output for C scanners, controlled by a
+   preprocessor symbol; see documentation for details
+
+* version 2.5.10 released 2002-7-24
+
+** yy_globals renamed to yyscanner and yy_globals_t renamed to
+   yy_guts_t
+
+** added dist-bzip2 option to Makefile.am so we now produce a bzip2'd
+   archive in addition to the standard gzip archive
+
+*  version 2.5.9
+
+** new tests in test suite: test-mem-{nr,r}, test-posix,
+   test-posixly-correct, test-debug-{nr,r}
+
+** made changes to work with gcc-3.2 development code
+
+** ability to choose which memory functions are used in flex
+
+** new yylex_destroy() function for the non-reentrant scanner
+
+** new handling of POSIXLY_CORRECT environment variable
+
+** the test suite now has its copyrights explicitly described
+
+** new ca, de, fr, ru, sv, tr translations
+
+* version 2.5.8
+
+** a new --posix option generates scanners with posix-style abc{1,3}
+   compatible parsing, see manual for the screwy details
+
+* version 2.5.7
+
+** configure.in now includes a call to AC_PREREQ to enforce the
+   requirement for autoconf at least 2.50 (This only effects you if
+   you're doing flex development.)
+
+** configure now uses autoconf's versioning information and configure
+   --help reports the bug-reporting address for flex
+
+** test suite now only reports success versus failure; reporting
+   skipped is problematic under the current setup
+
+** compilation with --disable-nls now works
+
+** flex can now be built in a separate directory
+
+* version 2.5.6
+
+** gettext support added (from gettext 0.11)
+
+*** translations for ca, da, de, es, fr, ko, ru, sv, tr included
+
+** distribution now built under automake 1.6 and autoconf 2.53
+
+** command-line option parsing happens differently now:
+
+*** Added long option parsing
+
+*** Options -n and -c, previously deprecated, now simply do nothing
+
+*** Options are now parsed left to right
+
+** added a number of new options
+
+*** All positive %options are now accessible from the command line
+
+*** Added option -D, to define a preprocessor symbol
+
+*** Added option --header=FILE to specify a C .h file to generate
+
+*** added option --yywrap to call yywrap on EOF
+
+*** added option --yylineno to track line count in yylineno
+
+*** --yyclass=NAME name of C++ class when generating c++ scanners
+
+*** for long option names which are associated with existing short
+options, see accompanying documentation
+
+*** new %option nounistd or command-line --nounistd added to prevent
+    flex from generating #include <unistd.h> on systems that don't
+    have that include file
+
+** Support for reentrant C scanners has been added
+
+*** Updated the manual with the new reentrant API
+
+*** Two new options %option reentrant (-R) and 
+%option reentrant-bison (-Rb)
+
+*** All globals optionally placed into struct yyglobals_t
+
+*** All access to globals replaced by macro invocations
+
+*** All functions optionally take one additional
+argument, yy_globals
+
+*** New style for invoking reentrant scanner:
+yylex_init(void** scanner );
+yylex( scanner );
+yylex_destroy( scanner );
+
+*** Added get/set functions for members of struct yy_globals_t
+e.g.,  yyget_text, yyget_leng, etc
+
+*** Prefix substitution added for new functions
+
+*** Macro shortcuts to the lengthy get/set functions
+provided for use in actions, e.g.,  yytext, yyleng, etc
+
+*** Arbitrary, user-defined data, "yyextra", may be added to scanner
+
+** %option nomain no longer implies %option yywrap
+But the inverse is still true
+
+** Developer test suite added
+
+*** TESTS/ directory has been added. Users can 
+'make test' in the TESTS directory to execute the test suite
+
+** Support for bison variables yylval and yylloc added
+
+** automake support for the build process
+
+** manual is now in texinfo/info format
+
+*** flex.1 removed from distribution
+
+** flex no longer generates C-language scanners with C++-style
+   comments
+
+** flex now generates scanners in c++ which are compatible with
+   recent c++ compilers
+
+** flex input scanner now recognizes '\r' as an EOL character
+
+See the file ONEWS for changes in earlier releases.
+
+Local Variables:
+mode: text
+mode: outline-minor
+end:
diff --git a/NOTICE b/NOTICE
new file mode 120000
index 0000000..d24842f
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1 @@
+COPYING
\ No newline at end of file
diff --git a/ONEWS b/ONEWS
new file mode 100644
index 0000000..3341577
--- /dev/null
+++ b/ONEWS
@@ -0,0 +1,1233 @@
+Changes between release 2.5.4 (11Sep96) and release 2.5.3:
+
+	- Fixed a bug introduced in 2.5.3 that blew it when a call
+	  to input() occurred at the end of an input file.
+
+	- Fixed scanner skeleton so the example in the man page of
+	  scanning strings using exclusive start conditions works.
+
+	- Minor Makefile tweaks.
+
+
+Changes between release 2.5.3 (29May96) and release 2.5.2:
+
+	- Some serious bugs in yymore() have been fixed.  In particular,
+	  when using AT&T-lex-compatibility or %array, you can intermix
+	  calls to input(), unput(), and yymore().  (This still doesn't
+	  work for %pointer, and isn't likely to in the future.)
+
+	- A bug in handling NUL's in the input stream of scanners using
+	  REJECT has been fixed.
+
+	- The default main() in libfl.a now repeatedly calls yylex() until
+	  it returns 0, rather than just calling it once.
+
+	- Minor tweak for Windows NT Makefile, MISC/NT/Makefile.
+
+
+Changes between release 2.5.2 (25Apr95) and release 2.5.1:
+
+	- The --prefix configuration option now works.
+
+	- A bug that completely broke the "-Cf" table compression
+	  option has been fixed.
+
+	- A major headache involving "const" declarators and Solaris
+	  systems has been fixed.
+
+	- An octal escape sequence in a flex regular expression must
+	  now contain only the digits 0-7.
+
+	- You can now use "--" on the flex command line to mark the
+	  end of flex options.
+
+	- You can now specify the filename '-' as a synonym for stdin.
+
+	- By default, the scanners generated by flex no longer
+	  statically initialize yyin and yyout to stdin and stdout.
+	  This change is necessary because in some ANSI environments,
+	  stdin and stdout are not compile-time constant.  You can
+	  force the initialization using "%option stdinit" in the first
+	  section of your flex input.
+
+	- "%option nounput" now correctly omits the unput() routine
+	  from the output.
+
+	- "make clean" now removes config.log, config.cache, and the
+	  flex binary.  The fact that it removes the flex binary means
+	  you should take care if making changes to scan.l, to make
+	  sure you don't wind up in a bootstrap problem.
+
+	- In general, the Makefile has been reworked somewhat (thanks
+	  to Francois Pinard) for added flexibility - more changes will
+	  follow in subsequent releases.
+
+	- The .texi and .info files in MISC/texinfo/ have been updated,
+	  thanks also to Francois Pinard.
+
+	- The FlexLexer::yylex(istream* new_in, ostream* new_out) method
+	  now does not have a default for the first argument, to disambiguate
+	  it from FlexLexer::yylex().
+
+	- A bug in destructing a FlexLexer object before doing any scanning
+	  with it has been fixed.
+
+	- A problem with including FlexLexer.h multiple times has been fixed.
+
+	- The alloca() chud necessary to accommodate bison has grown
+	  even uglier, but hopefully more correct.
+
+	- A portability tweak has been added to accommodate compilers that
+	  use char* generic pointers.
+
+	- EBCDIC contact information in the file MISC/EBCDIC has been updated.
+
+	- An OS/2 Makefile and config.h for flex 2.5 is now available in
+	  MISC/OS2/, contributed by Kai Uwe Rommel.
+
+	- The descrip.mms file for building flex under VMS has been updated,
+	  thanks to Pat Rankin.
+
+	- The notes on building flex for the Amiga have been updated for
+	  flex 2.5, contributed by Andreas Scherer.
+
+
+Changes between release 2.5.1 (28Mar95) and release 2.4.7:
+
+	- A new concept of "start condition" scope has been introduced.
+	  A start condition scope is begun with:
+
+		<SCs>{
+
+	  where SCs is a list of one or more start conditions.  Inside
+	  the start condition scope, every rule automatically has the
+	  prefix <SCs> applied to it, until a '}' which matches the
+	  initial '{'.  So, for example:
+
+		<ESC>{
+			"\\n"	return '\n';
+			"\\r"	return '\r';
+			"\\f"	return '\f';
+			"\\0"	return '\0';
+		}
+
+	  is equivalent to:
+
+		<ESC>"\\n"	return '\n';
+		<ESC>"\\r"	return '\r';
+		<ESC>"\\f"	return '\f';
+		<ESC>"\\0"	return '\0';
+
+	  As indicated in this example, rules inside start condition scopes
+	  (and any rule, actually, other than the first) can be indented,
+	  to better show the extent of the scope.
+
+	  Start condition scopes may be nested.
+
+	- The new %option directive can be used in the first section of
+	  a flex scanner to control scanner-generation options.  Most
+	  options are given simply as names, optionally preceded by the
+	  word "no" (with no intervening whitespace) to negate their
+	  meaning.  Some are equivalent to flex flags, so putting them
+	  in your scanner source is equivalent to always specifying
+	  the flag (%option's take precedence over flags):
+
+		7bit	-7 option
+		8bit	-8 option
+		align	-Ca option
+		backup	-b option
+		batch	-B option
+		c++	-+ option
+		caseful	opposite of -i option (caseful is the default);
+		case-sensitive	same as above
+		caseless	-i option;
+		case-insensitive	same as above
+		debug	-d option
+		default	opposite of -s option
+		ecs	-Ce option
+		fast	-F option
+		full	-f option
+		interactive	-I option
+		lex-compat	-l option
+		meta-ecs	-Cm option
+		perf-report	-p option
+		read	-Cr option
+		stdout	-t option
+		verbose	-v option
+		warn	opposite of -w option (so use "%option nowarn" for -w)
+
+		array	equivalent to "%array"
+		pointer	equivalent to "%pointer" (default)
+
+	  Some provide new features:
+
+		always-interactive	generate a scanner which always
+			considers its input "interactive" (no call to isatty()
+			will be made when the scanner runs)
+		main	supply a main program for the scanner, which
+			simply calls yylex().  Implies %option noyywrap.
+		never-interactive	generate a scanner which never
+			considers its input "interactive" (no call to isatty()
+			will be made when the scanner runs)
+		stack	if set, enable start condition stacks (see below)
+		stdinit	if unset ("%option nostdinit"), initialize yyin
+			and yyout statically to nil FILE* pointers, instead
+			of stdin and stdout
+		yylineno	if set, keep track of the current line
+			number in global yylineno (this option is expensive
+			in terms of performance).  The line number is available
+			to C++ scanning objects via the new member function
+			lineno().
+		yywrap	if unset ("%option noyywrap"), scanner does not
+			call yywrap() upon EOF but simply assumes there
+			are no more files to scan
+
+	  Flex scans your rule actions to determine whether you use the
+	  REJECT or yymore features (this is not new).  Two %options can be
+	  used to override its decision, either by setting them to indicate
+	  the feature is indeed used, or unsetting them to indicate it
+	  actually is not used:
+
+		reject
+		yymore
+
+	  Three %option's take string-delimited values, offset with '=':
+
+		outfile="<name>"	equivalent to -o<name>
+		prefix="<name>"		equivalent to -P<name>
+		yyclass="<name>"	set the name of the C++ scanning class
+					(see below)
+
+	  A number of %option's are available for lint purists who
+	  want to suppress the appearance of unneeded routines in
+	  the generated scanner.  Each of the following, if unset,
+	  results in the corresponding routine not appearing in the
+	  generated scanner:
+
+		input, unput
+		yy_push_state, yy_pop_state, yy_top_state
+		yy_scan_buffer, yy_scan_bytes, yy_scan_string
+
+	  You can specify multiple options with a single %option directive,
+	  and multiple directives in the first section of your flex input file.
+
+	- The new function:
+
+		YY_BUFFER_STATE yy_scan_string( const char *str )
+
+	  returns a YY_BUFFER_STATE (which also becomes the current input
+	  buffer) for scanning the given string, which occurs starting
+	  with the next call to yylex().  The string must be NUL-terminated.
+	  A related function:
+
+		YY_BUFFER_STATE yy_scan_bytes( const char *bytes, int len )
+
+	  creates a buffer for scanning "len" bytes (including possibly NUL's)
+	  starting at location "bytes".
+
+	  Note that both of these functions create and scan a *copy* of
+	  the string/bytes.  (This may be desirable, since yylex() modifies
+	  the contents of the buffer it is scanning.)  You can avoid the
+	  copy by using:
+
+		YY_BUFFER_STATE yy_scan_buffer( char *base, yy_size_t size )
+
+	  which scans in place the buffer starting at "base", consisting
+	  of "size" bytes, the last two bytes of which *must* be
+	  YY_END_OF_BUFFER_CHAR (these bytes are not scanned; thus, scanning
+	  consists of base[0] through base[size-2], inclusive).  If you
+	  fail to set up "base" in this manner, yy_scan_buffer returns a
+	  nil pointer instead of creating a new input buffer.
+
+	  The type yy_size_t is an integral type to which you can cast
+	  an integer expression reflecting the size of the buffer.
+
+	- Three new routines are available for manipulating stacks of
+	  start conditions:
+
+		void yy_push_state( int new_state )
+
+	  pushes the current start condition onto the top of the stack
+	  and BEGIN's "new_state" (recall that start condition names are
+	  also integers).
+
+		void yy_pop_state()
+
+	  pops the top of the stack and BEGIN's to it, and
+
+		int yy_top_state()
+
+	  returns the top of the stack without altering the stack's
+	  contents.
+
+	  The start condition stack grows dynamically and so has no built-in
+	  size limitation.  If memory is exhausted, program execution
+	  is aborted.
+
+	  To use start condition stacks, your scanner must include
+	  a "%option stack" directive.
+
+	- flex now supports POSIX character class expressions.  These
+	  are expressions enclosed inside "[:" and ":]" delimiters (which
+	  themselves must appear between the '[' and ']' of a character
+	  class; other elements may occur inside the character class, too).
+	  The expressions flex recognizes are:
+
+		[:alnum:] [:alpha:] [:blank:] [:cntrl:] [:digit:] [:graph:]	
+		[:lower:] [:print:] [:punct:] [:space:] [:upper:] [:xdigit:]
+
+	  These expressions all designate a set of characters equivalent to
+	  the corresponding isXXX function (for example, [:alnum:] designates
+	  those characters for which isalnum() returns true - i.e., any
+	  alphabetic or numeric).  Some systems don't provide isblank(),
+	  so flex defines [:blank:] as a blank or a tab.
+
+	  For example, the following character classes are all equivalent:
+
+		[[:alnum:]]
+		[[:alpha:][:digit:]
+		[[:alpha:]0-9]
+		[a-zA-Z0-9]
+
+	  If your scanner is case-insensitive (-i flag), then [:upper:]
+	  and [:lower:] are equivalent to [:alpha:].
+
+	- The promised rewrite of the C++ FlexLexer class has not yet
+	  been done.  Support for FlexLexer is limited at the moment to
+	  fixing show-stopper bugs, so, for example, the new functions
+	  yy_scan_string() & friends are not available to FlexLexer
+	  objects.
+
+	- The new macro
+
+		yy_set_interactive(is_interactive)
+
+	  can be used to control whether the current buffer is considered
+	  "interactive".  An interactive buffer is processed more slowly,
+	  but must be used when the scanner's input source is indeed
+	  interactive to avoid problems due to waiting to fill buffers
+	  (see the discussion of the -I flag in flex.1).  A non-zero value
+	  in the macro invocation marks the buffer as interactive, a zero
+	  value as non-interactive.  Note that use of this macro overrides
+	  "%option always-interactive" or "%option never-interactive".
+
+	  yy_set_interactive() must be invoked prior to beginning to
+	  scan the buffer.
+
+	- The new macro
+
+		yy_set_bol(at_bol)
+
+	  can be used to control whether the current buffer's scanning
+	  context for the next token match is done as though at the
+	  beginning of a line (non-zero macro argument; makes '^' anchored
+	  rules active) or not at the beginning of a line (zero argument,
+	  '^' rules inactive).
+
+	- Related to this change, the mechanism for determining when a scan is
+	  starting at the beginning of a line has changed.  It used to be
+	  that '^' was active iff the character prior to that at which the
+	  scan started was a newline.  The mechanism now is that '^' is
+	  active iff the last token ended in a newline (or the last call to
+	  input() returned a newline).  For most users, the difference in
+	  mechanisms is negligible.  Where it will make a difference,
+	  however, is if unput() or yyless() is used to alter the input
+	  stream.  When in doubt, use yy_set_bol().
+
+	- The new beginning-of-line mechanism involved changing some fairly
+	  twisted code, so it may have introduced bugs - beware ...
+
+	- The macro YY_AT_BOL() returns true if the next token scanned from
+	  the current buffer will have '^' rules active, false otherwise.
+
+	- The new function
+
+		void yy_flush_buffer( struct yy_buffer_state* b )
+
+	  flushes the contents of the current buffer (i.e., next time
+	  the scanner attempts to match a token using b as the current
+	  buffer, it will begin by invoking YY_INPUT to fill the buffer).
+	  This routine is also available to C++ scanners (unlike some
+	  of the other new routines).
+
+	  The related macro
+
+		YY_FLUSH_BUFFER
+
+	  flushes the contents of the current buffer.
+
+	- A new "-ooutput" option writes the generated scanner to "output".
+	  If used with -t, the scanner is still written to stdout, but
+	  its internal #line directives (see previous item) use "output".
+
+	- Flex now generates #line directives relating the code it
+	  produces to the output file; this means that error messages
+	  in the flex-generated code should be correctly pinpointed.
+
+	- When generating #line directives, filenames with embedded '\'s
+	  have those characters escaped (i.e., turned into '\\').  This
+	  feature helps with reporting filenames for some MS-DOS and OS/2
+	  systems.
+
+	- The FlexLexer class includes two new public member functions:
+
+		virtual void switch_streams( istream* new_in = 0,
+						ostream* new_out = 0 )
+
+	  reassigns yyin to new_in (if non-nil) and yyout to new_out
+	  (ditto), deleting the previous input buffer if yyin is
+	  reassigned.  It is used by:
+
+		int yylex( istream* new_in = 0, ostream* new_out = 0 )
+
+	  which first calls switch_streams() and then returns the value
+	  of calling yylex().
+
+	- C++ scanners now have yy_flex_debug as a member variable of
+	  FlexLexer rather than a global, and member functions for testing
+	  and setting it.
+
+	- When generating a C++ scanning class, you can now use
+
+		%option yyclass="foo"
+
+	  to inform flex that you have derived "foo" as a subclass of
+	  yyFlexLexer, so flex will place your actions in the member
+	  function foo::yylex() instead of yyFlexLexer::yylex().  It also
+	  generates a yyFlexLexer::yylex() member function that generates a
+	  run-time error if called (by invoking yyFlexLexer::LexerError()).
+	  This feature is necessary if your subclass "foo" introduces some
+	  additional member functions or variables that you need to access
+	  from yylex().
+
+	- Current texinfo files in MISC/texinfo, contributed by Francois
+	  Pinard.
+
+	- You can now change the name "flex" to something else (e.g., "lex")
+	  by redefining $(FLEX) in the Makefile.
+
+	- Two bugs (one serious) that could cause "bigcheck" to fail have
+	  been fixed.
+
+	- A number of portability/configuration changes have been made
+	  for easier portability.
+
+	- You can use "YYSTATE" in your scanner as an alias for YY_START
+	  (for AT&T lex compatibility).
+
+	- input() now maintains yylineno.
+
+	- input() no longer trashes yytext.
+
+	- interactive scanners now read characters in YY_INPUT up to a
+	  newline, a large performance gain.
+
+	- C++ scanner objects now work with the -P option.  You include
+	  <FlexLexer.h> once per scanner - see comments in <FlexLexer.h>
+	  (or flex.1) for details.
+
+	- C++ FlexLexer objects now use the "cerr" stream to report -d output
+	  instead of stdio.
+
+	- The -c flag now has its full glorious POSIX interpretation (do
+	  nothing), rather than being interpreted as an old-style -C flag.
+
+	- Scanners generated by flex now include two #define's giving
+	  the major and minor version numbers (YY_FLEX_MAJOR_VERSION,
+	  YY_FLEX_MINOR_VERSION).  These can then be tested to see
+	  whether certain flex features are available.
+
+	- Scanners generated using -l lex compatibility now have the symbol
+	  YY_FLEX_LEX_COMPAT #define'd.
+
+	- When initializing (i.e., yy_init is non-zero on entry to yylex()),
+	  generated scanners now set yy_init to zero before executing
+	  YY_USER_INIT.  This means that you can set yy_init back to a
+	  non-zero value in YY_USER_INIT if you need the scanner to be
+	  reinitialized on the next call.
+
+	- You can now use "#line" directives in the first section of your
+	  scanner specification.
+
+	- When generating full-table scanners (-Cf), flex now puts braces
+	  around each row of the 2-d array initialization, to silence warnings
+	  on over-zealous compilers.
+
+	- Improved support for MS-DOS.  The flex sources have been successfully
+	  built, unmodified, for Borland 4.02 (all that's required is a
+	  Borland Makefile and config.h file, which are supplied in
+	  MISC/Borland - contributed by Terrence O Kane).
+
+	- Improved support for Macintosh using Think C - the sources should
+	  build for this platform "out of the box".  Contributed by Scott
+	  Hofmann.
+
+	- Improved support for VMS, in MISC/VMS/, contributed by Pat Rankin.
+
+	- Support for the Amiga, in MISC/Amiga/, contributed by Andreas
+	  Scherer.  Note that the contributed files were developed for
+	  flex 2.4 and have not been tested with flex 2.5.
+
+	- Some notes on support for the NeXT, in MISC/NeXT, contributed
+	  by Raf Schietekat.
+
+	- The MISC/ directory now includes a preformatted version of flex.1
+	  in flex.man, and pre-yacc'd versions of parse.y in parse.{c,h}.
+
+	- The flex.1 and flexdoc.1 manual pages have been merged.  There
+	  is now just one document, flex.1, which includes an overview
+	  at the beginning to help you find the section you need.
+
+	- Documentation now clarifies that start conditions persist across
+	  switches to new input files or different input buffers.  If you
+	  want to e.g., return to INITIAL, you must explicitly do so.
+
+	- The "Performance Considerations" section of the manual has been
+	  updated.
+
+	- Documented the "yy_act" variable, which when YY_USER_ACTION is
+	  invoked holds the number of the matched rule, and added an
+	  example of using yy_act to profile how often each rule is matched.
+
+	- Added YY_NUM_RULES, a definition that gives the total number
+	  of rules in the file, including the default rule (even if you
+	  use -s).
+
+	- Documentation now clarifies that you can pass a nil FILE* pointer
+	  to yy_create_buffer() or yyrestart() if you've arrange YY_INPUT
+	  to not need yyin.
+
+	- Documentation now clarifies that YY_BUFFER_STATE is a pointer to
+	  an opaque "struct yy_buffer_state".
+
+	- Documentation now stresses that you gain the benefits of removing
+	  backing-up states only if you remove *all* of them.
+
+	- Documentation now points out that traditional lex allows you
+	  to put the action on a separate line from the rule pattern if
+	  the pattern has trailing whitespace (ugh!), but flex doesn't
+	  support this.
+
+	- A broken example in documentation of the difference between
+	  inclusive and exclusive start conditions is now fixed.
+
+	- Usage (-h) report now goes to stdout.
+
+	- Version (-V) info now goes to stdout.
+
+	- More #ifdef chud has been added to the parser in attempt to
+	  deal with bison's use of alloca().
+
+	- "make clean" no longer deletes emacs backup files (*~).
+
+	- Some memory leaks have been fixed.
+
+	- A bug was fixed in which dynamically-expanded buffers were
+	  reallocated a couple of bytes too small.
+
+	- A bug was fixed which could cause flex to read and write beyond
+	  the end of the input buffer.
+
+	- -S will not be going away.
+
+
+Changes between release 2.4.7 (03Aug94) and release 2.4.6:
+
+	- Fixed serious bug in reading multiple files.
+
+	- Fixed bug in scanning NUL's.
+
+	- Fixed bug in input() returning 8-bit characters.
+
+	- Fixed bug in matching text with embedded NUL's when
+	  using %array or lex compatibility.
+
+	- Fixed multiple invocations of YY_USER_ACTION when using '|'
+	  continuation action.
+
+	- Minor prototyping fixes.
+
+Changes between release 2.4.6 (04Jan94) and release 2.4.5:
+
+	- Linking with -lfl no longer required if your program includes
+	  its own yywrap() and main() functions.  (This change will cause
+	  problems if you have a non-ANSI compiler on a system for which
+	  sizeof(int) != sizeof(void*) or sizeof(int) != sizeof(size_t).)
+
+	- The use of 'extern "C++"' in FlexLexer.h has been modified to
+	  get around an incompatibility with g++'s header files.
+
+Changes between release 2.4.5 (11Dec93) and release 2.4.4:
+
+	- Fixed bug breaking C++ scanners that use REJECT or variable
+	  trailing context.
+
+	- Fixed serious input problem for interactive scanners on
+	  systems for which char is unsigned.
+
+	- Fixed bug in incorrectly treating '$' operator as variable
+	  trailing context.
+
+	- Fixed bug in -CF table representation that could lead to
+	  corrupt tables.
+
+	- Fixed fairly benign memory leak.
+
+	- Added `extern "C++"' wrapper to FlexLexer.h header.  This
+	  should overcome the g++ 2.5.X problems mentioned in the
+	  NEWS for release 2.4.3.
+
+	- Changed #include of FlexLexer.h to use <> instead of "".
+
+	- Added feature to control whether the scanner attempts to
+	  refill the input buffer once it's exhausted.  This feature
+	  will be documented in the 2.5 release.
+
+
+Changes between release 2.4.4 (07Dec93) and release 2.4.3:
+
+	- Fixed two serious bugs in scanning 8-bit characters.
+
+	- Fixed bug in YY_USER_ACTION that caused it to be executed
+	  inappropriately (on the scanner's own internal actions, and
+	  with incorrect yytext/yyleng values).
+
+	- Fixed bug in pointing yyin at a new file and resuming scanning.
+
+	- Portability fix regarding min/max/abs macros conflicting with
+	  function definitions in standard header files.
+
+	- Added a virtual LexerError() method to the C++ yyFlexLexer class
+	  for reporting error messages instead of always using cerr.
+
+	- Added warning in flexdoc that the C++ scanning class is presently
+	  experimental and subject to considerable change between major
+	  releases.
+
+
+Changes between release 2.4.3 (03Dec93) and release 2.4.2:
+
+	- Fixed bug causing fatal scanner messages to fail to print.
+
+	- Fixed things so FlexLexer.h can be included in other C++
+	  sources.  One side-effect of this change is that -+ and -CF
+	  are now incompatible.
+
+	- libfl.a now supplies private versions of the the <string.h>/
+	  <strings.h> string routines needed by flex and the scanners
+	  it generates, to enhance portability to some BSD systems.
+
+	- More robust solution to 2.4.2's flexfatal() bug fix.
+
+	- Added ranlib of installed libfl.a.
+
+	- Some lint tweaks.
+
+	- NOTE: problems have been encountered attempting to build flex
+	  C++ scanners using g++ version 2.5.X.  The problem is due to an
+	  unfortunate heuristic in g++ 2.5.X that attempts to discern between
+	  C and C++ headers.  Because FlexLexer.h is installed (by default)
+	  in /usr/local/include and not /usr/local/lib/g++-include, g++ 2.5.X
+	  decides that it's a C header :-(.  So if you have problems, install
+	  the header in /usr/local/lib/g++-include instead.
+
+
+Changes between release 2.4.2 (01Dec93) and release 2.4.1:
+
+	- Fixed bug in libfl.a referring to non-existent "flexfatal" function.
+
+	- Modified to produce both compress'd and gzip'd tar files for
+	  distributions (you probably don't care about this change!).
+
+
+Changes between release 2.4.1 (30Nov93) and release 2.3.8:
+
+	- The new '-+' flag instructs flex to generate a C++ scanner class
+	  (thanks to Kent Williams).  flex writes an implementation of the
+	  class defined in FlexLexer.h to lex.yy.cc.  You may include
+	  multiple scanner classes in your program using the -P flag.  Note
+	  that the scanner class also provides a mechanism for creating
+	  reentrant scanners.  The scanner class uses C++ streams for I/O
+	  instead of FILE*'s (thanks to Tom Epperly).  If the flex executable's
+	  name ends in '+' then the '-+' flag is automatically on, so creating
+	  a symlink or copy of "flex" to "flex++" results in a version of
+	  flex that can be used exclusively for C++ scanners.
+
+	  Note that without the '-+' flag, flex-generated scanners can still
+	  be compiled using C++ compilers, though they use FILE*'s for I/O
+	  instead of streams.
+
+	  See the "GENERATING C++ SCANNERS" section of flexdoc for details.
+
+	- The new '-l' flag turns on maximum AT&T lex compatibility.  In
+	  particular, -l includes support for "yylineno" and makes yytext
+	  be an array instead of a pointer.  It does not, however, do away
+	  with all incompatibilities.  See the "INCOMPATIBILITIES WITH LEX
+	  AND POSIX" section of flexdoc for details.
+
+	- The new '-P' option specifies a prefix to use other than "yy"
+	  for the scanner's globally-visible variables, and for the
+	  "lex.yy.c" filename.  Using -P you can link together multiple
+	  flex scanners in the same executable.
+
+	- The distribution includes a "texinfo" version of flexdoc.1,
+	  contributed by Roland Pesch (thanks also to Marq Kole, who
+	  contributed another version).  It has not been brought up to
+	  date, but reflects version 2.3.  See MISC/flex.texinfo.
+
+	  The flex distribution will soon include G.T. Nicol's flex
+	  manual; he is presently bringing it up-to-date for version 2.4.
+
+	- yywrap() is now a function, and you now *must* link flex scanners
+	  with libfl.a.
+
+	- Site-configuration is now done via an autoconf-generated
+	  "configure" script contributed by Francois Pinard.
+
+	- Scanners now use fread() (or getc(), if interactive) and not
+	  read() for input.  A new "table compression" option, -Cr,
+	  overrides this change and causes the scanner to use read()
+	  (because read() is a bit faster than fread()).  -f and -F
+	  are now equivalent to -Cfr and -CFr; i.e., they imply the
+	  -Cr option.
+
+	- In the blessed name of POSIX compliance, flex supports "%array"
+	  and "%pointer" directives in the definitions (first) section of
+	  the scanner specification.  The former specifies that yytext
+	  should be an array (of size YYLMAX), the latter, that it should
+	  be a pointer.  The array version of yytext is universally slower
+	  than the pointer version, but has the advantage that its contents
+	  remain unmodified across calls to input() and unput() (the pointer
+	  version of yytext is, still, trashed by such calls).
+
+	  "%array" cannot be used with the '-+' C++ scanner class option.
+
+	- The new '-Ca' option directs flex to trade off memory for
+	  natural alignment when generating a scanner's tables.  In
+	  particular, table entries that would otherwise be "short"
+	  become "long".
+
+	- The new '-h' option produces a summary of the flex flags.
+
+	- The new '-V' option reports the flex version number and exits.
+
+	- The new scanner macro YY_START returns an integer value
+	  corresponding to the current start condition.  You can return
+	  to that start condition by passing the value to a subsequent
+	  "BEGIN" action.  You also can implement "start condition stacks"
+	  by storing the values in an integer stack.
+
+	- You can now redefine macros such as YY_INPUT by just #define'ing
+	  them to some other value in the first section of the flex input;
+	  no need to first #undef them.
+
+	- flex now generates warnings for rules that can't be matched.
+	  These warnings can be turned off using the new '-w' flag.  If
+	  your scanner uses REJECT then you will not get these warnings.
+
+	- If you specify the '-s' flag but the default rule can be matched,
+	  flex now generates a warning.
+
+	- "yyleng" is now a global, and may be modified by the user (though
+	  doing so and then using yymore() will yield weird results).
+
+	- Name definitions in the first section of a scanner specification
+	  can now include a leading '^' or trailing '$' operator.  In this
+	  case, the definition is *not* pushed back inside of parentheses.
+
+	- Scanners with compressed tables are now "interactive" (-I option)
+	  by default.  You can suppress this attribute (which makes them
+	  run slightly slower) using the new '-B' flag.
+
+	- Flex now generates 8-bit scanners by default, unless you use the
+	  -Cf or -CF compression options (-Cfe  and -CFe result in 8-bit
+	  scanners).  You can force it to generate a 7-bit scanner using
+	  the new '-7' flag.  You can build flex to generate 8-bit scanners
+	  for -Cf and -CF, too, by adding -DDEFAULT_CSIZE=256 to CFLAGS
+	  in the Makefile.
+
+	- You no longer need to call the scanner routine yyrestart() to
+	  inform the scanner that you have switched to a new file after
+	  having seen an EOF on the current input file.  Instead, just
+	  point yyin at the new file and continue scanning.
+
+	- You no longer need to invoke YY_NEW_FILE in an <<EOF>> action
+	  to indicate you wish to continue scanning.  Simply point yyin
+	  at a new file.
+
+	- A leading '#' no longer introduces a comment in a flex input.
+
+	- flex no longer considers formfeed ('\f') a whitespace character.
+
+	- %t, I'm happy to report, has been nuked.
+
+	- The '-p' option may be given twice ('-pp') to instruct flex to
+	  report minor performance problems as well as major ones.
+
+	- The '-v' verbose output no longer includes start/finish time
+	  information.
+
+	- Newlines in flex inputs can optionally include leading or
+	  trailing carriage-returns ('\r'), in support of several PC/Mac
+	  run-time libraries that automatically include these.
+
+	- A start condition of the form "<*>" makes the following rule
+	  active in every start condition, whether exclusive or inclusive.
+
+	- The following items have been corrected in the flex documentation:
+
+		- '-C' table compression options *are* cumulative.
+
+		- You may modify yytext but not lengthen it by appending
+		  characters to the end.  Modifying its final character
+		  will affect '^' anchoring for the next rule matched
+		  if the character is changed to or from a newline.
+
+		- The term "backtracking" has been renamed "backing up",
+		  since it is a one-time repositioning and not a repeated
+		  search.  What used to be the "lex.backtrack" file is now
+		  "lex.backup".
+
+		- Unindented "/* ... */" comments are allowed in the first
+		  flex input section, but not in the second.
+
+		- yyless() can only be used in the flex input source, not
+		  externally.
+
+		- You can use "yyrestart(yyin)" to throw away the
+		  current contents of the input buffer.
+
+		- To write high-speed scanners, attempt to match as much
+		  text as possible with each rule.  See MISC/fastwc/README
+		  for more information.
+
+		- Using the beginning-of-line operator ('^') is fairly
+		  cheap.  Using unput() is expensive.  Using yyless() is
+		  cheap.
+
+		- An example of scanning strings with embedded escape
+		  sequences has been added.
+
+		- The example of backing-up in flexdoc was erroneous; it
+		  has been corrected.
+
+	- A flex scanner's internal buffer now dynamically grows if needed
+	  to match large tokens.  Note that growing the buffer presently
+	  requires rescanning the (large) token, so consuming a lot of
+	  text this way is a slow process.  Also note that presently the
+	  buffer does *not* grow if you unput() more text than can fit
+	  into the buffer.
+
+	- The MISC/ directory has been reorganized; see MISC/README for
+	  details.
+
+	- yyless() can now be used in the third (user action) section
+	  of a scanner specification, thanks to Ceriel Jacobs.  yyless()
+	  remains a macro and cannot be used outside of the scanner source.
+
+	- The skeleton file is no longer opened at run-time, but instead
+	  compiled into a large string array (thanks to John Gilmore and
+	  friends at Cygnus).  You can still use the -S flag to point flex
+	  at a different skeleton file.
+
+	- flex no longer uses a temporary file to store the scanner's
+	  actions.
+
+	- A number of changes have been made to decrease porting headaches.
+	  In particular, flex no longer uses memset() or ctime(), and
+	  provides a single simple mechanism for dealing with C compilers
+	  that still define malloc() as returning char* instead of void*.
+
+	- Flex now detects if the scanner specification requires the -8 flag
+	  but the flag was not given or on by default.
+
+	- A number of table-expansion fencepost bugs have been fixed,
+	  making flex more robust for generating large scanners.
+
+	- flex more consistently identifies the location of errors in
+	  its input.
+
+	- YY_USER_ACTION is now invoked only for "real" actions, not for
+	  internal actions used by the scanner for things like filling
+	  the buffer or handling EOF.
+
+	- The rule "[^]]" now matches any character other than a ']';
+	  formerly it matched any character at all followed by a ']'.
+	  This change was made for compatibility with AT&T lex.
+
+	- A large number of miscellaneous bugs have been found and fixed
+	  thanks to Gerhard Wilhelms.
+
+	- The source code has been heavily reformatted, making patches
+	  relative to previous flex releases no longer accurate.
+
+
+Changes between 2.3 Patch #8 (21Feb93) and 2.3 Patch #7:
+
+	- Fixed bugs in dynamic memory allocation leading to grievous
+	  fencepost problems when generating large scanners.
+	- Fixed bug causing infinite loops on character classes with 8-bit
+	  characters in them.
+	- Fixed bug in matching repetitions with a lower bound of 0.
+	- Fixed bug in scanning NUL characters using an "interactive" scanner.
+	- Fixed bug in using yymore() at the end of a file.
+	- Fixed bug in misrecognizing rules with variable trailing context.
+	- Fixed bug compiling flex on Suns using gcc 2.
+	- Fixed bug in not recognizing that input files with the character
+	  ASCII 128 in them require the -8 flag.
+	- Fixed bug that could cause an infinite loop writing out
+	  error messages.
+	- Fixed bug in not recognizing old-style lex % declarations if
+	  followed by a tab instead of a space.
+	- Fixed potential crash when flex terminated early (usually due
+	  to a bad flag) and the -v flag had been given.
+	- Added some missing declarations of void functions.
+	- Changed to only use '\a' for __STDC__ compilers.
+	- Updated mailing addresses.
+
+
+Changes between 2.3 Patch #7 (28Mar91) and 2.3 Patch #6:
+
+	- Fixed out-of-bounds array access that caused bad tables
+	  to be produced on machines where the bad reference happened
+	  to yield a 1.  This caused problems installing or running
+	  flex on some Suns, in particular.
+
+
+Changes between 2.3 Patch #6 (29Aug90) and 2.3 Patch #5:
+
+	- Fixed a serious bug in yymore() which basically made it
+	  completely broken.  Thanks goes to Jean Christophe of
+	  the Nethack development team for finding the problem
+	  and passing along the fix.
+
+
+Changes between 2.3 Patch #5 (16Aug90) and 2.3 Patch #4:
+
+	- An up-to-date version of initscan.c so "make test" will
+	  work after applying the previous patches
+
+
+Changes between 2.3 Patch #4 (14Aug90) and 2.3 Patch #3:
+
+	- Fixed bug in hexadecimal escapes which allowed only digits,
+	  not letters, in escapes
+	- Fixed bug in previous "Changes" file!
+
+
+Changes between 2.3 Patch #3 (03Aug90) and 2.3 Patch #2:
+
+	- Correction to patch #2 for gcc compilation; thanks goes to
+	  Paul Eggert for catching this.
+
+
+Changes between 2.3 Patch #2 (02Aug90) and original 2.3 release:
+
+	- Fixed (hopefully) headaches involving declaring malloc()
+	  and free() for gcc, which defines __STDC__ but (often) doesn't
+	  come with the standard include files such as <stdlib.h>.
+	  Reordered #ifdef maze in the scanner skeleton in the hope of
+	  getting the declarations right for cfront and g++, too.
+
+	- Note that this patch supercedes patch #1 for release 2.3,
+	  which was never announced but was available briefly for
+	  anonymous ftp.
+
+
+Changes between 2.3 (full) release of 28Jun90 and 2.2 (alpha) release:
+
+User-visible:
+
+	- A lone <<EOF>> rule (that is, one which is not qualified with
+	  a list of start conditions) now specifies the EOF action for
+	  *all* start conditions which haven't already had <<EOF>> actions
+	  given.  To specify an end-of-file action for just the initial
+	  state, use <INITIAL><<EOF>>.
+
+	- -d debug output is now contigent on the global yy_flex_debug
+	  being set to a non-zero value, which it is by default.
+
+	- A new macro, YY_USER_INIT, is provided for the user to specify
+	  initialization action to be taken on the first call to the
+	  scanner.  This action is done before the scanner does its
+	  own initialization.
+
+	- yy_new_buffer() has been added as an alias for yy_create_buffer()
+
+	- Comments beginning with '#' and extending to the end of the line
+	  now work, but have been deprecated (in anticipation of making
+	  flex recognize #line directives).
+
+	- The funky restrictions on when semi-colons could follow the
+	  YY_NEW_FILE and yyless macros have been removed.  They now
+	  behave identically to functions.
+
+	- A bug in the sample redefinition of YY_INPUT in the documentation
+	  has been corrected.
+
+	- A bug in the sample simple tokener in the documentation has
+	  been corrected.
+
+	- The documentation on the incompatibilities between flex and
+	  lex has been reordered so that the discussion of yylineno
+	  and input() come first, as it's anticipated that these will
+	  be the most common source of headaches.
+
+
+Things which didn't used to be documented but now are:
+
+	- flex interprets "^foo|bar" differently from lex.  flex interprets
+	  it as "match either a 'foo' or a 'bar', providing it comes at the
+	  beginning of a line", whereas lex interprets it as "match either
+	  a 'foo' at the beginning of a line, or a 'bar' anywhere".
+
+	- flex initializes the global "yyin" on the first call to the
+	  scanner, while lex initializes it at compile-time.
+
+	- yy_switch_to_buffer() can be used in the yywrap() macro/routine.
+
+	- flex scanners do not use stdio for their input, and hence when
+	  writing an interactive scanner one must explictly call fflush()
+	  after writing out a prompt.
+
+	- flex scanner can be made reentrant (after a fashion) by using
+	  "yyrestart( yyin );".  This is useful for interactive scanners
+	  which have interrupt handlers that long-jump out of the scanner.
+
+	- a defense of why yylineno is not supported is included, along
+	  with a suggestion on how to convert scanners which rely on it.
+
+
+Other changes:
+
+	- Prototypes and proper declarations of void routines have
+	  been added to the flex source code, courtesy of Kevin B. Kenny.
+
+	- Routines dealing with memory allocation now use void* pointers
+	  instead of char* - see Makefile for porting implications.
+
+	- Error-checking is now done when flex closes a file.
+
+	- Various lint tweaks were added to reduce the number of gripes.
+
+	- Makefile has been further parameterized to aid in porting.
+
+	- Support for SCO Unix added.
+
+	- Flex now sports the latest & greatest UC copyright notice
+	  (which is only slightly different from the previous one).
+
+	- A note has been added to flexdoc.1 mentioning work in progress
+	  on modifying flex to generate straight C code rather than a
+	  table-driven automaton, with an email address of whom to contact
+	  if you are working along similar lines.
+
+
+Changes between 2.2 Patch #3 (30Mar90) and 2.2 Patch #2:
+
+	- fixed bug which caused -I scanners to bomb
+
+
+Changes between 2.2 Patch #2 (27Mar90) and 2.2 Patch #1:
+
+	- fixed bug writing past end of input buffer in yyunput()
+	- fixed bug detecting NUL's at the end of a buffer
+
+
+Changes between 2.2 Patch #1 (23Mar90) and 2.2 (alpha) release:
+
+	- Makefile fixes: definition of MAKE variable for systems
+	  which don't have it; installation of flexdoc.1 along with
+	  flex.1; fixed two bugs which could cause "bigtest" to fail.
+
+	- flex.skel fix for compiling with g++.
+
+	- README and flexdoc.1 no longer list an out-of-date BITNET address
+	  for contacting me.
+
+	- minor typos and formatting changes to flex.1 and flexdoc.1.
+
+
+Changes between 2.2 (alpha) release of March '90 and previous release:
+
+User-visible:
+
+	- Full user documentation now available.
+
+	- Support for 8-bit scanners.
+
+	- Scanners now accept NUL's.
+
+	- A facility has been added for dealing with multiple
+	  input buffers.
+
+	- Two manual entries now.  One which fully describes flex
+	  (rather than just its differences from lex), and the
+	  other for quick(er) reference.
+
+	- A number of changes to bring flex closer into compliance
+	  with the latest POSIX lex draft:
+
+		%t support
+		flex now accepts multiple input files and concatenates
+		    them together to form its input
+		previous -c (compress) flag renamed -C
+		do-nothing -c and -n flags added
+		Any indented code or code within %{}'s in section 2 is
+		    now copied to the output
+
+	- yyleng is now a bona fide global integer.
+
+	- -d debug information now gives the line number of the
+	  matched rule instead of which number rule it was from
+	  the beginning of the file.
+
+	- -v output now includes a summary of the flags used to generate
+	  the scanner.
+
+	- unput() and yyrestart() are now globally callable.
+
+	- yyrestart() no longer closes the previous value of yyin.
+
+	- C++ support; generated scanners can be compiled with C++ compiler.
+
+	- Primitive -lfl library added, containing default main()
+	  which calls yylex().  A number of routines currently living
+	  in the scanner skeleton will probably migrate to here
+	  in the future (in particular, yywrap() will probably cease
+	  to be a macro and instead be a function in the -lfl library).
+
+	- Hexadecimal (\x) escape sequences added.
+
+	- Support for MS-DOS, VMS, and Turbo-C integrated.
+
+	- The %used/%unused operators have been deprecated.  They
+	  may go away soon.
+
+
+Other changes:
+
+	- Makefile enhanced for easier testing and installation.
+	- The parser has been tweaked to detect some erroneous
+	  constructions which previously were missed.
+	- Scanner input buffer overflow is now detected.
+	- Bugs with missing "const" declarations fixed.
+	- Out-of-date Minix/Atari patches provided.
+	- Scanners no longer require printf() unless FLEX_DEBUG is being used.
+	- A subtle input() bug has been fixed.
+	- Line numbers for "continued action" rules (those following
+	  the special '|' action) are now correct.
+	- unput() bug fixed; had been causing problems porting flex to VMS.
+	- yymore() handling rewritten to fix bug with interaction
+	  between yymore() and trailing context.
+	- EOF in actions now generates an error message.
+	- Bug involving -CFe and generating equivalence classes fixed.
+	- Bug which made -CF be treated as -Cf fixed.
+	- Support for SysV tmpnam() added.
+	- Unused #define's for scanner no longer generated.
+	- Error messages which are associated with a particular input
+	  line are now all identified with their input line in standard
+	  format.
+	- % directives which are valid to lex but not to flex are
+	  now ignored instead of generating warnings.
+	- -DSYS_V flag can now also be specified -DUSG for System V
+	  compilation.
+
+
+Changes between 2.1 beta-test release of June '89 and previous release:
+
+User-visible:
+
+	- -p flag generates a performance report to stderr.  The report
+	  consists of comments regarding features of the scanner rules
+	  which result in slower scanners.
+
+	- -b flag generates backtracking information to lex.backtrack.
+	  This is a list of scanner states which require backtracking
+	  and the characters on which they do so.  By adding rules
+	  one can remove backtracking states.  If all backtracking states
+	  are eliminated, the generated scanner will run faster.
+	  Backtracking is not yet documented in the manual entry.
+
+	- Variable trailing context now works, i.e., one can have
+	  rules like "(foo)*/[ \t]*bletch".  Some trailing context
+	  patterns still cannot be properly matched and generate
+	  error messages.  These are patterns where the ending of the
+	  first part of the rule matches the beginning of the second
+	  part, such as "zx*/xy*", where the 'x*' matches the 'x' at
+	  the beginning of the trailing context.  Lex won't get these
+	  patterns right either.
+
+	- Faster scanners.
+
+	- End-of-file rules.  The special rule "<<EOF>>" indicates
+	  actions which are to be taken when an end-of-file is
+	  encountered and yywrap() returns non-zero (i.e., indicates
+	  no further files to process).  See manual entry for example.
+
+	- The -r (reject used) flag is gone.  flex now scans the input
+	  for occurrences of the string "REJECT" to determine if the
+	  action is needed.  It tries to be intelligent about this but
+	  can be fooled.  One can force the presence or absence of
+	  REJECT by adding a line in the first section of the form
+	  "%used REJECT" or "%unused REJECT".
+
+	- yymore() has been implemented.  Similarly to REJECT, flex
+	  detects the use of yymore(), which can be overridden using
+	  "%used" or "%unused".
+
+	- Patterns like "x{0,3}" now work (i.e., with lower-limit == 0).
+
+	- Removed '\^x' for ctrl-x misfeature.
+
+	- Added '\a' and '\v' escape sequences.
+
+	- \<digits> now works for octal escape sequences; previously
+	  \0<digits> was required.
+
+	- Better error reporting; line numbers are associated with rules.
+
+	- yyleng is a macro; it cannot be accessed outside of the
+	  scanner source file.
+
+	- yytext and yyleng should not be modified within a flex action.
+
+	- Generated scanners #define the name FLEX_SCANNER.
+
+	- Rules are internally separated by YY_BREAK in lex.yy.c rather
+	  than break, to allow redefinition.
+
+	- The macro YY_USER_ACTION can be redefined to provide an action
+	  which is always executed prior to the matched rule's action.
+	
+	- yyrestart() is a new action which can be used to restart
+	  the scanner after it has seen an end-of-file (a "real" one,
+	  that is, one for which yywrap() returned non-zero).  It takes
+	  a FILE* argument indicating a new file to scan and sets
+	  things up so that a subsequent call to yylex() will start
+	  scanning that file.
+
+	- Internal scanner names all preceded by "yy_"
+
+	- lex.yy.c is deleted if errors are encountered during processing.
+
+	- Comments may be put in the first section of the input by preceding
+	  them with '#'.
+
+
+
+Other changes:
+
+	- Some portability-related bugs fixed, in particular for machines
+	  with unsigned characters or sizeof( int* ) != sizeof( int ).
+	  Also, tweaks for VMS and Microsoft C (MS-DOS), and identifiers all
+	  trimmed to be 31 or fewer characters.  Shortened file names
+	  for dinosaur OS's.  Checks for allocating > 64K memory
+	  on 16 bit'ers.  Amiga tweaks.  Compiles using gcc on a Sun-3.
+	- Compressed and fast scanner skeletons merged.
+	- Skeleton header files done away with.
+	- Generated scanner uses prototypes and "const" for __STDC__.
+	- -DSV flag is now -DSYS_V for System V compilation.
+	- Removed all references to FTL language.
+	- Software now covered by BSD Copyright.
+	- flex will replace lex in subsequent BSD releases.
diff --git a/README b/README
new file mode 100644
index 0000000..8906709
--- /dev/null
+++ b/README
@@ -0,0 +1,113 @@
+This is flex, the fast lexical analyzer generator.
+
+flex is a tool for generating scanners: programs which recognize
+lexical patterns in text.
+
+More information about flex as well as the latest official release of
+flex can be found at:
+
+http://flex.sourceforge.net/
+
+Bug reports should be submitted using the SourceForge Bug Tracker for
+flex at:
+
+http://sourceforge.net/tracker/?group_id=97492&atid=618177
+
+The flex codebase is kept in git at:
+
+https://github.com/westes/flex
+
+There are several mailing lists available as well:
+
+flex-announce@lists.sourceforge.net - where posts will be made
+announcing new releases of flex.
+
+flex-help@lists.sourceforge.net - where you can post questions about
+using flex
+
+flex-devel@lists.sourceforge.net - where you can discuss development of
+flex itself
+
+Find information on subscribing to the mailing lists at:
+
+http://sourceforge.net/mail/?group_id=97492
+
+The flex distribution contains the following files which may be of interest:
+
+README - This file.
+
+NEWS - current version number and list of user-visible changes.
+
+INSTALL - basic installation information.
+
+ABOUT-NLS - description of internationalization support in flex.
+
+COPYING - flex's copyright and license.
+
+doc/ - user documentation.
+
+examples/ - containing examples of some possible flex scanners and a
+few other things. See the file examples/README for more details.
+
+TODO - outstanding bug reports, desired features, etc.
+
+tests/ - regression tests. See TESTS/README for details.
+
+po/ - internationalization support files.
+
+You need the following tools to build flex from the maintainer's
+repository:
+
+compiler suite - flex is built with gcc
+bash, or a good Bourne-style shell
+m4 - m4 -p needs to work; GNU m4 and a few others are suitable
+GNU bison;  to generate parse.c from parse.y
+autoconf 2.69; for handling the build system
+automake 1.12.2; for Makefile generation
+gettext 0.18; fori18n support
+help2man 1.36; to generate the flex man page
+tar, gzip, etc.; for packaging of the source distribution
+GNU texinfo 498;  to build and test the flex manual
+GNU indent 2.8; for indenting the flex source the way we want it done
+
+Once you have all the necessary tools installed, life becomes
+simple. To prepare the flex tree for building, run the script:
+
+$ ./autogen.sh
+
+in the top level of the flex source tree.
+This script calls the various tools needed to get flex ready for the
+GNU-style configure script to be able to work.
+
+From this point on, building flex follows the usual configure, make,
+make install routine.
+
+This file is part of flex.
+
+This code is derived from software contributed to Berkeley by
+Vern Paxson.
+
+The United States Government has rights in this work pursuant
+to contract no. DE-AC03-76SF00098 between the United States
+Department of Energy and the University of California.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+Neither the name of the University nor the names of its contributors
+may be used to endorse or promote products derived from this software
+without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.
+
+
diff --git a/THANKS b/THANKS
new file mode 100644
index 0000000..ac91ef6
--- /dev/null
+++ b/THANKS
@@ -0,0 +1,53 @@
+Vern had the following things to say:
+
+Many thanks to the 2.5 beta-testers for finding bugs and helping test and
+increase portability: Stan Adermann, Scott David Daniels, Charles Elliott,
+Joe Gayda, Chris Meier, James Nordby, Terrence O'Kane, Karsten Pahnke,
+Francois Pinard, Pat Rankin, Andreas Scherer, Marc Wiese, Nathan Zelle.
+
+Thanks to the many flex beta-testers, feedbackers, and contributors,
+especially Francois Pinard, Casey Leedom, Robert Abramovitz, Stan
+Adermann, Terry Allen, David Barker-Plummer, John Basrai, Neal Becker,
+Nelson H.F. Beebe, benson@odi.com, Karl Berry, Peter A. Bigot, Simon
+Blanchard, Keith Bostic, Frederic Brehm, Ian Brockbank, Kin Cho, Nick
+Christopher, Brian Clapper, J.T. Conklin, Jason Coughlin, Bill Cox,
+Nick Cropper, Dave Curtis, Scott David Daniels, Chris G. Demetriou,
+Theo Deraadt, Mike Donahue, Chuck Doucette, Tom Epperly, Leo Eskin,
+Chris Faylor, Chris Flatters, Jon Forrest, Jeffrey Friedl, Joe Gayda,
+Kaveh R. Ghazi, Wolfgang Glunz, Eric Goldman, Christopher M. Gould,
+Ulrich Grepel, Peer Griebel, Jan Hajic, Charles Hemphill, NORO Hideo,
+Jarkko Hietaniemi, Scott Hofmann, Jeff Honig, Dana Hudes, Eric Hughes,
+John Interrante, Ceriel Jacobs, Michal Jaegermann, Sakari Jalovaara,
+Jeffrey R. Jones, Henry Juengst, Klaus Kaempf, Jonathan I. Kamens,
+Terrence O Kane, Amir Katz, ken@ken.hilco.com, Kevin B. Kenny, Steve
+Kirsch, Winfried Koenig, Marq Kole, Ronald Lamprecht, Greg Lee, Rohan
+Lenard, Craig Leres, John Levine, Steve Liddle, David Loffredo, Mike
+Long, Mohamed el Lozy, Brian Madsen, Malte, Joe Marshall, Bengt
+Martensson, Chris Metcalf, Luke Mewburn, Jim Meyering, R. Alexander
+Milowski, Erik Naggum, G.T. Nicol, Landon Noll, James Nordby, Marc
+Nozell, Richard Ohnemus, Karsten Pahnke, Sven Panne, Roland Pesch,
+Walter Pelissero, Gaumond Pierre, Esmond Pitt, Jef Poskanzer, Joe
+Rahmeh, Jarmo Raiha, Frederic Raimbault, Pat Rankin, Rick Richardson,
+Kevin Rodgers, Kai Uwe Rommel, Jim Roskind, Alberto Santini, Andreas
+Scherer, Darrell Schiebel, Raf Schietekat, Doug Schmidt, Philippe
+Schnoebelen, Andreas Schwab, Larry Schwimmer, Alex Siegel, Eckehard
+Stolz, Jan-Erik Strvmquist, Mike Stump, Paul Stuart, Dave Tallman, Ian
+Lance Taylor, Chris Thewalt, Richard M. Timoney, Jodi Tsai, Paul
+Tuinenga, Gary Weik, Frank Whaley, Gerhard Wilhelms, Kent Williams,
+Ken Yap, Ron Zellar, Nathan Zelle, David Zuhn, and those whose names
+have slipped my marginal mail-archiving skills but whose contributions
+are appreciated all the same.
+
+Thanks to Keith Bostic, Jon Forrest, Noah Friedman,
+John Gilmore, Craig Leres, John Levine, Bob Mulcahy, G.T.
+Nicol, Francois Pinard, Rich Salz, and Richard Stallman for help with various
+distribution headaches.
+
+Thanks to Esmond Pitt and Earle Horton for 8-bit character support; to
+Benson Margulies and Fred Burke for C++ support; to Kent Williams and Tom
+Epperly for C++ class support; to Ove Ewerlid for support of NUL's; and to
+Eric Hughes for support of multiple buffers.
+
+This work was primarily done when I was with the Real Time Systems Group
+at the Lawrence Berkeley Laboratory in Berkeley, CA.  Many thanks to all there
+for the support I received.
diff --git a/TODO b/TODO
new file mode 100644
index 0000000..702a6db
--- /dev/null
+++ b/TODO
@@ -0,0 +1,66 @@
+* the manual:
+
+** do an end-to-end proofread of the manual (this is under way, but is
+   going slowly)
+
+** pretty up the dvi output; overflows, etc.
+
+** faq
+
+*** clean up the faqs section. The information is good; the texinfo
+   could use some touching up.
+
+*** index the faq entries
+
+*** mention that it's possible to use a variable to scan matching
+    brackets, nested comments etc.
+
+*** include something about lexing/parsing fortran
+
+** create a section on flex design, features, etc.
+
+* getext
+
+** make sure all flex modules use gettext translation facilities
+
+*subdirectories
+
+** in examples/manual, integrate the Makefile.examples into the
+    Makefile.am
+
+* test suite
+
+** integrate the test suite into automake's framework (note that the
+   test suite can be run from the top level directory with "make
+   check". Still, we want to get it completely under automake's control.)
+
+** make test suite more complete
+
+* generic coding
+
+** move as much skeleton code as possible out of gen.c and into
+  flex.skl
+
+** figure out whether we want to add the capability to have
+  auto-generated backout rules
+
+** token-type and token buffer support
+
+** check if we still need to #undef macros at the end of a header
+
+** merge yylineno into support for location tracking
+
+** bug where yylineno is not decremented on REJECT
+
+** bug where yylineno is counted in trailing context
+
+* C++
+
+** have a separate skeleton for c++
+
+** revisit the C++ API. We get requests to make it more complete.
+
+Local Variables:
+Mode: text
+mode: outline-minor
+End:
diff --git a/autogen.sh b/autogen.sh
new file mode 100755
index 0000000..a298f07
--- /dev/null
+++ b/autogen.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+#  This file is part of flex.
+
+#  Redistribution and use in source and binary forms, with or without
+#  modification, are permitted provided that the following conditions
+#  are met:
+
+#  1. Redistributions of source code must retain the above copyright
+#     notice, this list of conditions and the following disclaimer.
+#  2. Redistributions in binary form must reproduce the above copyright
+#     notice, this list of conditions and the following disclaimer in the
+#     documentation and/or other materials provided with the distribution.
+
+#  Neither the name of the University nor the names of its contributors
+#  may be used to endorse or promote products derived from this software
+#  without specific prior written permission.
+
+#  THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+#  IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+#  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+#  PURPOSE.
+
+# If you see no configure script, then run ./autogen.sh to create it
+# and procede with the "normal" build procedures.
+
+#if we pretend to have a ChangeLog, then automake is less
+#worried. (Don't worry, we *do* have a ChangeLog, we just need the
+#Makefile first.)
+
+touch ChangeLog
+autoreconf --install --verbose --force
diff --git a/buf.c b/buf.c
new file mode 100644
index 0000000..e5deb4e
--- /dev/null
+++ b/buf.c
@@ -0,0 +1,273 @@
+/* flex - tool to generate fast lexical analyzers */
+
+/*  Copyright (c) 1990 The Regents of the University of California. */
+/*  All rights reserved. */
+
+/*  This code is derived from software contributed to Berkeley by */
+/*  Vern Paxson. */
+
+/*  The United States Government has rights in this work pursuant */
+/*  to contract no. DE-AC03-76SF00098 between the United States */
+/*  Department of Energy and the University of California. */
+
+/*  This file is part of flex. */
+
+/*  Redistribution and use in source and binary forms, with or without */
+/*  modification, are permitted provided that the following conditions */
+/*  are met: */
+
+/*  1. Redistributions of source code must retain the above copyright */
+/*     notice, this list of conditions and the following disclaimer. */
+/*  2. Redistributions in binary form must reproduce the above copyright */
+/*     notice, this list of conditions and the following disclaimer in the */
+/*     documentation and/or other materials provided with the distribution. */
+
+/*  Neither the name of the University nor the names of its contributors */
+/*  may be used to endorse or promote products derived from this software */
+/*  without specific prior written permission. */
+
+/*  THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR */
+/*  IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED */
+/*  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR */
+/*  PURPOSE. */
+
+#include "flexdef.h"
+
+/* Take note: The buffer object is sometimes used as a String buffer (one
+ * continuous string), and sometimes used as a list of strings, usually line by
+ * line.
+ * 
+ * The type is specified in buf_init by the elt_size. If the elt_size is
+ * sizeof(char), then the buffer should be treated as string buffer. If the
+ * elt_size is sizeof(char*), then the buffer should be treated as a list of
+ * strings.
+ *
+ * Certain functions are only appropriate for one type or the other. 
+ */
+
+/* global buffers. */
+struct Buf userdef_buf;		/**< for user #definitions triggered by cmd-line. */
+struct Buf defs_buf;		/**< for #define's autogenerated. List of strings. */
+struct Buf yydmap_buf;		/**< string buffer to hold yydmap elements */
+struct Buf m4defs_buf;          /**< m4 definitions. List of strings. */
+struct Buf top_buf;             /**< contains %top code. String buffer. */
+
+struct Buf *buf_print_strings(struct Buf * buf, FILE* out)
+{
+    int i;
+
+    if(!buf || !out)
+        return buf;
+
+    for (i=0; i < buf->nelts; i++){
+        const char * s = ((char**)buf->elts)[i];
+        if(s)
+            fprintf(out, "%s", s);
+    }
+    return buf;
+}
+
+/* Append a "%s" formatted string to a string buffer */
+struct Buf *buf_prints (struct Buf *buf, const char *fmt, const char *s)
+{
+	char   *t;
+        size_t tsz;
+
+	t = flex_alloc (tsz = strlen (fmt) + strlen (s) + 1);
+	if (!t)
+	    flexfatal (_("Allocation of buffer to print string failed"));
+	snprintf (t, tsz, fmt, s);
+	buf = buf_strappend (buf, t);
+	flex_free (t);
+	return buf;
+}
+
+/** Append a line directive to the string buffer.
+ * @param buf A string buffer.
+ * @param filename file name
+ * @param lineno line number
+ * @return buf
+ */
+struct Buf *buf_linedir (struct Buf *buf, const char* filename, int lineno)
+{
+    char *dst, *src, *t;
+
+    t = flex_alloc (strlen ("#line \"\"\n")          +   /* constant parts */
+                    2 * strlen (filename)            +   /* filename with possibly all backslashes escaped */
+                    (int) (1 + log10 (abs (lineno))) +   /* line number */
+                    1);                                  /* NUL */
+    if (!t)
+      flexfatal (_("Allocation of buffer for line directive failed"));
+    for (dst = t + sprintf (t, "#line %d \"", lineno), src = filename; *src; *dst++ = *src++)
+      if (*src == '\\')   /* escape backslashes */
+        *dst++ = '\\';
+    *dst++ = '"';
+    *dst++ = '\n';
+    *dst   = '\0';
+    buf = buf_strappend (buf, t);
+    flex_free (t);
+    return buf;
+}
+
+
+/** Append the contents of @a src to @a dest.
+ * @param @a dest the destination buffer
+ * @param @a dest the source buffer
+ * @return @a dest
+ */
+struct Buf *buf_concat(struct Buf* dest, const struct Buf* src)
+{
+    buf_append(dest, src->elts, src->nelts);
+    return dest;
+}
+
+
+/* Appends n characters in str to buf. */
+struct Buf *buf_strnappend (buf, str, n)
+     struct Buf *buf;
+     const char *str;
+     int n;
+{
+	buf_append (buf, str, n + 1);
+
+	/* "undo" the '\0' character that buf_append() already copied. */
+	buf->nelts--;
+
+	return buf;
+}
+
+/* Appends characters in str to buf. */
+struct Buf *buf_strappend (buf, str)
+     struct Buf *buf;
+     const char *str;
+{
+	return buf_strnappend (buf, str, strlen (str));
+}
+
+/* appends "#define str def\n" */
+struct Buf *buf_strdefine (buf, str, def)
+     struct Buf *buf;
+     const char *str;
+     const char *def;
+{
+	buf_strappend (buf, "#define ");
+	buf_strappend (buf, " ");
+	buf_strappend (buf, str);
+	buf_strappend (buf, " ");
+	buf_strappend (buf, def);
+	buf_strappend (buf, "\n");
+	return buf;
+}
+
+/** Pushes "m4_define( [[def]], [[val]])m4_dnl" to end of buffer.
+ * @param buf A buffer as a list of strings.
+ * @param def The m4 symbol to define.
+ * @param val The definition; may be NULL.
+ * @return buf
+ */
+struct Buf *buf_m4_define (struct Buf *buf, const char* def, const char* val)
+{
+    const char * fmt = "m4_define( [[%s]], [[%s]])m4_dnl\n";
+    char * str;
+    size_t strsz;
+
+    val = val?val:"";
+    str = (char*)flex_alloc(strsz = strlen(fmt) + strlen(def) + strlen(val) + 2);
+    if (!str)
+        flexfatal (_("Allocation of buffer for m4 def failed"));
+
+    snprintf(str, strsz, fmt, def, val);
+    buf_append(buf, &str, 1);
+    return buf;
+}
+
+/** Pushes "m4_undefine([[def]])m4_dnl" to end of buffer.
+ * @param buf A buffer as a list of strings.
+ * @param def The m4 symbol to undefine.
+ * @return buf
+ */
+struct Buf *buf_m4_undefine (struct Buf *buf, const char* def)
+{
+    const char * fmt = "m4_undefine( [[%s]])m4_dnl\n";
+    char * str;
+    size_t strsz;
+
+    str = (char*)flex_alloc(strsz = strlen(fmt) + strlen(def) + 2);
+    if (!str)
+        flexfatal (_("Allocation of buffer for m4 undef failed"));
+
+    snprintf(str, strsz, fmt, def);
+    buf_append(buf, &str, 1);
+    return buf;
+}
+
+/* create buf with 0 elements, each of size elem_size. */
+void buf_init (buf, elem_size)
+     struct Buf *buf;
+     size_t elem_size;
+{
+	buf->elts = (void *) 0;
+	buf->nelts = 0;
+	buf->elt_size = elem_size;
+	buf->nmax = 0;
+}
+
+/* frees memory */
+void buf_destroy (buf)
+     struct Buf *buf;
+{
+	if (buf && buf->elts)
+		flex_free (buf->elts);
+	buf->elts = (void *) 0;
+}
+
+
+/* appends ptr[] to buf, grow if necessary.
+ * n_elem is number of elements in ptr[], NOT bytes.
+ * returns buf.
+ * We grow by mod(512) boundaries.
+ */
+
+struct Buf *buf_append (buf, ptr, n_elem)
+     struct Buf *buf;
+     const void *ptr;
+     int n_elem;
+{
+	int     n_alloc = 0;
+
+	if (!ptr || n_elem == 0)
+		return buf;
+
+	/* May need to alloc more. */
+	if (n_elem + buf->nelts > buf->nmax) {
+
+		/* exact amount needed... */
+		n_alloc = (n_elem + buf->nelts) * buf->elt_size;
+
+		/* ...plus some extra */
+		if (((n_alloc * buf->elt_size) % 512) != 0
+		    && buf->elt_size < 512)
+			n_alloc +=
+				(512 -
+				 ((n_alloc * buf->elt_size) % 512)) /
+				buf->elt_size;
+
+		if (!buf->elts)
+			buf->elts =
+				allocate_array (n_alloc, buf->elt_size);
+		else
+			buf->elts =
+				reallocate_array (buf->elts, n_alloc,
+						  buf->elt_size);
+
+		buf->nmax = n_alloc;
+	}
+
+	memcpy ((char *) buf->elts + buf->nelts * buf->elt_size, ptr,
+		n_elem * buf->elt_size);
+	buf->nelts += n_elem;
+
+	return buf;
+}
+
+/* vim:set tabstop=8 softtabstop=4 shiftwidth=4: */
diff --git a/ccl.c b/ccl.c
new file mode 100644
index 0000000..8d66bb6
--- /dev/null
+++ b/ccl.c
@@ -0,0 +1,310 @@
+/* ccl - routines for character classes */
+
+/*  Copyright (c) 1990 The Regents of the University of California. */
+/*  All rights reserved. */
+
+/*  This code is derived from software contributed to Berkeley by */
+/*  Vern Paxson. */
+
+/*  The United States Government has rights in this work pursuant */
+/*  to contract no. DE-AC03-76SF00098 between the United States */
+ /*  Department of Energy and the University of California. */
+
+/*  This file is part of flex. */
+
+/*  Redistribution and use in source and binary forms, with or without */
+/*  modification, are permitted provided that the following conditions */
+/*  are met: */
+
+/*  1. Redistributions of source code must retain the above copyright */
+/*     notice, this list of conditions and the following disclaimer. */
+/*  2. Redistributions in binary form must reproduce the above copyright */
+/*     notice, this list of conditions and the following disclaimer in the */
+/*     documentation and/or other materials provided with the distribution. */
+
+/*  Neither the name of the University nor the names of its contributors */
+/*  may be used to endorse or promote products derived from this software */
+/*  without specific prior written permission. */
+
+/*  THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR */
+/*  IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED */
+/*  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR */
+/*  PURPOSE. */
+
+#include "flexdef.h"
+
+/* return true if the chr is in the ccl. Takes negation into account. */
+static bool
+ccl_contains (const int cclp, const int ch)
+{
+	int     ind, len, i;
+
+	len = ccllen[cclp];
+	ind = cclmap[cclp];
+
+	for (i = 0; i < len; ++i)
+		if (ccltbl[ind + i] == ch)
+			return !cclng[cclp];
+
+    return cclng[cclp];
+}
+
+
+/* ccladd - add a single character to a ccl */
+
+void    ccladd (cclp, ch)
+     int     cclp;
+     int     ch;
+{
+	int     ind, len, newpos, i;
+
+	check_char (ch);
+
+	len = ccllen[cclp];
+	ind = cclmap[cclp];
+
+	/* check to see if the character is already in the ccl */
+
+	for (i = 0; i < len; ++i)
+		if (ccltbl[ind + i] == ch)
+			return;
+
+	/* mark newlines */
+	if (ch == nlch)
+		ccl_has_nl[cclp] = true;
+
+	newpos = ind + len;
+
+	if (newpos >= current_max_ccl_tbl_size) {
+		current_max_ccl_tbl_size += MAX_CCL_TBL_SIZE_INCREMENT;
+
+		++num_reallocs;
+
+		ccltbl = reallocate_Character_array (ccltbl,
+						     current_max_ccl_tbl_size);
+	}
+
+	ccllen[cclp] = len + 1;
+	ccltbl[newpos] = ch;
+}
+
+/* dump_cclp - same thing as list_character_set, but for cclps.  */
+
+static void    dump_cclp (FILE* file, int cclp)
+{
+	register int i;
+
+	putc ('[', file);
+
+	for (i = 0; i < csize; ++i) {
+		if (ccl_contains(cclp, i)){
+			register int start_char = i;
+
+			putc (' ', file);
+
+			fputs (readable_form (i), file);
+
+			while (++i < csize && ccl_contains(cclp,i)) ;
+
+			if (i - 1 > start_char)
+				/* this was a run */
+				fprintf (file, "-%s",
+					 readable_form (i - 1));
+
+			putc (' ', file);
+		}
+	}
+
+	putc (']', file);
+}
+
+
+
+/* ccl_set_diff - create a new ccl as the set difference of the two given ccls. */
+int
+ccl_set_diff (int a, int b)
+{
+    int  d, ch;
+
+    /* create new class  */
+    d = cclinit();
+
+    /* In order to handle negation, we spin through all possible chars,
+     * addding each char in a that is not in b.
+     * (This could be O(n^2), but n is small and bounded.)
+     */
+	for ( ch = 0; ch < csize; ++ch )
+        if (ccl_contains (a, ch) && !ccl_contains(b, ch))
+            ccladd (d, ch);
+
+    /* debug */
+    if (0){
+        fprintf(stderr, "ccl_set_diff (");
+            fprintf(stderr, "\n    ");
+            dump_cclp (stderr, a);
+            fprintf(stderr, "\n    ");
+            dump_cclp (stderr, b);
+            fprintf(stderr, "\n    ");
+            dump_cclp (stderr, d);
+        fprintf(stderr, "\n)\n");
+    }
+    return d;
+}
+
+/* ccl_set_union - create a new ccl as the set union of the two given ccls. */
+int
+ccl_set_union (int a, int b)
+{
+    int  d, i;
+
+    /* create new class  */
+    d = cclinit();
+
+    /* Add all of a */
+    for (i = 0; i < ccllen[a]; ++i)
+		ccladd (d, ccltbl[cclmap[a] + i]);
+
+    /* Add all of b */
+    for (i = 0; i < ccllen[b]; ++i)
+		ccladd (d, ccltbl[cclmap[b] + i]);
+
+    /* debug */
+    if (0){
+        fprintf(stderr, "ccl_set_union (%d + %d = %d", a, b, d);
+            fprintf(stderr, "\n    ");
+            dump_cclp (stderr, a);
+            fprintf(stderr, "\n    ");
+            dump_cclp (stderr, b);
+            fprintf(stderr, "\n    ");
+            dump_cclp (stderr, d);
+        fprintf(stderr, "\n)\n");
+    }
+    return d;
+}
+
+
+/* cclinit - return an empty ccl */
+
+int     cclinit ()
+{
+	if (++lastccl >= current_maxccls) {
+		current_maxccls += MAX_CCLS_INCREMENT;
+
+		++num_reallocs;
+
+		cclmap =
+			reallocate_integer_array (cclmap, current_maxccls);
+		ccllen =
+			reallocate_integer_array (ccllen, current_maxccls);
+		cclng = reallocate_integer_array (cclng, current_maxccls);
+		ccl_has_nl =
+			reallocate_bool_array (ccl_has_nl,
+					       current_maxccls);
+	}
+
+	if (lastccl == 1)
+		/* we're making the first ccl */
+		cclmap[lastccl] = 0;
+
+	else
+		/* The new pointer is just past the end of the last ccl.
+		 * Since the cclmap points to the \first/ character of a
+		 * ccl, adding the length of the ccl to the cclmap pointer
+		 * will produce a cursor to the first free space.
+		 */
+		cclmap[lastccl] =
+			cclmap[lastccl - 1] + ccllen[lastccl - 1];
+
+	ccllen[lastccl] = 0;
+	cclng[lastccl] = 0;	/* ccl's start out life un-negated */
+	ccl_has_nl[lastccl] = false;
+
+	return lastccl;
+}
+
+
+/* cclnegate - negate the given ccl */
+
+void    cclnegate (cclp)
+     int     cclp;
+{
+	cclng[cclp] = 1;
+	ccl_has_nl[cclp] = !ccl_has_nl[cclp];
+}
+
+
+/* list_character_set - list the members of a set of characters in CCL form
+ *
+ * Writes to the given file a character-class representation of those
+ * characters present in the given CCL.  A character is present if it
+ * has a non-zero value in the cset array.
+ */
+
+void    list_character_set (file, cset)
+     FILE   *file;
+     int     cset[];
+{
+	register int i;
+
+	putc ('[', file);
+
+	for (i = 0; i < csize; ++i) {
+		if (cset[i]) {
+			register int start_char = i;
+
+			putc (' ', file);
+
+			fputs (readable_form (i), file);
+
+			while (++i < csize && cset[i]) ;
+
+			if (i - 1 > start_char)
+				/* this was a run */
+				fprintf (file, "-%s",
+					 readable_form (i - 1));
+
+			putc (' ', file);
+		}
+	}
+
+	putc (']', file);
+}
+
+/** Determines if the range [c1-c2] is unambiguous in a case-insensitive
+ * scanner.  Specifically, if a lowercase or uppercase character, x, is in the
+ * range [c1-c2], then we require that UPPERCASE(x) and LOWERCASE(x) must also
+ * be in the range. If not, then this range is ambiguous, and the function
+ * returns false.  For example, [@-_] spans [a-z] but not [A-Z].  Beware that
+ * [a-z] will be labeled ambiguous because it does not include [A-Z].
+ *
+ * @param c1 the lower end of the range
+ * @param c2 the upper end of the range
+ * @return true if [c1-c2] is not ambiguous for a caseless scanner.
+ */
+bool range_covers_case (int c1, int c2)
+{
+	int     i, o;
+
+	for (i = c1; i <= c2; i++) {
+		if (has_case (i)) {
+			o = reverse_case (i);
+			if (o < c1 || c2 < o)
+				return false;
+		}
+	}
+	return true;
+}
+
+/** Reverse the case of a character, if possible.
+ * @return c if case-reversal does not apply.
+ */
+int reverse_case (int c)
+{
+	return isupper (c) ? tolower (c) : (islower (c) ? toupper (c) : c);
+}
+
+/** Return true if c is uppercase or lowercase. */
+bool has_case (int c)
+{
+	return (isupper (c) || islower (c)) ? true : false;
+}
diff --git a/configure.ac b/configure.ac
new file mode 100644
index 0000000..2ef26a5
--- /dev/null
+++ b/configure.ac
@@ -0,0 +1,173 @@
+#                                               -*- Autoconf -*-
+# Process this file with autoconf to produce a configure script.
+
+# This file is part of flex.
+
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+
+# Neither the name of the University nor the names of its contributors
+# may be used to endorse or promote products derived from this software
+# without specific prior written permission.
+
+# THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE.
+
+# autoconf requirements and initialization
+
+AC_INIT([the fast lexical analyser generator], [2.5.38],
+[flex-help@lists.sourceforge.net], [flex])
+SHARED_VERSION_INFO="2:0:0"
+AC_SUBST(SHARED_VERSION_INFO)
+
+AC_CONFIG_SRCDIR([scan.l])
+AM_INIT_AUTOMAKE([gnu check-news std-options dist-bzip2 dist-xz 1.10])
+AC_CONFIG_HEADER([config.h:conf.in])
+AC_CONFIG_LIBOBJ_DIR([lib])
+
+# checks for programs
+
+AM_GNU_GETTEXT([external])
+AM_GNU_GETTEXT_VERSION([0.18.1])
+AC_PROG_YACC
+AM_PROG_LEX
+AC_PROG_CC
+AC_PROG_CXX
+AM_PROG_CC_C_O
+AC_PROG_LN_S
+LT_INIT
+AC_PROG_AWK
+AC_PROG_INSTALL
+
+AC_PATH_PROG(BISON, bison,bison)
+AC_PATH_PROG(HELP2MAN, help2man, help2man)
+
+# Check for a m4 that supports -P
+
+AC_CACHE_CHECK([for m4 that supports -P], [ac_cv_path_M4],
+[AC_PATH_PROGS_FEATURE_CHECK([M4], [gm4 gnum4 m4],
+              [[m4out=`echo 'm''4_divnum' | $ac_path_M4 -P`]
+                [test "x$m4out" = x0 \
+                && ac_cv_path_M4=$ac_path_M4 ac_path_M4_found=:]],
+              [AC_MSG_ERROR([could not find m4 that supports -P])])])
+AC_SUBST([M4], [$ac_cv_path_M4])
+AC_DEFINE_UNQUOTED([M4], ["$M4"], [Define to the m4 executable name.])
+
+AC_PATH_PROG(INDENT, indent, indent)
+# if INDENT is set to 'indent' then we didn't find indent
+if test "$INDENT" != indent ; then
+   AC_MSG_CHECKING(if $INDENT is GNU indent)
+   if $INDENT --version 2>/dev/null | head -n 1|grep "GNU indent" > /dev/null ; then
+      AC_MSG_RESULT(yes)
+   else
+      AC_MSG_RESULT(no)
+      AC_MSG_WARN($INDENT does not appear to be GNU indent.)
+   fi
+else
+   AC_MSG_WARN(no indent program found: make indent target will not function)
+fi
+
+# checks for headers
+
+AC_HEADER_STDC
+AC_HEADER_SYS_WAIT
+AC_CHECK_HEADERS([inttypes.h libintl.h limits.h locale.h malloc.h netinet/in.h regex.h stddef.h stdlib.h string.h strings.h unistd.h])
+
+# checks for libraries
+
+# The test test-pthread uses libpthread, so we check for it here, but
+# all we need is the preprocessor symbol defined since we don't need
+# LIBS to include libpthread for building flex.
+
+AC_CHECK_LIB(pthread, pthread_mutex_lock,
+AC_DEFINE([HAVE_LIBPTHREAD], 1, [pthread library] ),
+AC_DEFINE([HAVE_LIBPTHREAD], 0, [pthread library] )
+)
+AC_CHECK_HEADERS([pthread.h])
+
+AC_CHECK_LIB(m, log10)
+
+# Checks for typedefs, structures, and compiler characteristics.
+
+AC_HEADER_STDBOOL
+AC_C_CONST
+AC_TYPE_SIZE_T
+
+# Checks for library functions.
+
+AC_FUNC_ALLOCA
+AC_FUNC_FORK
+AC_FUNC_MALLOC
+AC_FUNC_REALLOC
+AC_CHECK_FUNCS([dup2 isascii memset pow regcomp setlocale strchr strtol])
+
+AC_CONFIG_FILES(
+Makefile
+doc/Makefile
+examples/Makefile
+examples/fastwc/Makefile
+examples/manual/Makefile
+lib/Makefile
+po/Makefile.in
+tests/Makefile
+tests/TEMPLATE/Makefile
+tests/test-array-nr/Makefile
+tests/test-array-r/Makefile
+tests/test-basic-nr/Makefile
+tests/test-basic-r/Makefile
+tests/test-bison-yylloc/Makefile
+tests/test-bison-yylval/Makefile
+tests/test-c-cpp-nr/Makefile
+tests/test-c-cpp-r/Makefile
+tests/test-header-nr/Makefile
+tests/test-header-r/Makefile
+tests/test-include-by-buffer/Makefile
+tests/test-include-by-push/Makefile
+tests/test-include-by-reentrant/Makefile
+tests/test-multiple-scanners-nr/Makefile
+tests/test-multiple-scanners-r/Makefile
+tests/test-noansi-nr/Makefile
+tests/test-noansi-r/Makefile
+tests/test-prefix-nr/Makefile
+tests/test-prefix-r/Makefile
+tests/test-pthread/Makefile
+tests/test-string-nr/Makefile
+tests/test-string-r/Makefile
+tests/test-yyextra/Makefile
+tests/test-alloc-extra/Makefile
+tests/test-lineno-nr/Makefile
+tests/test-lineno-trailing/Makefile
+tests/test-lineno-r/Makefile
+tests/test-linedir-r/Makefile
+tests/test-debug-r/Makefile
+tests/test-debug-nr/Makefile
+tests/test-mem-nr/Makefile
+tests/test-mem-r/Makefile
+tests/test-posix/Makefile
+tests/test-posixly-correct/Makefile
+tests/test-table-opts/Makefile
+tests/test-c++-basic/Makefile
+tests/test-bison-nr/Makefile
+tests/test-reject/Makefile
+tests/test-c++-multiple-scanners/Makefile
+tests/test-top/Makefile
+tests/test-rescan-nr/Makefile
+tests/test-rescan-r/Makefile
+tests/test-quotes/Makefile
+tests/test-ccl/Makefile
+tests/test-extended/Makefile
+tests/test-c++-yywrap/Makefile
+tests/test-concatenated-options/Makefile
+dnl --new-test-here-- This line is processed by tests/create-test.
+)
+
+AC_OUTPUT
diff --git a/devel/00EXTRACT-ALL-SYMS.sh b/devel/00EXTRACT-ALL-SYMS.sh
new file mode 100644
index 0000000..0f7d4ae
--- /dev/null
+++ b/devel/00EXTRACT-ALL-SYMS.sh
@@ -0,0 +1,57 @@
+# extract all table types from various scanners. We try to get the ones
+# that are declared outside '{'.
+# hopefully we'll hit them all.
+OUTFILE=/tmp/EXTRACT-FLEX-SYMS.$$
+echo > $OUTFILE
+
+for s in  \
+	yy_trans_info \
+	yy_NUL_trans \
+	yy_accept \
+	yy_base \
+	yy_chk \
+	yy_def \
+	yy_ec \
+	yy_meta \
+	yy_nxt \
+	yy_rule_can_match_eol \
+	yy_rule_linenum \
+	yy_start_state_list \
+	yy_state_type \
+	yy_transition
+do
+	perl -ne \
+	'
+		BEGIN{
+		   	$s = qq('$s');
+		}
+		s/\b(short|long)\s+int\b/$1/;
+		s/\b(static|(yy)?const)\b\s+//g;
+		next unless m/((?:struct\s*)?\w+)\W+$s\s*((?:\[\s*\d*\s*\])+)\s*=/;
+		$type=$1;
+		$arr = $2;
+		$arr =~ s/\d+//g;
+		$a{$s}->{$type . $arr}= $_;
+		
+		END{
+			for(sort keys %a){
+				print values %{$a{$_}}
+			}
+		}
+	' \
+	tests/test-*/*.c  >> $OUTFILE
+done
+
+grep -E 'typedef.*yy_state_type'  tests/test-*/*.c | 
+		gawk -F ':' '{print $2}' |
+		sort |
+		uniq >> $OUTFILE
+
+pcregrep  '(\w+)[\s*]*yy_(verify|nxt)\s*;'  tests/test-*/*.c |
+		perl -pe 's/^.*?\.c?[\-\:]//;' |
+		sort |
+		uniq >> $OUTFILE
+
+cat $OUTFILE
+rm -f $OUTFILE
+
diff --git a/devel/README b/devel/README
new file mode 100644
index 0000000..01bca86
--- /dev/null
+++ b/devel/README
@@ -0,0 +1,2 @@
+This directory contains flex developer junk.
+It should not be considered part of the official distribution.
diff --git a/devel/dump-tables.pl b/devel/dump-tables.pl
new file mode 100644
index 0000000..1515e18
--- /dev/null
+++ b/devel/dump-tables.pl
@@ -0,0 +1,22 @@
+#! /usr/bin/perl -w
+# vim:set ft=perl ai si et ts=4 sts=4 sw=4 tw=0:
+# USAGE: dump-tables.pl  FILE [max-data]
+use strict;
+
+my $dir = $0;
+$dir =~ s:/[^/]+$::;
+push @INC, $dir;
+require "tables.pl";
+
+@ARGV == 1 || @ARGV == 2 || die "\nUSAGE: dump-tables.pl FILE [max-data]\n";
+open FD, $ARGV[0] or die "$ARGV[0]: $!";
+
+dump_hdr(read_hdr());
+while(!eof FD){
+   dump_table(read_table(),$ARGV[1]);
+}
+close FD;
+
+1
+__END__
+
diff --git a/devel/tables.pl b/devel/tables.pl
new file mode 100644
index 0000000..6e7d507
--- /dev/null
+++ b/devel/tables.pl
@@ -0,0 +1,165 @@
+#! /usr/bin/perl -w
+# vim:set ft=perl ai si et ts=4 sts=4 sw=4 tw=0:
+# THIS FILE IS A LIBRARY, AND MEANT TO BE 'required' BY OTHER SCRIPTS
+use strict;
+use Class::Struct yytbl_hdr => [ 
+                    th_magic => '$', th_magic_offset => '$',
+                    th_hsize => '$', th_hsize_offset => '$',
+                    th_ssize => '$', th_ssize_offset => '$',
+                    th_flags => '$', th_flags_offset => '$',
+                    th_version => '$', th_version_offset => '$',
+                    th_name => '$', th_name_offset => '$'
+                 ];
+use Class::Struct yytbl_data => [
+                    td_id => '$', td_id_offset => '$',
+                    td_flags => '$', td_flags_offset => '$',
+                    td_hilen => '$', td_hilen_offset => '$',
+                    td_lolen => '$', td_lolen_offset => '$',
+                    td_data => '@', td_data_offset => '$'
+                  ];
+
+sub read_hdr {
+    my $th = new yytbl_hdr;
+    $th->th_magic_offset(tell FD);
+    $th->th_magic(read32());
+    $th->th_hsize_offset(tell FD);
+    $th->th_hsize(read32());
+    $th->th_ssize_offset(tell FD);
+    my @a = unpack 'Nna*',readn($th->th_hsize - 8);
+    $th->th_ssize($a[0]);
+    $th->th_flags_offset($th->th_ssize_offset + 4);
+    $th->th_flags($a[1]);
+    $th->th_version_offset($th->th_flags_offset + 2);
+    $th->th_version($a[2]);
+    $th->th_name((split /\0/, $th->th_version)[1]);
+    $th->th_version((split /\0/, $th->th_version)[0]);
+    $th->th_name_offset($th->th_version_offset + length($th->th_version) + 1);
+
+    return $th;
+}
+sub dump_hdr {
+    my $th = shift;
+    printf "%04X th_magic:   %08X\n",$th->th_magic_offset, $th->th_magic;
+    printf "%04X th_hsize:   \%d\n",$th->th_hsize_offset, $th->th_hsize;
+    printf "%04X th_ssize:   \%d\n",$th->th_ssize_offset, $th->th_ssize;
+    printf "%04X th_flags:   \%d\n",$th->th_flags_offset, $th->th_flags;
+    printf "%04X th_version: \%s\n",$th->th_version_offset, $th->th_version;
+    printf "%04X th_name:    \%s\n",$th->th_name_offset, $th->th_name;
+}
+
+
+sub readn { my $s; read FD, $s, shift; return $s }
+sub read32 { my $n; read FD, $n, 4; return unpack 'i', pack 'i', unpack 'N',$n }
+sub read16 { my $n; read FD, $n, 2; return unpack 's', pack 's', unpack 'n',$n }
+sub read8 { my $n; read FD, $n, 1;  return unpack 'c',$n }
+sub readstruct {
+    my $td=shift;
+    my $r = $::TFLAGS{$td->td_flags & 0x07}->[1];
+    return ($r->(), $r->());
+}
+sub getreader {
+    my $td = shift;
+    return \&readstruct if ($td->td_flags & 0x10);
+    return  $::TFLAGS{$td->td_flags & 0x07}->[1];
+}
+
+sub read_table {
+    my $td = new yytbl_data;
+    printf "HERE: %04X\n", tell FD;
+    $td->td_id_offset(tell FD);
+    $td->td_id(read16());
+    $td->td_flags_offset(tell FD);
+    $td->td_flags(read16());
+    $td->td_hilen_offset(tell FD);
+    $td->td_hilen(read32());
+    $td->td_lolen_offset(tell FD);
+    $td->td_lolen(read32());
+    $td->td_data_offset(tell FD);
+
+    my $read = getreader($td);
+    
+    my $tot = $td->td_lolen * ($td->td_hilen?$td->td_hilen:1);
+    $tot *=2 if $::TID{$td->td_id}eq 'YYTH_ID_TRANSITION';
+
+    for(1..$tot){
+        my @v = $read->($td);
+        push @{$td->td_data}, @v;
+    }
+    my $pad = pad64(tell FD);
+    while($pad-- > 0){ read8() }
+    return $td;
+}
+
+sub dump_table {
+    my $td = shift;
+    my $max = shift;
+    defined($max) or $max = 2**15;
+    printf "%04X td_id:    \%d (\%s)\n", $td->td_id_offset, $td->td_id, $::TID{$td->td_id};
+    printf "%04X td_flags: \%0x (\%s)\n",$td->td_flags_offset, $td->td_flags, "@{[TFLAGS($td->td_flags)]}";
+    printf "%04X td_hilen: \%d\n", $td->td_hilen_offset, $td->td_hilen;
+    printf "%04X td_lolen: \%d\n", $td->td_lolen_offset, $td->td_lolen;
+    
+    printf "%04X td_data: ", $td->td_data_offset;
+    foreach( @{$td->td_data} ){
+        last if $max-- == 0;
+        print "$_, ";
+    }
+    print "\n";
+    return $td;
+}
+
+
+sub TFLAGS {
+    my @s;
+    my $f = shift;
+    foreach(keys %::TFLAGS){
+        if ($f & $_){
+            push @s, $::TFLAGS{$_}->[0]
+        }
+    }
+    return join '|', @s;
+}
+
+sub pad64{ return ((8-((shift)%8))%8) }
+
+BEGIN {
+    %::TID = ( 
+	0x01 => 'YYTD_ID_ACCEPT' ,
+	0x02 => 'YYTD_ID_BASE' ,
+	0x03 => 'YYTD_ID_CHK' ,
+	0x04 => 'YYTD_ID_DEF' ,
+	0x05 => 'YYTD_ID_EC' ,
+	0x06 => 'YYTD_ID_META' ,
+	0x07 => 'YYTD_ID_NUL_TRANS' ,
+	0x08 => 'YYTD_ID_NXT' ,
+	0x09 => 'YYTD_ID_RULE_CAN_MATCH_EOL' ,
+	0x0A => 'YYTD_ID_START_STATE_LIST' ,
+	0x0B => 'YYTD_ID_TRANSITION',
+	0x0C => 'YYTD_ID_ACCLIST');
+
+    %::TFLAGS = (
+	0x01 => ['YYTD_DATA8',\&read8] ,
+	0x02 => ['YYTD_DATA16',\&read16] ,
+	0x04 => ['YYTD_DATA32',\&read32] ,
+	0x08 => ['YYTD_PTRANS',undef],
+	0x10 => ['YYTD_STRUCT',\&readstruct]);
+}
+
+# CODE TO EXTRACT TABLES FROM GENERATED C CODE
+sub extract_from_code {
+    my $nms = qr/yy_(?:accept|base|chk|
+                 def|ec|meta|NUL_trans|
+                 nxt|rule_can_match_eol|
+                 start_state_list|transition)/x;
+
+    while(<FD>){
+        next unless m/static\s+yyconst\s+.+yy_($nms)(\[\])?\[(\d+)\]\s*=/;
+        my $n = $1;
+        my $dim = defined($2)?2:1;
+        # TODO 
+    }
+}
+
+1
+__END__
+
diff --git a/dfa.c b/dfa.c
new file mode 100644
index 0000000..b8b68eb
--- /dev/null
+++ b/dfa.c
@@ -0,0 +1,1096 @@
+/* dfa - DFA construction routines */
+
+/*  Copyright (c) 1990 The Regents of the University of California. */
+/*  All rights reserved. */
+
+/*  This code is derived from software contributed to Berkeley by */
+/*  Vern Paxson. */
+
+/*  The United States Government has rights in this work pursuant */
+/*  to contract no. DE-AC03-76SF00098 between the United States */
+/*  Department of Energy and the University of California. */
+
+/*  Redistribution and use in source and binary forms, with or without */
+/*  modification, are permitted provided that the following conditions */
+/*  are met: */
+
+/*  1. Redistributions of source code must retain the above copyright */
+/*     notice, this list of conditions and the following disclaimer. */
+/*  2. Redistributions in binary form must reproduce the above copyright */
+/*     notice, this list of conditions and the following disclaimer in the */
+/*     documentation and/or other materials provided with the distribution. */
+
+/*  Neither the name of the University nor the names of its contributors */
+/*  may be used to endorse or promote products derived from this software */
+/*  without specific prior written permission. */
+
+/*  THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR */
+/*  IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED */
+/*  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR */
+/*  PURPOSE. */
+
+#include "flexdef.h"
+#include "tables.h"
+
+/* declare functions that have forward references */
+
+void dump_associated_rules PROTO ((FILE *, int));
+void dump_transitions PROTO ((FILE *, int[]));
+void sympartition PROTO ((int[], int, int[], int[]));
+int symfollowset PROTO ((int[], int, int, int[]));
+
+
+/* check_for_backing_up - check a DFA state for backing up
+ *
+ * synopsis
+ *     void check_for_backing_up( int ds, int state[numecs] );
+ *
+ * ds is the number of the state to check and state[] is its out-transitions,
+ * indexed by equivalence class.
+ */
+
+void check_for_backing_up (ds, state)
+     int ds;
+     int state[];
+{
+	if ((reject && !dfaacc[ds].dfaacc_set) || (!reject && !dfaacc[ds].dfaacc_state)) {	/* state is non-accepting */
+		++num_backing_up;
+
+		if (backing_up_report) {
+			fprintf (backing_up_file,
+				 _("State #%d is non-accepting -\n"), ds);
+
+			/* identify the state */
+			dump_associated_rules (backing_up_file, ds);
+
+			/* Now identify it further using the out- and
+			 * jam-transitions.
+			 */
+			dump_transitions (backing_up_file, state);
+
+			putc ('\n', backing_up_file);
+		}
+	}
+}
+
+
+/* check_trailing_context - check to see if NFA state set constitutes
+ *                          "dangerous" trailing context
+ *
+ * synopsis
+ *    void check_trailing_context( int nfa_states[num_states+1], int num_states,
+ *				int accset[nacc+1], int nacc );
+ *
+ * NOTES
+ *  Trailing context is "dangerous" if both the head and the trailing
+ *  part are of variable size \and/ there's a DFA state which contains
+ *  both an accepting state for the head part of the rule and NFA states
+ *  which occur after the beginning of the trailing context.
+ *
+ *  When such a rule is matched, it's impossible to tell if having been
+ *  in the DFA state indicates the beginning of the trailing context or
+ *  further-along scanning of the pattern.  In these cases, a warning
+ *  message is issued.
+ *
+ *    nfa_states[1 .. num_states] is the list of NFA states in the DFA.
+ *    accset[1 .. nacc] is the list of accepting numbers for the DFA state.
+ */
+
+void check_trailing_context (nfa_states, num_states, accset, nacc)
+     int    *nfa_states, num_states;
+     int    *accset;
+     int nacc;
+{
+	register int i, j;
+
+	for (i = 1; i <= num_states; ++i) {
+		int     ns = nfa_states[i];
+		register int type = state_type[ns];
+		register int ar = assoc_rule[ns];
+
+		if (type == STATE_NORMAL || rule_type[ar] != RULE_VARIABLE) {	/* do nothing */
+		}
+
+		else if (type == STATE_TRAILING_CONTEXT) {
+			/* Potential trouble.  Scan set of accepting numbers
+			 * for the one marking the end of the "head".  We
+			 * assume that this looping will be fairly cheap
+			 * since it's rare that an accepting number set
+			 * is large.
+			 */
+			for (j = 1; j <= nacc; ++j)
+				if (accset[j] & YY_TRAILING_HEAD_MASK) {
+					line_warning (_
+						      ("dangerous trailing context"),
+						      rule_linenum[ar]);
+					return;
+				}
+		}
+	}
+}
+
+
+/* dump_associated_rules - list the rules associated with a DFA state
+ *
+ * Goes through the set of NFA states associated with the DFA and
+ * extracts the first MAX_ASSOC_RULES unique rules, sorts them,
+ * and writes a report to the given file.
+ */
+
+void dump_associated_rules (file, ds)
+     FILE   *file;
+     int ds;
+{
+	register int i, j;
+	register int num_associated_rules = 0;
+	int     rule_set[MAX_ASSOC_RULES + 1];
+	int    *dset = dss[ds];
+	int     size = dfasiz[ds];
+
+	for (i = 1; i <= size; ++i) {
+		register int rule_num = rule_linenum[assoc_rule[dset[i]]];
+
+		for (j = 1; j <= num_associated_rules; ++j)
+			if (rule_num == rule_set[j])
+				break;
+
+		if (j > num_associated_rules) {	/* new rule */
+			if (num_associated_rules < MAX_ASSOC_RULES)
+				rule_set[++num_associated_rules] =
+					rule_num;
+		}
+	}
+
+	qsort (&rule_set [1], num_associated_rules, sizeof (rule_set [1]), intcmp);
+
+	fprintf (file, _(" associated rule line numbers:"));
+
+	for (i = 1; i <= num_associated_rules; ++i) {
+		if (i % 8 == 1)
+			putc ('\n', file);
+
+		fprintf (file, "\t%d", rule_set[i]);
+	}
+
+	putc ('\n', file);
+}
+
+
+/* dump_transitions - list the transitions associated with a DFA state
+ *
+ * synopsis
+ *     dump_transitions( FILE *file, int state[numecs] );
+ *
+ * Goes through the set of out-transitions and lists them in human-readable
+ * form (i.e., not as equivalence classes); also lists jam transitions
+ * (i.e., all those which are not out-transitions, plus EOF).  The dump
+ * is done to the given file.
+ */
+
+void dump_transitions (file, state)
+     FILE   *file;
+     int state[];
+{
+	register int i, ec;
+	int     out_char_set[CSIZE];
+
+	for (i = 0; i < csize; ++i) {
+		ec = ABS (ecgroup[i]);
+		out_char_set[i] = state[ec];
+	}
+
+	fprintf (file, _(" out-transitions: "));
+
+	list_character_set (file, out_char_set);
+
+	/* now invert the members of the set to get the jam transitions */
+	for (i = 0; i < csize; ++i)
+		out_char_set[i] = !out_char_set[i];
+
+	fprintf (file, _("\n jam-transitions: EOF "));
+
+	list_character_set (file, out_char_set);
+
+	putc ('\n', file);
+}
+
+
+/* epsclosure - construct the epsilon closure of a set of ndfa states
+ *
+ * synopsis
+ *    int *epsclosure( int t[num_states], int *numstates_addr,
+ *			int accset[num_rules+1], int *nacc_addr,
+ *			int *hashval_addr );
+ *
+ * NOTES
+ *  The epsilon closure is the set of all states reachable by an arbitrary
+ *  number of epsilon transitions, which themselves do not have epsilon
+ *  transitions going out, unioned with the set of states which have non-null
+ *  accepting numbers.  t is an array of size numstates of nfa state numbers.
+ *  Upon return, t holds the epsilon closure and *numstates_addr is updated.
+ *  accset holds a list of the accepting numbers, and the size of accset is
+ *  given by *nacc_addr.  t may be subjected to reallocation if it is not
+ *  large enough to hold the epsilon closure.
+ *
+ *  hashval is the hash value for the dfa corresponding to the state set.
+ */
+
+int    *epsclosure (t, ns_addr, accset, nacc_addr, hv_addr)
+     int    *t, *ns_addr, accset[], *nacc_addr, *hv_addr;
+{
+	register int stkpos, ns, tsp;
+	int     numstates = *ns_addr, nacc, hashval, transsym, nfaccnum;
+	int     stkend, nstate;
+	static int did_stk_init = false, *stk;
+
+#define MARK_STATE(state) \
+do{ trans1[state] = trans1[state] - MARKER_DIFFERENCE;} while(0)
+
+#define IS_MARKED(state) (trans1[state] < 0)
+
+#define UNMARK_STATE(state) \
+do{ trans1[state] = trans1[state] + MARKER_DIFFERENCE;} while(0)
+
+#define CHECK_ACCEPT(state) \
+do{ \
+nfaccnum = accptnum[state]; \
+if ( nfaccnum != NIL ) \
+accset[++nacc] = nfaccnum; \
+}while(0)
+
+#define DO_REALLOCATION() \
+do { \
+current_max_dfa_size += MAX_DFA_SIZE_INCREMENT; \
+++num_reallocs; \
+t = reallocate_integer_array( t, current_max_dfa_size ); \
+stk = reallocate_integer_array( stk, current_max_dfa_size ); \
+}while(0) \
+
+#define PUT_ON_STACK(state) \
+do { \
+if ( ++stkend >= current_max_dfa_size ) \
+DO_REALLOCATION(); \
+stk[stkend] = state; \
+MARK_STATE(state); \
+}while(0)
+
+#define ADD_STATE(state) \
+do { \
+if ( ++numstates >= current_max_dfa_size ) \
+DO_REALLOCATION(); \
+t[numstates] = state; \
+hashval += state; \
+}while(0)
+
+#define STACK_STATE(state) \
+do { \
+PUT_ON_STACK(state); \
+CHECK_ACCEPT(state); \
+if ( nfaccnum != NIL || transchar[state] != SYM_EPSILON ) \
+ADD_STATE(state); \
+}while(0)
+
+
+	if (!did_stk_init) {
+		stk = allocate_integer_array (current_max_dfa_size);
+		did_stk_init = true;
+	}
+
+	nacc = stkend = hashval = 0;
+
+	for (nstate = 1; nstate <= numstates; ++nstate) {
+		ns = t[nstate];
+
+		/* The state could be marked if we've already pushed it onto
+		 * the stack.
+		 */
+		if (!IS_MARKED (ns)) {
+			PUT_ON_STACK (ns);
+			CHECK_ACCEPT (ns);
+			hashval += ns;
+		}
+	}
+
+	for (stkpos = 1; stkpos <= stkend; ++stkpos) {
+		ns = stk[stkpos];
+		transsym = transchar[ns];
+
+		if (transsym == SYM_EPSILON) {
+			tsp = trans1[ns] + MARKER_DIFFERENCE;
+
+			if (tsp != NO_TRANSITION) {
+				if (!IS_MARKED (tsp))
+					STACK_STATE (tsp);
+
+				tsp = trans2[ns];
+
+				if (tsp != NO_TRANSITION
+				    && !IS_MARKED (tsp))
+					STACK_STATE (tsp);
+			}
+		}
+	}
+
+	/* Clear out "visit" markers. */
+
+	for (stkpos = 1; stkpos <= stkend; ++stkpos) {
+		if (IS_MARKED (stk[stkpos]))
+			UNMARK_STATE (stk[stkpos]);
+		else
+			flexfatal (_
+				   ("consistency check failed in epsclosure()"));
+	}
+
+	*ns_addr = numstates;
+	*hv_addr = hashval;
+	*nacc_addr = nacc;
+
+	return t;
+}
+
+
+/* increase_max_dfas - increase the maximum number of DFAs */
+
+void increase_max_dfas ()
+{
+	current_max_dfas += MAX_DFAS_INCREMENT;
+
+	++num_reallocs;
+
+	base = reallocate_integer_array (base, current_max_dfas);
+	def = reallocate_integer_array (def, current_max_dfas);
+	dfasiz = reallocate_integer_array (dfasiz, current_max_dfas);
+	accsiz = reallocate_integer_array (accsiz, current_max_dfas);
+	dhash = reallocate_integer_array (dhash, current_max_dfas);
+	dss = reallocate_int_ptr_array (dss, current_max_dfas);
+	dfaacc = reallocate_dfaacc_union (dfaacc, current_max_dfas);
+
+	if (nultrans)
+		nultrans =
+			reallocate_integer_array (nultrans,
+						  current_max_dfas);
+}
+
+
+/* ntod - convert an ndfa to a dfa
+ *
+ * Creates the dfa corresponding to the ndfa we've constructed.  The
+ * dfa starts out in state #1.
+ */
+
+void ntod ()
+{
+	int    *accset, ds, nacc, newds;
+	int     sym, hashval, numstates, dsize;
+	int     num_full_table_rows=0;	/* used only for -f */
+	int    *nset, *dset;
+	int     targptr, totaltrans, i, comstate, comfreq, targ;
+	int     symlist[CSIZE + 1];
+	int     num_start_states;
+	int     todo_head, todo_next;
+
+	struct yytbl_data *yynxt_tbl = 0;
+	flex_int32_t *yynxt_data = 0, yynxt_curr = 0;
+
+	/* Note that the following are indexed by *equivalence classes*
+	 * and not by characters.  Since equivalence classes are indexed
+	 * beginning with 1, even if the scanner accepts NUL's, this
+	 * means that (since every character is potentially in its own
+	 * equivalence class) these arrays must have room for indices
+	 * from 1 to CSIZE, so their size must be CSIZE + 1.
+	 */
+	int     duplist[CSIZE + 1], state[CSIZE + 1];
+	int     targfreq[CSIZE + 1], targstate[CSIZE + 1];
+
+	/* accset needs to be large enough to hold all of the rules present
+	 * in the input, *plus* their YY_TRAILING_HEAD_MASK variants.
+	 */
+	accset = allocate_integer_array ((num_rules + 1) * 2);
+	nset = allocate_integer_array (current_max_dfa_size);
+
+	/* The "todo" queue is represented by the head, which is the DFA
+	 * state currently being processed, and the "next", which is the
+	 * next DFA state number available (not in use).  We depend on the
+	 * fact that snstods() returns DFA's \in increasing order/, and thus
+	 * need only know the bounds of the dfas to be processed.
+	 */
+	todo_head = todo_next = 0;
+
+	for (i = 0; i <= csize; ++i) {
+		duplist[i] = NIL;
+		symlist[i] = false;
+	}
+
+	for (i = 0; i <= num_rules; ++i)
+		accset[i] = NIL;
+
+	if (trace) {
+		dumpnfa (scset[1]);
+		fputs (_("\n\nDFA Dump:\n\n"), stderr);
+	}
+
+	inittbl ();
+
+	/* Check to see whether we should build a separate table for
+	 * transitions on NUL characters.  We don't do this for full-speed
+	 * (-F) scanners, since for them we don't have a simple state
+	 * number lying around with which to index the table.  We also
+	 * don't bother doing it for scanners unless (1) NUL is in its own
+	 * equivalence class (indicated by a positive value of
+	 * ecgroup[NUL]), (2) NUL's equivalence class is the last
+	 * equivalence class, and (3) the number of equivalence classes is
+	 * the same as the number of characters.  This latter case comes
+	 * about when useecs is false or when it's true but every character
+	 * still manages to land in its own class (unlikely, but it's
+	 * cheap to check for).  If all these things are true then the
+	 * character code needed to represent NUL's equivalence class for
+	 * indexing the tables is going to take one more bit than the
+	 * number of characters, and therefore we won't be assured of
+	 * being able to fit it into a YY_CHAR variable.  This rules out
+	 * storing the transitions in a compressed table, since the code
+	 * for interpreting them uses a YY_CHAR variable (perhaps it
+	 * should just use an integer, though; this is worth pondering ...
+	 * ###).
+	 *
+	 * Finally, for full tables, we want the number of entries in the
+	 * table to be a power of two so the array references go fast (it
+	 * will just take a shift to compute the major index).  If
+	 * encoding NUL's transitions in the table will spoil this, we
+	 * give it its own table (note that this will be the case if we're
+	 * not using equivalence classes).
+	 */
+
+	/* Note that the test for ecgroup[0] == numecs below accomplishes
+	 * both (1) and (2) above
+	 */
+	if (!fullspd && ecgroup[0] == numecs) {
+		/* NUL is alone in its equivalence class, which is the
+		 * last one.
+		 */
+		int     use_NUL_table = (numecs == csize);
+
+		if (fulltbl && !use_NUL_table) {
+			/* We still may want to use the table if numecs
+			 * is a power of 2.
+			 */
+			int     power_of_two;
+
+			for (power_of_two = 1; power_of_two <= csize;
+			     power_of_two *= 2)
+				if (numecs == power_of_two) {
+					use_NUL_table = true;
+					break;
+				}
+		}
+
+		if (use_NUL_table)
+			nultrans =
+				allocate_integer_array (current_max_dfas);
+
+		/* From now on, nultrans != nil indicates that we're
+		 * saving null transitions for later, separate encoding.
+		 */
+	}
+
+
+	if (fullspd) {
+		for (i = 0; i <= numecs; ++i)
+			state[i] = 0;
+
+		place_state (state, 0, 0);
+		dfaacc[0].dfaacc_state = 0;
+	}
+
+	else if (fulltbl) {
+		if (nultrans)
+			/* We won't be including NUL's transitions in the
+			 * table, so build it for entries from 0 .. numecs - 1.
+			 */
+			num_full_table_rows = numecs;
+
+		else
+			/* Take into account the fact that we'll be including
+			 * the NUL entries in the transition table.  Build it
+			 * from 0 .. numecs.
+			 */
+			num_full_table_rows = numecs + 1;
+
+		/* Begin generating yy_nxt[][]
+		 * This spans the entire LONG function.
+		 * This table is tricky because we don't know how big it will be.
+		 * So we'll have to realloc() on the way...
+		 * we'll wait until we can calculate yynxt_tbl->td_hilen.
+		 */
+		yynxt_tbl =
+			(struct yytbl_data *) calloc (1,
+						      sizeof (struct
+							      yytbl_data));
+		yytbl_data_init (yynxt_tbl, YYTD_ID_NXT);
+		yynxt_tbl->td_hilen = 1;
+		yynxt_tbl->td_lolen = num_full_table_rows;
+		yynxt_tbl->td_data = yynxt_data =
+			(flex_int32_t *) calloc (yynxt_tbl->td_lolen *
+					    yynxt_tbl->td_hilen,
+					    sizeof (flex_int32_t));
+		yynxt_curr = 0;
+
+		buf_prints (&yydmap_buf,
+			    "\t{YYTD_ID_NXT, (void**)&yy_nxt, sizeof(%s)},\n",
+			    long_align ? "flex_int32_t" : "flex_int16_t");
+
+		/* Unless -Ca, declare it "short" because it's a real
+		 * long-shot that that won't be large enough.
+		 */
+		if (gentables)
+			out_str_dec
+				("static yyconst %s yy_nxt[][%d] =\n    {\n",
+				 long_align ? "flex_int32_t" : "flex_int16_t",
+				 num_full_table_rows);
+		else {
+			out_dec ("#undef YY_NXT_LOLEN\n#define YY_NXT_LOLEN (%d)\n", num_full_table_rows);
+			out_str ("static yyconst %s *yy_nxt =0;\n",
+				 long_align ? "flex_int32_t" : "flex_int16_t");
+		}
+
+
+		if (gentables)
+			outn ("    {");
+
+		/* Generate 0 entries for state #0. */
+		for (i = 0; i < num_full_table_rows; ++i) {
+			mk2data (0);
+			yynxt_data[yynxt_curr++] = 0;
+		}
+
+		dataflush ();
+		if (gentables)
+			outn ("    },\n");
+	}
+
+	/* Create the first states. */
+
+	num_start_states = lastsc * 2;
+
+	for (i = 1; i <= num_start_states; ++i) {
+		numstates = 1;
+
+		/* For each start condition, make one state for the case when
+		 * we're at the beginning of the line (the '^' operator) and
+		 * one for the case when we're not.
+		 */
+		if (i % 2 == 1)
+			nset[numstates] = scset[(i / 2) + 1];
+		else
+			nset[numstates] =
+				mkbranch (scbol[i / 2], scset[i / 2]);
+
+		nset = epsclosure (nset, &numstates, accset, &nacc,
+				   &hashval);
+
+		if (snstods (nset, numstates, accset, nacc, hashval, &ds)) {
+			numas += nacc;
+			totnst += numstates;
+			++todo_next;
+
+			if (variable_trailing_context_rules && nacc > 0)
+				check_trailing_context (nset, numstates,
+							accset, nacc);
+		}
+	}
+
+	if (!fullspd) {
+		if (!snstods (nset, 0, accset, 0, 0, &end_of_buffer_state))
+			flexfatal (_
+				   ("could not create unique end-of-buffer state"));
+
+		++numas;
+		++num_start_states;
+		++todo_next;
+	}
+
+
+	while (todo_head < todo_next) {
+		targptr = 0;
+		totaltrans = 0;
+
+		for (i = 1; i <= numecs; ++i)
+			state[i] = 0;
+
+		ds = ++todo_head;
+
+		dset = dss[ds];
+		dsize = dfasiz[ds];
+
+		if (trace)
+			fprintf (stderr, _("state # %d:\n"), ds);
+
+		sympartition (dset, dsize, symlist, duplist);
+
+		for (sym = 1; sym <= numecs; ++sym) {
+			if (symlist[sym]) {
+				symlist[sym] = 0;
+
+				if (duplist[sym] == NIL) {
+					/* Symbol has unique out-transitions. */
+					numstates =
+						symfollowset (dset, dsize,
+							      sym, nset);
+					nset = epsclosure (nset,
+							   &numstates,
+							   accset, &nacc,
+							   &hashval);
+
+					if (snstods
+					    (nset, numstates, accset, nacc,
+					     hashval, &newds)) {
+						totnst = totnst +
+							numstates;
+						++todo_next;
+						numas += nacc;
+
+						if (variable_trailing_context_rules && nacc > 0)
+							check_trailing_context
+								(nset,
+								 numstates,
+								 accset,
+								 nacc);
+					}
+
+					state[sym] = newds;
+
+					if (trace)
+						fprintf (stderr,
+							 "\t%d\t%d\n", sym,
+							 newds);
+
+					targfreq[++targptr] = 1;
+					targstate[targptr] = newds;
+					++numuniq;
+				}
+
+				else {
+					/* sym's equivalence class has the same
+					 * transitions as duplist(sym)'s
+					 * equivalence class.
+					 */
+					targ = state[duplist[sym]];
+					state[sym] = targ;
+
+					if (trace)
+						fprintf (stderr,
+							 "\t%d\t%d\n", sym,
+							 targ);
+
+					/* Update frequency count for
+					 * destination state.
+					 */
+
+					i = 0;
+					while (targstate[++i] != targ) ;
+
+					++targfreq[i];
+					++numdup;
+				}
+
+				++totaltrans;
+				duplist[sym] = NIL;
+			}
+		}
+
+
+		numsnpairs += totaltrans;
+
+		if (ds > num_start_states)
+			check_for_backing_up (ds, state);
+
+		if (nultrans) {
+			nultrans[ds] = state[NUL_ec];
+			state[NUL_ec] = 0;	/* remove transition */
+		}
+
+		if (fulltbl) {
+
+			/* Each time we hit here, it's another td_hilen, so we realloc. */
+			yynxt_tbl->td_hilen++;
+			yynxt_tbl->td_data = yynxt_data =
+				(flex_int32_t *) realloc (yynxt_data,
+						     yynxt_tbl->td_hilen *
+						     yynxt_tbl->td_lolen *
+						     sizeof (flex_int32_t));
+
+
+			if (gentables)
+				outn ("    {");
+
+			/* Supply array's 0-element. */
+			if (ds == end_of_buffer_state) {
+				mk2data (-end_of_buffer_state);
+				yynxt_data[yynxt_curr++] =
+					-end_of_buffer_state;
+			}
+			else {
+				mk2data (end_of_buffer_state);
+				yynxt_data[yynxt_curr++] =
+					end_of_buffer_state;
+			}
+
+			for (i = 1; i < num_full_table_rows; ++i) {
+				/* Jams are marked by negative of state
+				 * number.
+				 */
+				mk2data (state[i] ? state[i] : -ds);
+				yynxt_data[yynxt_curr++] =
+					state[i] ? state[i] : -ds;
+			}
+
+			dataflush ();
+			if (gentables)
+				outn ("    },\n");
+		}
+
+		else if (fullspd)
+			place_state (state, ds, totaltrans);
+
+		else if (ds == end_of_buffer_state)
+			/* Special case this state to make sure it does what
+			 * it's supposed to, i.e., jam on end-of-buffer.
+			 */
+			stack1 (ds, 0, 0, JAMSTATE);
+
+		else {		/* normal, compressed state */
+
+			/* Determine which destination state is the most
+			 * common, and how many transitions to it there are.
+			 */
+
+			comfreq = 0;
+			comstate = 0;
+
+			for (i = 1; i <= targptr; ++i)
+				if (targfreq[i] > comfreq) {
+					comfreq = targfreq[i];
+					comstate = targstate[i];
+				}
+
+			bldtbl (state, ds, totaltrans, comstate, comfreq);
+		}
+	}
+
+	if (fulltbl) {
+		dataend ();
+		if (tablesext) {
+			yytbl_data_compress (yynxt_tbl);
+			if (yytbl_data_fwrite (&tableswr, yynxt_tbl) < 0)
+				flexerror (_
+					   ("Could not write yynxt_tbl[][]"));
+		}
+		if (yynxt_tbl) {
+			yytbl_data_destroy (yynxt_tbl);
+			yynxt_tbl = 0;
+		}
+	}
+
+	else if (!fullspd) {
+		cmptmps ();	/* create compressed template entries */
+
+		/* Create tables for all the states with only one
+		 * out-transition.
+		 */
+		while (onesp > 0) {
+			mk1tbl (onestate[onesp], onesym[onesp],
+				onenext[onesp], onedef[onesp]);
+			--onesp;
+		}
+
+		mkdeftbl ();
+	}
+
+	flex_free ((void *) accset);
+	flex_free ((void *) nset);
+}
+
+
+/* snstods - converts a set of ndfa states into a dfa state
+ *
+ * synopsis
+ *    is_new_state = snstods( int sns[numstates], int numstates,
+ *				int accset[num_rules+1], int nacc,
+ *				int hashval, int *newds_addr );
+ *
+ * On return, the dfa state number is in newds.
+ */
+
+int snstods (sns, numstates, accset, nacc, hashval, newds_addr)
+     int sns[], numstates, accset[], nacc, hashval, *newds_addr;
+{
+	int     didsort = 0;
+	register int i, j;
+	int     newds, *oldsns;
+
+	for (i = 1; i <= lastdfa; ++i)
+		if (hashval == dhash[i]) {
+			if (numstates == dfasiz[i]) {
+				oldsns = dss[i];
+
+				if (!didsort) {
+					/* We sort the states in sns so we
+					 * can compare it to oldsns quickly.
+					 */
+					qsort (&sns [1], numstates, sizeof (sns [1]), intcmp);
+					didsort = 1;
+				}
+
+				for (j = 1; j <= numstates; ++j)
+					if (sns[j] != oldsns[j])
+						break;
+
+				if (j > numstates) {
+					++dfaeql;
+					*newds_addr = i;
+					return 0;
+				}
+
+				++hshcol;
+			}
+
+			else
+				++hshsave;
+		}
+
+	/* Make a new dfa. */
+
+	if (++lastdfa >= current_max_dfas)
+		increase_max_dfas ();
+
+	newds = lastdfa;
+
+	dss[newds] = allocate_integer_array (numstates + 1);
+
+	/* If we haven't already sorted the states in sns, we do so now,
+	 * so that future comparisons with it can be made quickly.
+	 */
+
+	if (!didsort)
+		qsort (&sns [1], numstates, sizeof (sns [1]), intcmp);
+
+	for (i = 1; i <= numstates; ++i)
+		dss[newds][i] = sns[i];
+
+	dfasiz[newds] = numstates;
+	dhash[newds] = hashval;
+
+	if (nacc == 0) {
+		if (reject)
+			dfaacc[newds].dfaacc_set = (int *) 0;
+		else
+			dfaacc[newds].dfaacc_state = 0;
+
+		accsiz[newds] = 0;
+	}
+
+	else if (reject) {
+		/* We sort the accepting set in increasing order so the
+		 * disambiguating rule that the first rule listed is considered
+		 * match in the event of ties will work.
+		 */
+
+		qsort (&accset [1], nacc, sizeof (accset [1]), intcmp);
+
+		dfaacc[newds].dfaacc_set =
+			allocate_integer_array (nacc + 1);
+
+		/* Save the accepting set for later */
+		for (i = 1; i <= nacc; ++i) {
+			dfaacc[newds].dfaacc_set[i] = accset[i];
+
+			if (accset[i] <= num_rules)
+				/* Who knows, perhaps a REJECT can yield
+				 * this rule.
+				 */
+				rule_useful[accset[i]] = true;
+		}
+
+		accsiz[newds] = nacc;
+	}
+
+	else {
+		/* Find lowest numbered rule so the disambiguating rule
+		 * will work.
+		 */
+		j = num_rules + 1;
+
+		for (i = 1; i <= nacc; ++i)
+			if (accset[i] < j)
+				j = accset[i];
+
+		dfaacc[newds].dfaacc_state = j;
+
+		if (j <= num_rules)
+			rule_useful[j] = true;
+	}
+
+	*newds_addr = newds;
+
+	return 1;
+}
+
+
+/* symfollowset - follow the symbol transitions one step
+ *
+ * synopsis
+ *    numstates = symfollowset( int ds[current_max_dfa_size], int dsize,
+ *				int transsym, int nset[current_max_dfa_size] );
+ */
+
+int symfollowset (ds, dsize, transsym, nset)
+     int ds[], dsize, transsym, nset[];
+{
+	int     ns, tsp, sym, i, j, lenccl, ch, numstates, ccllist;
+
+	numstates = 0;
+
+	for (i = 1; i <= dsize; ++i) {	/* for each nfa state ns in the state set of ds */
+		ns = ds[i];
+		sym = transchar[ns];
+		tsp = trans1[ns];
+
+		if (sym < 0) {	/* it's a character class */
+			sym = -sym;
+			ccllist = cclmap[sym];
+			lenccl = ccllen[sym];
+
+			if (cclng[sym]) {
+				for (j = 0; j < lenccl; ++j) {
+					/* Loop through negated character
+					 * class.
+					 */
+					ch = ccltbl[ccllist + j];
+
+					if (ch == 0)
+						ch = NUL_ec;
+
+					if (ch > transsym)
+						/* Transsym isn't in negated
+						 * ccl.
+						 */
+						break;
+
+					else if (ch == transsym)
+						/* next 2 */
+						goto bottom;
+				}
+
+				/* Didn't find transsym in ccl. */
+				nset[++numstates] = tsp;
+			}
+
+			else
+				for (j = 0; j < lenccl; ++j) {
+					ch = ccltbl[ccllist + j];
+
+					if (ch == 0)
+						ch = NUL_ec;
+
+					if (ch > transsym)
+						break;
+					else if (ch == transsym) {
+						nset[++numstates] = tsp;
+						break;
+					}
+				}
+		}
+
+		else if (sym == SYM_EPSILON) {	/* do nothing */
+		}
+
+		else if (ABS (ecgroup[sym]) == transsym)
+			nset[++numstates] = tsp;
+
+	      bottom:;
+	}
+
+	return numstates;
+}
+
+
+/* sympartition - partition characters with same out-transitions
+ *
+ * synopsis
+ *    sympartition( int ds[current_max_dfa_size], int numstates,
+ *			int symlist[numecs], int duplist[numecs] );
+ */
+
+void sympartition (ds, numstates, symlist, duplist)
+     int ds[], numstates;
+     int symlist[], duplist[];
+{
+	int     tch, i, j, k, ns, dupfwd[CSIZE + 1], lenccl, cclp, ich;
+
+	/* Partitioning is done by creating equivalence classes for those
+	 * characters which have out-transitions from the given state.  Thus
+	 * we are really creating equivalence classes of equivalence classes.
+	 */
+
+	for (i = 1; i <= numecs; ++i) {	/* initialize equivalence class list */
+		duplist[i] = i - 1;
+		dupfwd[i] = i + 1;
+	}
+
+	duplist[1] = NIL;
+	dupfwd[numecs] = NIL;
+
+	for (i = 1; i <= numstates; ++i) {
+		ns = ds[i];
+		tch = transchar[ns];
+
+		if (tch != SYM_EPSILON) {
+			if (tch < -lastccl || tch >= csize) {
+				flexfatal (_
+					   ("bad transition character detected in sympartition()"));
+			}
+
+			if (tch >= 0) {	/* character transition */
+				int     ec = ecgroup[tch];
+
+				mkechar (ec, dupfwd, duplist);
+				symlist[ec] = 1;
+			}
+
+			else {	/* character class */
+				tch = -tch;
+
+				lenccl = ccllen[tch];
+				cclp = cclmap[tch];
+				mkeccl (ccltbl + cclp, lenccl, dupfwd,
+					duplist, numecs, NUL_ec);
+
+				if (cclng[tch]) {
+					j = 0;
+
+					for (k = 0; k < lenccl; ++k) {
+						ich = ccltbl[cclp + k];
+
+						if (ich == 0)
+							ich = NUL_ec;
+
+						for (++j; j < ich; ++j)
+							symlist[j] = 1;
+					}
+
+					for (++j; j <= numecs; ++j)
+						symlist[j] = 1;
+				}
+
+				else
+					for (k = 0; k < lenccl; ++k) {
+						ich = ccltbl[cclp + k];
+
+						if (ich == 0)
+							ich = NUL_ec;
+
+						symlist[ich] = 1;
+					}
+			}
+		}
+	}
+}
diff --git a/doc/.gitignore b/doc/.gitignore
new file mode 100644
index 0000000..c74bc7b
--- /dev/null
+++ b/doc/.gitignore
@@ -0,0 +1,29 @@
+*.aux
+*.cp
+*.cps
+*.dvi
+*.fn
+*.fns
+*.hk
+*.hks
+*.info*
+*.ky
+*.log
+*.op
+*.ops
+*.pg
+*.toc
+*.tp
+*.tps
+*.vr
+*.vrs
+Makefile
+Makefile.in
+flex.1
+flex.pdf
+flex.ps
+mdate-sh
+stamp-*
+version.texi
+flex.html
+texinfo.tex
diff --git a/doc/Makefile.am b/doc/Makefile.am
new file mode 100644
index 0000000..0e23dca
--- /dev/null
+++ b/doc/Makefile.am
@@ -0,0 +1,31 @@
+help2man = @HELP2MAN@
+
+info_TEXINFOS =	flex.texi
+dist_man_MANS = flex.1
+dist_doc_DATA= flex.pdf
+
+CLEANFILES = \
+	flex.aux \
+	flex.cp \
+	flex.cps \
+	flex.fn \
+	flex.fns \
+	flex.hk \
+	flex.hks \
+	flex.ky \
+	flex.log \
+	flex.op \
+	flex.ops \
+	flex.pg \
+	flex.toc \
+	flex.tp \
+	flex.tps \
+	flex.vr \
+	flex.vrs
+
+$(dist_man_MANS): $(top_srcdir)/main.c
+	for i in $(dist_man_MANS) ; do \
+	$(help2man) --name='$(PACKAGE_NAME)' \
+	--section=`echo $$i | sed -e 's/.*\.\([^.]*\)$$/\1/'` \
+	 ../flex$(EXEEXT) > $$i || rm -f $$i ; \
+	done
diff --git a/doc/flex.texi b/doc/flex.texi
new file mode 100644
index 0000000..07ce3ac
--- /dev/null
+++ b/doc/flex.texi
@@ -0,0 +1,8615 @@
+\input texinfo.tex @c -*-texinfo-*-
+@c %**start of header
+@setfilename flex.info
+@include version.texi
+@settitle Lexical Analysis With Flex, for Flex @value{VERSION}
+@set authors Vern Paxson, Will Estes and John Millaway
+@c  "Macro Hooks" index
+@defindex hk
+@c  "Options" index
+@defindex op
+@dircategory Programming
+@direntry
+* flex: (flex).      Fast lexical analyzer generator (lex replacement).
+@end direntry
+@c %**end of header
+
+@copying
+
+The flex manual is placed under the same licensing conditions as the
+rest of flex:
+
+Copyright @copyright{} 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2012
+The Flex Project.
+
+Copyright @copyright{} 1990, 1997 The Regents of the University of California.
+All rights reserved.
+
+This code is derived from software contributed to Berkeley by
+Vern Paxson.
+
+The United States Government has rights in this work pursuant
+to contract no. DE-AC03-76SF00098 between the United States
+Department of Energy and the University of California.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+@enumerate
+@item
+ Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+
+@item
+Redistributions in binary form must reproduce the above copyright
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution.
+@end enumerate
+
+Neither the name of the University nor the names of its contributors
+may be used to endorse or promote products derived from this software
+without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.
+@end copying
+
+@titlepage
+@title Lexical Analysis with Flex
+@subtitle Edition @value{EDITION}, @value{UPDATED}
+@author @value{authors}
+@page
+@vskip 0pt plus 1filll
+@insertcopying
+@end titlepage
+@contents
+@ifnottex
+@node Top, Copyright, (dir), (dir)
+@top flex
+
+This manual describes @code{flex}, a tool for generating programs that
+perform pattern-matching on text.  The manual includes both tutorial and
+reference sections.
+
+This edition of @cite{The flex Manual} documents @code{flex} version
+@value{VERSION}. It was last updated on @value{UPDATED}.
+
+This manual was written by @value{authors}.
+
+@menu
+* Copyright::                   
+* Reporting Bugs::              
+* Introduction::                
+* Simple Examples::             
+* Format::                      
+* Patterns::                    
+* Matching::                    
+* Actions::                     
+* Generated Scanner::           
+* Start Conditions::            
+* Multiple Input Buffers::      
+* EOF::                         
+* Misc Macros::                 
+* User Values::                 
+* Yacc::                        
+* Scanner Options::             
+* Performance::                 
+* Cxx::                         
+* Reentrant::                   
+* Lex and Posix::               
+* Memory Management::           
+* Serialized Tables::           
+* Diagnostics::                 
+* Limitations::                 
+* Bibliography::                
+* FAQ::                         
+* Appendices::                  
+* Indices::                     
+
+@detailmenu
+ --- The Detailed Node Listing ---
+
+Format of the Input File
+
+* Definitions Section::         
+* Rules Section::               
+* User Code Section::           
+* Comments in the Input::       
+
+Scanner Options
+
+* Options for Specifying Filenames::  
+* Options Affecting Scanner Behavior::  
+* Code-Level And API Options::  
+* Options for Scanner Speed and Size::  
+* Debugging Options::           
+* Miscellaneous Options::       
+
+Reentrant C Scanners
+
+* Reentrant Uses::              
+* Reentrant Overview::          
+* Reentrant Example::           
+* Reentrant Detail::            
+* Reentrant Functions::         
+
+The Reentrant API in Detail
+
+* Specify Reentrant::           
+* Extra Reentrant Argument::    
+* Global Replacement::          
+* Init and Destroy Functions::  
+* Accessor Methods::            
+* Extra Data::                  
+* About yyscan_t::              
+
+Memory Management
+
+* The Default Memory Management::  
+* Overriding The Default Memory Management::  
+* A Note About yytext And Memory::  
+
+Serialized Tables
+
+* Creating Serialized Tables::  
+* Loading and Unloading Serialized Tables::  
+* Tables File Format::          
+
+FAQ
+
+* When was flex born?::         
+* How do I expand backslash-escape sequences in C-style quoted strings?::  
+* Why do flex scanners call fileno if it is not ANSI compatible?::  
+* Does flex support recursive pattern definitions?::  
+* How do I skip huge chunks of input (tens of megabytes) while using flex?::  
+* Flex is not matching my patterns in the same order that I defined them.::  
+* My actions are executing out of order or sometimes not at all.::  
+* How can I have multiple input sources feed into the same scanner at the same time?::  
+* Can I build nested parsers that work with the same input file?::  
+* How can I match text only at the end of a file?::  
+* How can I make REJECT cascade across start condition boundaries?::  
+* Why cant I use fast or full tables with interactive mode?::  
+* How much faster is -F or -f than -C?::  
+* If I have a simple grammar cant I just parse it with flex?::  
+* Why doesn't yyrestart() set the start state back to INITIAL?::  
+* How can I match C-style comments?::  
+* The period isn't working the way I expected.::  
+* Can I get the flex manual in another format?::  
+* Does there exist a "faster" NDFA->DFA algorithm?::  
+* How does flex compile the DFA so quickly?::  
+* How can I use more than 8192 rules?::  
+* How do I abandon a file in the middle of a scan and switch to a new file?::  
+* How do I execute code only during initialization (only before the first scan)?::  
+* How do I execute code at termination?::  
+* Where else can I find help?::  
+* Can I include comments in the "rules" section of the file?::  
+* I get an error about undefined yywrap().::  
+* How can I change the matching pattern at run time?::  
+* How can I expand macros in the input?::  
+* How can I build a two-pass scanner?::  
+* How do I match any string not matched in the preceding rules?::  
+* I am trying to port code from AT&T lex that uses yysptr and yysbuf.::  
+* Is there a way to make flex treat NULL like a regular character?::  
+* Whenever flex can not match the input it says "flex scanner jammed".::  
+* Why doesn't flex have non-greedy operators like perl does?::  
+* Memory leak - 16386 bytes allocated by malloc.::  
+* How do I track the byte offset for lseek()?::  
+* How do I use my own I/O classes in a C++ scanner?::  
+* How do I skip as many chars as possible?::  
+* deleteme00::              
+* Are certain equivalent patterns faster than others?::              
+* Is backing up a big deal?::              
+* Can I fake multi-byte character support?::              
+* deleteme01::              
+* Can you discuss some flex internals?::              
+* unput() messes up yy_at_bol::              
+* The | operator is not doing what I want::              
+* Why can't flex understand this variable trailing context pattern?::              
+* The ^ operator isn't working::              
+* Trailing context is getting confused with trailing optional patterns::              
+* Is flex GNU or not?::              
+* ERASEME53::              
+* I need to scan if-then-else blocks and while loops::              
+* ERASEME55::              
+* ERASEME56::              
+* ERASEME57::              
+* Is there a repository for flex scanners?::              
+* How can I conditionally compile or preprocess my flex input file?::              
+* Where can I find grammars for lex and yacc?::              
+* I get an end-of-buffer message for each character scanned.::              
+* unnamed-faq-62::              
+* unnamed-faq-63::              
+* unnamed-faq-64::              
+* unnamed-faq-65::              
+* unnamed-faq-66::              
+* unnamed-faq-67::              
+* unnamed-faq-68::              
+* unnamed-faq-69::              
+* unnamed-faq-70::              
+* unnamed-faq-71::              
+* unnamed-faq-72::              
+* unnamed-faq-73::              
+* unnamed-faq-74::              
+* unnamed-faq-75::              
+* unnamed-faq-76::              
+* unnamed-faq-77::              
+* unnamed-faq-78::              
+* unnamed-faq-79::              
+* unnamed-faq-80::              
+* unnamed-faq-81::              
+* unnamed-faq-82::              
+* unnamed-faq-83::              
+* unnamed-faq-84::              
+* unnamed-faq-85::              
+* unnamed-faq-86::              
+* unnamed-faq-87::              
+* unnamed-faq-88::              
+* unnamed-faq-90::              
+* unnamed-faq-91::              
+* unnamed-faq-92::              
+* unnamed-faq-93::              
+* unnamed-faq-94::              
+* unnamed-faq-95::              
+* unnamed-faq-96::              
+* unnamed-faq-97::              
+* unnamed-faq-98::              
+* unnamed-faq-99::              
+* unnamed-faq-100::             
+* unnamed-faq-101::             
+* What is the difference between YYLEX_PARAM and YY_DECL?::
+* Why do I get "conflicting types for yylex" error?::
+* How do I access the values set in a Flex action from within a Bison action?::
+
+Appendices
+
+* Makefiles and Flex::          
+* Bison Bridge::                
+* M4 Dependency::               
+* Common Patterns::               
+
+Indices
+
+* Concept Index::               
+* Index of Functions and Macros::  
+* Index of Variables::          
+* Index of Data Types::         
+* Index of Hooks::              
+* Index of Scanner Options::    
+
+@end detailmenu
+@end menu
+@end ifnottex
+@node Copyright, Reporting Bugs, Top, Top
+@chapter Copyright
+
+@cindex copyright of flex
+@cindex distributing flex
+@insertcopying
+
+@node Reporting Bugs, Introduction, Copyright, Top
+@chapter Reporting Bugs
+
+@cindex bugs, reporting
+@cindex reporting bugs
+
+If you find a bug in @code{flex}, please report it using
+the SourceForge Bug Tracking facilities which can be found on
+@url{http://sourceforge.net/projects/flex,flex's SourceForge Page}.
+
+@node Introduction, Simple Examples, Reporting Bugs, Top
+@chapter Introduction
+
+@cindex scanner, definition of
+@code{flex} is a tool for generating @dfn{scanners}.  A scanner is a
+program which recognizes lexical patterns in text.  The @code{flex}
+program reads the given input files, or its standard input if no file
+names are given, for a description of a scanner to generate.  The
+description is in the form of pairs of regular expressions and C code,
+called @dfn{rules}. @code{flex} generates as output a C source file,
+@file{lex.yy.c} by default, which defines a routine @code{yylex()}.
+This file can be compiled and linked with the flex runtime library to
+produce an executable.  When the executable is run, it analyzes its
+input for occurrences of the regular expressions.  Whenever it finds
+one, it executes the corresponding C code.
+
+@node Simple Examples, Format, Introduction, Top
+@chapter Some Simple Examples
+
+First some simple examples to get the flavor of how one uses
+@code{flex}.
+
+@cindex username expansion
+The following @code{flex} input specifies a scanner which, when it
+encounters the string @samp{username} will replace it with the user's
+login name:
+
+@example
+@verbatim
+    %%
+    username    printf( "%s", getlogin() );
+@end verbatim
+@end example
+
+@cindex default rule
+@cindex rules, default
+By default, any text not matched by a @code{flex} scanner is copied to
+the output, so the net effect of this scanner is to copy its input file
+to its output with each occurrence of @samp{username} expanded.  In this
+input, there is just one rule.  @samp{username} is the @dfn{pattern} and
+the @samp{printf} is the @dfn{action}.  The @samp{%%} symbol marks the
+beginning of the rules.
+
+Here's another simple example:
+
+@cindex counting characters and lines
+@example
+@verbatim
+            int num_lines = 0, num_chars = 0;
+
+    %%
+    \n      ++num_lines; ++num_chars;
+    .       ++num_chars;
+
+    %%
+
+    int main()
+            {
+            yylex();
+            printf( "# of lines = %d, # of chars = %d\n",
+                    num_lines, num_chars );
+            }
+@end verbatim
+@end example
+
+This scanner counts the number of characters and the number of lines in
+its input. It produces no output other than the final report on the
+character and line counts.  The first line declares two globals,
+@code{num_lines} and @code{num_chars}, which are accessible both inside
+@code{yylex()} and in the @code{main()} routine declared after the
+second @samp{%%}.  There are two rules, one which matches a newline
+(@samp{\n}) and increments both the line count and the character count,
+and one which matches any character other than a newline (indicated by
+the @samp{.} regular expression).
+
+A somewhat more complicated example:
+
+@cindex Pascal-like language
+@example
+@verbatim
+    /* scanner for a toy Pascal-like language */
+
+    %{
+    /* need this for the call to atof() below */
+    #include <math.h>
+    %}
+
+    DIGIT    [0-9]
+    ID       [a-z][a-z0-9]*
+
+    %%
+
+    {DIGIT}+    {
+                printf( "An integer: %s (%d)\n", yytext,
+                        atoi( yytext ) );
+                }
+
+    {DIGIT}+"."{DIGIT}*        {
+                printf( "A float: %s (%g)\n", yytext,
+                        atof( yytext ) );
+                }
+
+    if|then|begin|end|procedure|function        {
+                printf( "A keyword: %s\n", yytext );
+                }
+
+    {ID}        printf( "An identifier: %s\n", yytext );
+
+    "+"|"-"|"*"|"/"   printf( "An operator: %s\n", yytext );
+
+    "{"[\^{}}\n]*"}"     /* eat up one-line comments */
+
+    [ \t\n]+          /* eat up whitespace */
+
+    .           printf( "Unrecognized character: %s\n", yytext );
+
+    %%
+
+    int main( int argc, char **argv )
+        {
+        ++argv, --argc;  /* skip over program name */
+        if ( argc > 0 )
+                yyin = fopen( argv[0], "r" );
+        else
+                yyin = stdin;
+
+        yylex();
+        }
+@end verbatim
+@end example
+
+This is the beginnings of a simple scanner for a language like Pascal.
+It identifies different types of @dfn{tokens} and reports on what it has
+seen.
+
+The details of this example will be explained in the following
+sections.
+
+@node Format, Patterns, Simple Examples, Top
+@chapter Format of the Input File
+
+
+@cindex format of flex input
+@cindex input, format of
+@cindex file format
+@cindex sections of flex input
+
+The @code{flex} input file consists of three sections, separated by a
+line containing only @samp{%%}.
+
+@cindex format of input file
+@example
+@verbatim
+    definitions
+    %%
+    rules
+    %%
+    user code
+@end verbatim
+@end example
+
+@menu
+* Definitions Section::         
+* Rules Section::               
+* User Code Section::           
+* Comments in the Input::       
+@end menu
+
+@node Definitions Section, Rules Section, Format, Format
+@section Format of the Definitions Section
+
+@cindex input file, Definitions section
+@cindex Definitions, in flex input
+The @dfn{definitions section} contains declarations of simple @dfn{name}
+definitions to simplify the scanner specification, and declarations of
+@dfn{start conditions}, which are explained in a later section.
+
+@cindex aliases, how to define
+@cindex pattern aliases, how to define
+Name definitions have the form:
+
+@example
+@verbatim
+    name definition
+@end verbatim
+@end example
+
+The @samp{name} is a word beginning with a letter or an underscore
+(@samp{_}) followed by zero or more letters, digits, @samp{_}, or
+@samp{-} (dash).  The definition is taken to begin at the first
+non-whitespace character following the name and continuing to the end of
+the line.  The definition can subsequently be referred to using
+@samp{@{name@}}, which will expand to @samp{(definition)}.  For example,
+
+@cindex pattern aliases, defining
+@cindex defining pattern aliases
+@example
+@verbatim
+    DIGIT    [0-9]
+    ID       [a-z][a-z0-9]*
+@end verbatim
+@end example
+
+Defines @samp{DIGIT} to be a regular expression which matches a single
+digit, and @samp{ID} to be a regular expression which matches a letter
+followed by zero-or-more letters-or-digits.  A subsequent reference to
+
+@cindex pattern aliases, use of
+@example
+@verbatim
+    {DIGIT}+"."{DIGIT}*
+@end verbatim
+@end example
+
+is identical to
+
+@example
+@verbatim
+    ([0-9])+"."([0-9])*
+@end verbatim
+@end example
+
+and matches one-or-more digits followed by a @samp{.} followed by
+zero-or-more digits.
+
+@cindex comments in flex input
+An unindented comment (i.e., a line
+beginning with @samp{/*}) is copied verbatim to the output up
+to the next @samp{*/}.
+
+@cindex %@{ and %@}, in Definitions Section
+@cindex embedding C code in flex input
+@cindex C code in flex input
+Any @emph{indented} text or text enclosed in @samp{%@{} and @samp{%@}}
+is also copied verbatim to the output (with the %@{ and %@} symbols
+removed).  The %@{ and %@} symbols must appear unindented on lines by
+themselves.
+
+@cindex %top
+
+A @code{%top} block is similar to a @samp{%@{} ... @samp{%@}} block, except
+that the code in a @code{%top} block is relocated to the @emph{top} of the
+generated file, before any flex definitions @footnote{Actually,
+@code{yyIN_HEADER} is defined before the @samp{%top} block.}. 
+The @code{%top} block is useful when you want certain preprocessor macros to be
+defined or certain files to be included before the generated code.
+The single characters, @samp{@{}  and @samp{@}} are used to delimit the
+@code{%top} block, as show in the example below:
+
+@example
+@verbatim
+    %top{
+        /* This code goes at the "top" of the generated file. */
+        #include <stdint.h>
+        #include <inttypes.h>
+    }
+@end verbatim
+@end example
+
+Multiple @code{%top} blocks are allowed, and their order is preserved.
+
+@node Rules Section, User Code Section, Definitions Section, Format
+@section Format of the Rules Section
+
+@cindex input file, Rules Section
+@cindex rules, in flex input
+The @dfn{rules} section of the @code{flex} input contains a series of
+rules of the form:
+
+@example
+@verbatim
+    pattern   action
+@end verbatim
+@end example
+
+where the pattern must be unindented and the action must begin
+on the same line.
+@xref{Patterns}, for a further description of patterns and actions.
+
+In the rules section, any indented or %@{ %@} enclosed text appearing
+before the first rule may be used to declare variables which are local
+to the scanning routine and (after the declarations) code which is to be
+executed whenever the scanning routine is entered.  Other indented or
+%@{ %@} text in the rule section is still copied to the output, but its
+meaning is not well-defined and it may well cause compile-time errors
+(this feature is present for @acronym{POSIX} compliance. @xref{Lex and
+Posix}, for other such features).
+
+Any @emph{indented} text or text enclosed in @samp{%@{} and @samp{%@}}
+is copied verbatim to the output (with the %@{ and %@} symbols removed).
+The %@{ and %@} symbols must appear unindented on lines by themselves.
+
+@node User Code Section, Comments in the Input, Rules Section, Format
+@section Format of the User Code Section
+
+@cindex input file, user code Section
+@cindex user code, in flex input
+The user code section is simply copied to @file{lex.yy.c} verbatim.  It
+is used for companion routines which call or are called by the scanner.
+The presence of this section is optional; if it is missing, the second
+@samp{%%} in the input file may be skipped, too.
+
+@node Comments in the Input,  , User Code Section, Format
+@section Comments in the Input
+
+@cindex comments, syntax of
+Flex supports C-style comments, that is, anything between @samp{/*} and
+@samp{*/} is
+considered a comment. Whenever flex encounters a comment, it copies the
+entire comment verbatim to the generated source code. Comments may
+appear just about anywhere, but with the following exceptions:
+
+@itemize
+@cindex comments, in rules section
+@item
+Comments may not appear in the Rules Section wherever flex is expecting
+a regular expression. This means comments may not appear at the
+beginning of a line, or immediately following a list of scanner states.
+@item
+Comments may not appear on an @samp{%option} line in the Definitions
+Section.
+@end itemize
+
+If you want to follow a simple rule, then always begin a comment on a
+new line, with one or more whitespace characters before the initial
+@samp{/*}).  This rule will work anywhere in the input file.
+
+All the comments in the following example are valid:
+
+@cindex comments, valid uses of
+@cindex comments in the input
+@example
+@verbatim
+%{
+/* code block */
+%}
+
+/* Definitions Section */
+%x STATE_X
+
+%%
+    /* Rules Section */
+ruleA   /* after regex */ { /* code block */ } /* after code block */
+        /* Rules Section (indented) */
+<STATE_X>{
+ruleC   ECHO;
+ruleD   ECHO;
+%{
+/* code block */
+%}
+}
+%%
+/* User Code Section */
+
+@end verbatim
+@end example
+
+@node Patterns, Matching, Format, Top
+@chapter Patterns
+
+@cindex patterns, in rules section
+@cindex regular expressions, in patterns
+The patterns in the input (see @ref{Rules Section}) are written using an
+extended set of regular expressions.  These are:
+
+@cindex patterns, syntax
+@cindex patterns, syntax
+@table @samp
+@item x
+match the character 'x'
+
+@item .
+any character (byte) except newline
+
+@cindex [] in patterns
+@cindex character classes in patterns, syntax of
+@cindex POSIX, character classes in patterns, syntax of
+@item [xyz]
+a @dfn{character class}; in this case, the pattern
+matches either an 'x', a 'y', or a 'z'
+
+@cindex ranges in patterns
+@item [abj-oZ]
+a "character class" with a range in it; matches
+an 'a', a 'b', any letter from 'j' through 'o',
+or a 'Z'
+
+@cindex ranges in patterns, negating
+@cindex negating ranges in patterns
+@item [^A-Z]
+a "negated character class", i.e., any character
+but those in the class.  In this case, any
+character EXCEPT an uppercase letter.
+
+@item [^A-Z\n]
+any character EXCEPT an uppercase letter or
+a newline
+
+@item [a-z]@{-@}[aeiou]
+the lowercase consonants
+
+@item r*
+zero or more r's, where r is any regular expression
+
+@item r+
+one or more r's
+
+@item r?
+zero or one r's (that is, ``an optional r'')
+
+@cindex braces in patterns
+@item r@{2,5@}
+anywhere from two to five r's
+
+@item r@{2,@}
+two or more r's
+
+@item r@{4@}
+exactly 4 r's
+
+@cindex pattern aliases, expansion of
+@item @{name@}
+the expansion of the @samp{name} definition
+(@pxref{Format}).
+
+@cindex literal text in patterns, syntax of
+@cindex verbatim text in patterns, syntax of
+@item "[xyz]\"foo"
+the literal string: @samp{[xyz]"foo}
+
+@cindex escape sequences in patterns, syntax of
+@item \X
+if X is @samp{a}, @samp{b}, @samp{f}, @samp{n}, @samp{r}, @samp{t}, or
+@samp{v}, then the ANSI-C interpretation of @samp{\x}.  Otherwise, a
+literal @samp{X} (used to escape operators such as @samp{*})
+
+@cindex NULL character in patterns, syntax of
+@item \0
+a NUL character (ASCII code 0)
+
+@cindex octal characters in patterns
+@item \123
+the character with octal value 123
+
+@item \x2a
+the character with hexadecimal value 2a
+
+@item (r)
+match an @samp{r}; parentheses are used to override precedence (see below)
+
+@item (?r-s:pattern)
+apply option @samp{r} and omit option @samp{s} while interpreting pattern.
+Options may be zero or more of the characters @samp{i}, @samp{s}, or @samp{x}.
+
+@samp{i} means case-insensitive. @samp{-i} means case-sensitive.
+
+@samp{s} alters the meaning of the @samp{.} syntax to match any single byte whatsoever.
+@samp{-s} alters the meaning of @samp{.} to match any byte except @samp{\n}.
+
+@samp{x} ignores comments and whitespace in patterns. Whitespace is ignored unless
+it is backslash-escaped, contained within @samp{""}s, or appears inside a 
+character class.
+
+The following are all valid:
+
+@verbatim
+(?:foo)         same as  (foo)
+(?i:ab7)        same as  ([aA][bB]7)
+(?-i:ab)        same as  (ab)
+(?s:.)          same as  [\x00-\xFF]
+(?-s:.)         same as  [^\n]
+(?ix-s: a . b)  same as  ([Aa][^\n][bB])
+(?x:a  b)       same as  ("ab")
+(?x:a\ b)       same as  ("a b")
+(?x:a" "b)      same as  ("a b")
+(?x:a[ ]b)      same as  ("a b")
+(?x:a
+    /* comment */
+    b
+    c)          same as  (abc)
+@end verbatim
+
+@item (?# comment )
+omit everything within @samp{()}. The first @samp{)}
+character encountered ends the pattern. It is not possible to for the comment
+to contain a @samp{)} character. The comment may span lines.
+
+@cindex concatenation, in patterns
+@item rs
+the regular expression @samp{r} followed by the regular expression @samp{s}; called
+@dfn{concatenation}
+
+@item r|s
+either an @samp{r} or an @samp{s}
+
+@cindex trailing context, in patterns
+@item r/s
+an @samp{r} but only if it is followed by an @samp{s}.  The text matched by @samp{s} is
+included when determining whether this rule is the longest match, but is
+then returned to the input before the action is executed.  So the action
+only sees the text matched by @samp{r}.  This type of pattern is called
+@dfn{trailing context}.  (There are some combinations of @samp{r/s} that flex
+cannot match correctly. @xref{Limitations}, regarding dangerous trailing
+context.)
+
+@cindex beginning of line, in patterns
+@cindex BOL, in patterns
+@item ^r
+an @samp{r}, but only at the beginning of a line (i.e.,
+when just starting to scan, or right after a
+newline has been scanned).
+
+@cindex end of line, in patterns
+@cindex EOL, in patterns
+@item r$
+an @samp{r}, but only at the end of a line (i.e., just before a
+newline).  Equivalent to @samp{r/\n}.
+
+@cindex newline, matching in patterns
+Note that @code{flex}'s notion of ``newline'' is exactly
+whatever the C compiler used to compile @code{flex}
+interprets @samp{\n} as; in particular, on some DOS
+systems you must either filter out @samp{\r}s in the
+input yourself, or explicitly use @samp{r/\r\n} for @samp{r$}.
+
+@cindex start conditions, in patterns
+@item <s>r
+an @samp{r}, but only in start condition @code{s} (see @ref{Start
+Conditions} for discussion of start conditions).
+
+@item <s1,s2,s3>r
+same, but in any of start conditions @code{s1}, @code{s2}, or @code{s3}.
+
+@item <*>r
+an @samp{r} in any start condition, even an exclusive one.
+
+@cindex end of file, in patterns
+@cindex EOF in patterns, syntax of
+@item <<EOF>>
+an end-of-file.
+
+@item <s1,s2><<EOF>>
+an end-of-file when in start condition @code{s1} or @code{s2}
+@end table
+
+Note that inside of a character class, all regular expression operators
+lose their special meaning except escape (@samp{\}) and the character class
+operators, @samp{-}, @samp{]]}, and, at the beginning of the class, @samp{^}.
+
+@cindex patterns, precedence of operators
+The regular expressions listed above are grouped according to
+precedence, from highest precedence at the top to lowest at the bottom.
+Those grouped together have equal precedence (see special note on the
+precedence of the repeat operator, @samp{@{@}}, under the documentation
+for the @samp{--posix} POSIX compliance option).  For example,
+
+@cindex patterns, grouping and precedence
+@example
+@verbatim
+    foo|bar*
+@end verbatim
+@end example
+
+is the same as
+
+@example
+@verbatim
+    (foo)|(ba(r*))
+@end verbatim
+@end example
+
+since the @samp{*} operator has higher precedence than concatenation,
+and concatenation higher than alternation (@samp{|}).  This pattern
+therefore matches @emph{either} the string @samp{foo} @emph{or} the
+string @samp{ba} followed by zero-or-more @samp{r}'s.  To match
+@samp{foo} or zero-or-more repetitions of the string @samp{bar}, use:
+
+@example
+@verbatim
+    foo|(bar)*
+@end verbatim
+@end example
+
+And to match a sequence of zero or more repetitions of @samp{foo} and
+@samp{bar}:
+
+@cindex patterns, repetitions with grouping
+@example
+@verbatim
+    (foo|bar)*
+@end verbatim
+@end example
+
+@cindex character classes in patterns
+In addition to characters and ranges of characters, character classes
+can also contain @dfn{character class expressions}.  These are
+expressions enclosed inside @samp{[}: and @samp{:]} delimiters (which
+themselves must appear between the @samp{[} and @samp{]} of the
+character class. Other elements may occur inside the character class,
+too).  The valid expressions are:
+
+@cindex patterns, valid character classes
+@example
+@verbatim
+    [:alnum:] [:alpha:] [:blank:]
+    [:cntrl:] [:digit:] [:graph:]
+    [:lower:] [:print:] [:punct:]
+    [:space:] [:upper:] [:xdigit:]
+@end verbatim
+@end example
+
+These expressions all designate a set of characters equivalent to the
+corresponding standard C @code{isXXX} function.  For example,
+@samp{[:alnum:]} designates those characters for which @code{isalnum()}
+returns true - i.e., any alphabetic or numeric character.  Some systems
+don't provide @code{isblank()}, so flex defines @samp{[:blank:]} as a
+blank or a tab.
+
+For example, the following character classes are all equivalent:
+
+@cindex character classes, equivalence of
+@cindex patterns, character class equivalence
+@example
+@verbatim
+    [[:alnum:]]
+    [[:alpha:][:digit:]]
+    [[:alpha:][0-9]]
+    [a-zA-Z0-9]
+@end verbatim
+@end example
+
+A word of caution. Character classes are expanded immediately when seen in the @code{flex} input. 
+This means the character classes are sensitive to the locale in which @code{flex}
+is executed, and the resulting scanner will not be sensitive to the runtime locale.
+This may or may not be desirable.
+
+
+@itemize
+@cindex case-insensitive, effect on character classes
+@item If your scanner is case-insensitive (the @samp{-i} flag), then
+@samp{[:upper:]} and @samp{[:lower:]} are equivalent to
+@samp{[:alpha:]}.
+
+@anchor{case and character ranges}
+@item Character classes with ranges, such as @samp{[a-Z]}, should be used with
+caution in a case-insensitive scanner if the range spans upper or lowercase
+characters. Flex does not know if you want to fold all upper and lowercase
+characters together, or if you want the literal numeric range specified (with
+no case folding). When in doubt, flex will assume that you meant the literal
+numeric range, and will issue a warning. The exception to this rule is a
+character range such as @samp{[a-z]} or @samp{[S-W]} where it is obvious that you
+want case-folding to occur. Here are some examples with the @samp{-i} flag
+enabled:
+
+@multitable {@samp{[a-zA-Z]}} {ambiguous} {@samp{[A-Z\[\\\]_`a-t]}} {@samp{[@@A-Z\[\\\]_`abc]}}
+@item Range @tab Result @tab Literal Range @tab Alternate Range
+@item @samp{[a-t]} @tab ok @tab @samp{[a-tA-T]} @tab
+@item @samp{[A-T]} @tab ok @tab @samp{[a-tA-T]} @tab
+@item @samp{[A-t]} @tab ambiguous @tab @samp{[A-Z\[\\\]_`a-t]} @tab @samp{[a-tA-T]}
+@item @samp{[_-@{]} @tab ambiguous @tab @samp{[_`a-z@{]} @tab @samp{[_`a-zA-Z@{]}
+@item @samp{[@@-C]} @tab ambiguous @tab @samp{[@@ABC]} @tab @samp{[@@A-Z\[\\\]_`abc]}
+@end multitable
+
+@cindex end of line, in negated character classes
+@cindex EOL, in negated character classes
+@item
+A negated character class such as the example @samp{[^A-Z]} above
+@emph{will} match a newline unless @samp{\n} (or an equivalent escape
+sequence) is one of the characters explicitly present in the negated
+character class (e.g., @samp{[^A-Z\n]}).  This is unlike how many other
+regular expression tools treat negated character classes, but
+unfortunately the inconsistency is historically entrenched.  Matching
+newlines means that a pattern like @samp{[^"]*} can match the entire
+input unless there's another quote in the input.
+
+Flex allows negation of character class expressions by prepending @samp{^} to
+the POSIX character class name.
+
+@example
+@verbatim
+    [:^alnum:] [:^alpha:] [:^blank:]
+    [:^cntrl:] [:^digit:] [:^graph:]
+    [:^lower:] [:^print:] [:^punct:]
+    [:^space:] [:^upper:] [:^xdigit:]
+@end verbatim
+@end example
+
+Flex will issue a warning if the expressions @samp{[:^upper:]} and
+@samp{[:^lower:]} appear in a case-insensitive scanner, since their meaning is
+unclear. The current behavior is to skip them entirely, but this may change
+without notice in future revisions of flex.
+
+@item
+
+The @samp{@{-@}} operator computes the difference of two character classes. For
+example, @samp{[a-c]@{-@}[b-z]} represents all the characters in the class
+@samp{[a-c]} that are not in the class @samp{[b-z]} (which in this case, is
+just the single character @samp{a}). The @samp{@{-@}} operator is left
+associative, so @samp{[abc]@{-@}[b]@{-@}[c]} is the same as @samp{[a]}. Be careful
+not to accidentally create an empty set, which will never match.
+
+@item
+
+The @samp{@{+@}} operator computes the union of two character classes. For
+example, @samp{[a-z]@{+@}[0-9]} is the same as @samp{[a-z0-9]}. This operator
+is useful when preceded by the result of a difference operation, as in,
+@samp{[[:alpha:]]@{-@}[[:lower:]]@{+@}[q]}, which is equivalent to
+@samp{[A-Zq]} in the "C" locale.
+
+@cindex trailing context, limits of
+@cindex ^ as non-special character in patterns
+@cindex $ as normal character in patterns
+@item
+A rule can have at most one instance of trailing context (the @samp{/} operator
+or the @samp{$} operator).  The start condition, @samp{^}, and @samp{<<EOF>>} patterns
+can only occur at the beginning of a pattern, and, as well as with @samp{/} and @samp{$},
+cannot be grouped inside parentheses.  A @samp{^} which does not occur at
+the beginning of a rule or a @samp{$} which does not occur at the end of
+a rule loses its special properties and is treated as a normal character.
+
+@item
+The following are invalid:
+
+@cindex patterns, invalid trailing context
+@example
+@verbatim
+    foo/bar$
+    <sc1>foo<sc2>bar
+@end verbatim
+@end example
+
+Note that the first of these can be written @samp{foo/bar\n}.
+
+@item
+The following will result in @samp{$} or @samp{^} being treated as a normal character:
+
+@cindex patterns, special characters treated as non-special
+@example
+@verbatim
+    foo|(bar$)
+    foo|^bar
+@end verbatim
+@end example
+
+If the desired meaning is a @samp{foo} or a
+@samp{bar}-followed-by-a-newline, the following could be used (the
+special @code{|} action is explained below, @pxref{Actions}):
+
+@cindex patterns, end of line
+@example
+@verbatim
+    foo      |
+    bar$     /* action goes here */
+@end verbatim
+@end example
+
+A similar trick will work for matching a @samp{foo} or a
+@samp{bar}-at-the-beginning-of-a-line.
+@end itemize
+
+@node Matching, Actions, Patterns, Top
+@chapter How the Input Is Matched
+
+@cindex patterns, matching
+@cindex input, matching
+@cindex trailing context, matching
+@cindex matching, and trailing context
+@cindex matching, length of
+@cindex matching, multiple matches
+When the generated scanner is run, it analyzes its input looking for
+strings which match any of its patterns.  If it finds more than one
+match, it takes the one matching the most text (for trailing context
+rules, this includes the length of the trailing part, even though it
+will then be returned to the input).  If it finds two or more matches of
+the same length, the rule listed first in the @code{flex} input file is
+chosen.
+
+@cindex token
+@cindex yytext
+@cindex yyleng
+Once the match is determined, the text corresponding to the match
+(called the @dfn{token}) is made available in the global character
+pointer @code{yytext}, and its length in the global integer
+@code{yyleng}.  The @dfn{action} corresponding to the matched pattern is
+then executed (@pxref{Actions}), and then the remaining input is scanned
+for another match.
+
+@cindex default rule
+If no match is found, then the @dfn{default rule} is executed: the next
+character in the input is considered matched and copied to the standard
+output.  Thus, the simplest valid @code{flex} input is:
+
+@cindex minimal scanner
+@example
+@verbatim
+    %%
+@end verbatim
+@end example
+
+which generates a scanner that simply copies its input (one character at
+a time) to its output.
+
+@cindex yytext, two types of
+@cindex %array, use of
+@cindex %pointer, use of
+@vindex yytext
+Note that @code{yytext} can be defined in two different ways: either as
+a character @emph{pointer} or as a character @emph{array}. You can
+control which definition @code{flex} uses by including one of the
+special directives @code{%pointer} or @code{%array} in the first
+(definitions) section of your flex input.  The default is
+@code{%pointer}, unless you use the @samp{-l} lex compatibility option,
+in which case @code{yytext} will be an array.  The advantage of using
+@code{%pointer} is substantially faster scanning and no buffer overflow
+when matching very large tokens (unless you run out of dynamic memory).
+The disadvantage is that you are restricted in how your actions can
+modify @code{yytext} (@pxref{Actions}), and calls to the @code{unput()}
+function destroys the present contents of @code{yytext}, which can be a
+considerable porting headache when moving between different @code{lex}
+versions.
+
+@cindex %array, advantages of
+The advantage of @code{%array} is that you can then modify @code{yytext}
+to your heart's content, and calls to @code{unput()} do not destroy
+@code{yytext} (@pxref{Actions}).  Furthermore, existing @code{lex}
+programs sometimes access @code{yytext} externally using declarations of
+the form:
+
+@example
+@verbatim
+    extern char yytext[];
+@end verbatim
+@end example
+
+This definition is erroneous when used with @code{%pointer}, but correct
+for @code{%array}.
+
+The @code{%array} declaration defines @code{yytext} to be an array of
+@code{YYLMAX} characters, which defaults to a fairly large value.  You
+can change the size by simply #define'ing @code{YYLMAX} to a different
+value in the first section of your @code{flex} input.  As mentioned
+above, with @code{%pointer} yytext grows dynamically to accommodate
+large tokens.  While this means your @code{%pointer} scanner can
+accommodate very large tokens (such as matching entire blocks of
+comments), bear in mind that each time the scanner must resize
+@code{yytext} it also must rescan the entire token from the beginning,
+so matching such tokens can prove slow.  @code{yytext} presently does
+@emph{not} dynamically grow if a call to @code{unput()} results in too
+much text being pushed back; instead, a run-time error results.
+
+@cindex %array, with C++
+Also note that you cannot use @code{%array} with C++ scanner classes
+(@pxref{Cxx}).
+
+@node Actions, Generated Scanner, Matching, Top
+@chapter Actions
+
+@cindex actions
+Each pattern in a rule has a corresponding @dfn{action}, which can be
+any arbitrary C statement.  The pattern ends at the first non-escaped
+whitespace character; the remainder of the line is its action.  If the
+action is empty, then when the pattern is matched the input token is
+simply discarded.  For example, here is the specification for a program
+which deletes all occurrences of @samp{zap me} from its input:
+
+@cindex deleting lines from input
+@example
+@verbatim
+    %%
+    "zap me"
+@end verbatim
+@end example
+
+This example will copy all other characters in the input to the output
+since they will be matched by the default rule.
+
+Here is a program which compresses multiple blanks and tabs down to a
+single blank, and throws away whitespace found at the end of a line:
+
+@cindex whitespace, compressing
+@cindex compressing whitespace
+@example
+@verbatim
+    %%
+    [ \t]+        putchar( ' ' );
+    [ \t]+$       /* ignore this token */
+@end verbatim
+@end example
+
+@cindex %@{ and %@}, in Rules Section
+@cindex actions, use of @{ and @}
+@cindex actions, embedded C strings
+@cindex C-strings, in actions
+@cindex comments, in actions
+If the action contains a @samp{@{}, then the action spans till the
+balancing @samp{@}} is found, and the action may cross multiple lines.
+@code{flex} knows about C strings and comments and won't be fooled by
+braces found within them, but also allows actions to begin with
+@samp{%@{} and will consider the action to be all the text up to the
+next @samp{%@}} (regardless of ordinary braces inside the action).
+
+@cindex |, in actions
+An action consisting solely of a vertical bar (@samp{|}) means ``same as the
+action for the next rule''.  See below for an illustration.
+
+Actions can include arbitrary C code, including @code{return} statements
+to return a value to whatever routine called @code{yylex()}.  Each time
+@code{yylex()} is called it continues processing tokens from where it
+last left off until it either reaches the end of the file or executes a
+return.
+
+@cindex yytext, modification of
+Actions are free to modify @code{yytext} except for lengthening it
+(adding characters to its end--these will overwrite later characters in
+the input stream).  This however does not apply when using @code{%array}
+(@pxref{Matching}). In that case, @code{yytext} may be freely modified
+in any way.
+
+@cindex yyleng, modification of
+@cindex yymore, and yyleng
+Actions are free to modify @code{yyleng} except they should not do so if
+the action also includes use of @code{yymore()} (see below).
+
+@cindex preprocessor macros, for use in actions
+There are a number of special directives which can be included within an
+action:
+
+@table @code
+@item  ECHO
+@cindex ECHO
+copies yytext to the scanner's output.
+
+@item  BEGIN
+@cindex BEGIN
+followed by the name of a start condition places the scanner in the
+corresponding start condition (see below).
+
+@item  REJECT
+@cindex REJECT
+directs the scanner to proceed on to the ``second best'' rule which
+matched the input (or a prefix of the input).  The rule is chosen as
+described above in @ref{Matching}, and @code{yytext} and @code{yyleng}
+set up appropriately.  It may either be one which matched as much text
+as the originally chosen rule but came later in the @code{flex} input
+file, or one which matched less text.  For example, the following will
+both count the words in the input and call the routine @code{special()}
+whenever @samp{frob} is seen:
+
+@example
+@verbatim
+            int word_count = 0;
+    %%
+
+    frob        special(); REJECT;
+    [^ \t\n]+   ++word_count;
+@end verbatim
+@end example
+
+Without the @code{REJECT}, any occurrences of @samp{frob} in the input
+would not be counted as words, since the scanner normally executes only
+one action per token.  Multiple uses of @code{REJECT} are allowed, each
+one finding the next best choice to the currently active rule.  For
+example, when the following scanner scans the token @samp{abcd}, it will
+write @samp{abcdabcaba} to the output:
+
+@cindex REJECT, calling multiple times
+@cindex |, use of
+@example
+@verbatim
+    %%
+    a        |
+    ab       |
+    abc      |
+    abcd     ECHO; REJECT;
+    .|\n     /* eat up any unmatched character */
+@end verbatim
+@end example
+
+The first three rules share the fourth's action since they use the
+special @samp{|} action.
+
+@code{REJECT} is a particularly expensive feature in terms of scanner
+performance; if it is used in @emph{any} of the scanner's actions it
+will slow down @emph{all} of the scanner's matching.  Furthermore,
+@code{REJECT} cannot be used with the @samp{-Cf} or @samp{-CF} options
+(@pxref{Scanner Options}).
+
+Note also that unlike the other special actions, @code{REJECT} is a
+@emph{branch}.  Code immediately following it in the action will
+@emph{not} be executed.
+
+@item  yymore()
+@cindex yymore()
+tells the scanner that the next time it matches a rule, the
+corresponding token should be @emph{appended} onto the current value of
+@code{yytext} rather than replacing it.  For example, given the input
+@samp{mega-kludge} the following will write @samp{mega-mega-kludge} to
+the output:
+
+@cindex yymore(), mega-kludge
+@cindex yymore() to append token to previous token
+@example
+@verbatim
+    %%
+    mega-    ECHO; yymore();
+    kludge   ECHO;
+@end verbatim
+@end example
+
+First @samp{mega-} is matched and echoed to the output.  Then @samp{kludge}
+is matched, but the previous @samp{mega-} is still hanging around at the
+beginning of
+@code{yytext}
+so the
+@code{ECHO}
+for the @samp{kludge} rule will actually write @samp{mega-kludge}.
+@end table
+
+@cindex yymore, performance penalty of
+Two notes regarding use of @code{yymore()}.  First, @code{yymore()}
+depends on the value of @code{yyleng} correctly reflecting the size of
+the current token, so you must not modify @code{yyleng} if you are using
+@code{yymore()}.  Second, the presence of @code{yymore()} in the
+scanner's action entails a minor performance penalty in the scanner's
+matching speed.
+
+@cindex yyless()
+@code{yyless(n)} returns all but the first @code{n} characters of the
+current token back to the input stream, where they will be rescanned
+when the scanner looks for the next match.  @code{yytext} and
+@code{yyleng} are adjusted appropriately (e.g., @code{yyleng} will now
+be equal to @code{n}).  For example, on the input @samp{foobar} the
+following will write out @samp{foobarbar}:
+
+@cindex yyless(), pushing back characters
+@cindex pushing back characters with yyless
+@example
+@verbatim
+    %%
+    foobar    ECHO; yyless(3);
+    [a-z]+    ECHO;
+@end verbatim
+@end example
+
+An argument of 0 to @code{yyless()} will cause the entire current input
+string to be scanned again.  Unless you've changed how the scanner will
+subsequently process its input (using @code{BEGIN}, for example), this
+will result in an endless loop.
+
+Note that @code{yyless()} is a macro and can only be used in the flex
+input file, not from other source files.
+
+@cindex unput()
+@cindex pushing back characters with unput
+@code{unput(c)} puts the character @code{c} back onto the input stream.
+It will be the next character scanned.  The following action will take
+the current token and cause it to be rescanned enclosed in parentheses.
+
+@cindex unput(), pushing back characters
+@cindex pushing back characters with unput()
+@example
+@verbatim
+    {
+    int i;
+    /* Copy yytext because unput() trashes yytext */
+    char *yycopy = strdup( yytext );
+    unput( ')' );
+    for ( i = yyleng - 1; i >= 0; --i )
+        unput( yycopy[i] );
+    unput( '(' );
+    free( yycopy );
+    }
+@end verbatim
+@end example
+
+Note that since each @code{unput()} puts the given character back at the
+@emph{beginning} of the input stream, pushing back strings must be done
+back-to-front.
+
+@cindex %pointer, and unput()
+@cindex unput(), and %pointer
+An important potential problem when using @code{unput()} is that if you
+are using @code{%pointer} (the default), a call to @code{unput()}
+@emph{destroys} the contents of @code{yytext}, starting with its
+rightmost character and devouring one character to the left with each
+call.  If you need the value of @code{yytext} preserved after a call to
+@code{unput()} (as in the above example), you must either first copy it
+elsewhere, or build your scanner using @code{%array} instead
+(@pxref{Matching}).
+
+@cindex pushing back EOF
+@cindex EOF, pushing back
+Finally, note that you cannot put back @samp{EOF} to attempt to mark the
+input stream with an end-of-file.
+
+@cindex input()
+@code{input()} reads the next character from the input stream.  For
+example, the following is one way to eat up C comments:
+
+@cindex comments, discarding
+@cindex discarding C comments
+@example
+@verbatim
+    %%
+    "/*"        {
+                register int c;
+
+                for ( ; ; )
+                    {
+                    while ( (c = input()) != '*' &&
+                            c != EOF )
+                        ;    /* eat up text of comment */
+
+                    if ( c == '*' )
+                        {
+                        while ( (c = input()) == '*' )
+                            ;
+                        if ( c == '/' )
+                            break;    /* found the end */
+                        }
+
+                    if ( c == EOF )
+                        {
+                        error( "EOF in comment" );
+                        break;
+                        }
+                    }
+                }
+@end verbatim
+@end example
+
+@cindex input(), and C++
+@cindex yyinput()
+(Note that if the scanner is compiled using @code{C++}, then
+@code{input()} is instead referred to as @b{yyinput()}, in order to
+avoid a name clash with the @code{C++} stream by the name of
+@code{input}.)
+
+@cindex flushing the internal buffer
+@cindex YY_FLUSH_BUFFER
+@code{YY_FLUSH_BUFFER;} flushes the scanner's internal buffer so that
+the next time the scanner attempts to match a token, it will first
+refill the buffer using @code{YY_INPUT()} (@pxref{Generated Scanner}).
+This action is a special case of the more general
+@code{yy_flush_buffer;} function, described below (@pxref{Multiple
+Input Buffers})
+
+@cindex yyterminate()
+@cindex terminating with yyterminate()
+@cindex exiting with yyterminate()
+@cindex halting with yyterminate()
+@code{yyterminate()} can be used in lieu of a return statement in an
+action.  It terminates the scanner and returns a 0 to the scanner's
+caller, indicating ``all done''.  By default, @code{yyterminate()} is
+also called when an end-of-file is encountered.  It is a macro and may
+be redefined.
+
+@node Generated Scanner, Start Conditions, Actions, Top
+@chapter The Generated Scanner
+
+@cindex yylex(), in generated scanner
+The output of @code{flex} is the file @file{lex.yy.c}, which contains
+the scanning routine @code{yylex()}, a number of tables used by it for
+matching tokens, and a number of auxiliary routines and macros.  By
+default, @code{yylex()} is declared as follows:
+
+@example
+@verbatim
+    int yylex()
+        {
+        ... various definitions and the actions in here ...
+        }
+@end verbatim
+@end example
+
+@cindex yylex(), overriding
+(If your environment supports function prototypes, then it will be
+@code{int yylex( void )}.)  This definition may be changed by defining
+the @code{YY_DECL} macro.  For example, you could use:
+
+@cindex yylex, overriding the prototype of
+@example
+@verbatim
+    #define YY_DECL float lexscan( a, b ) float a, b;
+@end verbatim
+@end example
+
+to give the scanning routine the name @code{lexscan}, returning a float,
+and taking two floats as arguments.  Note that if you give arguments to
+the scanning routine using a K&R-style/non-prototyped function
+declaration, you must terminate the definition with a semi-colon (;).
+
+@code{flex} generates @samp{C99} function definitions by
+default. However flex does have the ability to generate obsolete, er,
+@samp{traditional}, function definitions. This is to support
+bootstrapping gcc on old systems.  Unfortunately, traditional
+definitions prevent us from using any standard data types smaller than
+int (such as short, char, or bool) as function arguments.  For this
+reason, future versions of @code{flex} may generate standard C99 code
+only, leaving K&R-style functions to the historians.  Currently, if you
+do @strong{not} want @samp{C99} definitions, then you must use 
+@code{%option noansi-definitions}.
+
+@cindex stdin, default for yyin
+@cindex yyin
+Whenever @code{yylex()} is called, it scans tokens from the global input
+file @file{yyin} (which defaults to stdin).  It continues until it
+either reaches an end-of-file (at which point it returns the value 0) or
+one of its actions executes a @code{return} statement.
+
+@cindex EOF and yyrestart()
+@cindex end-of-file, and yyrestart()
+@cindex yyrestart()
+If the scanner reaches an end-of-file, subsequent calls are undefined
+unless either @file{yyin} is pointed at a new input file (in which case
+scanning continues from that file), or @code{yyrestart()} is called.
+@code{yyrestart()} takes one argument, a @code{FILE *} pointer (which
+can be NULL, if you've set up @code{YY_INPUT} to scan from a source other
+than @code{yyin}), and initializes @file{yyin} for scanning from that
+file.  Essentially there is no difference between just assigning
+@file{yyin} to a new input file or using @code{yyrestart()} to do so;
+the latter is available for compatibility with previous versions of
+@code{flex}, and because it can be used to switch input files in the
+middle of scanning.  It can also be used to throw away the current input
+buffer, by calling it with an argument of @file{yyin}; but it would be
+better to use @code{YY_FLUSH_BUFFER} (@pxref{Actions}).  Note that
+@code{yyrestart()} does @emph{not} reset the start condition to
+@code{INITIAL} (@pxref{Start Conditions}).
+
+@cindex RETURN, within actions
+If @code{yylex()} stops scanning due to executing a @code{return}
+statement in one of the actions, the scanner may then be called again
+and it will resume scanning where it left off.
+
+@cindex YY_INPUT
+By default (and for purposes of efficiency), the scanner uses
+block-reads rather than simple @code{getc()} calls to read characters
+from @file{yyin}.  The nature of how it gets its input can be controlled
+by defining the @code{YY_INPUT} macro.  The calling sequence for
+@code{YY_INPUT()} is @code{YY_INPUT(buf,result,max_size)}.  Its action
+is to place up to @code{max_size} characters in the character array
+@code{buf} and return in the integer variable @code{result} either the
+number of characters read or the constant @code{YY_NULL} (0 on Unix
+systems) to indicate @samp{EOF}.  The default @code{YY_INPUT} reads from
+the global file-pointer @file{yyin}.
+
+@cindex YY_INPUT, overriding
+Here is a sample definition of @code{YY_INPUT} (in the definitions
+section of the input file):
+
+@example
+@verbatim
+    %{
+    #define YY_INPUT(buf,result,max_size) \
+        { \
+        int c = getchar(); \
+        result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
+        }
+    %}
+@end verbatim
+@end example
+
+This definition will change the input processing to occur one character
+at a time.
+
+@cindex yywrap()
+When the scanner receives an end-of-file indication from YY_INPUT, it
+then checks the @code{yywrap()} function.  If @code{yywrap()} returns
+false (zero), then it is assumed that the function has gone ahead and
+set up @file{yyin} to point to another input file, and scanning
+continues.  If it returns true (non-zero), then the scanner terminates,
+returning 0 to its caller.  Note that in either case, the start
+condition remains unchanged; it does @emph{not} revert to
+@code{INITIAL}.
+
+@cindex yywrap, default for
+@cindex noyywrap, %option
+@cindex %option noyywrapp
+If you do not supply your own version of @code{yywrap()}, then you must
+either use @code{%option noyywrap} (in which case the scanner behaves as
+though @code{yywrap()} returned 1), or you must link with @samp{-lfl} to
+obtain the default version of the routine, which always returns 1.
+
+For scanning from in-memory buffers (e.g., scanning strings), see
+@ref{Scanning Strings}. @xref{Multiple Input Buffers}.
+
+@cindex ECHO, and yyout
+@cindex yyout
+@cindex stdout, as default for yyout
+The scanner writes its @code{ECHO} output to the @file{yyout} global
+(default, @file{stdout}), which may be redefined by the user simply by
+assigning it to some other @code{FILE} pointer.
+
+@node Start Conditions, Multiple Input Buffers, Generated Scanner, Top
+@chapter Start Conditions
+
+@cindex start conditions
+@code{flex} provides a mechanism for conditionally activating rules.
+Any rule whose pattern is prefixed with @samp{<sc>} will only be active
+when the scanner is in the @dfn{start condition} named @code{sc}.  For
+example,
+
+@c proofread edit stopped here
+@example
+@verbatim
+    <STRING>[^"]*        { /* eat up the string body ... */
+                ...
+                }
+@end verbatim
+@end example
+
+will be active only when the scanner is in the @code{STRING} start
+condition, and
+
+@cindex start conditions, multiple
+@example
+@verbatim
+    <INITIAL,STRING,QUOTE>\.        { /* handle an escape ... */
+                ...
+                }
+@end verbatim
+@end example
+
+will be active only when the current start condition is either
+@code{INITIAL}, @code{STRING}, or @code{QUOTE}.
+
+@cindex start conditions, inclusive v.s.@: exclusive
+Start conditions are declared in the definitions (first) section of the
+input using unindented lines beginning with either @samp{%s} or
+@samp{%x} followed by a list of names.  The former declares
+@dfn{inclusive} start conditions, the latter @dfn{exclusive} start
+conditions.  A start condition is activated using the @code{BEGIN}
+action.  Until the next @code{BEGIN} action is executed, rules with the
+given start condition will be active and rules with other start
+conditions will be inactive.  If the start condition is inclusive, then
+rules with no start conditions at all will also be active.  If it is
+exclusive, then @emph{only} rules qualified with the start condition
+will be active.  A set of rules contingent on the same exclusive start
+condition describe a scanner which is independent of any of the other
+rules in the @code{flex} input.  Because of this, exclusive start
+conditions make it easy to specify ``mini-scanners'' which scan portions
+of the input that are syntactically different from the rest (e.g.,
+comments).
+
+If the distinction between inclusive and exclusive start conditions
+is still a little vague, here's a simple example illustrating the
+connection between the two.  The set of rules:
+
+@cindex start conditions, inclusive
+@example
+@verbatim
+    %s example
+    %%
+
+    <example>foo   do_something();
+
+    bar            something_else();
+@end verbatim
+@end example
+
+is equivalent to
+
+@cindex start conditions, exclusive
+@example
+@verbatim
+    %x example
+    %%
+
+    <example>foo   do_something();
+
+    <INITIAL,example>bar    something_else();
+@end verbatim
+@end example
+
+Without the @code{<INITIAL,example>} qualifier, the @code{bar} pattern in
+the second example wouldn't be active (i.e., couldn't match) when in
+start condition @code{example}.  If we just used @code{<example>} to
+qualify @code{bar}, though, then it would only be active in
+@code{example} and not in @code{INITIAL}, while in the first example
+it's active in both, because in the first example the @code{example}
+start condition is an inclusive @code{(%s)} start condition.
+
+@cindex start conditions, special wildcard condition
+Also note that the special start-condition specifier
+@code{<*>}
+matches every start condition.  Thus, the above example could also
+have been written:
+
+@cindex start conditions, use of wildcard condition (<*>)
+@example
+@verbatim
+    %x example
+    %%
+
+    <example>foo   do_something();
+
+    <*>bar    something_else();
+@end verbatim
+@end example
+
+The default rule (to @code{ECHO} any unmatched character) remains active
+in start conditions.  It is equivalent to:
+
+@cindex start conditions, behavior of default rule
+@example
+@verbatim
+    <*>.|\n     ECHO;
+@end verbatim
+@end example
+
+@cindex BEGIN, explanation
+@findex BEGIN
+@vindex INITIAL
+@code{BEGIN(0)} returns to the original state where only the rules with
+no start conditions are active.  This state can also be referred to as
+the start-condition @code{INITIAL}, so @code{BEGIN(INITIAL)} is
+equivalent to @code{BEGIN(0)}.  (The parentheses around the start
+condition name are not required but are considered good style.)
+
+@code{BEGIN} actions can also be given as indented code at the beginning
+of the rules section.  For example, the following will cause the scanner
+to enter the @code{SPECIAL} start condition whenever @code{yylex()} is
+called and the global variable @code{enter_special} is true:
+
+@cindex start conditions, using BEGIN
+@example
+@verbatim
+            int enter_special;
+
+    %x SPECIAL
+    %%
+            if ( enter_special )
+                BEGIN(SPECIAL);
+
+    <SPECIAL>blahblahblah
+    ...more rules follow...
+@end verbatim
+@end example
+
+To illustrate the uses of start conditions, here is a scanner which
+provides two different interpretations of a string like @samp{123.456}.
+By default it will treat it as three tokens, the integer @samp{123}, a
+dot (@samp{.}), and the integer @samp{456}.  But if the string is
+preceded earlier in the line by the string @samp{expect-floats} it will
+treat it as a single token, the floating-point number @samp{123.456}:
+
+@cindex start conditions, for different interpretations of same input
+@example
+@verbatim
+    %{
+    #include <math.h>
+    %}
+    %s expect
+
+    %%
+    expect-floats        BEGIN(expect);
+
+    <expect>[0-9]+.[0-9]+      {
+                printf( "found a float, = %f\n",
+                        atof( yytext ) );
+                }
+    <expect>\n           {
+                /* that's the end of the line, so
+                 * we need another "expect-number"
+                 * before we'll recognize any more
+                 * numbers
+                 */
+                BEGIN(INITIAL);
+                }
+
+    [0-9]+      {
+                printf( "found an integer, = %d\n",
+                        atoi( yytext ) );
+                }
+
+    "."         printf( "found a dot\n" );
+@end verbatim
+@end example
+
+@cindex comments, example of scanning C comments
+Here is a scanner which recognizes (and discards) C comments while
+maintaining a count of the current input line.
+
+@cindex recognizing C comments
+@example
+@verbatim
+    %x comment
+    %%
+            int line_num = 1;
+
+    "/*"         BEGIN(comment);
+
+    <comment>[^*\n]*        /* eat anything that's not a '*' */
+    <comment>"*"+[^*/\n]*   /* eat up '*'s not followed by '/'s */
+    <comment>\n             ++line_num;
+    <comment>"*"+"/"        BEGIN(INITIAL);
+@end verbatim
+@end example
+
+This scanner goes to a bit of trouble to match as much
+text as possible with each rule.  In general, when attempting to write
+a high-speed scanner try to match as much possible in each rule, as
+it's a big win.
+
+Note that start-conditions names are really integer values and
+can be stored as such.  Thus, the above could be extended in the
+following fashion:
+
+@cindex start conditions, integer values
+@cindex using integer values of start condition names
+@example
+@verbatim
+    %x comment foo
+    %%
+            int line_num = 1;
+            int comment_caller;
+
+    "/*"         {
+                 comment_caller = INITIAL;
+                 BEGIN(comment);
+                 }
+
+    ...
+
+    <foo>"/*"    {
+                 comment_caller = foo;
+                 BEGIN(comment);
+                 }
+
+    <comment>[^*\n]*        /* eat anything that's not a '*' */
+    <comment>"*"+[^*/\n]*   /* eat up '*'s not followed by '/'s */
+    <comment>\n             ++line_num;
+    <comment>"*"+"/"        BEGIN(comment_caller);
+@end verbatim
+@end example
+
+@cindex YY_START, example
+Furthermore, you can access the current start condition using the
+integer-valued @code{YY_START} macro.  For example, the above
+assignments to @code{comment_caller} could instead be written
+
+@cindex getting current start state with YY_START
+@example
+@verbatim
+    comment_caller = YY_START;
+@end verbatim
+@end example
+
+@vindex YY_START
+Flex provides @code{YYSTATE} as an alias for @code{YY_START} (since that
+is what's used by AT&T @code{lex}).
+
+For historical reasons, start conditions do not have their own
+name-space within the generated scanner. The start condition names are
+unmodified in the generated scanner and generated header.
+@xref{option-header}. @xref{option-prefix}.
+
+
+
+Finally, here's an example of how to match C-style quoted strings using
+exclusive start conditions, including expanded escape sequences (but
+not including checking for a string that's too long):
+
+@cindex matching C-style double-quoted strings
+@example
+@verbatim
+    %x str
+
+    %%
+            char string_buf[MAX_STR_CONST];
+            char *string_buf_ptr;
+
+
+    \"      string_buf_ptr = string_buf; BEGIN(str);
+
+    <str>\"        { /* saw closing quote - all done */
+            BEGIN(INITIAL);
+            *string_buf_ptr = '\0';
+            /* return string constant token type and
+             * value to parser
+             */
+            }
+
+    <str>\n        {
+            /* error - unterminated string constant */
+            /* generate error message */
+            }
+
+    <str>\\[0-7]{1,3} {
+            /* octal escape sequence */
+            int result;
+
+            (void) sscanf( yytext + 1, "%o", &result );
+
+            if ( result > 0xff )
+                    /* error, constant is out-of-bounds */
+
+            *string_buf_ptr++ = result;
+            }
+
+    <str>\\[0-9]+ {
+            /* generate error - bad escape sequence; something
+             * like '\48' or '\0777777'
+             */
+            }
+
+    <str>\\n  *string_buf_ptr++ = '\n';
+    <str>\\t  *string_buf_ptr++ = '\t';
+    <str>\\r  *string_buf_ptr++ = '\r';
+    <str>\\b  *string_buf_ptr++ = '\b';
+    <str>\\f  *string_buf_ptr++ = '\f';
+
+    <str>\\(.|\n)  *string_buf_ptr++ = yytext[1];
+
+    <str>[^\\\n\"]+        {
+            char *yptr = yytext;
+
+            while ( *yptr )
+                    *string_buf_ptr++ = *yptr++;
+            }
+@end verbatim
+@end example
+
+@cindex start condition, applying to multiple patterns
+Often, such as in some of the examples above, you wind up writing a
+whole bunch of rules all preceded by the same start condition(s).  Flex
+makes this a little easier and cleaner by introducing a notion of start
+condition @dfn{scope}.  A start condition scope is begun with:
+
+@example
+@verbatim
+    <SCs>{
+@end verbatim
+@end example
+
+where @code{SCs} is a list of one or more start conditions.  Inside the
+start condition scope, every rule automatically has the prefix
+@code{SCs>} applied to it, until a @samp{@}} which matches the initial
+@samp{@{}.  So, for example,
+
+@cindex extended scope of start conditions
+@example
+@verbatim
+    <ESC>{
+        "\\n"   return '\n';
+        "\\r"   return '\r';
+        "\\f"   return '\f';
+        "\\0"   return '\0';
+    }
+@end verbatim
+@end example
+
+is equivalent to:
+
+@example
+@verbatim
+    <ESC>"\\n"  return '\n';
+    <ESC>"\\r"  return '\r';
+    <ESC>"\\f"  return '\f';
+    <ESC>"\\0"  return '\0';
+@end verbatim
+@end example
+
+Start condition scopes may be nested.
+
+@cindex stacks, routines for manipulating
+@cindex start conditions, use of a stack
+
+The following routines are available for manipulating stacks of start conditions:
+
+@deftypefun  void yy_push_state ( int @code{new_state} )
+pushes the current start condition onto the top of the start condition
+stack and switches to
+@code{new_state}
+as though you had used
+@code{BEGIN new_state}
+(recall that start condition names are also integers).
+@end deftypefun
+
+@deftypefun void yy_pop_state ()
+pops the top of the stack and switches to it via
+@code{BEGIN}.
+@end deftypefun
+
+@deftypefun int yy_top_state ()
+returns the top of the stack without altering the stack's contents.
+@end deftypefun
+
+@cindex memory, for start condition stacks
+The start condition stack grows dynamically and so has no built-in size
+limitation.  If memory is exhausted, program execution aborts.
+
+To use start condition stacks, your scanner must include a @code{%option
+stack} directive (@pxref{Scanner Options}).
+
+@node Multiple Input Buffers, EOF, Start Conditions, Top
+@chapter Multiple Input Buffers
+
+@cindex multiple input streams
+Some scanners (such as those which support ``include'' files) require
+reading from several input streams.  As @code{flex} scanners do a large
+amount of buffering, one cannot control where the next input will be
+read from by simply writing a @code{YY_INPUT()} which is sensitive to
+the scanning context.  @code{YY_INPUT()} is only called when the scanner
+reaches the end of its buffer, which may be a long time after scanning a
+statement such as an @code{include} statement which requires switching
+the input source.
+
+To negotiate these sorts of problems, @code{flex} provides a mechanism
+for creating and switching between multiple input buffers.  An input
+buffer is created by using:
+
+@cindex memory, allocating input buffers
+@deftypefun YY_BUFFER_STATE yy_create_buffer ( FILE *file, int size )
+@end deftypefun
+
+which takes a @code{FILE} pointer and a size and creates a buffer
+associated with the given file and large enough to hold @code{size}
+characters (when in doubt, use @code{YY_BUF_SIZE} for the size).  It
+returns a @code{YY_BUFFER_STATE} handle, which may then be passed to
+other routines (see below).
+@tindex YY_BUFFER_STATE
+The @code{YY_BUFFER_STATE} type is a
+pointer to an opaque @code{struct yy_buffer_state} structure, so you may
+safely initialize @code{YY_BUFFER_STATE} variables to @code{((YY_BUFFER_STATE)
+0)} if you wish, and also refer to the opaque structure in order to
+correctly declare input buffers in source files other than that of your
+scanner.  Note that the @code{FILE} pointer in the call to
+@code{yy_create_buffer} is only used as the value of @file{yyin} seen by
+@code{YY_INPUT}.  If you redefine @code{YY_INPUT()} so it no longer uses
+@file{yyin}, then you can safely pass a NULL @code{FILE} pointer to
+@code{yy_create_buffer}.  You select a particular buffer to scan from
+using:
+
+@deftypefun void yy_switch_to_buffer ( YY_BUFFER_STATE new_buffer )
+@end deftypefun
+
+The above function switches the scanner's input buffer so subsequent tokens
+will come from @code{new_buffer}.  Note that @code{yy_switch_to_buffer()} may
+be used by @code{yywrap()} to set things up for continued scanning, instead of
+opening a new file and pointing @file{yyin} at it. If you are looking for a
+stack of input buffers, then you want to use @code{yypush_buffer_state()}
+instead of this function. Note also that switching input sources via either
+@code{yy_switch_to_buffer()} or @code{yywrap()} does @emph{not} change the
+start condition.
+
+@cindex memory, deleting input buffers
+@deftypefun void yy_delete_buffer ( YY_BUFFER_STATE buffer )
+@end deftypefun
+
+is used to reclaim the storage associated with a buffer.  (@code{buffer}
+can be NULL, in which case the routine does nothing.)  You can also clear
+the current contents of a buffer using:
+
+@cindex pushing an input buffer
+@cindex stack, input buffer push
+@deftypefun void yypush_buffer_state ( YY_BUFFER_STATE buffer )
+@end deftypefun
+
+This function pushes the new buffer state onto an internal stack. The pushed
+state becomes the new current state. The stack is maintained by flex and will
+grow as required. This function is intended to be used instead of
+@code{yy_switch_to_buffer}, when you want to change states, but preserve the
+current state for later use. 
+
+@cindex popping an input buffer
+@cindex stack, input buffer pop
+@deftypefun void yypop_buffer_state ( )
+@end deftypefun
+
+This function removes the current state from the top of the stack, and deletes
+it by calling @code{yy_delete_buffer}.  The next state on the stack, if any,
+becomes the new current state.
+
+@cindex clearing an input buffer
+@cindex flushing an input buffer
+@deftypefun void yy_flush_buffer ( YY_BUFFER_STATE buffer )
+@end deftypefun
+
+This function discards the buffer's contents,
+so the next time the scanner attempts to match a token from the
+buffer, it will first fill the buffer anew using
+@code{YY_INPUT()}.
+
+@deftypefun YY_BUFFER_STATE yy_new_buffer ( FILE *file, int size )
+@end deftypefun
+
+is an alias for @code{yy_create_buffer()},
+provided for compatibility with the C++ use of @code{new} and
+@code{delete} for creating and destroying dynamic objects.
+
+@cindex YY_CURRENT_BUFFER, and multiple buffers Finally, the macro
+@code{YY_CURRENT_BUFFER} macro returns a @code{YY_BUFFER_STATE} handle to the
+current buffer. It should not be used as an lvalue.
+
+@cindex EOF, example using multiple input buffers
+Here are two examples of using these features for writing a scanner
+which expands include files (the
+@code{<<EOF>>}
+feature is discussed below).
+
+This first example uses yypush_buffer_state and yypop_buffer_state. Flex
+maintains the stack internally.
+
+@cindex handling include files with multiple input buffers
+@example
+@verbatim
+    /* the "incl" state is used for picking up the name
+     * of an include file
+     */
+    %x incl
+    %%
+    include             BEGIN(incl);
+
+    [a-z]+              ECHO;
+    [^a-z\n]*\n?        ECHO;
+
+    <incl>[ \t]*      /* eat the whitespace */
+    <incl>[^ \t\n]+   { /* got the include file name */
+            yyin = fopen( yytext, "r" );
+
+            if ( ! yyin )
+                error( ... );
+
+			yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE ));
+
+            BEGIN(INITIAL);
+            }
+
+    <<EOF>> {
+			yypop_buffer_state();
+
+            if ( !YY_CURRENT_BUFFER )
+                {
+                yyterminate();
+                }
+            }
+@end verbatim
+@end example
+
+The second example, below, does the same thing as the previous example did, but
+manages its own input buffer stack manually (instead of letting flex do it).
+
+@cindex handling include files with multiple input buffers
+@example
+@verbatim
+    /* the "incl" state is used for picking up the name
+     * of an include file
+     */
+    %x incl
+
+    %{
+    #define MAX_INCLUDE_DEPTH 10
+    YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
+    int include_stack_ptr = 0;
+    %}
+
+    %%
+    include             BEGIN(incl);
+
+    [a-z]+              ECHO;
+    [^a-z\n]*\n?        ECHO;
+
+    <incl>[ \t]*      /* eat the whitespace */
+    <incl>[^ \t\n]+   { /* got the include file name */
+            if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
+                {
+                fprintf( stderr, "Includes nested too deeply" );
+                exit( 1 );
+                }
+
+            include_stack[include_stack_ptr++] =
+                YY_CURRENT_BUFFER;
+
+            yyin = fopen( yytext, "r" );
+
+            if ( ! yyin )
+                error( ... );
+
+            yy_switch_to_buffer(
+                yy_create_buffer( yyin, YY_BUF_SIZE ) );
+
+            BEGIN(INITIAL);
+            }
+
+    <<EOF>> {
+            if ( --include_stack_ptr  0 )
+                {
+                yyterminate();
+                }
+
+            else
+                {
+                yy_delete_buffer( YY_CURRENT_BUFFER );
+                yy_switch_to_buffer(
+                     include_stack[include_stack_ptr] );
+                }
+            }
+@end verbatim
+@end example
+
+@anchor{Scanning Strings}
+@cindex strings, scanning strings instead of files
+The following routines are available for setting up input buffers for
+scanning in-memory strings instead of files.  All of them create a new
+input buffer for scanning the string, and return a corresponding
+@code{YY_BUFFER_STATE} handle (which you should delete with
+@code{yy_delete_buffer()} when done with it).  They also switch to the
+new buffer using @code{yy_switch_to_buffer()}, so the next call to
+@code{yylex()} will start scanning the string.
+
+@deftypefun YY_BUFFER_STATE yy_scan_string ( const char *str )
+scans a NUL-terminated string.
+@end deftypefun
+
+@deftypefun YY_BUFFER_STATE yy_scan_bytes ( const char *bytes, int len )
+scans @code{len} bytes (including possibly @code{NUL}s) starting at location
+@code{bytes}.
+@end deftypefun
+
+Note that both of these functions create and scan a @emph{copy} of the
+string or bytes.  (This may be desirable, since @code{yylex()} modifies
+the contents of the buffer it is scanning.)  You can avoid the copy by
+using:
+
+@vindex YY_END_OF_BUFFER_CHAR
+@deftypefun YY_BUFFER_STATE yy_scan_buffer (char *base, yy_size_t size)
+which scans in place the buffer starting at @code{base}, consisting of
+@code{size} bytes, the last two bytes of which @emph{must} be
+@code{YY_END_OF_BUFFER_CHAR} (ASCII NUL).  These last two bytes are not
+scanned; thus, scanning consists of @code{base[0]} through
+@code{base[size-2]}, inclusive.
+@end deftypefun
+
+If you fail to set up @code{base} in this manner (i.e., forget the final
+two @code{YY_END_OF_BUFFER_CHAR} bytes), then @code{yy_scan_buffer()}
+returns a NULL pointer instead of creating a new input buffer.
+
+@deftp  {Data type} yy_size_t
+is an integral type to which you can cast an integer expression
+reflecting the size of the buffer.
+@end deftp
+
+@node EOF, Misc Macros, Multiple Input Buffers, Top
+@chapter End-of-File Rules
+
+@cindex EOF, explanation
+The special rule @code{<<EOF>>} indicates
+actions which are to be taken when an end-of-file is
+encountered and @code{yywrap()} returns non-zero (i.e., indicates
+no further files to process).  The action must finish
+by doing one of the following things:
+
+@itemize
+@item
+@findex YY_NEW_FILE  (now obsolete)
+assigning @file{yyin} to a new input file (in previous versions of
+@code{flex}, after doing the assignment you had to call the special
+action @code{YY_NEW_FILE}.  This is no longer necessary.)
+
+@item
+executing a @code{return} statement;
+
+@item
+executing the special @code{yyterminate()} action.
+
+@item
+or, switching to a new buffer using @code{yy_switch_to_buffer()} as
+shown in the example above.
+@end itemize
+
+<<EOF>> rules may not be used with other patterns; they may only be
+qualified with a list of start conditions.  If an unqualified <<EOF>>
+rule is given, it applies to @emph{all} start conditions which do not
+already have <<EOF>> actions.  To specify an <<EOF>> rule for only the
+initial start condition, use:
+
+@example
+@verbatim
+    <INITIAL><<EOF>>
+@end verbatim
+@end example
+
+These rules are useful for catching things like unclosed comments.  An
+example:
+
+@cindex <<EOF>>, use of
+@example
+@verbatim
+    %x quote
+    %%
+
+    ...other rules for dealing with quotes...
+
+    <quote><<EOF>>   {
+             error( "unterminated quote" );
+             yyterminate();
+             }
+   <<EOF>>  {
+             if ( *++filelist )
+                 yyin = fopen( *filelist, "r" );
+             else
+                yyterminate();
+             }
+@end verbatim
+@end example
+
+@node Misc Macros, User Values, EOF, Top
+@chapter Miscellaneous Macros
+
+@hkindex YY_USER_ACTION
+The macro @code{YY_USER_ACTION} can be defined to provide an action
+which is always executed prior to the matched rule's action.  For
+example, it could be #define'd to call a routine to convert yytext to
+lower-case.  When @code{YY_USER_ACTION} is invoked, the variable
+@code{yy_act} gives the number of the matched rule (rules are numbered
+starting with 1).  Suppose you want to profile how often each of your
+rules is matched.  The following would do the trick:
+
+@cindex YY_USER_ACTION to track each time a rule is matched
+@example
+@verbatim
+    #define YY_USER_ACTION ++ctr[yy_act]
+@end verbatim
+@end example
+
+@vindex YY_NUM_RULES
+where @code{ctr} is an array to hold the counts for the different rules.
+Note that the macro @code{YY_NUM_RULES} gives the total number of rules
+(including the default rule), even if you use @samp{-s)}, so a correct
+declaration for @code{ctr} is:
+
+@example
+@verbatim
+    int ctr[YY_NUM_RULES];
+@end verbatim
+@end example
+
+@hkindex YY_USER_INIT
+The macro @code{YY_USER_INIT} may be defined to provide an action which
+is always executed before the first scan (and before the scanner's
+internal initializations are done).  For example, it could be used to
+call a routine to read in a data table or open a logging file.
+
+@findex yy_set_interactive
+The macro @code{yy_set_interactive(is_interactive)} can be used to
+control whether the current buffer is considered @dfn{interactive}.  An
+interactive buffer is processed more slowly, but must be used when the
+scanner's input source is indeed interactive to avoid problems due to
+waiting to fill buffers (see the discussion of the @samp{-I} flag in
+@ref{Scanner Options}).  A non-zero value in the macro invocation marks
+the buffer as interactive, a zero value as non-interactive.  Note that
+use of this macro overrides @code{%option always-interactive} or
+@code{%option never-interactive} (@pxref{Scanner Options}).
+@code{yy_set_interactive()} must be invoked prior to beginning to scan
+the buffer that is (or is not) to be considered interactive.
+
+@cindex BOL, setting it
+@findex yy_set_bol
+The macro @code{yy_set_bol(at_bol)} can be used to control whether the
+current buffer's scanning context for the next token match is done as
+though at the beginning of a line.  A non-zero macro argument makes
+rules anchored with @samp{^} active, while a zero argument makes
+@samp{^} rules inactive.
+
+@cindex BOL, checking the BOL flag
+@findex YY_AT_BOL
+The macro @code{YY_AT_BOL()} returns true if the next token scanned from
+the current buffer will have @samp{^} rules active, false otherwise.
+
+@cindex actions, redefining YY_BREAK
+@hkindex YY_BREAK
+In the generated scanner, the actions are all gathered in one large
+switch statement and separated using @code{YY_BREAK}, which may be
+redefined.  By default, it is simply a @code{break}, to separate each
+rule's action from the following rule's.  Redefining @code{YY_BREAK}
+allows, for example, C++ users to #define YY_BREAK to do nothing (while
+being very careful that every rule ends with a @code{break} or a
+@code{return}!) to avoid suffering from unreachable statement warnings
+where because a rule's action ends with @code{return}, the
+@code{YY_BREAK} is inaccessible.
+
+@node User Values, Yacc, Misc Macros, Top
+@chapter Values Available To the User
+
+This chapter summarizes the various values available to the user in the
+rule actions.
+
+@table @code
+@vindex yytext
+@item  char *yytext
+holds the text of the current token.  It may be modified but not
+lengthened (you cannot append characters to the end).
+
+@cindex yytext, default array size
+@cindex array, default size for yytext
+@vindex YYLMAX
+If the special directive @code{%array} appears in the first section of
+the scanner description, then @code{yytext} is instead declared
+@code{char yytext[YYLMAX]}, where @code{YYLMAX} is a macro definition
+that you can redefine in the first section if you don't like the default
+value (generally 8KB).  Using @code{%array} results in somewhat slower
+scanners, but the value of @code{yytext} becomes immune to calls to
+@code{unput()}, which potentially destroy its value when @code{yytext} is
+a character pointer.  The opposite of @code{%array} is @code{%pointer},
+which is the default.
+
+@cindex C++ and %array
+You cannot use @code{%array} when generating C++ scanner classes (the
+@samp{-+} flag).
+
+@vindex yyleng
+@item  int yyleng
+holds the length of the current token.
+
+@vindex yyin
+@item  FILE *yyin
+is the file which by default @code{flex} reads from.  It may be
+redefined but doing so only makes sense before scanning begins or after
+an EOF has been encountered.  Changing it in the midst of scanning will
+have unexpected results since @code{flex} buffers its input; use
+@code{yyrestart()} instead.  Once scanning terminates because an
+end-of-file has been seen, you can assign @file{yyin} at the new input
+file and then call the scanner again to continue scanning.
+
+@findex yyrestart
+@item  void yyrestart( FILE *new_file )
+may be called to point @file{yyin} at the new input file.  The
+switch-over to the new file is immediate (any previously buffered-up
+input is lost).  Note that calling @code{yyrestart()} with @file{yyin}
+as an argument thus throws away the current input buffer and continues
+scanning the same input file.
+
+@vindex yyout
+@item  FILE *yyout
+is the file to which @code{ECHO} actions are done.  It can be reassigned
+by the user.
+
+@vindex YY_CURRENT_BUFFER
+@item  YY_CURRENT_BUFFER
+returns a @code{YY_BUFFER_STATE} handle to the current buffer.
+
+@vindex YY_START
+@item  YY_START
+returns an integer value corresponding to the current start condition.
+You can subsequently use this value with @code{BEGIN} to return to that
+start condition.
+@end table
+
+@node Yacc, Scanner Options, User Values, Top
+@chapter Interfacing with Yacc
+
+@cindex yacc, interface
+
+@vindex yylval, with yacc
+One of the main uses of @code{flex} is as a companion to the @code{yacc}
+parser-generator.  @code{yacc} parsers expect to call a routine named
+@code{yylex()} to find the next input token.  The routine is supposed to
+return the type of the next token as well as putting any associated
+value in the global @code{yylval}.  To use @code{flex} with @code{yacc},
+one specifies the @samp{-d} option to @code{yacc} to instruct it to
+generate the file @file{y.tab.h} containing definitions of all the
+@code{%tokens} appearing in the @code{yacc} input.  This file is then
+included in the @code{flex} scanner.  For example, if one of the tokens
+is @code{TOK_NUMBER}, part of the scanner might look like:
+
+@cindex yacc interface
+@example
+@verbatim
+    %{
+    #include "y.tab.h"
+    %}
+
+    %%
+
+    [0-9]+        yylval = atoi( yytext ); return TOK_NUMBER;
+@end verbatim
+@end example
+
+@node Scanner Options, Performance, Yacc, Top
+@chapter Scanner Options
+
+@cindex command-line options
+@cindex options, command-line
+@cindex arguments, command-line
+
+The various @code{flex} options are categorized by function in the following
+menu. If you want to lookup a particular option by name, @xref{Index of Scanner Options}.
+
+@menu
+* Options for Specifying Filenames::  
+* Options Affecting Scanner Behavior::  
+* Code-Level And API Options::  
+* Options for Scanner Speed and Size::  
+* Debugging Options::           
+* Miscellaneous Options::       
+@end menu
+
+Even though there are many scanner options, a typical scanner might only
+specify the following options:
+
+@example
+@verbatim
+%option   8bit reentrant bison-bridge
+%option   warn nodefault
+%option   yylineno
+%option   outfile="scanner.c" header-file="scanner.h"
+@end verbatim
+@end example
+
+The first line specifies the general type of scanner we want. The second line
+specifies that we are being careful. The third line asks flex to track line
+numbers. The last line tells flex what to name the files. (The options can be
+specified in any order. We just divided them.)
+
+@code{flex} also provides a mechanism for controlling options within the
+scanner specification itself, rather than from the flex command-line.
+This is done by including @code{%option} directives in the first section
+of the scanner specification.  You can specify multiple options with a
+single @code{%option} directive, and multiple directives in the first
+section of your flex input file.
+
+Most options are given simply as names, optionally preceded by the
+word @samp{no} (with no intervening whitespace) to negate their meaning.
+The names are the same as their long-option equivalents (but without the
+leading @samp{--} ).
+
+@code{flex} scans your rule actions to determine whether you use the
+@code{REJECT} or @code{yymore()} features.  The @code{REJECT} and
+@code{yymore} options are available to override its decision as to
+whether you use the options, either by setting them (e.g., @code{%option
+reject)} to indicate the feature is indeed used, or unsetting them to
+indicate it actually is not used (e.g., @code{%option noyymore)}.
+
+
+A number of options are available for lint purists who want to suppress
+the appearance of unneeded routines in the generated scanner.  Each of
+the following, if unset (e.g., @code{%option nounput}), results in the
+corresponding routine not appearing in the generated scanner:
+
+@example
+@verbatim
+    input, unput
+    yy_push_state, yy_pop_state, yy_top_state
+    yy_scan_buffer, yy_scan_bytes, yy_scan_string
+
+    yyget_extra, yyset_extra, yyget_leng, yyget_text,
+    yyget_lineno, yyset_lineno, yyget_in, yyset_in,
+    yyget_out, yyset_out, yyget_lval, yyset_lval,
+    yyget_lloc, yyset_lloc, yyget_debug, yyset_debug
+@end verbatim
+@end example
+
+(though @code{yy_push_state()} and friends won't appear anyway unless
+you use @code{%option stack)}.
+
+@node Options for Specifying Filenames, Options Affecting Scanner Behavior, Scanner Options, Scanner Options
+@section Options for Specifying Filenames
+
+@table @samp
+
+@anchor{option-header}
+@opindex ---header-file
+@opindex header-file
+@item --header-file=FILE, @code{%option header-file="FILE"}
+instructs flex to write a C header to @file{FILE}. This file contains
+function prototypes, extern variables, and types used by the scanner.
+Only the external API is exported by the header file. Many macros that
+are usable from within scanner actions are not exported to the header
+file. This is due to namespace problems and the goal of a clean
+external API.
+
+While in the header, the macro @code{yyIN_HEADER} is defined, where @samp{yy}
+is substituted with the appropriate prefix.
+
+The @samp{--header-file} option is not compatible with the @samp{--c++} option,
+since the C++ scanner provides its own header in @file{yyFlexLexer.h}.
+
+
+
+@anchor{option-outfile}
+@opindex -o
+@opindex ---outfile
+@opindex outfile
+@item -oFILE, --outfile=FILE, @code{%option outfile="FILE"}
+directs flex to write the scanner to the file @file{FILE} instead of
+@file{lex.yy.c}.  If you combine @samp{--outfile} with the @samp{--stdout} option,
+then the scanner is written to @file{stdout} but its @code{#line}
+directives (see the @samp{-l} option above) refer to the file
+@file{FILE}.
+
+
+
+@anchor{option-stdout}
+@opindex -t
+@opindex ---stdout
+@opindex stdout
+@item -t, --stdout, @code{%option stdout}
+instructs @code{flex} to write the scanner it generates to standard
+output instead of @file{lex.yy.c}.
+
+
+
+@opindex ---skel
+@item -SFILE, --skel=FILE
+overrides the default skeleton file from which
+@code{flex}
+constructs its scanners.  You'll never need this option unless you are doing
+@code{flex}
+maintenance or development.
+
+@opindex ---tables-file
+@opindex tables-file
+@item --tables-file=FILE
+Write serialized scanner dfa tables to FILE. The generated scanner will not
+contain the tables, and requires them to be loaded at runtime.
+@xref{serialization}.
+
+@opindex ---tables-verify
+@opindex tables-verify
+@item --tables-verify
+This option is for flex development. We document it here in case you stumble
+upon it by accident or in case you suspect some inconsistency in the serialized
+tables.  Flex will serialize the scanner dfa tables but will also generate the
+in-code tables as it normally does. At runtime, the scanner will verify that
+the serialized tables match the in-code tables, instead of loading them. 
+
+@end table
+
+@node Options Affecting Scanner Behavior, Code-Level And API Options, Options for Specifying Filenames, Scanner Options
+@section Options Affecting Scanner Behavior
+
+@table @samp
+@anchor{option-case-insensitive}
+@opindex -i
+@opindex ---case-insensitive
+@opindex case-insensitive
+@item -i, --case-insensitive, @code{%option case-insensitive}
+instructs @code{flex} to generate a @dfn{case-insensitive} scanner.  The
+case of letters given in the @code{flex} input patterns will be ignored,
+and tokens in the input will be matched regardless of case.  The matched
+text given in @code{yytext} will have the preserved case (i.e., it will
+not be folded).  For tricky behavior, see @ref{case and character ranges}.
+
+
+
+@anchor{option-lex-compat}
+@opindex -l
+@opindex ---lex-compat
+@opindex lex-compat
+@item -l, --lex-compat, @code{%option lex-compat}
+turns on maximum compatibility with the original AT&T @code{lex}
+implementation.  Note that this does not mean @emph{full} compatibility.
+Use of this option costs a considerable amount of performance, and it
+cannot be used with the @samp{--c++}, @samp{--full}, @samp{--fast}, @samp{-Cf}, or
+@samp{-CF} options.  For details on the compatibilities it provides, see
+@ref{Lex and Posix}.  This option also results in the name
+@code{YY_FLEX_LEX_COMPAT} being @code{#define}'d in the generated scanner.
+
+
+
+@anchor{option-batch}
+@opindex -B
+@opindex ---batch
+@opindex batch
+@item -B, --batch, @code{%option batch}
+instructs @code{flex} to generate a @dfn{batch} scanner, the opposite of
+@emph{interactive} scanners generated by @samp{--interactive} (see below).  In
+general, you use @samp{-B} when you are @emph{certain} that your scanner
+will never be used interactively, and you want to squeeze a
+@emph{little} more performance out of it.  If your goal is instead to
+squeeze out a @emph{lot} more performance, you should be using the
+@samp{-Cf} or @samp{-CF} options, which turn on @samp{--batch} automatically
+anyway.
+
+
+
+@anchor{option-interactive}
+@opindex -I
+@opindex ---interactive
+@opindex interactive
+@item -I, --interactive, @code{%option interactive}
+instructs @code{flex} to generate an @i{interactive} scanner.  An
+interactive scanner is one that only looks ahead to decide what token
+has been matched if it absolutely must.  It turns out that always
+looking one extra character ahead, even if the scanner has already seen
+enough text to disambiguate the current token, is a bit faster than only
+looking ahead when necessary.  But scanners that always look ahead give
+dreadful interactive performance; for example, when a user types a
+newline, it is not recognized as a newline token until they enter
+@emph{another} token, which often means typing in another whole line.
+
+@code{flex} scanners default to @code{interactive} unless you use the
+@samp{-Cf} or @samp{-CF} table-compression options
+(@pxref{Performance}).  That's because if you're looking for
+high-performance you should be using one of these options, so if you
+didn't, @code{flex} assumes you'd rather trade off a bit of run-time
+performance for intuitive interactive behavior.  Note also that you
+@emph{cannot} use @samp{--interactive} in conjunction with @samp{-Cf} or
+@samp{-CF}.  Thus, this option is not really needed; it is on by default
+for all those cases in which it is allowed.
+
+You can force a scanner to
+@emph{not}
+be interactive by using
+@samp{--batch}
+
+
+
+@anchor{option-7bit}
+@opindex -7
+@opindex ---7bit
+@opindex 7bit
+@item -7, --7bit, @code{%option 7bit}
+instructs @code{flex} to generate a 7-bit scanner, i.e., one which can
+only recognize 7-bit characters in its input.  The advantage of using
+@samp{--7bit} is that the scanner's tables can be up to half the size of
+those generated using the @samp{--8bit}.  The disadvantage is that such
+scanners often hang or crash if their input contains an 8-bit character.
+
+Note, however, that unless you generate your scanner using the
+@samp{-Cf} or @samp{-CF} table compression options, use of @samp{--7bit}
+will save only a small amount of table space, and make your scanner
+considerably less portable.  @code{Flex}'s default behavior is to
+generate an 8-bit scanner unless you use the @samp{-Cf} or @samp{-CF},
+in which case @code{flex} defaults to generating 7-bit scanners unless
+your site was always configured to generate 8-bit scanners (as will
+often be the case with non-USA sites).  You can tell whether flex
+generated a 7-bit or an 8-bit scanner by inspecting the flag summary in
+the @samp{--verbose} output as described above.
+
+Note that if you use @samp{-Cfe} or @samp{-CFe} @code{flex} still
+defaults to generating an 8-bit scanner, since usually with these
+compression options full 8-bit tables are not much more expensive than
+7-bit tables.
+
+
+
+@anchor{option-8bit}
+@opindex -8
+@opindex ---8bit
+@opindex 8bit
+@item -8, --8bit, @code{%option 8bit}
+instructs @code{flex} to generate an 8-bit scanner, i.e., one which can
+recognize 8-bit characters.  This flag is only needed for scanners
+generated using @samp{-Cf} or @samp{-CF}, as otherwise flex defaults to
+generating an 8-bit scanner anyway.
+
+See the discussion of
+@samp{--7bit}
+above for @code{flex}'s default behavior and the tradeoffs between 7-bit
+and 8-bit scanners.
+
+
+
+@anchor{option-default}
+@opindex ---default
+@opindex default
+@item --default, @code{%option default}
+generate the default rule.
+
+
+
+@anchor{option-always-interactive}
+@opindex ---always-interactive
+@opindex always-interactive
+@item --always-interactive, @code{%option always-interactive}
+instructs flex to generate a scanner which always considers its input
+@emph{interactive}.  Normally, on each new input file the scanner calls
+@code{isatty()} in an attempt to determine whether the scanner's input
+source is interactive and thus should be read a character at a time.
+When this option is used, however, then no such call is made.
+
+
+
+@opindex ---never-interactive
+@item --never-interactive, @code{--never-interactive}
+instructs flex to generate a scanner which never considers its input
+interactive.  This is the opposite of @code{always-interactive}.
+
+
+@anchor{option-posix}
+@opindex -X
+@opindex ---posix
+@opindex posix
+@item -X, --posix, @code{%option posix}
+turns on maximum compatibility with the POSIX 1003.2-1992 definition of
+@code{lex}.  Since @code{flex} was originally designed to implement the
+POSIX definition of @code{lex} this generally involves very few changes
+in behavior.  At the current writing the known differences between
+@code{flex} and the POSIX standard are:
+
+@itemize
+@item
+In POSIX and AT&T @code{lex}, the repeat operator, @samp{@{@}}, has lower
+precedence than concatenation (thus @samp{ab@{3@}} yields @samp{ababab}).
+Most POSIX utilities use an Extended Regular Expression (ERE) precedence
+that has the precedence of the repeat operator higher than concatenation
+(which causes @samp{ab@{3@}} to yield @samp{abbb}).  By default, @code{flex}
+places the precedence of the repeat operator higher than concatenation
+which matches the ERE processing of other POSIX utilities.  When either
+@samp{--posix} or @samp{-l} are specified, @code{flex} will use the
+traditional AT&T and POSIX-compliant precedence for the repeat operator
+where concatenation has higher precedence than the repeat operator.
+@end itemize
+
+
+@anchor{option-stack}
+@opindex ---stack
+@opindex stack
+@item --stack, @code{%option stack}
+enables the use of
+start condition stacks (@pxref{Start Conditions}).
+
+
+
+@anchor{option-stdinit}
+@opindex ---stdinit
+@opindex stdinit
+@item --stdinit, @code{%option stdinit}
+if set (i.e., @b{%option stdinit)} initializes @code{yyin} and
+@code{yyout} to @file{stdin} and @file{stdout}, instead of the default of
+@file{NULL}.  Some existing @code{lex} programs depend on this behavior,
+even though it is not compliant with ANSI C, which does not require
+@file{stdin} and @file{stdout} to be compile-time constant. In a
+reentrant scanner, however, this is not a problem since initialization
+is performed in @code{yylex_init} at runtime.
+
+
+
+@anchor{option-yylineno}
+@opindex ---yylineno
+@opindex yylineno
+@item --yylineno, @code{%option yylineno}
+directs @code{flex} to generate a scanner
+that maintains the number of the current line read from its input in the
+global variable @code{yylineno}.  This option is implied by @code{%option
+lex-compat}.  In a reentrant C scanner, the macro @code{yylineno} is
+accessible regardless of the value of @code{%option yylineno}, however, its
+value is not modified by @code{flex} unless @code{%option yylineno} is enabled.
+
+
+
+@anchor{option-yywrap}
+@opindex ---yywrap
+@opindex yywrap
+@item --yywrap, @code{%option yywrap}
+if unset (i.e., @code{--noyywrap)}, makes the scanner not call
+@code{yywrap()} upon an end-of-file, but simply assume that there are no
+more files to scan (until the user points @file{yyin} at a new file and
+calls @code{yylex()} again).
+
+@end table
+
+@node Code-Level And API Options, Options for Scanner Speed and Size, Options Affecting Scanner Behavior, Scanner Options
+@section Code-Level And API Options
+
+@table @samp
+
+@anchor{option-ansi-definitions}
+@opindex ---option-ansi-definitions
+@opindex ansi-definitions
+@item --ansi-definitions, @code{%option ansi-definitions}
+instruct flex to generate ANSI C99 definitions for functions.
+This option is enabled by default.
+If @code{%option noansi-definitions} is specified, then the obsolete style
+is generated.
+
+@anchor{option-ansi-prototypes}
+@opindex ---option-ansi-prototypes
+@opindex ansi-prototypes
+@item --ansi-prototypes, @code{%option ansi-prototypes}
+instructs flex to generate ANSI C99 prototypes for functions. 
+This option is enabled by default.
+If @code{noansi-prototypes} is specified, then
+prototypes will have empty parameter lists.
+
+@anchor{option-bison-bridge}
+@opindex ---bison-bridge
+@opindex bison-bridge
+@item --bison-bridge, @code{%option bison-bridge}
+instructs flex to generate a C scanner that is
+meant to be called by a
+@code{GNU bison}
+parser. The scanner has minor API changes for
+@code{bison}
+compatibility. In particular, the declaration of
+@code{yylex}
+is modified to take an additional parameter,
+@code{yylval}.
+@xref{Bison Bridge}.
+
+@anchor{option-bison-locations}
+@opindex ---bison-locations
+@opindex bison-locations
+@item --bison-locations, @code{%option bison-locations}
+instruct flex that 
+@code{GNU bison} @code{%locations} are being used.
+This means @code{yylex} will be passed
+an additional parameter, @code{yylloc}. This option
+implies @code{%option bison-bridge}.
+@xref{Bison Bridge}.
+
+@anchor{option-noline}
+@opindex -L
+@opindex ---noline
+@opindex noline
+@item -L, --noline, @code{%option noline}
+instructs
+@code{flex}
+not to generate
+@code{#line}
+directives.  Without this option,
+@code{flex}
+peppers the generated scanner
+with @code{#line} directives so error messages in the actions will be correctly
+located with respect to either the original
+@code{flex}
+input file (if the errors are due to code in the input file), or
+@file{lex.yy.c}
+(if the errors are
+@code{flex}'s
+fault -- you should report these sorts of errors to the email address
+given in @ref{Reporting Bugs}).
+
+
+
+@anchor{option-reentrant}
+@opindex -R
+@opindex ---reentrant
+@opindex reentrant
+@item -R, --reentrant, @code{%option reentrant}
+instructs flex to generate a reentrant C scanner.  The generated scanner
+may safely be used in a multi-threaded environment. The API for a
+reentrant scanner is different than for a non-reentrant scanner
+@pxref{Reentrant}).  Because of the API difference between
+reentrant and non-reentrant @code{flex} scanners, non-reentrant flex
+code must be modified before it is suitable for use with this option.
+This option is not compatible with the @samp{--c++} option.
+
+The option @samp{--reentrant} does not affect the performance of
+the scanner.
+
+
+
+@anchor{option-c++}
+@opindex -+
+@opindex ---c++
+@opindex c++
+@item -+, --c++, @code{%option c++}
+specifies that you want flex to generate a C++
+scanner class.  @xref{Cxx}, for
+details.
+
+
+
+@anchor{option-array}
+@opindex ---array
+@opindex array
+@item --array, @code{%option array}
+specifies that you want yytext to be an array instead of a char*
+
+
+
+@anchor{option-pointer}
+@opindex ---pointer
+@opindex pointer
+@item --pointer, @code{%option pointer}
+specify that  @code{yytext} should be a @code{char *}, not an array.
+This default is @code{char *}.
+
+
+
+@anchor{option-prefix}
+@opindex -P
+@opindex ---prefix
+@opindex prefix
+@item -PPREFIX, --prefix=PREFIX, @code{%option prefix="PREFIX"}
+changes the default @samp{yy} prefix used by @code{flex} for all
+globally-visible variable and function names to instead be
+@samp{PREFIX}.  For example, @samp{--prefix=foo} changes the name of
+@code{yytext} to @code{footext}.  It also changes the name of the default
+output file from @file{lex.yy.c} to @file{lex.foo.c}.  Here is a partial
+list of the names affected:
+
+@example
+@verbatim
+    yy_create_buffer
+    yy_delete_buffer
+    yy_flex_debug
+    yy_init_buffer
+    yy_flush_buffer
+    yy_load_buffer_state
+    yy_switch_to_buffer
+    yyin
+    yyleng
+    yylex
+    yylineno
+    yyout
+    yyrestart
+    yytext
+    yywrap
+    yyalloc
+    yyrealloc
+    yyfree
+@end verbatim
+@end example
+
+(If you are using a C++ scanner, then only @code{yywrap} and
+@code{yyFlexLexer} are affected.)  Within your scanner itself, you can
+still refer to the global variables and functions using either version
+of their name; but externally, they have the modified name.
+
+This option lets you easily link together multiple
+@code{flex}
+programs into the same executable.  Note, though, that using this
+option also renames
+@code{yywrap()},
+so you now
+@emph{must}
+either
+provide your own (appropriately-named) version of the routine for your
+scanner, or use
+@code{%option noyywrap},
+as linking with
+@samp{-lfl}
+no longer provides one for you by default.
+
+
+
+@anchor{option-main}
+@opindex ---main
+@opindex main
+@item --main, @code{%option main}
+ directs flex to provide a default @code{main()} program for the
+scanner, which simply calls @code{yylex()}.  This option implies
+@code{noyywrap} (see below).
+
+
+
+@anchor{option-nounistd}
+@opindex ---nounistd
+@opindex nounistd
+@item --nounistd, @code{%option nounistd}
+suppresses inclusion of the non-ANSI header file @file{unistd.h}. This option
+is meant to target environments in which @file{unistd.h} does not exist. Be aware
+that certain options may cause flex to generate code that relies on functions
+normally found in @file{unistd.h}, (e.g. @code{isatty()}, @code{read()}.)
+If you wish to use these functions, you will have to inform your compiler where
+to find them.
+@xref{option-always-interactive}. @xref{option-read}.
+
+
+
+@anchor{option-yyclass}
+@opindex ---yyclass
+@opindex yyclass
+@item --yyclass=NAME, @code{%option yyclass="NAME"}
+only applies when generating a C++ scanner (the @samp{--c++} option).  It
+informs @code{flex} that you have derived @code{NAME} as a subclass of
+@code{yyFlexLexer}, so @code{flex} will place your actions in the member
+function @code{foo::yylex()} instead of @code{yyFlexLexer::yylex()}.  It
+also generates a @code{yyFlexLexer::yylex()} member function that emits
+a run-time error (by invoking @code{yyFlexLexer::LexerError())} if
+called.  @xref{Cxx}.
+
+@end table
+
+@node Options for Scanner Speed and Size, Debugging Options, Code-Level And API Options, Scanner Options
+@section Options for Scanner Speed and Size
+
+@table @samp
+
+@item -C[aefFmr]
+controls the degree of table compression and, more generally, trade-offs
+between small scanners and fast scanners.
+
+@table @samp
+@opindex -C
+@item -C
+A lone @samp{-C} specifies that the scanner tables should be compressed
+but neither equivalence classes nor meta-equivalence classes should be
+used.
+
+@anchor{option-align}
+@opindex -Ca
+@opindex ---align
+@opindex align
+@item -Ca, --align, @code{%option align}
+(``align'') instructs flex to trade off larger tables in the
+generated scanner for faster performance because the elements of
+the tables are better aligned for memory access and computation.  On some
+RISC architectures, fetching and manipulating longwords is more efficient
+than with smaller-sized units such as shortwords.  This option can
+quadruple the size of the tables used by your scanner.
+
+@anchor{option-ecs}
+@opindex -Ce
+@opindex ---ecs
+@opindex ecs
+@item -Ce, --ecs, @code{%option ecs}
+directs @code{flex} to construct @dfn{equivalence classes}, i.e., sets
+of characters which have identical lexical properties (for example, if
+the only appearance of digits in the @code{flex} input is in the
+character class ``[0-9]'' then the digits '0', '1', ..., '9' will all be
+put in the same equivalence class).  Equivalence classes usually give
+dramatic reductions in the final table/object file sizes (typically a
+factor of 2-5) and are pretty cheap performance-wise (one array look-up
+per character scanned).
+
+@opindex -Cf
+@item -Cf
+specifies that the @dfn{full} scanner tables should be generated -
+@code{flex} should not compress the tables by taking advantages of
+similar transition functions for different states.
+
+@opindex -CF
+@item -CF
+specifies that the alternate fast scanner representation (described
+above under the @samp{--fast} flag) should be used.  This option cannot be
+used with @samp{--c++}.
+
+@anchor{option-meta-ecs}
+@opindex -Cm
+@opindex ---meta-ecs
+@opindex meta-ecs
+@item -Cm, --meta-ecs, @code{%option meta-ecs}
+directs
+@code{flex}
+to construct
+@dfn{meta-equivalence classes},
+which are sets of equivalence classes (or characters, if equivalence
+classes are not being used) that are commonly used together.  Meta-equivalence
+classes are often a big win when using compressed tables, but they
+have a moderate performance impact (one or two @code{if} tests and one
+array look-up per character scanned).
+
+@anchor{option-read}
+@opindex -Cr
+@opindex ---read
+@opindex read
+@item -Cr, --read, @code{%option read}
+causes the generated scanner to @emph{bypass} use of the standard I/O
+library (@code{stdio}) for input.  Instead of calling @code{fread()} or
+@code{getc()}, the scanner will use the @code{read()} system call,
+resulting in a performance gain which varies from system to system, but
+in general is probably negligible unless you are also using @samp{-Cf}
+or @samp{-CF}.  Using @samp{-Cr} can cause strange behavior if, for
+example, you read from @file{yyin} using @code{stdio} prior to calling
+the scanner (because the scanner will miss whatever text your previous
+reads left in the @code{stdio} input buffer).  @samp{-Cr} has no effect
+if you define @code{YY_INPUT()} (@pxref{Generated Scanner}).
+@end table
+
+The options @samp{-Cf} or @samp{-CF} and @samp{-Cm} do not make sense
+together - there is no opportunity for meta-equivalence classes if the
+table is not being compressed.  Otherwise the options may be freely
+mixed, and are cumulative.
+
+The default setting is @samp{-Cem}, which specifies that @code{flex}
+should generate equivalence classes and meta-equivalence classes.  This
+setting provides the highest degree of table compression.  You can trade
+off faster-executing scanners at the cost of larger tables with the
+following generally being true:
+
+@example
+@verbatim
+    slowest & smallest
+          -Cem
+          -Cm
+          -Ce
+          -C
+          -C{f,F}e
+          -C{f,F}
+          -C{f,F}a
+    fastest & largest
+@end verbatim
+@end example
+
+Note that scanners with the smallest tables are usually generated and
+compiled the quickest, so during development you will usually want to
+use the default, maximal compression.
+
+@samp{-Cfe} is often a good compromise between speed and size for
+production scanners.
+
+@anchor{option-full}
+@opindex -f
+@opindex ---full
+@opindex full
+@item -f, --full, @code{%option full}
+specifies
+@dfn{fast scanner}.
+No table compression is done and @code{stdio} is bypassed.
+The result is large but fast.  This option is equivalent to
+@samp{--Cfr}
+
+
+@anchor{option-fast}
+@opindex -F
+@opindex ---fast
+@opindex fast
+@item -F, --fast, @code{%option fast}
+specifies that the @emph{fast} scanner table representation should be
+used (and @code{stdio} bypassed).  This representation is about as fast
+as the full table representation @samp{--full}, and for some sets of
+patterns will be considerably smaller (and for others, larger).  In
+general, if the pattern set contains both @emph{keywords} and a
+catch-all, @emph{identifier} rule, such as in the set:
+
+@example
+@verbatim
+    "case"    return TOK_CASE;
+    "switch"  return TOK_SWITCH;
+    ...
+    "default" return TOK_DEFAULT;
+    [a-z]+    return TOK_ID;
+@end verbatim
+@end example
+
+then you're better off using the full table representation.  If only
+the @emph{identifier} rule is present and you then use a hash table or some such
+to detect the keywords, you're better off using
+@samp{--fast}.
+
+This option is equivalent to @samp{-CFr}.  It cannot be used
+with @samp{--c++}.
+
+@end table
+
+@node Debugging Options, Miscellaneous Options, Options for Scanner Speed and Size, Scanner Options
+@section Debugging Options
+
+@table @samp
+
+@anchor{option-backup}
+@opindex -b
+@opindex ---backup
+@opindex backup
+@item -b, --backup, @code{%option backup}
+Generate backing-up information to @file{lex.backup}.  This is a list of
+scanner states which require backing up and the input characters on
+which they do so.  By adding rules one can remove backing-up states.  If
+@emph{all} backing-up states are eliminated and @samp{-Cf} or @code{-CF}
+is used, the generated scanner will run faster (see the @samp{--perf-report} flag).
+Only users who wish to squeeze every last cycle out of their scanners
+need worry about this option.  (@pxref{Performance}).
+
+
+
+@anchor{option-debug}
+@opindex -d
+@opindex ---debug
+@opindex debug
+@item -d, --debug, @code{%option debug}
+makes the generated scanner run in @dfn{debug} mode.  Whenever a pattern
+is recognized and the global variable @code{yy_flex_debug} is non-zero
+(which is the default), the scanner will write to @file{stderr} a line
+of the form:
+
+@example
+@verbatim
+    -accepting rule at line 53 ("the matched text")
+@end verbatim
+@end example
+
+The line number refers to the location of the rule in the file defining
+the scanner (i.e., the file that was fed to flex).  Messages are also
+generated when the scanner backs up, accepts the default rule, reaches
+the end of its input buffer (or encounters a NUL; at this point, the two
+look the same as far as the scanner's concerned), or reaches an
+end-of-file.
+
+
+
+@anchor{option-perf-report}
+@opindex -p
+@opindex ---perf-report
+@opindex perf-report
+@item -p, --perf-report, @code{%option perf-report}
+generates a performance report to @file{stderr}.  The report consists of
+comments regarding features of the @code{flex} input file which will
+cause a serious loss of performance in the resulting scanner.  If you
+give the flag twice, you will also get comments regarding features that
+lead to minor performance losses.
+
+Note that the use of @code{REJECT}, and
+variable trailing context (@pxref{Limitations}) entails a substantial
+performance penalty; use of @code{yymore()}, the @samp{^} operator, and
+the @samp{--interactive} flag entail minor performance penalties.
+
+
+
+@anchor{option-nodefault}
+@opindex -s
+@opindex ---nodefault
+@opindex nodefault
+@item -s, --nodefault, @code{%option nodefault}
+causes the @emph{default rule} (that unmatched scanner input is echoed
+to @file{stdout)} to be suppressed.  If the scanner encounters input
+that does not match any of its rules, it aborts with an error.  This
+option is useful for finding holes in a scanner's rule set.
+
+
+
+@anchor{option-trace}
+@opindex -T
+@opindex ---trace
+@opindex trace
+@item -T, --trace, @code{%option trace}
+makes @code{flex} run in @dfn{trace} mode.  It will generate a lot of
+messages to @file{stderr} concerning the form of the input and the
+resultant non-deterministic and deterministic finite automata.  This
+option is mostly for use in maintaining @code{flex}.
+
+
+
+@anchor{option-nowarn}
+@opindex -w
+@opindex ---nowarn
+@opindex nowarn
+@item -w, --nowarn, @code{%option nowarn}
+suppresses warning messages.
+
+
+
+@anchor{option-verbose}
+@opindex -v
+@opindex ---verbose
+@opindex verbose
+@item -v, --verbose, @code{%option verbose}
+specifies that @code{flex} should write to @file{stderr} a summary of
+statistics regarding the scanner it generates.  Most of the statistics
+are meaningless to the casual @code{flex} user, but the first line
+identifies the version of @code{flex} (same as reported by @samp{--version}),
+and the next line the flags used when generating the scanner, including
+those that are on by default.
+
+
+
+@anchor{option-warn}
+@opindex ---warn
+@opindex warn
+@item --warn, @code{%option warn}
+warn about certain things. In particular, if the default rule can be
+matched but no default rule has been given, the flex will warn you.
+We recommend using this option always.
+
+@end table
+
+@node Miscellaneous Options,  , Debugging Options, Scanner Options
+@section Miscellaneous Options
+
+@table @samp
+@opindex -c
+@item -c
+A do-nothing option included for POSIX compliance.
+
+@opindex -h
+@opindex ---help
+@item -h, -?, --help
+generates a ``help'' summary of @code{flex}'s options to @file{stdout}
+and then exits.
+
+@opindex -n
+@item -n
+Another do-nothing option included for
+POSIX compliance.
+
+@opindex -V
+@opindex ---version
+@item -V, --version
+prints the version number to @file{stdout} and exits.
+
+@end table
+
+
+@node Performance, Cxx, Scanner Options, Top
+@chapter Performance Considerations
+
+@cindex performance, considerations
+The main design goal of @code{flex} is that it generate high-performance
+scanners.  It has been optimized for dealing well with large sets of
+rules.  Aside from the effects on scanner speed of the table compression
+@samp{-C} options outlined above, there are a number of options/actions
+which degrade performance.  These are, from most expensive to least:
+
+@cindex REJECT, performance costs
+@cindex yylineno, performance costs
+@cindex trailing context, performance costs
+@example
+@verbatim
+    REJECT
+    arbitrary trailing context
+
+    pattern sets that require backing up
+    %option yylineno
+    %array
+
+    %option interactive
+    %option always-interactive
+
+    ^ beginning-of-line operator
+    yymore()
+@end verbatim
+@end example
+
+with the first two all being quite expensive and the last two being
+quite cheap.  Note also that @code{unput()} is implemented as a routine
+call that potentially does quite a bit of work, while @code{yyless()} is
+a quite-cheap macro. So if you are just putting back some excess text
+you scanned, use @code{yyless()}.
+
+@code{REJECT} should be avoided at all costs when performance is
+important.  It is a particularly expensive option.
+
+There is one case when @code{%option yylineno} can be expensive. That is when
+your patterns match long tokens that could @emph{possibly} contain a newline
+character. There is no performance penalty for rules that can not possibly
+match newlines, since flex does not need to check them for newlines.  In
+general, you should avoid rules such as @code{[^f]+}, which match very long
+tokens, including newlines, and may possibly match your entire file! A better
+approach is to separate @code{[^f]+} into two rules:
+
+@example
+@verbatim
+%option yylineno
+%%
+    [^f\n]+
+    \n+
+@end verbatim
+@end example
+
+The above scanner does not incur a performance penalty.
+
+@cindex patterns, tuning for performance
+@cindex performance, backing up
+@cindex backing up, example of eliminating
+Getting rid of backing up is messy and often may be an enormous amount
+of work for a complicated scanner.  In principal, one begins by using
+the @samp{-b} flag to generate a @file{lex.backup} file.  For example,
+on the input:
+
+@cindex backing up, eliminating
+@example
+@verbatim
+    %%
+    foo        return TOK_KEYWORD;
+    foobar     return TOK_KEYWORD;
+@end verbatim
+@end example
+
+the file looks like:
+
+@example
+@verbatim
+    State #6 is non-accepting -
+     associated rule line numbers:
+           2       3
+     out-transitions: [ o ]
+     jam-transitions: EOF [ \001-n  p-\177 ]
+
+    State #8 is non-accepting -
+     associated rule line numbers:
+           3
+     out-transitions: [ a ]
+     jam-transitions: EOF [ \001-`  b-\177 ]
+
+    State #9 is non-accepting -
+     associated rule line numbers:
+           3
+     out-transitions: [ r ]
+     jam-transitions: EOF [ \001-q  s-\177 ]
+
+    Compressed tables always back up.
+@end verbatim
+@end example
+
+The first few lines tell us that there's a scanner state in which it can
+make a transition on an 'o' but not on any other character, and that in
+that state the currently scanned text does not match any rule.  The
+state occurs when trying to match the rules found at lines 2 and 3 in
+the input file.  If the scanner is in that state and then reads
+something other than an 'o', it will have to back up to find a rule
+which is matched.  With a bit of headscratching one can see that this
+must be the state it's in when it has seen @samp{fo}.  When this has
+happened, if anything other than another @samp{o} is seen, the scanner
+will have to back up to simply match the @samp{f} (by the default rule).
+
+The comment regarding State #8 indicates there's a problem when
+@samp{foob} has been scanned.  Indeed, on any character other than an
+@samp{a}, the scanner will have to back up to accept "foo".  Similarly,
+the comment for State #9 concerns when @samp{fooba} has been scanned and
+an @samp{r} does not follow.
+
+The final comment reminds us that there's no point going to all the
+trouble of removing backing up from the rules unless we're using
+@samp{-Cf} or @samp{-CF}, since there's no performance gain doing so
+with compressed scanners.
+
+@cindex error rules, to eliminate backing up
+The way to remove the backing up is to add ``error'' rules:
+
+@cindex backing up, eliminating by adding error rules
+@example
+@verbatim
+    %%
+    foo         return TOK_KEYWORD;
+    foobar      return TOK_KEYWORD;
+
+    fooba       |
+    foob        |
+    fo          {
+                /* false alarm, not really a keyword */
+                return TOK_ID;
+                }
+@end verbatim
+@end example
+
+Eliminating backing up among a list of keywords can also be done using a
+``catch-all'' rule:
+
+@cindex backing up, eliminating with catch-all rule
+@example
+@verbatim
+    %%
+    foo         return TOK_KEYWORD;
+    foobar      return TOK_KEYWORD;
+
+    [a-z]+      return TOK_ID;
+@end verbatim
+@end example
+
+This is usually the best solution when appropriate.
+
+Backing up messages tend to cascade.  With a complicated set of rules
+it's not uncommon to get hundreds of messages.  If one can decipher
+them, though, it often only takes a dozen or so rules to eliminate the
+backing up (though it's easy to make a mistake and have an error rule
+accidentally match a valid token.  A possible future @code{flex} feature
+will be to automatically add rules to eliminate backing up).
+
+It's important to keep in mind that you gain the benefits of eliminating
+backing up only if you eliminate @emph{every} instance of backing up.
+Leaving just one means you gain nothing.
+
+@emph{Variable} trailing context (where both the leading and trailing
+parts do not have a fixed length) entails almost the same performance
+loss as @code{REJECT} (i.e., substantial).  So when possible a rule
+like:
+
+@cindex trailing context, variable length
+@example
+@verbatim
+    %%
+    mouse|rat/(cat|dog)   run();
+@end verbatim
+@end example
+
+is better written:
+
+@example
+@verbatim
+    %%
+    mouse/cat|dog         run();
+    rat/cat|dog           run();
+@end verbatim
+@end example
+
+or as
+
+@example
+@verbatim
+    %%
+    mouse|rat/cat         run();
+    mouse|rat/dog         run();
+@end verbatim
+@end example
+
+Note that here the special '|' action does @emph{not} provide any
+savings, and can even make things worse (@pxref{Limitations}).
+
+Another area where the user can increase a scanner's performance (and
+one that's easier to implement) arises from the fact that the longer the
+tokens matched, the faster the scanner will run.  This is because with
+long tokens the processing of most input characters takes place in the
+(short) inner scanning loop, and does not often have to go through the
+additional work of setting up the scanning environment (e.g.,
+@code{yytext}) for the action.  Recall the scanner for C comments:
+
+@cindex performance optimization, matching longer tokens
+@example
+@verbatim
+    %x comment
+    %%
+            int line_num = 1;
+
+    "/*"         BEGIN(comment);
+
+    <comment>[^*\n]*
+    <comment>"*"+[^*/\n]*
+    <comment>\n             ++line_num;
+    <comment>"*"+"/"        BEGIN(INITIAL);
+@end verbatim
+@end example
+
+This could be sped up by writing it as:
+
+@example
+@verbatim
+    %x comment
+    %%
+            int line_num = 1;
+
+    "/*"         BEGIN(comment);
+
+    <comment>[^*\n]*
+    <comment>[^*\n]*\n      ++line_num;
+    <comment>"*"+[^*/\n]*
+    <comment>"*"+[^*/\n]*\n ++line_num;
+    <comment>"*"+"/"        BEGIN(INITIAL);
+@end verbatim
+@end example
+
+Now instead of each newline requiring the processing of another action,
+recognizing the newlines is distributed over the other rules to keep the
+matched text as long as possible.  Note that @emph{adding} rules does
+@emph{not} slow down the scanner!  The speed of the scanner is
+independent of the number of rules or (modulo the considerations given
+at the beginning of this section) how complicated the rules are with
+regard to operators such as @samp{*} and @samp{|}.
+
+@cindex keywords, for performance
+@cindex performance, using keywords
+A final example in speeding up a scanner: suppose you want to scan
+through a file containing identifiers and keywords, one per line
+and with no other extraneous characters, and recognize all the
+keywords.  A natural first approach is:
+
+@cindex performance optimization, recognizing keywords
+@example
+@verbatim
+    %%
+    asm      |
+    auto     |
+    break    |
+    ... etc ...
+    volatile |
+    while    /* it's a keyword */
+
+    .|\n     /* it's not a keyword */
+@end verbatim
+@end example
+
+To eliminate the back-tracking, introduce a catch-all rule:
+
+@example
+@verbatim
+    %%
+    asm      |
+    auto     |
+    break    |
+    ... etc ...
+    volatile |
+    while    /* it's a keyword */
+
+    [a-z]+   |
+    .|\n     /* it's not a keyword */
+@end verbatim
+@end example
+
+Now, if it's guaranteed that there's exactly one word per line, then we
+can reduce the total number of matches by a half by merging in the
+recognition of newlines with that of the other tokens:
+
+@example
+@verbatim
+    %%
+    asm\n    |
+    auto\n   |
+    break\n  |
+    ... etc ...
+    volatile\n |
+    while\n  /* it's a keyword */
+
+    [a-z]+\n |
+    .|\n     /* it's not a keyword */
+@end verbatim
+@end example
+
+One has to be careful here, as we have now reintroduced backing up
+into the scanner.  In particular, while
+@emph{we}
+know that there will never be any characters in the input stream
+other than letters or newlines,
+@code{flex}
+can't figure this out, and it will plan for possibly needing to back up
+when it has scanned a token like @samp{auto} and then the next character
+is something other than a newline or a letter.  Previously it would
+then just match the @samp{auto} rule and be done, but now it has no @samp{auto}
+rule, only a @samp{auto\n} rule.  To eliminate the possibility of backing up,
+we could either duplicate all rules but without final newlines, or,
+since we never expect to encounter such an input and therefore don't
+how it's classified, we can introduce one more catch-all rule, this
+one which doesn't include a newline:
+
+@example
+@verbatim
+    %%
+    asm\n    |
+    auto\n   |
+    break\n  |
+    ... etc ...
+    volatile\n |
+    while\n  /* it's a keyword */
+
+    [a-z]+\n |
+    [a-z]+   |
+    .|\n     /* it's not a keyword */
+@end verbatim
+@end example
+
+Compiled with @samp{-Cf}, this is about as fast as one can get a
+@code{flex} scanner to go for this particular problem.
+
+A final note: @code{flex} is slow when matching @code{NUL}s,
+particularly when a token contains multiple @code{NUL}s.  It's best to
+write rules which match @emph{short} amounts of text if it's anticipated
+that the text will often include @code{NUL}s.
+
+Another final note regarding performance: as mentioned in
+@ref{Matching}, dynamically resizing @code{yytext} to accommodate huge
+tokens is a slow process because it presently requires that the (huge)
+token be rescanned from the beginning.  Thus if performance is vital,
+you should attempt to match ``large'' quantities of text but not
+``huge'' quantities, where the cutoff between the two is at about 8K
+characters per token.
+
+@node Cxx, Reentrant, Performance, Top
+@chapter Generating C++ Scanners
+
+@cindex c++, experimental form of scanner class
+@cindex experimental form of c++ scanner class
+@strong{IMPORTANT}: the present form of the scanning class is @emph{experimental}
+and may change considerably between major releases.
+
+@cindex C++
+@cindex member functions, C++
+@cindex methods, c++
+@code{flex} provides two different ways to generate scanners for use
+with C++.  The first way is to simply compile a scanner generated by
+@code{flex} using a C++ compiler instead of a C compiler.  You should
+not encounter any compilation errors (@pxref{Reporting Bugs}).  You can
+then use C++ code in your rule actions instead of C code.  Note that the
+default input source for your scanner remains @file{yyin}, and default
+echoing is still done to @file{yyout}.  Both of these remain @code{FILE
+*} variables and not C++ @emph{streams}.
+
+You can also use @code{flex} to generate a C++ scanner class, using the
+@samp{-+} option (or, equivalently, @code{%option c++)}, which is
+automatically specified if the name of the @code{flex} executable ends
+in a '+', such as @code{flex++}.  When using this option, @code{flex}
+defaults to generating the scanner to the file @file{lex.yy.cc} instead
+of @file{lex.yy.c}.  The generated scanner includes the header file
+@file{FlexLexer.h}, which defines the interface to two C++ classes.
+
+The first class,
+@code{FlexLexer},
+provides an abstract base class defining the general scanner class
+interface.  It provides the following member functions:
+
+@table @code
+@findex YYText (C++ only)
+@item const char* YYText()
+returns the text of the most recently matched token, the equivalent of
+@code{yytext}.
+
+@findex YYLeng (C++ only)
+@item int YYLeng()
+returns the length of the most recently matched token, the equivalent of
+@code{yyleng}.
+
+@findex lineno (C++ only)
+@item int lineno() const
+returns the current input line number (see @code{%option yylineno)}, or
+@code{1} if @code{%option yylineno} was not used.
+
+@findex set_debug (C++ only)
+@item void set_debug( int flag )
+sets the debugging flag for the scanner, equivalent to assigning to
+@code{yy_flex_debug} (@pxref{Scanner Options}).  Note that you must build
+the scanner using @code{%option debug} to include debugging information
+in it.
+
+@findex  debug (C++ only)
+@item int debug() const
+returns the current setting of the debugging flag.
+@end table
+
+Also provided are member functions equivalent to
+@code{yy_switch_to_buffer()}, @code{yy_create_buffer()} (though the
+first argument is an @code{istream*} object pointer and not a
+@code{FILE*)}, @code{yy_flush_buffer()}, @code{yy_delete_buffer()}, and
+@code{yyrestart()} (again, the first argument is a @code{istream*}
+object pointer).
+
+@tindex yyFlexLexer (C++ only)
+@tindex FlexLexer (C++ only)
+The second class defined in @file{FlexLexer.h} is @code{yyFlexLexer},
+which is derived from @code{FlexLexer}.  It defines the following
+additional member functions:
+
+@table @code
+@findex yyFlexLexer constructor (C++ only)
+@item yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 )
+constructs a @code{yyFlexLexer} object using the given streams for input
+and output.  If not specified, the streams default to @code{cin} and
+@code{cout}, respectively.
+
+@findex yylex (C++ version)
+@item virtual int yylex()
+performs the same role is @code{yylex()} does for ordinary @code{flex}
+scanners: it scans the input stream, consuming tokens, until a rule's
+action returns a value.  If you derive a subclass @code{S} from
+@code{yyFlexLexer} and want to access the member functions and variables
+of @code{S} inside @code{yylex()}, then you need to use @code{%option
+yyclass="S"} to inform @code{flex} that you will be using that subclass
+instead of @code{yyFlexLexer}.  In this case, rather than generating
+@code{yyFlexLexer::yylex()}, @code{flex} generates @code{S::yylex()}
+(and also generates a dummy @code{yyFlexLexer::yylex()} that calls
+@code{yyFlexLexer::LexerError()} if called).
+
+@findex switch_streams (C++ only)
+@item virtual void switch_streams(istream* new_in = 0, ostream* new_out = 0)
+reassigns @code{yyin} to @code{new_in} (if non-null) and @code{yyout} to
+@code{new_out} (if non-null), deleting the previous input buffer if
+@code{yyin} is reassigned.
+
+@item int yylex( istream* new_in, ostream* new_out = 0 )
+first switches the input streams via @code{switch_streams( new_in,
+new_out )} and then returns the value of @code{yylex()}.
+@end table
+
+In addition, @code{yyFlexLexer} defines the following protected virtual
+functions which you can redefine in derived classes to tailor the
+scanner:
+
+@table @code
+@findex LexerInput (C++ only)
+@item virtual int LexerInput( char* buf, int max_size )
+reads up to @code{max_size} characters into @code{buf} and returns the
+number of characters read.  To indicate end-of-input, return 0
+characters.  Note that @code{interactive} scanners (see the @samp{-B}
+and @samp{-I} flags in @ref{Scanner Options}) define the macro
+@code{YY_INTERACTIVE}.  If you redefine @code{LexerInput()} and need to
+take different actions depending on whether or not the scanner might be
+scanning an interactive input source, you can test for the presence of
+this name via @code{#ifdef} statements.
+
+@findex LexerOutput (C++ only)
+@item virtual void LexerOutput( const char* buf, int size )
+writes out @code{size} characters from the buffer @code{buf}, which, while
+@code{NUL}-terminated, may also contain internal @code{NUL}s if the
+scanner's rules can match text with @code{NUL}s in them.
+
+@cindex error reporting, in C++
+@findex LexerError (C++ only)
+@item virtual void LexerError( const char* msg )
+reports a fatal error message.  The default version of this function
+writes the message to the stream @code{cerr} and exits.
+@end table
+
+Note that a @code{yyFlexLexer} object contains its @emph{entire}
+scanning state.  Thus you can use such objects to create reentrant
+scanners, but see also @ref{Reentrant}.  You can instantiate multiple
+instances of the same @code{yyFlexLexer} class, and you can also combine
+multiple C++ scanner classes together in the same program using the
+@samp{-P} option discussed above.
+
+Finally, note that the @code{%array} feature is not available to C++
+scanner classes; you must use @code{%pointer} (the default).
+
+Here is an example of a simple C++ scanner:
+
+@cindex C++ scanners, use of
+@example
+@verbatim
+     // An example of using the flex C++ scanner class.
+
+    %{
+    #include <iostream>
+    using namespace std;
+    int mylineno = 0;
+    %}
+
+    %option noyywrap
+
+    string  \"[^\n"]+\"
+
+    ws      [ \t]+
+
+    alpha   [A-Za-z]
+    dig     [0-9]
+    name    ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])*
+    num1    [-+]?{dig}+\.?([eE][-+]?{dig}+)?
+    num2    [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)?
+    number  {num1}|{num2}
+
+    %%
+
+    {ws}    /* skip blanks and tabs */
+
+    "/*"    {
+            int c;
+
+            while((c = yyinput()) != 0)
+                {
+                if(c == '\n')
+                    ++mylineno;
+
+                else if(c == '*')
+                    {
+                    if((c = yyinput()) == '/')
+                        break;
+                    else
+                        unput(c);
+                    }
+                }
+            }
+
+    {number}  cout << "number " << YYText() << '\n';
+
+    \n        mylineno++;
+
+    {name}    cout << "name " << YYText() << '\n';
+
+    {string}  cout << "string " << YYText() << '\n';
+
+    %%
+
+    int main( int /* argc */, char** /* argv */ )
+    {
+        FlexLexer* lexer = new yyFlexLexer;
+        while(lexer->yylex() != 0)
+            ;
+        return 0;
+    }
+@end verbatim
+@end example
+
+@cindex C++, multiple different scanners
+If you want to create multiple (different) lexer classes, you use the
+@samp{-P} flag (or the @code{prefix=} option) to rename each
+@code{yyFlexLexer} to some other @samp{xxFlexLexer}.  You then can
+include @file{<FlexLexer.h>} in your other sources once per lexer class,
+first renaming @code{yyFlexLexer} as follows:
+
+@cindex include files, with C++
+@cindex header files, with C++
+@cindex C++ scanners, including multiple scanners
+@example
+@verbatim
+    #undef yyFlexLexer
+    #define yyFlexLexer xxFlexLexer
+    #include <FlexLexer.h>
+
+    #undef yyFlexLexer
+    #define yyFlexLexer zzFlexLexer
+    #include <FlexLexer.h>
+@end verbatim
+@end example
+
+if, for example, you used @code{%option prefix="xx"} for one of your
+scanners and @code{%option prefix="zz"} for the other.
+
+@node Reentrant, Lex and Posix, Cxx, Top
+@chapter Reentrant C Scanners
+
+@cindex reentrant, explanation
+@code{flex} has the ability to generate a reentrant C scanner. This is
+accomplished by specifying @code{%option reentrant} (@samp{-R}) The generated
+scanner is both portable, and safe to use in one or more separate threads of
+control.  The most common use for reentrant scanners is from within
+multi-threaded applications.  Any thread may create and execute a reentrant
+@code{flex} scanner without the need for synchronization with other threads.
+
+@menu
+* Reentrant Uses::              
+* Reentrant Overview::          
+* Reentrant Example::           
+* Reentrant Detail::            
+* Reentrant Functions::         
+@end menu
+
+@node Reentrant Uses, Reentrant Overview, Reentrant, Reentrant
+@section Uses for Reentrant Scanners
+
+However, there are other uses for a reentrant scanner.  For example, you
+could scan two or more files simultaneously to implement a @code{diff} at
+the token level (i.e., instead of at the character level):
+
+@cindex reentrant scanners, multiple interleaved scanners
+@example
+@verbatim
+    /* Example of maintaining more than one active scanner. */
+
+    do {
+        int tok1, tok2;
+
+        tok1 = yylex( scanner_1 );
+        tok2 = yylex( scanner_2 );
+
+        if( tok1 != tok2 )
+            printf("Files are different.");
+
+   } while ( tok1 && tok2 );
+@end verbatim
+@end example
+
+Another use for a reentrant scanner is recursion.
+(Note that a recursive scanner can also be created using a non-reentrant scanner and
+buffer states. @xref{Multiple Input Buffers}.)
+
+The following crude scanner supports the @samp{eval} command by invoking
+another instance of itself.
+
+@cindex reentrant scanners, recursive invocation
+@example
+@verbatim
+    /* Example of recursive invocation. */
+
+    %option reentrant
+
+    %%
+    "eval(".+")"  {
+                      yyscan_t scanner;
+                      YY_BUFFER_STATE buf;
+
+                      yylex_init( &scanner );
+                      yytext[yyleng-1] = ' ';
+
+                      buf = yy_scan_string( yytext + 5, scanner );
+                      yylex( scanner );
+
+                      yy_delete_buffer(buf,scanner);
+                      yylex_destroy( scanner );
+                 }
+    ...
+    %%
+@end verbatim
+@end example
+
+@node Reentrant Overview, Reentrant Example, Reentrant Uses, Reentrant
+@section An Overview of the Reentrant API
+
+@cindex reentrant, API explanation
+The API for reentrant scanners is different than for non-reentrant
+scanners. Here is a quick overview of the API:
+
+@itemize
+@code{%option reentrant} must be specified.
+
+@item
+All functions take one additional argument: @code{yyscanner}
+
+@item
+All global variables are replaced by their macro equivalents.
+(We tell you this because it may be important to you during debugging.)
+
+@item
+@code{yylex_init} and @code{yylex_destroy} must be called before and
+after @code{yylex}, respectively.
+
+@item
+Accessor methods (get/set functions) provide access to common
+@code{flex} variables.
+
+@item
+User-specific data can be stored in @code{yyextra}.
+@end itemize
+
+@node Reentrant Example, Reentrant Detail, Reentrant Overview, Reentrant
+@section Reentrant Example
+
+First, an example of a reentrant scanner:
+@cindex reentrant, example of
+@example
+@verbatim
+    /* This scanner prints "//" comments. */
+
+    %option reentrant stack noyywrap
+    %x COMMENT
+
+    %%
+
+    "//"                 yy_push_state( COMMENT, yyscanner);
+    .|\n
+
+    <COMMENT>\n          yy_pop_state( yyscanner );
+    <COMMENT>[^\n]+      fprintf( yyout, "%s\n", yytext);
+
+    %%
+
+    int main ( int argc, char * argv[] )
+    {
+        yyscan_t scanner;
+
+        yylex_init ( &scanner );
+        yylex ( scanner );
+        yylex_destroy ( scanner );
+    return 0;
+   }
+@end verbatim
+@end example
+
+@node Reentrant Detail, Reentrant Functions, Reentrant Example, Reentrant
+@section The Reentrant API in Detail
+
+Here are the things you need to do or know to use the reentrant C API of
+@code{flex}.
+
+@menu
+* Specify Reentrant::           
+* Extra Reentrant Argument::    
+* Global Replacement::          
+* Init and Destroy Functions::  
+* Accessor Methods::            
+* Extra Data::                  
+* About yyscan_t::              
+@end menu
+
+@node Specify Reentrant, Extra Reentrant Argument, Reentrant Detail, Reentrant Detail
+@subsection Declaring a Scanner As Reentrant
+
+ %option reentrant (--reentrant) must be specified.
+
+Notice that @code{%option reentrant} is specified in the above example
+(@pxref{Reentrant Example}. Had this option not been specified,
+@code{flex} would have happily generated a non-reentrant scanner without
+complaining. You may explicitly specify @code{%option noreentrant}, if
+you do @emph{not} want a reentrant scanner, although it is not
+necessary. The default is to generate a non-reentrant scanner.
+
+@node Extra Reentrant Argument, Global Replacement, Specify Reentrant, Reentrant Detail
+@subsection The Extra Argument
+
+@cindex reentrant, calling functions
+@vindex yyscanner (reentrant only)
+All functions take one additional argument: @code{yyscanner}.
+
+Notice that the calls to @code{yy_push_state} and @code{yy_pop_state}
+both have an argument, @code{yyscanner} , that is not present in a
+non-reentrant scanner.  Here are the declarations of
+@code{yy_push_state} and @code{yy_pop_state} in the reentrant scanner:
+
+@example
+@verbatim
+    static void yy_push_state  ( int new_state , yyscan_t yyscanner ) ;
+    static void yy_pop_state  ( yyscan_t yyscanner  ) ;
+@end verbatim
+@end example
+
+Notice that the argument @code{yyscanner} appears in the declaration of
+both functions.  In fact, all @code{flex} functions in a reentrant
+scanner have this additional argument.  It is always the last argument
+in the argument list, it is always of type @code{yyscan_t} (which is
+typedef'd to @code{void *}) and it is
+always named @code{yyscanner}.  As you may have guessed,
+@code{yyscanner} is a pointer to an opaque data structure encapsulating
+the current state of the scanner.  For a list of function declarations,
+see @ref{Reentrant Functions}. Note that preprocessor macros, such as
+@code{BEGIN}, @code{ECHO}, and @code{REJECT}, do not take this
+additional argument.
+
+@node Global Replacement, Init and Destroy Functions, Extra Reentrant Argument, Reentrant Detail
+@subsection Global Variables Replaced By Macros
+
+@cindex reentrant, accessing flex variables
+All global variables in traditional flex have been replaced by macro equivalents.
+
+Note that in the above example, @code{yyout} and @code{yytext} are
+not plain variables. These are macros that will expand to their equivalent lvalue.
+All of the familiar @code{flex} globals have been replaced by their macro
+equivalents. In particular, @code{yytext}, @code{yyleng}, @code{yylineno},
+@code{yyin}, @code{yyout}, @code{yyextra}, @code{yylval}, and @code{yylloc}
+are macros. You may safely use these macros in actions as if they were plain
+variables. We only tell you this so you don't expect to link to these variables
+externally. Currently, each macro expands to a member of an internal struct, e.g.,
+
+@example
+@verbatim
+#define yytext (((struct yyguts_t*)yyscanner)->yytext_r)
+@end verbatim
+@end example
+
+One important thing to remember about
+@code{yytext}
+and friends is that
+@code{yytext}
+is not a global variable in a reentrant
+scanner, you can not access it directly from outside an action or from
+other functions. You must use an accessor method, e.g.,
+@code{yyget_text},
+to accomplish this. (See below).
+
+@node Init and Destroy Functions, Accessor Methods, Global Replacement, Reentrant Detail
+@subsection Init and Destroy Functions
+
+@cindex memory, considerations for reentrant scanners
+@cindex reentrant, initialization
+@findex yylex_init
+@findex yylex_destroy
+
+@code{yylex_init} and @code{yylex_destroy} must be called before and
+after @code{yylex}, respectively.
+
+@example
+@verbatim
+    int yylex_init ( yyscan_t * ptr_yy_globals ) ;
+    int yylex_init_extra ( YY_EXTRA_TYPE user_defined, yyscan_t * ptr_yy_globals ) ;
+    int yylex ( yyscan_t yyscanner ) ;
+    int yylex_destroy ( yyscan_t yyscanner ) ;
+@end verbatim
+@end example
+
+The function @code{yylex_init} must be called before calling any other
+function. The argument to @code{yylex_init} is the address of an
+uninitialized pointer to be filled in by @code{yylex_init}, overwriting
+any previous contents. The function @code{yylex_init_extra} may be used
+instead, taking as its first argument a variable of type @code{YY_EXTRA_TYPE}.
+See the section on yyextra, below, for more details.
+
+The value stored in @code{ptr_yy_globals} should
+thereafter be passed to @code{yylex} and @code{yylex_destroy}.  Flex
+does not save the argument passed to @code{yylex_init}, so it is safe to
+pass the address of a local pointer to @code{yylex_init} so long as it remains
+in scope for the duration of all calls to the scanner, up to and including
+the call to @code{yylex_destroy}.
+
+The function
+@code{yylex} should be familiar to you by now. The reentrant version
+takes one argument, which is the value returned (via an argument) by
+@code{yylex_init}.  Otherwise, it behaves the same as the non-reentrant
+version of @code{yylex}.
+
+Both @code{yylex_init} and @code{yylex_init_extra} returns 0 (zero) on success,
+or non-zero on failure, in which case errno is set to one of the following values:
+
+@itemize
+@item ENOMEM
+Memory allocation error. @xref{memory-management}.
+@item EINVAL
+Invalid argument.
+@end itemize
+
+
+The function @code{yylex_destroy} should be
+called to free resources used by the scanner. After @code{yylex_destroy}
+is called, the contents of @code{yyscanner} should not be used.  Of
+course, there is no need to destroy a scanner if you plan to reuse it.
+A @code{flex} scanner (both reentrant and non-reentrant) may be
+restarted by calling @code{yyrestart}.
+
+Below is an example of a program that creates a scanner, uses it, then destroys
+it when done:
+
+@example
+@verbatim
+    int main ()
+    {
+        yyscan_t scanner;
+        int tok;
+
+        yylex_init(&scanner);
+
+        while ((tok=yylex(scanner)) > 0)
+            printf("tok=%d  yytext=%s\n", tok, yyget_text(scanner));
+
+        yylex_destroy(scanner);
+        return 0;
+    }
+@end verbatim
+@end example
+
+@node Accessor Methods, Extra Data, Init and Destroy Functions, Reentrant Detail
+@subsection Accessing Variables with Reentrant Scanners
+
+@cindex reentrant, accessor functions
+Accessor methods (get/set functions) provide access to common
+@code{flex} variables.
+
+Many scanners that you build will be part of a larger project. Portions
+of your project will need access to @code{flex} values, such as
+@code{yytext}.  In a non-reentrant scanner, these values are global, so
+there is no problem accessing them. However, in a reentrant scanner, there are no
+global @code{flex} values. You can not access them directly.  Instead,
+you must access @code{flex} values using accessor methods (get/set
+functions). Each accessor method is named @code{yyget_NAME} or
+@code{yyset_NAME}, where @code{NAME} is the name of the @code{flex}
+variable you want. For example:
+
+@cindex accessor functions, use of
+@example
+@verbatim
+    /* Set the last character of yytext to NULL. */
+    void chop ( yyscan_t scanner )
+    {
+        int len = yyget_leng( scanner );
+        yyget_text( scanner )[len - 1] = '\0';
+    }
+@end verbatim
+@end example
+
+The above code may be called from within an action like this:
+
+@example
+@verbatim
+    %%
+    .+\n    { chop( yyscanner );}
+@end verbatim
+@end example
+
+You may find that @code{%option header-file} is particularly useful for generating
+prototypes of all the accessor functions. @xref{option-header}.
+
+@node Extra Data, About yyscan_t, Accessor Methods, Reentrant Detail
+@subsection Extra Data
+
+@cindex reentrant, extra data
+@vindex yyextra
+User-specific data can be stored in @code{yyextra}.
+
+In a reentrant scanner, it is unwise to use global variables to
+communicate with or maintain state between different pieces of your program.
+However, you may need access to external data or invoke external functions
+from within the scanner actions.
+Likewise, you may need to pass information to your scanner
+(e.g., open file descriptors, or database connections).
+In a non-reentrant scanner, the only way to do this would be through the
+use of global variables.
+@code{Flex} allows you to store arbitrary, ``extra'' data in a scanner.
+This data is accessible through the accessor methods
+@code{yyget_extra} and @code{yyset_extra}
+from outside the scanner, and through the shortcut macro
+@code{yyextra}
+from within the scanner itself. They are defined as follows:
+
+@tindex YY_EXTRA_TYPE (reentrant only)
+@findex yyget_extra
+@findex yyset_extra
+@example
+@verbatim
+    #define YY_EXTRA_TYPE  void*
+    YY_EXTRA_TYPE  yyget_extra ( yyscan_t scanner );
+    void           yyset_extra ( YY_EXTRA_TYPE arbitrary_data , yyscan_t scanner);
+@end verbatim
+@end example
+
+In addition, an extra form of @code{yylex_init} is provided,
+@code{yylex_init_extra}. This function is provided so that the yyextra value can
+be accessed from within the very first yyalloc, used to allocate
+the scanner itself.
+
+By default, @code{YY_EXTRA_TYPE} is defined as type @code{void *}.  You
+may redefine this type using @code{%option extra-type="your_type"} in 
+the scanner:
+
+@cindex YY_EXTRA_TYPE, defining your own type
+@example
+@verbatim
+    /* An example of overriding YY_EXTRA_TYPE. */
+    %{
+    #include <sys/stat.h>
+    #include <unistd.h>
+    %}
+    %option reentrant
+    %option extra-type="struct stat *"
+    %%
+
+    __filesize__     printf( "%ld", yyextra->st_size  );
+    __lastmod__      printf( "%ld", yyextra->st_mtime );
+    %%
+    void scan_file( char* filename )
+    {
+        yyscan_t scanner;
+        struct stat buf;
+        FILE *in;
+
+        in = fopen( filename, "r" );
+        stat( filename, &buf );
+
+        yylex_init_extra( buf, &scanner );
+        yyset_in( in, scanner );
+        yylex( scanner );
+        yylex_destroy( scanner );
+
+        fclose( in );
+   }
+@end verbatim
+@end example
+
+
+@node About yyscan_t,  , Extra Data, Reentrant Detail
+@subsection About yyscan_t
+
+@tindex yyscan_t (reentrant only)
+@code{yyscan_t} is defined as:
+
+@example
+@verbatim
+     typedef void* yyscan_t;
+@end verbatim
+@end example
+
+It is initialized by @code{yylex_init()} to point to
+an internal structure. You should never access this value
+directly. In particular, you should never attempt to free it
+(use @code{yylex_destroy()} instead.)
+
+@node Reentrant Functions,  , Reentrant Detail, Reentrant
+@section Functions and Macros Available in Reentrant C Scanners
+
+The following Functions are available in a reentrant scanner:
+
+@findex yyget_text
+@findex yyget_leng
+@findex yyget_in
+@findex yyget_out
+@findex yyget_lineno
+@findex yyset_in
+@findex yyset_out
+@findex yyset_lineno
+@findex yyget_debug
+@findex yyset_debug
+@findex yyget_extra
+@findex yyset_extra
+
+@example
+@verbatim
+    char *yyget_text ( yyscan_t scanner );
+    int yyget_leng ( yyscan_t scanner );
+    FILE *yyget_in ( yyscan_t scanner );
+    FILE *yyget_out ( yyscan_t scanner );
+    int yyget_lineno ( yyscan_t scanner );
+    YY_EXTRA_TYPE yyget_extra ( yyscan_t scanner );
+    int  yyget_debug ( yyscan_t scanner );
+
+    void yyset_debug ( int flag, yyscan_t scanner );
+    void yyset_in  ( FILE * in_str , yyscan_t scanner );
+    void yyset_out  ( FILE * out_str , yyscan_t scanner );
+    void yyset_lineno ( int line_number , yyscan_t scanner );
+    void yyset_extra ( YY_EXTRA_TYPE user_defined , yyscan_t scanner );
+@end verbatim
+@end example
+
+There are no ``set'' functions for yytext and yyleng. This is intentional.
+
+The following Macro shortcuts are available in actions in a reentrant
+scanner:
+
+@example
+@verbatim
+    yytext
+    yyleng
+    yyin
+    yyout
+    yylineno
+    yyextra
+    yy_flex_debug
+@end verbatim
+@end example
+
+@cindex yylineno, in a reentrant scanner
+In a reentrant C scanner, support for yylineno is always present
+(i.e., you may access yylineno), but the value is never modified by
+@code{flex} unless @code{%option yylineno} is enabled. This is to allow
+the user to maintain the line count independently of @code{flex}.
+
+@anchor{bison-functions}
+The following functions and macros are made available when @code{%option
+bison-bridge} (@samp{--bison-bridge}) is specified:
+
+@example
+@verbatim
+    YYSTYPE * yyget_lval ( yyscan_t scanner );
+    void yyset_lval ( YYSTYPE * yylvalp , yyscan_t scanner );
+    yylval
+@end verbatim
+@end example
+
+The following functions and macros are made available
+when @code{%option bison-locations} (@samp{--bison-locations}) is specified:
+
+@example
+@verbatim
+    YYLTYPE *yyget_lloc ( yyscan_t scanner );
+    void yyset_lloc ( YYLTYPE * yyllocp , yyscan_t scanner );
+    yylloc
+@end verbatim
+@end example
+
+Support for yylval assumes that @code{YYSTYPE} is a valid type.  Support for
+yylloc assumes that @code{YYSLYPE} is a valid type.  Typically, these types are
+generated by @code{bison}, and are included in section 1 of the @code{flex}
+input.
+
+@node Lex and Posix, Memory Management, Reentrant, Top
+@chapter Incompatibilities with Lex and Posix
+
+@cindex POSIX and lex
+@cindex lex (traditional) and POSIX
+
+@code{flex} is a rewrite of the AT&T Unix @emph{lex} tool (the two
+implementations do not share any code, though), with some extensions and
+incompatibilities, both of which are of concern to those who wish to
+write scanners acceptable to both implementations.  @code{flex} is fully
+compliant with the POSIX @code{lex} specification, except that when
+using @code{%pointer} (the default), a call to @code{unput()} destroys
+the contents of @code{yytext}, which is counter to the POSIX
+specification.  In this section we discuss all of the known areas of
+incompatibility between @code{flex}, AT&T @code{lex}, and the POSIX
+specification.  @code{flex}'s @samp{-l} option turns on maximum
+compatibility with the original AT&T @code{lex} implementation, at the
+cost of a major loss in the generated scanner's performance.  We note
+below which incompatibilities can be overcome using the @samp{-l}
+option.  @code{flex} is fully compatible with @code{lex} with the
+following exceptions:
+
+@itemize
+@item
+The undocumented @code{lex} scanner internal variable @code{yylineno} is
+not supported unless @samp{-l} or @code{%option yylineno} is used.
+
+@item
+@code{yylineno} should be maintained on a per-buffer basis, rather than
+a per-scanner (single global variable) basis.
+
+@item
+@code{yylineno} is not part of the POSIX specification.
+
+@item
+The @code{input()} routine is not redefinable, though it may be called
+to read characters following whatever has been matched by a rule.  If
+@code{input()} encounters an end-of-file the normal @code{yywrap()}
+processing is done.  A ``real'' end-of-file is returned by
+@code{input()} as @code{EOF}.
+
+@item
+Input is instead controlled by defining the @code{YY_INPUT()} macro.
+
+@item
+The @code{flex} restriction that @code{input()} cannot be redefined is
+in accordance with the POSIX specification, which simply does not
+specify any way of controlling the scanner's input other than by making
+an initial assignment to @file{yyin}.
+
+@item
+The @code{unput()} routine is not redefinable.  This restriction is in
+accordance with POSIX.
+
+@item
+@code{flex} scanners are not as reentrant as @code{lex} scanners.  In
+particular, if you have an interactive scanner and an interrupt handler
+which long-jumps out of the scanner, and the scanner is subsequently
+called again, you may get the following message:
+
+@cindex error messages, end of buffer missed
+@example
+@verbatim
+    fatal flex scanner internal error--end of buffer missed
+@end verbatim
+@end example
+
+To reenter the scanner, first use:
+
+@cindex restarting the scanner
+@example
+@verbatim
+    yyrestart( yyin );
+@end verbatim
+@end example
+
+Note that this call will throw away any buffered input; usually this
+isn't a problem with an interactive scanner. @xref{Reentrant}, for
+@code{flex}'s reentrant API.
+
+@item
+Also note that @code{flex} C++ scanner classes
+@emph{are}
+reentrant, so if using C++ is an option for you, you should use
+them instead.  @xref{Cxx}, and @ref{Reentrant}  for details.
+
+@item
+@code{output()} is not supported.  Output from the @b{ECHO} macro is
+done to the file-pointer @code{yyout} (default @file{stdout)}.
+
+@item
+@code{output()} is not part of the POSIX specification.
+
+@item
+@code{lex} does not support exclusive start conditions (%x), though they
+are in the POSIX specification.
+
+@item
+When definitions are expanded, @code{flex} encloses them in parentheses.
+With @code{lex}, the following:
+
+@cindex name definitions, not POSIX
+@example
+@verbatim
+    NAME    [A-Z][A-Z0-9]*
+    %%
+    foo{NAME}?      printf( "Found it\n" );
+    %%
+@end verbatim
+@end example
+
+will not match the string @samp{foo} because when the macro is expanded
+the rule is equivalent to @samp{foo[A-Z][A-Z0-9]*?}  and the precedence
+is such that the @samp{?} is associated with @samp{[A-Z0-9]*}.  With
+@code{flex}, the rule will be expanded to @samp{foo([A-Z][A-Z0-9]*)?}
+and so the string @samp{foo} will match.
+
+@item
+Note that if the definition begins with @samp{^} or ends with @samp{$}
+then it is @emph{not} expanded with parentheses, to allow these
+operators to appear in definitions without losing their special
+meanings.  But the @samp{<s>}, @samp{/}, and @code{<<EOF>>} operators
+cannot be used in a @code{flex} definition.
+
+@item
+Using @samp{-l} results in the @code{lex} behavior of no parentheses
+around the definition.
+
+@item
+The POSIX specification is that the definition be enclosed in parentheses.
+
+@item
+Some implementations of @code{lex} allow a rule's action to begin on a
+separate line, if the rule's pattern has trailing whitespace:
+
+@cindex patterns and actions on different lines
+@example
+@verbatim
+    %%
+    foo|bar<space here>
+      { foobar_action();}
+@end verbatim
+@end example
+
+@code{flex} does not support this feature.
+
+@item
+The @code{lex} @code{%r} (generate a Ratfor scanner) option is not
+supported.  It is not part of the POSIX specification.
+
+@item
+After a call to @code{unput()}, @emph{yytext} is undefined until the
+next token is matched, unless the scanner was built using @code{%array}.
+This is not the case with @code{lex} or the POSIX specification.  The
+@samp{-l} option does away with this incompatibility.
+
+@item
+The precedence of the @samp{@{,@}} (numeric range) operator is
+different.  The AT&T and POSIX specifications of @code{lex}
+interpret @samp{abc@{1,3@}} as match one, two,
+or three occurrences of @samp{abc}'', whereas @code{flex} interprets it
+as ``match @samp{ab} followed by one, two, or three occurrences of
+@samp{c}''.  The @samp{-l} and @samp{--posix} options do away with this
+incompatibility.
+
+@item
+The precedence of the @samp{^} operator is different.  @code{lex}
+interprets @samp{^foo|bar} as ``match either 'foo' at the beginning of a
+line, or 'bar' anywhere'', whereas @code{flex} interprets it as ``match
+either @samp{foo} or @samp{bar} if they come at the beginning of a
+line''.  The latter is in agreement with the POSIX specification.
+
+@item
+The special table-size declarations such as @code{%a} supported by
+@code{lex} are not required by @code{flex} scanners..  @code{flex}
+ignores them.
+@item
+The name @code{FLEX_SCANNER} is @code{#define}'d so scanners may be
+written for use with either @code{flex} or @code{lex}.  Scanners also
+include @code{YY_FLEX_MAJOR_VERSION},  @code{YY_FLEX_MINOR_VERSION}
+and @code{YY_FLEX_SUBMINOR_VERSION}
+indicating which version of @code{flex} generated the scanner. For
+example, for the 2.5.22 release, these defines would be 2,  5 and 22
+respectively. If the version of @code{flex} being used is a beta
+version, then the symbol @code{FLEX_BETA} is defined.
+
+@item
+The symbols @samp{[[} and @samp{]]} in the code sections of the input
+may conflict with the m4 delimiters. @xref{M4 Dependency}.
+
+
+@end itemize
+
+@cindex POSIX comp;compliance
+@cindex non-POSIX features of flex
+The following @code{flex} features are not included in @code{lex} or the
+POSIX specification:
+
+@itemize
+@item
+C++ scanners
+@item
+%option
+@item
+start condition scopes
+@item
+start condition stacks
+@item
+interactive/non-interactive scanners
+@item
+yy_scan_string() and friends
+@item
+yyterminate()
+@item
+yy_set_interactive()
+@item
+yy_set_bol()
+@item
+YY_AT_BOL()
+   <<EOF>>
+@item
+<*>
+@item
+YY_DECL
+@item
+YY_START
+@item
+YY_USER_ACTION
+@item
+YY_USER_INIT
+@item
+#line directives
+@item
+%@{@}'s around actions
+@item
+reentrant C API
+@item
+multiple actions on a line
+@item
+almost all of the @code{flex} command-line options
+@end itemize
+
+The feature ``multiple actions on a line''
+refers to the fact that with @code{flex} you can put multiple actions on
+the same line, separated with semi-colons, while with @code{lex}, the
+following:
+
+@example
+@verbatim
+    foo    handle_foo(); ++num_foos_seen;
+@end verbatim
+@end example
+
+is (rather surprisingly) truncated to
+
+@example
+@verbatim
+    foo    handle_foo();
+@end verbatim
+@end example
+
+@code{flex} does not truncate the action.  Actions that are not enclosed
+in braces are simply terminated at the end of the line.
+
+@node Memory Management, Serialized Tables, Lex and Posix, Top
+@chapter Memory Management
+
+@cindex memory management
+@anchor{memory-management}
+This chapter describes how flex handles dynamic memory, and how you can
+override the default behavior.
+
+@menu
+* The Default Memory Management::  
+* Overriding The Default Memory Management::  
+* A Note About yytext And Memory::  
+@end menu
+
+@node The Default Memory Management, Overriding The Default Memory Management, Memory Management, Memory Management
+@section The Default Memory Management
+
+Flex allocates dynamic memory during initialization, and once in a while from
+within a call to yylex(). Initialization takes place during the first call to
+yylex(). Thereafter, flex may reallocate more memory if it needs to enlarge a
+buffer. As of version 2.5.9 Flex will clean up all memory when you call @code{yylex_destroy}
+@xref{faq-memory-leak}.
+
+Flex allocates dynamic memory for four purposes, listed below @footnote{The
+quantities given here are approximate, and may vary due to host architecture,
+compiler configuration, or due to future enhancements to flex.} 
+
+@table @asis
+
+@item 16kB for the input buffer.
+Flex allocates memory for the character buffer used to perform pattern
+matching.  Flex must read ahead from the input stream and store it in a large
+character buffer.  This buffer is typically the largest chunk of dynamic memory
+flex consumes. This buffer will grow if necessary, doubling the size each time.
+Flex frees this memory when you call yylex_destroy().  The default size of this
+buffer (16384 bytes) is almost always too large.  The ideal size for this
+buffer is the length of the longest token expected, in bytes, plus a little more.  Flex will allocate a few
+extra bytes for housekeeping. Currently, to override the size of the input buffer
+you must @code{#define YY_BUF_SIZE} to whatever number of bytes you want. We don't plan
+to change this in the near future, but we reserve the right to do so if we ever add a more robust memory management
+API. 
+
+@item 64kb for the REJECT state. This will only be allocated if you use REJECT.
+The size is  large enough to hold the same number of states as characters in the input buffer. If you override the size of the
+input buffer (via @code{YY_BUF_SIZE}), then you automatically override the size of this buffer as well.
+
+@item 100 bytes for the start condition stack.
+Flex allocates memory for the start condition stack. This is the stack used
+for pushing start states, i.e., with yy_push_state(). It will grow if
+necessary.  Since the states are simply integers, this stack doesn't consume
+much memory.  This stack is not present if @code{%option stack} is not
+specified.  You will rarely need to tune this buffer. The ideal size for this
+stack is the maximum depth expected.  The memory for this stack is
+automatically destroyed when you call yylex_destroy(). @xref{option-stack}.
+
+@item 40 bytes for each YY_BUFFER_STATE.
+Flex allocates memory for each YY_BUFFER_STATE. The buffer state itself
+is about 40 bytes, plus an additional large character buffer (described above.)
+The initial buffer state is created during initialization, and with each call
+to yy_create_buffer(). You can't tune the size of this, but you can tune the
+character buffer as described above. Any buffer state that you explicitly
+create by calling yy_create_buffer() is @emph{NOT} destroyed automatically. You
+must call yy_delete_buffer() to free the memory. The exception to this rule is
+that flex will delete the current buffer automatically when you call
+yylex_destroy(). If you delete the current buffer, be sure to set it to NULL.
+That way, flex will not try to delete the buffer a second time (possibly
+crashing your program!) At the time of this writing, flex does not provide a
+growable stack for the buffer states.  You have to manage that yourself.
+@xref{Multiple Input Buffers}.
+
+@item 84 bytes for the reentrant scanner guts
+Flex allocates about 84 bytes for the reentrant scanner structure when
+you call yylex_init(). It is destroyed when the user calls yylex_destroy().
+
+@end table
+
+
+@node Overriding The Default Memory Management, A Note About yytext And Memory, The Default Memory Management, Memory Management
+@section Overriding The Default Memory Management
+
+@cindex yyalloc, overriding
+@cindex yyrealloc, overriding
+@cindex yyfree, overriding
+
+Flex calls the functions @code{yyalloc}, @code{yyrealloc}, and @code{yyfree}
+when it needs to allocate or free memory. By default, these functions are
+wrappers around the standard C functions, @code{malloc}, @code{realloc}, and
+@code{free}, respectively. You can override the default implementations by telling
+flex that you will provide your own implementations.
+
+To override the default implementations, you must do two things:
+
+@enumerate
+
+@item Suppress the default implementations by specifying one or more of the
+following options:
+
+@itemize
+@opindex noyyalloc
+@item @code{%option noyyalloc}
+@item @code{%option noyyrealloc}
+@item @code{%option noyyfree}.
+@end itemize
+
+@item Provide your own implementation of the following functions: @footnote{It
+is not necessary to override all (or any) of the memory management routines.
+You may, for example, override @code{yyrealloc}, but not @code{yyfree} or
+@code{yyalloc}.}
+
+@example
+@verbatim
+// For a non-reentrant scanner
+void * yyalloc (size_t bytes);
+void * yyrealloc (void * ptr, size_t bytes);
+void   yyfree (void * ptr);
+
+// For a reentrant scanner
+void * yyalloc (size_t bytes, void * yyscanner);
+void * yyrealloc (void * ptr, size_t bytes, void * yyscanner);
+void   yyfree (void * ptr, void * yyscanner);
+@end verbatim
+@end example
+
+@end enumerate
+
+In the following example, we will override all three memory routines. We assume
+that there is a custom allocator with garbage collection. In order to make this
+example interesting, we will use a reentrant scanner, passing a pointer to the
+custom allocator through @code{yyextra}.
+
+@cindex overriding the memory routines
+@example
+@verbatim
+%{
+#include "some_allocator.h"
+%}
+
+/* Suppress the default implementations. */
+%option noyyalloc noyyrealloc noyyfree
+%option reentrant
+
+/* Initialize the allocator. */
+#define YY_EXTRA_TYPE  struct allocator*
+#define YY_USER_INIT  yyextra = allocator_create();
+
+%%
+.|\n   ;
+%%
+
+/* Provide our own implementations. */
+void * yyalloc (size_t bytes, void* yyscanner) {
+    return allocator_alloc (yyextra, bytes);
+}
+
+void * yyrealloc (void * ptr, size_t bytes, void* yyscanner) {
+    return allocator_realloc (yyextra, bytes);
+}
+
+void yyfree (void * ptr, void * yyscanner) {      
+    /* Do nothing -- we leave it to the garbage collector. */
+}
+
+@end verbatim
+@end example
+
+
+@node A Note About yytext And Memory,  , Overriding The Default Memory Management, Memory Management
+@section A Note About yytext And Memory
+
+@cindex yytext, memory considerations
+
+When flex finds a match, @code{yytext} points to the first character of the
+match in the input buffer. The string itself is part of the input buffer, and
+is @emph{NOT} allocated separately. The value of yytext will be overwritten the next
+time yylex() is called. In short, the value of yytext is only valid from within
+the matched rule's action.
+
+Often, you want the value of yytext to persist for later processing, i.e., by a
+parser with non-zero lookahead. In order to preserve yytext, you will have to
+copy it with strdup() or a similar function. But this introduces some headache
+because your parser is now responsible for freeing the copy of yytext. If you
+use a yacc or bison parser, (commonly used with flex), you will discover that
+the error recovery mechanisms can cause memory to be leaked.
+
+To prevent memory leaks from strdup'd yytext, you will have to track the memory
+somehow. Our experience has shown that a garbage collection mechanism or a
+pooled memory mechanism will save you a lot of grief when writing parsers.
+
+@node Serialized Tables, Diagnostics, Memory Management, Top
+@chapter Serialized Tables
+@cindex serialization
+@cindex memory, serialized tables
+
+@anchor{serialization}
+A @code{flex} scanner has the ability to save the DFA tables to a file, and
+load them at runtime when needed.  The motivation for this feature is to reduce
+the runtime memory footprint.  Traditionally, these tables have been compiled into
+the scanner as C arrays, and are sometimes quite large.  Since the tables are
+compiled into the scanner, the memory used by the tables can never be freed.
+This is a waste of memory, especially if an application uses several scanners,
+but none of them at the same time.
+
+The serialization feature allows the tables to be loaded at runtime, before
+scanning begins. The tables may be discarded when scanning is finished.
+
+@menu
+* Creating Serialized Tables::  
+* Loading and Unloading Serialized Tables::  
+* Tables File Format::          
+@end menu
+
+@node Creating Serialized Tables, Loading and Unloading Serialized Tables, Serialized Tables, Serialized Tables
+@section Creating Serialized Tables
+@cindex tables, creating serialized
+@cindex serialization of tables
+
+You may create a scanner with serialized tables by specifying:
+
+@example
+@verbatim
+    %option tables-file=FILE
+or
+    --tables-file=FILE
+@end verbatim
+@end example
+
+These options instruct flex to save the DFA tables to the file @var{FILE}. The tables
+will @emph{not} be embedded in the generated scanner. The scanner will not
+function on its own. The scanner will be dependent upon the serialized tables. You must
+load the tables from this file at runtime before you can scan anything. 
+
+If you do not specify a filename to @code{--tables-file}, the tables will be
+saved to @file{lex.yy.tables}, where @samp{yy} is the appropriate prefix.
+
+If your project uses several different scanners, you can concatenate the
+serialized tables into one file, and flex will find the correct set of tables,
+using the scanner prefix as part of the lookup key. An example follows:
+
+@cindex serialized tables, multiple scanners
+@example
+@verbatim
+$ flex --tables-file --prefix=cpp cpp.l
+$ flex --tables-file --prefix=c   c.l
+$ cat lex.cpp.tables lex.c.tables  >  all.tables
+@end verbatim
+@end example
+
+The above example created two scanners, @samp{cpp}, and @samp{c}. Since we did
+not specify a filename, the tables were serialized to @file{lex.c.tables} and
+@file{lex.cpp.tables}, respectively. Then, we concatenated the two files
+together into @file{all.tables}, which we will distribute with our project. At
+runtime, we will open the file and tell flex to load the tables from it.  Flex
+will find the correct tables automatically. (See next section).
+
+@node Loading and Unloading Serialized Tables, Tables File Format, Creating Serialized Tables, Serialized Tables
+@section Loading and Unloading Serialized Tables
+@cindex tables, loading and unloading
+@cindex loading tables at runtime
+@cindex tables, freeing
+@cindex freeing tables
+@cindex memory, serialized tables
+
+If you've built your scanner with @code{%option tables-file}, then you must
+load the scanner tables at runtime. This can be accomplished with the following
+function:
+
+@deftypefun int yytables_fload (FILE* @var{fp} [, yyscan_t @var{scanner}])
+Locates scanner tables in the stream pointed to by @var{fp} and loads them.
+Memory for the tables is allocated via @code{yyalloc}.  You must call this
+function before the first call to @code{yylex}. The argument @var{scanner}
+only appears in the reentrant scanner.
+This function returns @samp{0} (zero) on success, or non-zero on error.
+@end deftypefun
+
+The loaded tables are @strong{not} automatically destroyed (unloaded) when you
+call @code{yylex_destroy}. The reason is that you may create several scanners
+of the same type (in a reentrant scanner), each of which needs access to these
+tables.  To avoid a nasty memory leak, you must call the following function:
+
+@deftypefun int yytables_destroy ([yyscan_t @var{scanner}])
+Unloads the scanner tables. The tables must be loaded again before you can scan
+any more data.  The argument @var{scanner} only appears in the reentrant
+scanner.  This function returns @samp{0} (zero) on success, or non-zero on
+error.
+@end deftypefun
+
+@strong{The functions @code{yytables_fload} and @code{yytables_destroy} are not
+thread-safe.} You must ensure that these functions are called exactly once (for
+each scanner type) in a threaded program, before any thread calls @code{yylex}.
+After the tables are loaded, they are never written to, and no thread
+protection is required thereafter -- until you destroy them.
+
+@node Tables File Format,  , Loading and Unloading Serialized Tables, Serialized Tables
+@section Tables File Format
+@cindex tables, file format
+@cindex file format, serialized tables
+
+This section defines the file format of serialized @code{flex} tables.
+
+The tables format allows for one or more sets of tables to be
+specified, where each set corresponds to a given scanner. Scanners are
+indexed by name, as described below. The file format is as follows:
+
+@example
+@verbatim
+                 TABLE SET 1
+                +-------------------------------+
+        Header  | uint32          th_magic;     |
+                | uint32          th_hsize;     |
+                | uint32          th_ssize;     |
+                | uint16          th_flags;     |
+                | char            th_version[]; |
+                | char            th_name[];    |
+                | uint8           th_pad64[];   |
+                +-------------------------------+
+        Table 1 | uint16          td_id;        |
+                | uint16          td_flags;     |
+                | uint32          td_hilen;     |
+                | uint32          td_lolen;     |
+                | void            td_data[];    |
+                | uint8           td_pad64[];   |
+                +-------------------------------+
+        Table 2 |                               |
+           .    .                               .
+           .    .                               .
+           .    .                               .
+           .    .                               .
+        Table n |                               |
+                +-------------------------------+
+                 TABLE SET 2
+                      .
+                      .
+                      .
+                 TABLE SET N
+@end verbatim
+@end example
+
+The above diagram shows that a complete set of tables consists of a header
+followed by multiple individual tables. Furthermore, multiple complete sets may
+be present in the same file, each set with its own header and tables. The sets
+are contiguous in the file. The only way to know if another set follows is to
+check the next four bytes for the magic number (or check for EOF). The header
+and tables sections are padded to 64-bit boundaries. Below we describe each
+field in detail. This format does not specify how the scanner will expand the
+given data, i.e., data may be serialized as int8, but expanded to an int32
+array at runtime. This is to reduce the size of the serialized data where
+possible.  Remember, @emph{all integer values are in network byte order}. 
+
+@noindent
+Fields of a table header:
+
+@table @code
+@item th_magic
+Magic number, always 0xF13C57B1.
+
+@item th_hsize
+Size of this entire header, in bytes, including all fields plus any padding.
+
+@item th_ssize
+Size of this entire set, in bytes, including the header, all tables, plus
+any padding.
+
+@item th_flags
+Bit flags for this table set. Currently unused.
+
+@item th_version[]
+Flex version in NULL-terminated string format. e.g., @samp{2.5.13a}. This is
+the version of flex that was used to create the serialized tables.
+
+@item th_name[]
+Contains the name of this table set. The default is @samp{yytables},
+and is prefixed accordingly, e.g., @samp{footables}. Must be NULL-terminated.
+
+@item th_pad64[]
+Zero or more NULL bytes, padding the entire header to the next 64-bit boundary
+as calculated from the beginning of the header.
+@end table
+
+@noindent
+Fields of a table:
+
+@table @code
+@item td_id
+Specifies the table identifier. Possible values are:
+@table @code
+@item YYTD_ID_ACCEPT (0x01)
+@code{yy_accept}
+@item YYTD_ID_BASE   (0x02)
+@code{yy_base}
+@item YYTD_ID_CHK    (0x03)
+@code{yy_chk}
+@item YYTD_ID_DEF    (0x04)
+@code{yy_def}
+@item YYTD_ID_EC     (0x05)
+@code{yy_ec }
+@item YYTD_ID_META   (0x06)
+@code{yy_meta}
+@item YYTD_ID_NUL_TRANS (0x07)
+@code{yy_NUL_trans}
+@item YYTD_ID_NXT (0x08)
+@code{yy_nxt}. This array may be two dimensional. See the @code{td_hilen}
+field below.
+@item YYTD_ID_RULE_CAN_MATCH_EOL (0x09)
+@code{yy_rule_can_match_eol}
+@item YYTD_ID_START_STATE_LIST (0x0A)
+@code{yy_start_state_list}. This array is handled specially because it is an
+array of pointers to structs. See the @code{td_flags} field below.
+@item YYTD_ID_TRANSITION (0x0B)
+@code{yy_transition}. This array is handled specially because it is an array of
+structs. See the @code{td_lolen} field below.
+@item YYTD_ID_ACCLIST (0x0C)
+@code{yy_acclist}
+@end table
+
+@item td_flags
+Bit flags describing how to interpret the data in @code{td_data}.
+The data arrays are one-dimensional by default, but may be
+two dimensional as specified in the @code{td_hilen} field.
+
+@table @code
+@item YYTD_DATA8 (0x01)
+The data is serialized as an array of type int8.
+@item YYTD_DATA16 (0x02)
+The data is serialized as an array of type int16.
+@item YYTD_DATA32 (0x04)
+The data is serialized as an array of type int32.
+@item YYTD_PTRANS (0x08)
+The data is a list of indexes of entries in the expanded @code{yy_transition}
+array.  Each index should be expanded to a pointer to the corresponding entry
+in the @code{yy_transition} array. We count on the fact that the
+@code{yy_transition} array has already been seen.
+@item YYTD_STRUCT (0x10)
+The data is a list of yy_trans_info structs, each of which consists of
+two integers. There is no padding between struct elements or between structs.
+The type of each member is determined by the @code{YYTD_DATA*} bits.
+@end table
+
+@item td_hilen
+If @code{td_hilen} is non-zero, then the data is a two-dimensional array.
+Otherwise, the data is a one-dimensional array. @code{td_hilen} contains the
+number of elements in the higher dimensional array, and @code{td_lolen} contains
+the number of elements in the lowest dimension.
+
+Conceptually, @code{td_data} is either @code{sometype td_data[td_lolen]}, or
+@code{sometype td_data[td_hilen][td_lolen]}, where @code{sometype} is specified
+by the @code{td_flags} field.  It is possible for both @code{td_lolen} and
+@code{td_hilen} to be zero, in which case @code{td_data} is a zero length
+array, and no data is loaded, i.e., this table is simply skipped. Flex does not
+currently generate tables of zero length.
+
+@item td_lolen
+Specifies the number of elements in the lowest dimension array. If this is
+a one-dimensional array, then it is simply the number of elements in this array.
+The element size is determined by the @code{td_flags} field.
+
+@item td_data[]
+The table data. This array may be a one- or two-dimensional array, of type
+@code{int8}, @code{int16}, @code{int32}, @code{struct yy_trans_info}, or
+@code{struct yy_trans_info*},  depending upon the values in the
+@code{td_flags}, @code{td_hilen}, and @code{td_lolen} fields.
+
+@item td_pad64[]
+Zero or more NULL bytes, padding the entire table to the next 64-bit boundary as
+calculated from the beginning of this table.
+@end table
+
+@node Diagnostics, Limitations, Serialized Tables, Top
+@chapter Diagnostics
+
+@cindex error reporting, diagnostic messages
+@cindex warnings, diagnostic messages
+
+The following is a list of @code{flex} diagnostic messages:
+
+@itemize
+@item
+@samp{warning, rule cannot be matched} indicates that the given rule
+cannot be matched because it follows other rules that will always match
+the same text as it.  For example, in the following @samp{foo} cannot be
+matched because it comes after an identifier ``catch-all'' rule:
+
+@cindex warning, rule cannot be matched
+@example
+@verbatim
+    [a-z]+    got_identifier();
+    foo       got_foo();
+@end verbatim
+@end example
+
+Using @code{REJECT} in a scanner suppresses this warning.
+
+@item
+@samp{warning, -s option given but default rule can be matched} means
+that it is possible (perhaps only in a particular start condition) that
+the default rule (match any single character) is the only one that will
+match a particular input.  Since @samp{-s} was given, presumably this is
+not intended.
+
+@item
+@code{reject_used_but_not_detected undefined} or
+@code{yymore_used_but_not_detected undefined}. These errors can occur
+at compile time.  They indicate that the scanner uses @code{REJECT} or
+@code{yymore()} but that @code{flex} failed to notice the fact, meaning
+that @code{flex} scanned the first two sections looking for occurrences
+of these actions and failed to find any, but somehow you snuck some in
+(via a #include file, for example).  Use @code{%option reject} or
+@code{%option yymore} to indicate to @code{flex} that you really do use
+these features.
+
+@item
+@samp{flex scanner jammed}. a scanner compiled with
+@samp{-s} has encountered an input string which wasn't matched by any of
+its rules.  This error can also occur due to internal problems.
+
+@item
+@samp{token too large, exceeds YYLMAX}. your scanner uses @code{%array}
+and one of its rules matched a string longer than the @code{YYLMAX}
+constant (8K bytes by default).  You can increase the value by
+#define'ing @code{YYLMAX} in the definitions section of your @code{flex}
+input.
+
+@item
+@samp{scanner requires -8 flag to use the character 'x'}. Your scanner
+specification includes recognizing the 8-bit character @samp{'x'} and
+you did not specify the -8 flag, and your scanner defaulted to 7-bit
+because you used the @samp{-Cf} or @samp{-CF} table compression options.
+See the discussion of the @samp{-7} flag, @ref{Scanner Options}, for
+details.
+
+@item
+@samp{flex scanner push-back overflow}. you used @code{unput()} to push
+back so much text that the scanner's buffer could not hold both the
+pushed-back text and the current token in @code{yytext}.  Ideally the
+scanner should dynamically resize the buffer in this case, but at
+present it does not.
+
+@item
+@samp{input buffer overflow, can't enlarge buffer because scanner uses
+REJECT}.  the scanner was working on matching an extremely large token
+and needed to expand the input buffer.  This doesn't work with scanners
+that use @code{REJECT}.
+
+@item
+@samp{fatal flex scanner internal error--end of buffer missed}. This can
+occur in a scanner which is reentered after a long-jump has jumped out
+(or over) the scanner's activation frame.  Before reentering the
+scanner, use:
+@example
+@verbatim
+    yyrestart( yyin );
+@end verbatim
+@end example
+or, as noted above, switch to using the C++ scanner class.
+
+@item
+@samp{too many start conditions in <> construct!}  you listed more start
+conditions in a <> construct than exist (so you must have listed at
+least one of them twice).
+@end itemize
+
+@node Limitations, Bibliography, Diagnostics, Top
+@chapter Limitations
+
+@cindex limitations of flex
+
+Some trailing context patterns cannot be properly matched and generate
+warning messages (@samp{dangerous trailing context}).  These are
+patterns where the ending of the first part of the rule matches the
+beginning of the second part, such as @samp{zx*/xy*}, where the 'x*'
+matches the 'x' at the beginning of the trailing context.  (Note that
+the POSIX draft states that the text matched by such patterns is
+undefined.)  For some trailing context rules, parts which are actually
+fixed-length are not recognized as such, leading to the abovementioned
+performance loss.  In particular, parts using @samp{|} or @samp{@{n@}}
+(such as @samp{foo@{3@}}) are always considered variable-length.
+Combining trailing context with the special @samp{|} action can result
+in @emph{fixed} trailing context being turned into the more expensive
+@emph{variable} trailing context.  For example, in the following:
+
+@cindex warning, dangerous trailing context
+@example
+@verbatim
+    %%
+    abc      |
+    xyz/def
+@end verbatim
+@end example
+
+Use of @code{unput()} invalidates yytext and yyleng, unless the
+@code{%array} directive or the @samp{-l} option has been used.
+Pattern-matching of @code{NUL}s is substantially slower than matching
+other characters.  Dynamic resizing of the input buffer is slow, as it
+entails rescanning all the text matched so far by the current (generally
+huge) token.  Due to both buffering of input and read-ahead, you cannot
+intermix calls to @file{<stdio.h>} routines, such as, @b{getchar()},
+with @code{flex} rules and expect it to work.  Call @code{input()}
+instead.  The total table entries listed by the @samp{-v} flag excludes
+the number of table entries needed to determine what rule has been
+matched.  The number of entries is equal to the number of DFA states if
+the scanner does not use @code{REJECT}, and somewhat greater than the
+number of states if it does.  @code{REJECT} cannot be used with the
+@samp{-f} or @samp{-F} options.
+
+The @code{flex} internal algorithms need documentation.
+
+@node Bibliography, FAQ, Limitations, Top
+@chapter Additional Reading
+
+You may wish to read more about the following programs:
+@itemize
+@item lex
+@item yacc
+@item sed
+@item awk
+@end itemize
+
+The following books may contain material of interest:
+
+John Levine, Tony Mason, and Doug Brown,
+@emph{Lex & Yacc},
+O'Reilly and Associates.  Be sure to get the 2nd edition.
+
+M. E. Lesk and E. Schmidt,
+@emph{LEX -- Lexical Analyzer Generator}
+
+Alfred Aho, Ravi Sethi and Jeffrey Ullman, @emph{Compilers: Principles,
+Techniques and Tools}, Addison-Wesley (1986).  Describes the
+pattern-matching techniques used by @code{flex} (deterministic finite
+automata).
+
+@node FAQ, Appendices, Bibliography, Top
+@unnumbered FAQ
+
+From time to time, the @code{flex} maintainer receives certain
+questions. Rather than repeat answers to well-understood problems, we
+publish them here.
+
+@menu
+* When was flex born?::         
+* How do I expand backslash-escape sequences in C-style quoted strings?::  
+* Why do flex scanners call fileno if it is not ANSI compatible?::  
+* Does flex support recursive pattern definitions?::  
+* How do I skip huge chunks of input (tens of megabytes) while using flex?::  
+* Flex is not matching my patterns in the same order that I defined them.::  
+* My actions are executing out of order or sometimes not at all.::  
+* How can I have multiple input sources feed into the same scanner at the same time?::  
+* Can I build nested parsers that work with the same input file?::  
+* How can I match text only at the end of a file?::  
+* How can I make REJECT cascade across start condition boundaries?::  
+* Why cant I use fast or full tables with interactive mode?::  
+* How much faster is -F or -f than -C?::  
+* If I have a simple grammar cant I just parse it with flex?::  
+* Why doesn't yyrestart() set the start state back to INITIAL?::  
+* How can I match C-style comments?::  
+* The period isn't working the way I expected.::  
+* Can I get the flex manual in another format?::  
+* Does there exist a "faster" NDFA->DFA algorithm?::  
+* How does flex compile the DFA so quickly?::  
+* How can I use more than 8192 rules?::  
+* How do I abandon a file in the middle of a scan and switch to a new file?::  
+* How do I execute code only during initialization (only before the first scan)?::  
+* How do I execute code at termination?::  
+* Where else can I find help?::  
+* Can I include comments in the "rules" section of the file?::  
+* I get an error about undefined yywrap().::  
+* How can I change the matching pattern at run time?::  
+* How can I expand macros in the input?::  
+* How can I build a two-pass scanner?::  
+* How do I match any string not matched in the preceding rules?::  
+* I am trying to port code from AT&T lex that uses yysptr and yysbuf.::  
+* Is there a way to make flex treat NULL like a regular character?::  
+* Whenever flex can not match the input it says "flex scanner jammed".::  
+* Why doesn't flex have non-greedy operators like perl does?::  
+* Memory leak - 16386 bytes allocated by malloc.::  
+* How do I track the byte offset for lseek()?::  
+* How do I use my own I/O classes in a C++ scanner?::  
+* How do I skip as many chars as possible?::  
+* deleteme00::              
+* Are certain equivalent patterns faster than others?::              
+* Is backing up a big deal?::              
+* Can I fake multi-byte character support?::              
+* deleteme01::              
+* Can you discuss some flex internals?::              
+* unput() messes up yy_at_bol::              
+* The | operator is not doing what I want::              
+* Why can't flex understand this variable trailing context pattern?::              
+* The ^ operator isn't working::              
+* Trailing context is getting confused with trailing optional patterns::              
+* Is flex GNU or not?::              
+* ERASEME53::              
+* I need to scan if-then-else blocks and while loops::              
+* ERASEME55::              
+* ERASEME56::              
+* ERASEME57::              
+* Is there a repository for flex scanners?::              
+* How can I conditionally compile or preprocess my flex input file?::              
+* Where can I find grammars for lex and yacc?::              
+* I get an end-of-buffer message for each character scanned.::              
+* unnamed-faq-62::              
+* unnamed-faq-63::              
+* unnamed-faq-64::              
+* unnamed-faq-65::              
+* unnamed-faq-66::              
+* unnamed-faq-67::              
+* unnamed-faq-68::              
+* unnamed-faq-69::              
+* unnamed-faq-70::              
+* unnamed-faq-71::              
+* unnamed-faq-72::              
+* unnamed-faq-73::              
+* unnamed-faq-74::              
+* unnamed-faq-75::              
+* unnamed-faq-76::              
+* unnamed-faq-77::              
+* unnamed-faq-78::              
+* unnamed-faq-79::              
+* unnamed-faq-80::              
+* unnamed-faq-81::              
+* unnamed-faq-82::              
+* unnamed-faq-83::              
+* unnamed-faq-84::              
+* unnamed-faq-85::              
+* unnamed-faq-86::              
+* unnamed-faq-87::              
+* unnamed-faq-88::              
+* unnamed-faq-90::              
+* unnamed-faq-91::              
+* unnamed-faq-92::              
+* unnamed-faq-93::              
+* unnamed-faq-94::              
+* unnamed-faq-95::              
+* unnamed-faq-96::              
+* unnamed-faq-97::              
+* unnamed-faq-98::              
+* unnamed-faq-99::              
+* unnamed-faq-100::             
+* unnamed-faq-101::             
+* What is the difference between YYLEX_PARAM and YY_DECL?::
+* Why do I get "conflicting types for yylex" error?::
+* How do I access the values set in a Flex action from within a Bison action?::
+@end menu
+
+@node  When was flex born?
+@unnumberedsec When was flex born?
+
+Vern Paxson took over
+the @cite{Software Tools} lex project from Jef Poskanzer in 1982.  At that point it
+was written in Ratfor.  Around 1987 or so, Paxson translated it into C, and
+a legend was born :-).
+
+@node How do I expand backslash-escape sequences in C-style quoted strings?
+@unnumberedsec How do I expand backslash-escape sequences in C-style quoted strings?
+
+A key point when scanning quoted strings is that you cannot (easily) write
+a single rule that will precisely match the string if you allow things
+like embedded escape sequences and newlines.  If you try to match strings
+with a single rule then you'll wind up having to rescan the string anyway
+to find any escape sequences.
+
+Instead you can use exclusive start conditions and a set of rules, one for
+matching non-escaped text, one for matching a single escape, one for
+matching an embedded newline, and one for recognizing the end of the
+string.  Each of these rules is then faced with the question of where to
+put its intermediary results.  The best solution is for the rules to
+append their local value of @code{yytext} to the end of a ``string literal''
+buffer.  A rule like the escape-matcher will append to the buffer the
+meaning of the escape sequence rather than the literal text in @code{yytext}.
+In this way, @code{yytext} does not need to be modified at all.
+
+@node  Why do flex scanners call fileno if it is not ANSI compatible?
+@unnumberedsec Why do flex scanners call fileno if it is not ANSI compatible?
+
+Flex scanners call @code{fileno()} in order to get the file descriptor
+corresponding to @code{yyin}. The file descriptor may be passed to
+@code{isatty()} or @code{read()}, depending upon which @code{%options} you specified.
+If your system does not have @code{fileno()} support, to get rid of the
+@code{read()} call, do not specify @code{%option read}. To get rid of the @code{isatty()}
+call, you must specify one of @code{%option always-interactive} or
+@code{%option never-interactive}.
+
+@node  Does flex support recursive pattern definitions?
+@unnumberedsec Does flex support recursive pattern definitions?
+
+e.g.,
+
+@example
+@verbatim
+%%
+block   "{"({block}|{statement})*"}"
+@end verbatim
+@end example
+
+No. You cannot have recursive definitions.  The pattern-matching power of
+regular expressions in general (and therefore flex scanners, too) is
+limited.  In particular, regular expressions cannot ``balance'' parentheses
+to an arbitrary degree.  For example, it's impossible to write a regular
+expression that matches all strings containing the same number of '@{'s
+as '@}'s.  For more powerful pattern matching, you need a parser, such
+as @cite{GNU bison}.
+
+@node  How do I skip huge chunks of input (tens of megabytes) while using flex?
+@unnumberedsec How do I skip huge chunks of input (tens of megabytes) while using flex?
+
+Use @code{fseek()} (or @code{lseek()}) to position yyin, then call @code{yyrestart()}.
+
+@node  Flex is not matching my patterns in the same order that I defined them.
+@unnumberedsec Flex is not matching my patterns in the same order that I defined them.
+
+@code{flex} picks the
+rule that matches the most text (i.e., the longest possible input string).
+This is because @code{flex} uses an entirely different matching technique
+(``deterministic finite automata'') that actually does all of the matching
+simultaneously, in parallel.  (Seems impossible, but it's actually a fairly
+simple technique once you understand the principles.)
+
+A side-effect of this parallel matching is that when the input matches more
+than one rule, @code{flex} scanners pick the rule that matched the @emph{most} text. This
+is explained further in the manual, in the section @xref{Matching}.
+
+If you want @code{flex} to choose a shorter match, then you can work around this
+behavior by expanding your short
+rule to match more text, then put back the extra:
+
+@example
+@verbatim
+data_.*        yyless( 5 ); BEGIN BLOCKIDSTATE;
+@end verbatim
+@end example
+
+Another fix would be to make the second rule active only during the
+@code{<BLOCKIDSTATE>} start condition, and make that start condition exclusive
+by declaring it with @code{%x} instead of @code{%s}.
+
+A final fix is to change the input language so that the ambiguity for
+@samp{data_} is removed, by adding characters to it that don't match the
+identifier rule, or by removing characters (such as @samp{_}) from the
+identifier rule so it no longer matches @samp{data_}.  (Of course, you might
+also not have the option of changing the input language.)
+
+@node  My actions are executing out of order or sometimes not at all.
+@unnumberedsec My actions are executing out of order or sometimes not at all.
+
+Most likely, you have (in error) placed the opening @samp{@{} of the action
+block on a different line than the rule, e.g.,
+
+@example
+@verbatim
+^(foo|bar)
+{  <<<--- WRONG!
+
+}
+@end verbatim
+@end example
+
+@code{flex} requires that the opening @samp{@{} of an action associated with a rule
+begin on the same line as does the rule.  You need instead to write your rules
+as follows:
+
+@example
+@verbatim
+^(foo|bar)   {  // CORRECT!
+
+}
+@end verbatim
+@end example
+
+@node  How can I have multiple input sources feed into the same scanner at the same time?
+@unnumberedsec How can I have multiple input sources feed into the same scanner at the same time?
+
+If @dots{}
+@itemize
+@item
+your scanner is free of backtracking (verified using @code{flex}'s @samp{-b} flag),
+@item
+AND you run your scanner interactively (@samp{-I} option; default unless using special table
+compression options),
+@item
+AND you feed it one character at a time by redefining @code{YY_INPUT} to do so,
+@end itemize
+
+then every time it matches a token, it will have exhausted its input
+buffer (because the scanner is free of backtracking).  This means you
+can safely use @code{select()} at the point and only call @code{yylex()} for another
+token if @code{select()} indicates there's data available.
+
+That is, move the @code{select()} out from the input function to a point where
+it determines whether @code{yylex()} gets called for the next token.
+
+With this approach, you will still have problems if your input can arrive
+piecemeal; @code{select()} could inform you that the beginning of a token is
+available, you call @code{yylex()} to get it, but it winds up blocking waiting
+for the later characters in the token.
+
+Here's another way:  Move your input multiplexing inside of @code{YY_INPUT}.  That
+is, whenever @code{YY_INPUT} is called, it @code{select()}'s to see where input is
+available.  If input is available for the scanner, it reads and returns the
+next byte.  If input is available from another source, it calls whatever
+function is responsible for reading from that source.  (If no input is
+available, it blocks until some input is available.)  I've used this technique in an
+interpreter I wrote that both reads keyboard input using a @code{flex} scanner and
+IPC traffic from sockets, and it works fine.
+
+@node  Can I build nested parsers that work with the same input file?
+@unnumberedsec Can I build nested parsers that work with the same input file?
+
+This is not going to work without some additional effort.  The reason is
+that @code{flex} block-buffers the input it reads from @code{yyin}.  This means that the
+``outermost'' @code{yylex()}, when called, will automatically slurp up the first 8K
+of input available on yyin, and subsequent calls to other @code{yylex()}'s won't
+see that input.  You might be tempted to work around this problem by
+redefining @code{YY_INPUT} to only return a small amount of text, but it turns out
+that that approach is quite difficult.  Instead, the best solution is to
+combine all of your scanners into one large scanner, using a different
+exclusive start condition for each.
+
+@node  How can I match text only at the end of a file?
+@unnumberedsec How can I match text only at the end of a file?
+
+There is no way to write a rule which is ``match this text, but only if
+it comes at the end of the file''.  You can fake it, though, if you happen
+to have a character lying around that you don't allow in your input.
+Then you redefine @code{YY_INPUT} to call your own routine which, if it sees
+an @samp{EOF}, returns the magic character first (and remembers to return a
+real @code{EOF} next time it's called).  Then you could write:
+
+@example
+@verbatim
+<COMMENT>(.|\n)*{EOF_CHAR}    /* saw comment at EOF */
+@end verbatim
+@end example
+
+@node  How can I make REJECT cascade across start condition boundaries?
+@unnumberedsec How can I make REJECT cascade across start condition boundaries?
+
+You can do this as follows.  Suppose you have a start condition @samp{A}, and
+after exhausting all of the possible matches in @samp{<A>}, you want to try
+matches in @samp{<INITIAL>}.  Then you could use the following:
+
+@example
+@verbatim
+%x A
+%%
+<A>rule_that_is_long    ...; REJECT;
+<A>rule                 ...; REJECT; /* shorter rule */
+<A>etc.
+...
+<A>.|\n  {
+/* Shortest and last rule in <A>, so
+* cascaded REJECTs will eventually
+* wind up matching this rule.  We want
+* to now switch to the initial state
+* and try matching from there instead.
+*/
+yyless(0);    /* put back matched text */
+BEGIN(INITIAL);
+}
+@end verbatim
+@end example
+
+@node  Why cant I use fast or full tables with interactive mode?
+@unnumberedsec Why can't I use fast or full tables with interactive mode?
+
+One of the assumptions
+flex makes is that interactive applications are inherently slow (they're
+waiting on a human after all).
+It has to do with how the scanner detects that it must be finished scanning
+a token.  For interactive scanners, after scanning each character the current
+state is looked up in a table (essentially) to see whether there's a chance
+of another input character possibly extending the length of the match.  If
+not, the scanner halts.  For non-interactive scanners, the end-of-token test
+is much simpler, basically a compare with 0, so no memory bus cycles.  Since
+the test occurs in the innermost scanning loop, one would like to make it go
+as fast as possible.
+
+Still, it seems reasonable to allow the user to choose to trade off a bit
+of performance in this area to gain the corresponding flexibility.  There
+might be another reason, though, why fast scanners don't support the
+interactive option.
+
+@node  How much faster is -F or -f than -C?
+@unnumberedsec How much faster is -F or -f than -C?
+
+Much faster (factor of 2-3).
+
+@node  If I have a simple grammar cant I just parse it with flex?
+@unnumberedsec If I have a simple grammar can't I just parse it with flex?
+
+Is your grammar recursive? That's almost always a sign that you're
+better off using a parser/scanner rather than just trying to use a scanner
+alone.
+
+@node  Why doesn't yyrestart() set the start state back to INITIAL?
+@unnumberedsec Why doesn't yyrestart() set the start state back to INITIAL?
+
+There are two reasons.  The first is that there might
+be programs that rely on the start state not changing across file changes.
+The second is that beginning with @code{flex} version 2.4, use of @code{yyrestart()} is no longer required,
+so fixing the problem there doesn't solve the more general problem.
+
+@node  How can I match C-style comments?
+@unnumberedsec How can I match C-style comments?
+
+You might be tempted to try something like this:
+
+@example
+@verbatim
+"/*".*"*/"       // WRONG!
+@end verbatim
+@end example
+
+or, worse, this:
+
+@example
+@verbatim
+"/*"(.|\n)"*/"   // WRONG!
+@end verbatim
+@end example
+
+The above rules will eat too much input, and blow up on things like:
+
+@example
+@verbatim
+/* a comment */ do_my_thing( "oops */" );
+@end verbatim
+@end example
+
+Here is one way which allows you to track line information:
+
+@example
+@verbatim
+<INITIAL>{
+"/*"              BEGIN(IN_COMMENT);
+}
+<IN_COMMENT>{
+"*/"      BEGIN(INITIAL);
+[^*\n]+   // eat comment in chunks
+"*"       // eat the lone star
+\n        yylineno++;
+}
+@end verbatim
+@end example
+
+@node  The period isn't working the way I expected.
+@unnumberedsec The '.' isn't working the way I expected.
+
+Here are some tips for using @samp{.}:
+
+@itemize
+@item
+A common mistake is to place the grouping parenthesis AFTER an operator, when
+you really meant to place the parenthesis BEFORE the operator, e.g., you
+probably want this @code{(foo|bar)+} and NOT this @code{(foo|bar+)}.
+
+The first pattern matches the words @samp{foo} or @samp{bar} any number of
+times, e.g., it matches the text @samp{barfoofoobarfoo}. The
+second pattern matches a single instance of @code{foo} or a single instance of
+@code{bar} followed by one or more @samp{r}s, e.g., it matches the text @code{barrrr} .
+@item
+A @samp{.} inside @samp{[]}'s just means a literal@samp{.} (period),
+and NOT ``any character except newline''.
+@item
+Remember that @samp{.} matches any character EXCEPT @samp{\n} (and @samp{EOF}).
+If you really want to match ANY character, including newlines, then use @code{(.|\n)}
+Beware that the regex @code{(.|\n)+} will match your entire input!
+@item
+Finally, if you want to match a literal @samp{.} (a period), then use @samp{[.]} or @samp{"."}
+@end itemize
+
+@node  Can I get the flex manual in another format?
+@unnumberedsec Can I get the flex manual in another format?
+
+The @code{flex} source distribution  includes a texinfo manual. You are
+free to convert that texinfo into whatever format you desire. The
+@code{texinfo} package includes tools for conversion to a number of formats.
+
+@node  Does there exist a "faster" NDFA->DFA algorithm?
+@unnumberedsec Does there exist a "faster" NDFA->DFA algorithm?
+
+There's no way around the potential exponential running time - it
+can take you exponential time just to enumerate all of the DFA states.
+In practice, though, the running time is closer to linear, or sometimes
+quadratic.
+
+@node  How does flex compile the DFA so quickly?
+@unnumberedsec How does flex compile the DFA so quickly?
+
+There are two big speed wins that @code{flex} uses:
+
+@enumerate
+@item
+It analyzes the input rules to construct equivalence classes for those
+characters that always make the same transitions.  It then rewrites the NFA
+using equivalence classes for transitions instead of characters.  This cuts
+down the NFA->DFA computation time dramatically, to the point where, for
+uncompressed DFA tables, the DFA generation is often I/O bound in writing out
+the tables.
+@item
+It maintains hash values for previously computed DFA states, so testing
+whether a newly constructed DFA state is equivalent to a previously constructed
+state can be done very quickly, by first comparing hash values.
+@end enumerate
+
+@node  How can I use more than 8192 rules?
+@unnumberedsec How can I use more than 8192 rules?
+
+@code{Flex} is compiled with an upper limit of 8192 rules per scanner.
+If you need more than 8192 rules in your scanner, you'll have to recompile @code{flex}
+with the following changes in @file{flexdef.h}:
+
+@example
+@verbatim
+<    #define YY_TRAILING_MASK 0x2000
+<    #define YY_TRAILING_HEAD_MASK 0x4000
+--
+>    #define YY_TRAILING_MASK 0x20000000
+>    #define YY_TRAILING_HEAD_MASK 0x40000000
+@end verbatim
+@end example
+
+This should work okay as long as your C compiler uses 32 bit integers.
+But you might want to think about whether using such a huge number of rules
+is the best way to solve your problem.
+
+The following may also be relevant:
+
+With luck, you should be able to increase the definitions in flexdef.h for:
+
+@example
+@verbatim
+#define JAMSTATE -32766 /* marks a reference to the state that always jams */
+#define MAXIMUM_MNS 31999
+#define BAD_SUBSCRIPT -32767
+@end verbatim
+@end example
+
+recompile everything, and it'll all work.  Flex only has these 16-bit-like
+values built into it because a long time ago it was developed on a machine
+with 16-bit ints.  I've given this advice to others in the past but haven't
+heard back from them whether it worked okay or not...
+
+@node  How do I abandon a file in the middle of a scan and switch to a new file?
+@unnumberedsec How do I abandon a file in the middle of a scan and switch to a new file?
+
+Just call @code{yyrestart(newfile)}. Be sure to reset the start state if you want a
+``fresh start, since @code{yyrestart} does NOT reset the start state back to @code{INITIAL}.
+
+@node  How do I execute code only during initialization (only before the first scan)?
+@unnumberedsec How do I execute code only during initialization (only before the first scan)?
+
+You can specify an initial action by defining the macro @code{YY_USER_INIT} (though
+note that @code{yyout} may not be available at the time this macro is executed).  Or you
+can add to the beginning of your rules section:
+
+@example
+@verbatim
+%%
+    /* Must be indented! */
+    static int did_init = 0;
+
+    if ( ! did_init ){
+do_my_init();
+        did_init = 1;
+    }
+@end verbatim
+@end example
+
+@node  How do I execute code at termination?
+@unnumberedsec How do I execute code at termination?
+
+You can specify an action for the @code{<<EOF>>} rule.
+
+@node  Where else can I find help?
+@unnumberedsec Where else can I find help?
+
+You can find the flex homepage on the web at
+@uref{http://flex.sourceforge.net/}. See that page for details about flex
+mailing lists as well.
+
+@node  Can I include comments in the "rules" section of the file?
+@unnumberedsec Can I include comments in the "rules" section of the file?
+
+Yes, just about anywhere you want to. See the manual for the specific syntax.
+
+@node  I get an error about undefined yywrap().
+@unnumberedsec I get an error about undefined yywrap().
+
+You must supply a @code{yywrap()} function of your own, or link to @file{libfl.a}
+(which provides one), or use
+
+@example
+@verbatim
+%option noyywrap
+@end verbatim
+@end example
+
+in your source to say you don't want a @code{yywrap()} function.
+
+@node  How can I change the matching pattern at run time?
+@unnumberedsec How can I change the matching pattern at run time?
+
+You can't, it's compiled into a static table when flex builds the scanner.
+
+@node How can I expand macros in the input?
+@unnumberedsec How can I expand macros in the input?
+
+The best way to approach this problem is at a higher level, e.g., in the parser.
+
+However, you can do this using multiple input buffers.
+
+@example
+@verbatim
+%%
+macro/[a-z]+	{
+/* Saw the macro "macro" followed by extra stuff. */
+main_buffer = YY_CURRENT_BUFFER;
+expansion_buffer = yy_scan_string(expand(yytext));
+yy_switch_to_buffer(expansion_buffer);
+}
+
+<<EOF>>	{
+if ( expansion_buffer )
+{
+// We were doing an expansion, return to where
+// we were.
+yy_switch_to_buffer(main_buffer);
+yy_delete_buffer(expansion_buffer);
+expansion_buffer = 0;
+}
+else
+yyterminate();
+}
+@end verbatim
+@end example
+
+You probably will want a stack of expansion buffers to allow nested macros.
+From the above though hopefully the idea is clear.
+
+@node How can I build a two-pass scanner?
+@unnumberedsec How can I build a two-pass scanner?
+
+One way to do it is to filter the first pass to a temporary file,
+then process the temporary file on the second pass. You will probably see a
+performance hit, due to all the disk I/O.
+
+When you need to look ahead far forward like this, it almost always means
+that the right solution is to build a parse tree of the entire input, then
+walk it after the parse in order to generate the output.  In a sense, this
+is a two-pass approach, once through the text and once through the parse
+tree, but the performance hit for the latter is usually an order of magnitude
+smaller, since everything is already classified, in binary format, and
+residing in memory.
+
+@node How do I match any string not matched in the preceding rules?
+@unnumberedsec How do I match any string not matched in the preceding rules?
+
+One way to assign precedence, is to place the more specific rules first. If
+two rules would match the same input (same sequence of characters) then the
+first rule listed in the @code{flex} input wins, e.g.,
+
+@example
+@verbatim
+%%
+foo[a-zA-Z_]+    return FOO_ID;
+bar[a-zA-Z_]+    return BAR_ID;
+[a-zA-Z_]+       return GENERIC_ID;
+@end verbatim
+@end example
+
+Note that the rule @code{[a-zA-Z_]+} must come *after* the others.  It will match the
+same amount of text as the more specific rules, and in that case the
+@code{flex} scanner will pick the first rule listed in your scanner as the
+one to match.
+
+@node I am trying to port code from AT&T lex that uses yysptr and yysbuf.
+@unnumberedsec I am trying to port code from AT&T lex that uses yysptr and yysbuf.
+
+Those are internal variables pointing into the AT&T scanner's input buffer.  I
+imagine they're being manipulated in user versions of the @code{input()} and @code{unput()}
+functions.  If so, what you need to do is analyze those functions to figure out
+what they're doing, and then replace @code{input()} with an appropriate definition of
+@code{YY_INPUT}.  You shouldn't need to (and must not) replace
+@code{flex}'s @code{unput()} function.
+
+@node Is there a way to make flex treat NULL like a regular character?
+@unnumberedsec Is there a way to make flex treat NULL like a regular character?
+
+Yes, @samp{\0} and @samp{\x00} should both do the trick.  Perhaps you have an ancient
+version of @code{flex}.  The latest release is version @value{VERSION}.
+
+@node Whenever flex can not match the input it says "flex scanner jammed".
+@unnumberedsec Whenever flex can not match the input it says "flex scanner jammed".
+
+You need to add a rule that matches the otherwise-unmatched text,
+e.g.,
+
+@example
+@verbatim
+%option yylineno
+%%
+[[a bunch of rules here]]
+
+.	printf("bad input character '%s' at line %d\n", yytext, yylineno);
+@end verbatim
+@end example
+
+See @code{%option default} for more information.
+
+@node Why doesn't flex have non-greedy operators like perl does?
+@unnumberedsec Why doesn't flex have non-greedy operators like perl does?
+
+A DFA can do a non-greedy match by stopping
+the first time it enters an accepting state, instead of consuming input until
+it determines that no further matching is possible (a ``jam'' state).  This
+is actually easier to implement than longest leftmost match (which flex does).
+
+But it's also much less useful than longest leftmost match.  In general,
+when you find yourself wishing for non-greedy matching, that's usually a
+sign that you're trying to make the scanner do some parsing.  That's
+generally the wrong approach, since it lacks the power to do a decent job.
+Better is to either introduce a separate parser, or to split the scanner
+into multiple scanners using (exclusive) start conditions.
+
+You might have
+a separate start state once you've seen the @samp{BEGIN}. In that state, you
+might then have a regex that will match @samp{END} (to kick you out of the
+state), and perhaps @samp{(.|\n)} to get a single character within the chunk ...
+
+This approach also has much better error-reporting properties.
+
+@node Memory leak - 16386 bytes allocated by malloc.
+@unnumberedsec Memory leak - 16386 bytes allocated by malloc.
+@anchor{faq-memory-leak}
+
+UPDATED 2002-07-10: As of @code{flex} version 2.5.9, this leak means that you did not
+call @code{yylex_destroy()}. If you are using an earlier version of @code{flex}, then read
+on.
+
+The leak is about 16426 bytes.  That is, (8192 * 2 + 2) for the read-buffer, and
+about 40 for @code{struct yy_buffer_state} (depending upon alignment). The leak is in
+the non-reentrant C scanner only (NOT in the reentrant scanner, NOT in the C++
+scanner). Since @code{flex} doesn't know when you are done, the buffer is never freed.
+
+However, the leak won't multiply since the buffer is reused no matter how many
+times you call @code{yylex()}.
+
+If you want to reclaim the memory when you are completely done scanning, then
+you might try this:
+
+@example
+@verbatim
+/* For non-reentrant C scanner only. */
+yy_delete_buffer(YY_CURRENT_BUFFER);
+yy_init = 1;
+@end verbatim
+@end example
+
+Note: @code{yy_init} is an "internal variable", and hasn't been tested in this
+situation. It is possible that some other globals may need resetting as well.
+
+@node How do I track the byte offset for lseek()?
+@unnumberedsec How do I track the byte offset for lseek()?
+
+@example
+@verbatim
+>   We thought that it would be possible to have this number through the
+>   evaluation of the following expression:
+>
+>   seek_position = (no_buffers)*YY_READ_BUF_SIZE + yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf
+@end verbatim
+@end example
+
+While this is the right idea, it has two problems.  The first is that
+it's possible that @code{flex} will request less than @code{YY_READ_BUF_SIZE} during
+an invocation of @code{YY_INPUT} (or that your input source will return less
+even though @code{YY_READ_BUF_SIZE} bytes were requested).  The second problem
+is that when refilling its internal buffer, @code{flex} keeps some characters
+from the previous buffer (because usually it's in the middle of a match,
+and needs those characters to construct @code{yytext} for the match once it's
+done).  Because of this, @code{yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf} won't
+be exactly the number of characters already read from the current buffer.
+
+An alternative solution is to count the number of characters you've matched
+since starting to scan.  This can be done by using @code{YY_USER_ACTION}.  For
+example,
+
+@example
+@verbatim
+#define YY_USER_ACTION num_chars += yyleng;
+@end verbatim
+@end example
+
+(You need to be careful to update your bookkeeping if you use @code{yymore(}),
+@code{yyless()}, @code{unput()}, or @code{input()}.)
+
+@node How do I use my own I/O classes in a C++ scanner?
+@section How do I use my own I/O classes in a C++ scanner?
+
+When the flex C++ scanning class rewrite finally happens, then this sort of thing should become much easier.
+
+@cindex LexerOutput, overriding
+@cindex LexerInput, overriding
+@cindex overriding LexerOutput
+@cindex overriding LexerInput
+@cindex customizing I/O in C++ scanners
+@cindex C++ I/O, customizing
+You can do this by passing the various functions (such as @code{LexerInput()}
+and @code{LexerOutput()}) NULL @code{iostream*}'s, and then
+dealing with your own I/O classes surreptitiously (i.e., stashing them in
+special member variables).  This works because the only assumption about
+the lexer regarding what's done with the iostream's is that they're
+ultimately passed to @code{LexerInput()} and @code{LexerOutput}, which then do whatever
+is necessary with them.
+
+@c faq edit stopped here
+@node How do I skip as many chars as possible?
+@unnumberedsec How do I skip as many chars as possible?
+
+How do I skip as many chars as possible -- without interfering with the other
+patterns?
+
+In the example below, we want to skip over characters until we see the phrase
+"endskip". The following will @emph{NOT} work correctly (do you see why not?)
+
+@example
+@verbatim
+/* INCORRECT SCANNER */
+%x SKIP
+%%
+<INITIAL>startskip   BEGIN(SKIP);
+...
+<SKIP>"endskip"       BEGIN(INITIAL);
+<SKIP>.*             ;
+@end verbatim
+@end example
+
+The problem is that the pattern .* will eat up the word "endskip."
+The simplest (but slow) fix is:
+
+@example
+@verbatim
+<SKIP>"endskip"      BEGIN(INITIAL);
+<SKIP>.              ;
+@end verbatim
+@end example
+
+The fix involves making the second rule match more, without
+making it match "endskip" plus something else.  So for example:
+
+@example
+@verbatim
+<SKIP>"endskip"     BEGIN(INITIAL);
+<SKIP>[^e]+         ;
+<SKIP>.		        ;/* so you eat up e's, too */
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node deleteme00
+@unnumberedsec deleteme00
+@example
+@verbatim
+QUESTION:
+When was flex born?
+
+Vern Paxson took over
+the Software Tools lex project from Jef Poskanzer in 1982.  At that point it
+was written in Ratfor.  Around 1987 or so, Paxson translated it into C, and
+a legend was born :-).
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Are certain equivalent patterns faster than others?
+@unnumberedsec Are certain equivalent patterns faster than others?
+@example
+@verbatim
+To: Adoram Rogel <adoram@orna.hybridge.com>
+Subject: Re: Flex 2.5.2 performance questions
+In-reply-to: Your message of Wed, 18 Sep 96 11:12:17 EDT.
+Date: Wed, 18 Sep 96 10:51:02 PDT
+From: Vern Paxson <vern>
+
+[Note, the most recent flex release is 2.5.4, which you can get from
+ftp.ee.lbl.gov.  It has bug fixes over 2.5.2 and 2.5.3.]
+
+> 1. Using the pattern
+>    ([Ff](oot)?)?[Nn](ote)?(\.)?
+>    instead of
+>    (((F|f)oot(N|n)ote)|((N|n)ote)|((N|n)\.)|((F|f)(N|n)(\.)))
+>    (in a very complicated flex program) caused the program to slow from
+>    300K+/min to 100K/min (no other changes were done).
+
+These two are not equivalent.  For example, the first can match "footnote."
+but the second can only match "footnote".  This is almost certainly the
+cause in the discrepancy - the slower scanner run is matching more tokens,
+and/or having to do more backing up.
+
+> 2. Which of these two are better: [Ff]oot or (F|f)oot ?
+
+From a performance point of view, they're equivalent (modulo presumably
+minor effects such as memory cache hit rates; and the presence of trailing
+context, see below).  From a space point of view, the first is slightly
+preferable.
+
+> 3. I have a pattern that look like this:
+>    pats {p1}|{p2}|{p3}|...|{p50}     (50 patterns ORd)
+>
+>    running yet another complicated program that includes the following rule:
+>    <snext>{and}/{no4}{bb}{pats}
+>
+>    gets me to "too complicated - over 32,000 states"...
+
+I can't tell from this example whether the trailing context is variable-length
+or fixed-length (it could be the latter if {and} is fixed-length).  If it's
+variable length, which flex -p will tell you, then this reflects a basic
+performance problem, and if you can eliminate it by restructuring your
+scanner, you will see significant improvement.
+
+>    so I divided {pats} to {pats1}, {pats2},..., {pats5} each consists of about
+>    10 patterns and changed the rule to be 5 rules.
+>    This did compile, but what is the rule of thumb here ?
+
+The rule is to avoid trailing context other than fixed-length, in which for
+a/b, either the 'a' pattern or the 'b' pattern have a fixed length.  Use
+of the '|' operator automatically makes the pattern variable length, so in
+this case '[Ff]oot' is preferred to '(F|f)oot'.
+
+> 4. I changed a rule that looked like this:
+>    <snext8>{and}{bb}/{ROMAN}[^A-Za-z] { BEGIN...
+>
+>    to the next 2 rules:
+>    <snext8>{and}{bb}/{ROMAN}[A-Za-z] { ECHO;}
+>    <snext8>{and}{bb}/{ROMAN}         { BEGIN...
+>
+>    Again, I understand the using [^...] will cause a great performance loss
+
+Actually, it doesn't cause any sort of performance loss.  It's a surprising
+fact about regular expressions that they always match in linear time
+regardless of how complex they are.
+
+>    but are there any specific rules about it ?
+
+See the "Performance Considerations" section of the man page, and also
+the example in MISC/fastwc/.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Is backing up a big deal?
+@unnumberedsec Is backing up a big deal?
+@example
+@verbatim
+To: Adoram Rogel <adoram@hybridge.com>
+Subject: Re: Flex 2.5.2 performance questions
+In-reply-to: Your message of Thu, 19 Sep 96 10:16:04 EDT.
+Date: Thu, 19 Sep 96 09:58:00 PDT
+From: Vern Paxson <vern>
+
+> a lot about the backing up problem.
+> I believe that there lies my biggest problem, and I'll try to improve
+> it.
+
+Since you have variable trailing context, this is a bigger performance
+problem.  Fixing it is usually easier than fixing backing up, which in a
+complicated scanner (yours seems to fit the bill) can be extremely
+difficult to do correctly.
+
+You also don't mention what flags you are using for your scanner.
+-f makes a large speed difference, and -Cfe buys you nearly as much
+speed but the resulting scanner is considerably smaller.
+
+> I have an | operator in {and} and in {pats} so both of them are variable
+> length.
+
+-p should have reported this.
+
+> Is changing one of them to fixed-length is enough ?
+
+Yes.
+
+> Is it possible to change the 32,000 states limit ?
+
+Yes.  I've appended instructions on how.  Before you make this change,
+though, you should think about whether there are ways to fundamentally
+simplify your scanner - those are certainly preferable!
+
+		Vern
+
+To increase the 32K limit (on a machine with 32 bit integers), you increase
+the magnitude of the following in flexdef.h:
+
+#define JAMSTATE -32766 /* marks a reference to the state that always jams */
+#define MAXIMUM_MNS 31999
+#define BAD_SUBSCRIPT -32767
+#define MAX_SHORT 32700
+
+Adding a 0 or two after each should do the trick.
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Can I fake multi-byte character support?
+@unnumberedsec Can I fake multi-byte character support?
+@example
+@verbatim
+To: Heeman_Lee@hp.com
+Subject: Re: flex - multi-byte support?
+In-reply-to: Your message of Thu, 03 Oct 1996 17:24:04 PDT.
+Date: Fri, 04 Oct 1996 11:42:18 PDT
+From: Vern Paxson <vern>
+
+>      I assume as long as my *.l file defines the
+>      range of expected character code values (in octal format), flex will
+>      scan the file and read multi-byte characters correctly. But I have no
+>      confidence in this assumption.
+
+Your lack of confidence is justified - this won't work.
+
+Flex has in it a widespread assumption that the input is processed
+one byte at a time.  Fixing this is on the to-do list, but is involved,
+so it won't happen any time soon.  In the interim, the best I can suggest
+(unless you want to try fixing it yourself) is to write your rules in
+terms of pairs of bytes, using definitions in the first section:
+
+	X	\xfe\xc2
+	...
+	%%
+	foo{X}bar	found_foo_fe_c2_bar();
+
+etc.  Definitely a pain - sorry about that.
+
+By the way, the email address you used for me is ancient, indicating you
+have a very old version of flex.  You can get the most recent, 2.5.4, from
+ftp.ee.lbl.gov.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node deleteme01
+@unnumberedsec deleteme01
+@example
+@verbatim
+To: moleary@primus.com
+Subject: Re: Flex / Unicode compatibility question
+In-reply-to: Your message of Tue, 22 Oct 1996 10:15:42 PDT.
+Date: Tue, 22 Oct 1996 11:06:13 PDT
+From: Vern Paxson <vern>
+
+Unfortunately flex at the moment has a widespread assumption within it
+that characters are processed 8 bits at a time.  I don't see any easy
+fix for this (other than writing your rules in terms of double characters -
+a pain).  I also don't know of a wider lex, though you might try surfing
+the Plan 9 stuff because I know it's a Unicode system, and also the PCCT
+toolkit (try searching say Alta Vista for "Purdue Compiler Construction
+Toolkit").
+
+Fixing flex to handle wider characters is on the long-term to-do list.
+But since flex is a strictly spare-time project these days, this probably
+won't happen for quite a while, unless someone else does it first.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Can you discuss some flex internals?
+@unnumberedsec Can you discuss some flex internals?
+@example
+@verbatim
+To: Johan Linde <jl@theophys.kth.se>
+Subject: Re: translation of flex
+In-reply-to: Your message of Sun, 10 Nov 1996 09:16:36 PST.
+Date: Mon, 11 Nov 1996 10:33:50 PST
+From: Vern Paxson <vern>
+
+> I'm working for the Swedish team translating GNU program, and I'm currently
+> working with flex. I have a few questions about some of the messages which
+> I hope you can answer.
+
+All of the things you're wondering about, by the way, concerning flex
+internals - probably the only person who understands what they mean in
+English is me!  So I wouldn't worry too much about getting them right.
+That said ...
+
+> #: main.c:545
+> msgid "  %d protos created\n"
+>
+> Does proto mean prototype?
+
+Yes - prototypes of state compression tables.
+
+> #: main.c:539
+> msgid "  %d/%d (peak %d) template nxt-chk entries created\n"
+>
+> Here I'm mainly puzzled by 'nxt-chk'. I guess it means 'next-check'. (?)
+> However, 'template next-check entries' doesn't make much sense to me. To be
+> able to find a good translation I need to know a little bit more about it.
+
+There is a scheme in the Aho/Sethi/Ullman compiler book for compressing
+scanner tables.  It involves creating two pairs of tables.  The first has
+"base" and "default" entries, the second has "next" and "check" entries.
+The "base" entry is indexed by the current state and yields an index into
+the next/check table.  The "default" entry gives what to do if the state
+transition isn't found in next/check.  The "next" entry gives the next
+state to enter, but only if the "check" entry verifies that this entry is
+correct for the current state.  Flex creates templates of series of
+next/check entries and then encodes differences from these templates as a
+way to compress the tables.
+
+> #: main.c:533
+> msgid "  %d/%d base-def entries created\n"
+>
+> The same problem here for 'base-def'.
+
+See above.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unput() messes up yy_at_bol
+@unnumberedsec unput() messes up yy_at_bol
+@example
+@verbatim
+To: Xinying Li <xli@npac.syr.edu>
+Subject: Re: FLEX ?
+In-reply-to: Your message of Wed, 13 Nov 1996 17:28:38 PST.
+Date: Wed, 13 Nov 1996 19:51:54 PST
+From: Vern Paxson <vern>
+
+> "unput()" them to input flow, question occurs. If I do this after I scan
+> a carriage, the variable "YY_CURRENT_BUFFER->yy_at_bol" is changed. That
+> means the carriage flag has gone.
+
+You can control this by calling yy_set_bol().  It's described in the manual.
+
+>      And if in pre-reading it goes to the end of file, is anything done
+> to control the end of curren buffer and end of file?
+
+No, there's no way to put back an end-of-file.
+
+>      By the way I am using flex 2.5.2 and using the "-l".
+
+The latest release is 2.5.4, by the way.  It fixes some bugs in 2.5.2 and
+2.5.3.  You can get it from ftp.ee.lbl.gov.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node The | operator is not doing what I want
+@unnumberedsec The | operator is not doing what I want
+@example
+@verbatim
+To: Alain.ISSARD@st.com
+Subject: Re: Start condition with FLEX
+In-reply-to: Your message of Mon, 18 Nov 1996 09:45:02 PST.
+Date: Mon, 18 Nov 1996 10:41:34 PST
+From: Vern Paxson <vern>
+
+> I am not able to use the start condition scope and to use the | (OR) with
+> rules having start conditions.
+
+The problem is that if you use '|' as a regular expression operator, for
+example "a|b" meaning "match either 'a' or 'b'", then it must *not* have
+any blanks around it.  If you instead want the special '|' *action* (which
+from your scanner appears to be the case), which is a way of giving two
+different rules the same action:
+
+	foo	|
+	bar	matched_foo_or_bar();
+
+then '|' *must* be separated from the first rule by whitespace and *must*
+be followed by a new line.  You *cannot* write it as:
+
+	foo | bar	matched_foo_or_bar();
+
+even though you might think you could because yacc supports this syntax.
+The reason for this unfortunately incompatibility is historical, but it's
+unlikely to be changed.
+
+Your problems with start condition scope are simply due to syntax errors
+from your use of '|' later confusing flex.
+
+Let me know if you still have problems.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Why can't flex understand this variable trailing context pattern?
+@unnumberedsec Why can't flex understand this variable trailing context pattern?
+@example
+@verbatim
+To: Gregory Margo <gmargo@newton.vip.best.com>
+Subject: Re: flex-2.5.3 bug report
+In-reply-to: Your message of Sat, 23 Nov 1996 16:50:09 PST.
+Date: Sat, 23 Nov 1996 17:07:32 PST
+From: Vern Paxson <vern>
+
+> Enclosed is a lex file that "real" lex will process, but I cannot get
+> flex to process it.  Could you try it and maybe point me in the right direction?
+
+Your problem is that some of the definitions in the scanner use the '/'
+trailing context operator, and have it enclosed in ()'s.  Flex does not
+allow this operator to be enclosed in ()'s because doing so allows undefined
+regular expressions such as "(a/b)+".  So the solution is to remove the
+parentheses.  Note that you must also be building the scanner with the -l
+option for AT&T lex compatibility.  Without this option, flex automatically
+encloses the definitions in parentheses.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node The ^ operator isn't working
+@unnumberedsec The ^ operator isn't working
+@example
+@verbatim
+To: Thomas Hadig <hadig@toots.physik.rwth-aachen.de>
+Subject: Re: Flex Bug ?
+In-reply-to: Your message of Tue, 26 Nov 1996 14:35:01 PST.
+Date: Tue, 26 Nov 1996 11:15:05 PST
+From: Vern Paxson <vern>
+
+> In my lexer code, i have the line :
+> ^\*.*          { }
+>
+> Thus all lines starting with an astrix (*) are comment lines.
+> This does not work !
+
+I can't get this problem to reproduce - it works fine for me.  Note
+though that if what you have is slightly different:
+
+	COMMENT	^\*.*
+	%%
+	{COMMENT}	{ }
+
+then it won't work, because flex pushes back macro definitions enclosed
+in ()'s, so the rule becomes
+
+	(^\*.*)		{ }
+
+and now that the '^' operator is not at the immediate beginning of the
+line, it's interpreted as just a regular character.  You can avoid this
+behavior by using the "-l" lex-compatibility flag, or "%option lex-compat".
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Trailing context is getting confused with trailing optional patterns
+@unnumberedsec Trailing context is getting confused with trailing optional patterns
+@example
+@verbatim
+To: Adoram Rogel <adoram@hybridge.com>
+Subject: Re: Flex 2.5.4 BOF ???
+In-reply-to: Your message of Tue, 26 Nov 1996 16:10:41 PST.
+Date: Wed, 27 Nov 1996 10:56:25 PST
+From: Vern Paxson <vern>
+
+>     Organization(s)?/[a-z]
+>
+> This matched "Organizations" (looking in debug mode, the trailing s
+> was matched with trailing context instead of the optional (s) in the
+> end of the word.
+
+That should only happen with lex.  Flex can properly match this pattern.
+(That might be what you're saying, I'm just not sure.)
+
+> Is there a way to avoid this dangerous trailing context problem ?
+
+Unfortunately, there's no easy way.  On the other hand, I don't see why
+it should be a problem.  Lex's matching is clearly wrong, and I'd hope
+that usually the intent remains the same as expressed with the pattern,
+so flex's matching will be correct.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Is flex GNU or not?
+@unnumberedsec Is flex GNU or not?
+@example
+@verbatim
+To: Cameron MacKinnon <mackin@interlog.com>
+Subject: Re: Flex documentation bug
+In-reply-to: Your message of Mon, 02 Dec 1996 00:07:08 PST.
+Date: Sun, 01 Dec 1996 22:29:39 PST
+From: Vern Paxson <vern>
+
+> I'm not sure how or where to submit bug reports (documentation or
+> otherwise) for the GNU project stuff ...
+
+Well, strictly speaking flex isn't part of the GNU project.  They just
+distribute it because no one's written a decent GPL'd lex replacement.
+So you should send bugs directly to me.  Those sent to the GNU folks
+sometimes find there way to me, but some may drop between the cracks.
+
+> In GNU Info, under the section 'Start Conditions', and also in the man
+> page (mine's dated April '95) is a nice little snippet showing how to
+> parse C quoted strings into a buffer, defined to be MAX_STR_CONST in
+> size. Unfortunately, no overflow checking is ever done ...
+
+This is already mentioned in the manual:
+
+Finally, here's an example of how to  match  C-style  quoted
+strings using exclusive start conditions, including expanded
+escape sequences (but not including checking  for  a  string
+that's too long):
+
+The reason for not doing the overflow checking is that it will needlessly
+clutter up an example whose main purpose is just to demonstrate how to
+use flex.
+
+The latest release is 2.5.4, by the way, available from ftp.ee.lbl.gov.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node ERASEME53
+@unnumberedsec ERASEME53
+@example
+@verbatim
+To: tsv@cs.UManitoba.CA
+Subject: Re: Flex (reg)..
+In-reply-to: Your message of Thu, 06 Mar 1997 23:50:16 PST.
+Date: Thu, 06 Mar 1997 15:54:19 PST
+From: Vern Paxson <vern>
+
+> [:alpha:] ([:alnum:] | \\_)*
+
+If your rule really has embedded blanks as shown above, then it won't
+work, as the first blank delimits the rule from the action.  (It wouldn't
+even compile ...)  You need instead:
+
+[:alpha:]([:alnum:]|\\_)*
+
+and that should work fine - there's no restriction on what can go inside
+of ()'s except for the trailing context operator, '/'.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node I need to scan if-then-else blocks and while loops
+@unnumberedsec I need to scan if-then-else blocks and while loops
+@example
+@verbatim
+To: "Mike Stolnicki" <mstolnic@ford.com>
+Subject: Re: FLEX help
+In-reply-to: Your message of Fri, 30 May 1997 13:33:27 PDT.
+Date: Fri, 30 May 1997 10:46:35 PDT
+From: Vern Paxson <vern>
+
+> We'd like to add "if-then-else", "while", and "for" statements to our
+> language ...
+> We've investigated many possible solutions.  The one solution that seems
+> the most reasonable involves knowing the position of a TOKEN in yyin.
+
+I strongly advise you to instead build a parse tree (abstract syntax tree)
+and loop over that instead.  You'll find this has major benefits in keeping
+your interpreter simple and extensible.
+
+That said, the functionality you mention for get_position and set_position
+have been on the to-do list for a while.  As flex is a purely spare-time
+project for me, no guarantees when this will be added (in particular, it
+for sure won't be for many months to come).
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node ERASEME55
+@unnumberedsec ERASEME55
+@example
+@verbatim
+To: Colin Paul Adams <colin@colina.demon.co.uk>
+Subject: Re: Flex C++ classes and Bison
+In-reply-to: Your message of 09 Aug 1997 17:11:41 PDT.
+Date: Fri, 15 Aug 1997 10:48:19 PDT
+From: Vern Paxson <vern>
+
+> #define YY_DECL   int yylex (YYSTYPE *lvalp, struct parser_control
+> *parm)
+>
+> I have been trying  to get this to work as a C++ scanner, but it does
+> not appear to be possible (warning that it matches no declarations in
+> yyFlexLexer, or something like that).
+>
+> Is this supposed to be possible, or is it being worked on (I DID
+> notice the comment that scanner classes are still experimental, so I'm
+> not too hopeful)?
+
+What you need to do is derive a subclass from yyFlexLexer that provides
+the above yylex() method, squirrels away lvalp and parm into member
+variables, and then invokes yyFlexLexer::yylex() to do the regular scanning.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node ERASEME56
+@unnumberedsec ERASEME56
+@example
+@verbatim
+To: Mikael.Latvala@lmf.ericsson.se
+Subject: Re: Possible mistake in Flex v2.5 document
+In-reply-to: Your message of Fri, 05 Sep 1997 16:07:24 PDT.
+Date: Fri, 05 Sep 1997 10:01:54 PDT
+From: Vern Paxson <vern>
+
+> In that example you show how to count comment lines when using
+> C style /* ... */ comments. My question is, shouldn't you take into
+> account a scenario where end of a comment marker occurs inside
+> character or string literals?
+
+The scanner certainly needs to also scan character and string literals.
+However it does that (there's an example in the man page for strings), the
+lexer will recognize the beginning of the literal before it runs across the
+embedded "/*".  Consequently, it will finish scanning the literal before it
+even considers the possibility of matching "/*".
+
+Example:
+
+	'([^']*|{ESCAPE_SEQUENCE})'
+
+will match all the text between the ''s (inclusive).  So the lexer
+considers this as a token beginning at the first ', and doesn't even
+attempt to match other tokens inside it.
+
+I thinnk this subtlety is not worth putting in the manual, as I suspect
+it would confuse more people than it would enlighten.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node ERASEME57
+@unnumberedsec ERASEME57
+@example
+@verbatim
+To: "Marty Leisner" <leisner@sdsp.mc.xerox.com>
+Subject: Re: flex limitations
+In-reply-to: Your message of Sat, 06 Sep 1997 11:27:21 PDT.
+Date: Mon, 08 Sep 1997 11:38:08 PDT
+From: Vern Paxson <vern>
+
+> %%
+> [a-zA-Z]+       /* skip a line */
+>                 {  printf("got %s\n", yytext); }
+> %%
+
+What version of flex are you using?  If I feed this to 2.5.4, it complains:
+
+	"bug.l", line 5: EOF encountered inside an action
+	"bug.l", line 5: unrecognized rule
+	"bug.l", line 5: fatal parse error
+
+Not the world's greatest error message, but it manages to flag the problem.
+
+(With the introduction of start condition scopes, flex can't accommodate
+an action on a separate line, since it's ambiguous with an indented rule.)
+
+You can get 2.5.4 from ftp.ee.lbl.gov.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node Is there a repository for flex scanners?
+@unnumberedsec Is there a repository for flex scanners?
+
+Not that we know of. You might try asking on comp.compilers.
+
+@c TODO: Evaluate this faq.
+@node How can I conditionally compile or preprocess my flex input file?
+@unnumberedsec How can I conditionally compile or preprocess my flex input file?
+
+
+Flex doesn't have a preprocessor like C does.  You might try using m4, or the C
+preprocessor plus a sed script to clean up the result.
+
+
+@c TODO: Evaluate this faq.
+@node Where can I find grammars for lex and yacc?
+@unnumberedsec Where can I find grammars for lex and yacc?
+
+In the sources for flex and bison.
+
+@c TODO: Evaluate this faq.
+@node I get an end-of-buffer message for each character scanned.
+@unnumberedsec I get an end-of-buffer message for each character scanned.
+
+This will happen if your LexerInput() function returns only one character
+at a time, which can happen either if you're scanner is "interactive", or
+if the streams library on your platform always returns 1 for yyin->gcount().
+
+Solution: override LexerInput() with a version that returns whole buffers.
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-62
+@unnumberedsec unnamed-faq-62
+@example
+@verbatim
+To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
+Subject: Re: Flex maximums
+In-reply-to: Your message of Mon, 17 Nov 1997 17:16:06 PST.
+Date: Mon, 17 Nov 1997 17:16:15 PST
+From: Vern Paxson <vern>
+
+> I took a quick look into the flex-sources and altered some #defines in
+> flexdefs.h:
+>
+> 	#define INITIAL_MNS 64000
+> 	#define MNS_INCREMENT 1024000
+> 	#define MAXIMUM_MNS 64000
+
+The things to fix are to add a couple of zeroes to:
+
+#define JAMSTATE -32766 /* marks a reference to the state that always jams */
+#define MAXIMUM_MNS 31999
+#define BAD_SUBSCRIPT -32767
+#define MAX_SHORT 32700
+
+and, if you get complaints about too many rules, make the following change too:
+
+	#define YY_TRAILING_MASK 0x200000
+	#define YY_TRAILING_HEAD_MASK 0x400000
+
+- Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-63
+@unnumberedsec unnamed-faq-63
+@example
+@verbatim
+To: jimmey@lexis-nexis.com (Jimmey Todd)
+Subject: Re: FLEX question regarding istream vs ifstream
+In-reply-to: Your message of Mon, 08 Dec 1997 15:54:15 PST.
+Date: Mon, 15 Dec 1997 13:21:35 PST
+From: Vern Paxson <vern>
+
+>         stdin_handle = YY_CURRENT_BUFFER;
+>         ifstream fin( "aFile" );
+>         yy_switch_to_buffer( yy_create_buffer( fin, YY_BUF_SIZE ) );
+>
+> What I'm wanting to do, is pass the contents of a file thru one set
+> of rules and then pass stdin thru another set... It works great if, I
+> don't use the C++ classes. But since everything else that I'm doing is
+> in C++, I thought I'd be consistent.
+>
+> The problem is that 'yy_create_buffer' is expecting an istream* as it's
+> first argument (as stated in the man page). However, fin is a ifstream
+> object. Any ideas on what I might be doing wrong? Any help would be
+> appreciated. Thanks!!
+
+You need to pass &fin, to turn it into an ifstream* instead of an ifstream.
+Then its type will be compatible with the expected istream*, because ifstream
+is derived from istream.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-64
+@unnumberedsec unnamed-faq-64
+@example
+@verbatim
+To: Enda Fadian <fadiane@piercom.ie>
+Subject: Re: Question related to Flex man page?
+In-reply-to: Your message of Tue, 16 Dec 1997 15:17:34 PST.
+Date: Tue, 16 Dec 1997 14:17:09 PST
+From: Vern Paxson <vern>
+
+> Can you explain to me what is ment by a long-jump in relation to flex?
+
+Using the longjmp() function while inside yylex() or a routine called by it.
+
+> what is the flex activation frame.
+
+Just yylex()'s stack frame.
+
+> As far as I can see yyrestart will bring me back to the sart of the input
+> file and using flex++ isnot really an option!
+
+No, yyrestart() doesn't imply a rewind, even though its name might sound
+like it does.  It tells the scanner to flush its internal buffers and
+start reading from the given file at its present location.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-65
+@unnumberedsec unnamed-faq-65
+@example
+@verbatim
+To: hassan@larc.info.uqam.ca (Hassan Alaoui)
+Subject: Re: Need urgent Help
+In-reply-to: Your message of Sat, 20 Dec 1997 19:38:19 PST.
+Date: Sun, 21 Dec 1997 21:30:46 PST
+From: Vern Paxson <vern>
+
+> /usr/lib/yaccpar: In function `int yyparse()':
+> /usr/lib/yaccpar:184: warning: implicit declaration of function `int yylex(...)'
+>
+> ld: Undefined symbol
+>    _yylex
+>    _yyparse
+>    _yyin
+
+This is a known problem with Solaris C++ (and/or Solaris yacc).  I believe
+the fix is to explicitly insert some 'extern "C"' statements for the
+corresponding routines/symbols.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-66
+@unnumberedsec unnamed-faq-66
+@example
+@verbatim
+To: mc0307@mclink.it
+Cc: gnu@prep.ai.mit.edu
+Subject: Re: [mc0307@mclink.it: Help request]
+In-reply-to: Your message of Fri, 12 Dec 1997 17:57:29 PST.
+Date: Sun, 21 Dec 1997 22:33:37 PST
+From: Vern Paxson <vern>
+
+> This is my definition for float and integer types:
+> . . .
+> NZD          [1-9]
+> ...
+> I've tested my program on other lex version (on UNIX Sun Solaris an HP
+> UNIX) and it work well, so I think that my definitions are correct.
+> There are any differences between Lex and Flex?
+
+There are indeed differences, as discussed in the man page.  The one
+you are probably running into is that when flex expands a name definition,
+it puts parentheses around the expansion, while lex does not.  There's
+an example in the man page of how this can lead to different matching.
+Flex's behavior complies with the POSIX standard (or at least with the
+last POSIX draft I saw).
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-67
+@unnumberedsec unnamed-faq-67
+@example
+@verbatim
+To: hassan@larc.info.uqam.ca (Hassan Alaoui)
+Subject: Re: Thanks
+In-reply-to: Your message of Mon, 22 Dec 1997 16:06:35 PST.
+Date: Mon, 22 Dec 1997 14:35:05 PST
+From: Vern Paxson <vern>
+
+> Thank you very much for your help. I compile and link well with C++ while
+> declaring 'yylex ...' extern, But a little problem remains. I get a
+> segmentation default when executing ( I linked with lfl library) while it
+> works well when using LEX instead of flex. Do you have some ideas about the
+> reason for this ?
+
+The one possible reason for this that comes to mind is if you've defined
+yytext as "extern char yytext[]" (which is what lex uses) instead of
+"extern char *yytext" (which is what flex uses).  If it's not that, then
+I'm afraid I don't know what the problem might be.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-68
+@unnumberedsec unnamed-faq-68
+@example
+@verbatim
+To: "Bart Niswonger" <NISWONGR@almaden.ibm.com>
+Subject: Re: flex 2.5: c++ scanners & start conditions
+In-reply-to: Your message of Tue, 06 Jan 1998 10:34:21 PST.
+Date: Tue, 06 Jan 1998 19:19:30 PST
+From: Vern Paxson <vern>
+
+> The problem is that when I do this (using %option c++) start
+> conditions seem to not apply.
+
+The BEGIN macro modifies the yy_start variable.  For C scanners, this
+is a static with scope visible through the whole file.  For C++ scanners,
+it's a member variable, so it only has visible scope within a member
+function.  Your lexbegin() routine is not a member function when you
+build a C++ scanner, so it's not modifying the correct yy_start.  The
+diagnostic that indicates this is that you found you needed to add
+a declaration of yy_start in order to get your scanner to compile when
+using C++; instead, the correct fix is to make lexbegin() a member
+function (by deriving from yyFlexLexer).
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-69
+@unnumberedsec unnamed-faq-69
+@example
+@verbatim
+To: "Boris Zinin" <boris@ippe.rssi.ru>
+Subject: Re: current position in flex buffer
+In-reply-to: Your message of Mon, 12 Jan 1998 18:58:23 PST.
+Date: Mon, 12 Jan 1998 12:03:15 PST
+From: Vern Paxson <vern>
+
+> The problem is how to determine the current position in flex active
+> buffer when a rule is matched....
+
+You will need to keep track of this explicitly, such as by redefining
+YY_USER_ACTION to count the number of characters matched.
+
+The latest flex release, by the way, is 2.5.4, available from ftp.ee.lbl.gov.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-70
+@unnumberedsec unnamed-faq-70
+@example
+@verbatim
+To: Bik.Dhaliwal@bis.org
+Subject: Re: Flex question
+In-reply-to: Your message of Mon, 26 Jan 1998 13:05:35 PST.
+Date: Tue, 27 Jan 1998 22:41:52 PST
+From: Vern Paxson <vern>
+
+> That requirement involves knowing
+> the character position at which a particular token was matched
+> in the lexer.
+
+The way you have to do this is by explicitly keeping track of where
+you are in the file, by counting the number of characters scanned
+for each token (available in yyleng).  It may prove convenient to
+do this by redefining YY_USER_ACTION, as described in the manual.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-71
+@unnumberedsec unnamed-faq-71
+@example
+@verbatim
+To: Vladimir Alexiev <vladimir@cs.ualberta.ca>
+Subject: Re: flex: how to control start condition from parser?
+In-reply-to: Your message of Mon, 26 Jan 1998 05:50:16 PST.
+Date: Tue, 27 Jan 1998 22:45:37 PST
+From: Vern Paxson <vern>
+
+> It seems useful for the parser to be able to tell the lexer about such
+> context dependencies, because then they don't have to be limited to
+> local or sequential context.
+
+One way to do this is to have the parser call a stub routine that's
+included in the scanner's .l file, and consequently that has access ot
+BEGIN.  The only ugliness is that the parser can't pass in the state
+it wants, because those aren't visible - but if you don't have many
+such states, then using a different set of names doesn't seem like
+to much of a burden.
+
+While generating a .h file like you suggests is certainly cleaner,
+flex development has come to a virtual stand-still :-(, so a workaround
+like the above is much more pragmatic than waiting for a new feature.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-72
+@unnumberedsec unnamed-faq-72
+@example
+@verbatim
+To: Barbara Denny <denny@3com.com>
+Subject: Re: freebsd flex bug?
+In-reply-to: Your message of Fri, 30 Jan 1998 12:00:43 PST.
+Date: Fri, 30 Jan 1998 12:42:32 PST
+From: Vern Paxson <vern>
+
+> lex.yy.c:1996: parse error before `='
+
+This is the key, identifying this error.  (It may help to pinpoint
+it by using flex -L, so it doesn't generate #line directives in its
+output.)  I will bet you heavy money that you have a start condition
+name that is also a variable name, or something like that; flex spits
+out #define's for each start condition name, mapping them to a number,
+so you can wind up with:
+
+	%x foo
+	%%
+		...
+	%%
+	void bar()
+		{
+		int foo = 3;
+		}
+
+and the penultimate will turn into "int 1 = 3" after C preprocessing,
+since flex will put "#define foo 1" in the generated scanner.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-73
+@unnumberedsec unnamed-faq-73
+@example
+@verbatim
+To: Maurice Petrie <mpetrie@infoscigroup.com>
+Subject: Re: Lost flex .l file
+In-reply-to: Your message of Mon, 02 Feb 1998 14:10:01 PST.
+Date: Mon, 02 Feb 1998 11:15:12 PST
+From: Vern Paxson <vern>
+
+> I am curious as to
+> whether there is a simple way to backtrack from the generated source to
+> reproduce the lost list of tokens we are searching on.
+
+In theory, it's straight-forward to go from the DFA representation
+back to a regular-expression representation - the two are isomorphic.
+In practice, a huge headache, because you have to unpack all the tables
+back into a single DFA representation, and then write a program to munch
+on that and translate it into an RE.
+
+Sorry for the less-than-happy news ...
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-74
+@unnumberedsec unnamed-faq-74
+@example
+@verbatim
+To: jimmey@lexis-nexis.com (Jimmey Todd)
+Subject: Re: Flex performance question
+In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST.
+Date: Thu, 19 Feb 1998 08:48:51 PST
+From: Vern Paxson <vern>
+
+> What I have found, is that the smaller the data chunk, the faster the
+> program executes. This is the opposite of what I expected. Should this be
+> happening this way?
+
+This is exactly what will happen if your input file has embedded NULs.
+From the man page:
+
+A final note: flex is slow when matching NUL's, particularly
+when  a  token  contains multiple NUL's.  It's best to write
+rules which match short amounts of text if it's  anticipated
+that the text will often include NUL's.
+
+So that's the first thing to look for.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-75
+@unnumberedsec unnamed-faq-75
+@example
+@verbatim
+To: jimmey@lexis-nexis.com (Jimmey Todd)
+Subject: Re: Flex performance question
+In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST.
+Date: Thu, 19 Feb 1998 15:42:25 PST
+From: Vern Paxson <vern>
+
+So there are several problems.
+
+First, to go fast, you want to match as much text as possible, which
+your scanners don't in the case that what they're scanning is *not*
+a <RN> tag.  So you want a rule like:
+
+	[^<]+
+
+Second, C++ scanners are particularly slow if they're interactive,
+which they are by default.  Using -B speeds it up by a factor of 3-4
+on my workstation.
+
+Third, C++ scanners that use the istream interface are slow, because
+of how poorly implemented istream's are.  I built two versions of
+the following scanner:
+
+	%%
+	.*\n
+	.*
+	%%
+
+and the C version inhales a 2.5MB file on my workstation in 0.8 seconds.
+The C++ istream version, using -B, takes 3.8 seconds.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-76
+@unnumberedsec unnamed-faq-76
+@example
+@verbatim
+To: "Frescatore, David (CRD, TAD)" <frescatore@exc01crdge.crd.ge.com>
+Subject: Re: FLEX 2.5 & THE YEAR 2000
+In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT.
+Date: Wed, 03 Jun 1998 10:22:26 PDT
+From: Vern Paxson <vern>
+
+> I am researching the Y2K problem with General Electric R&D
+> and need to know if there are any known issues concerning
+> the above mentioned software and Y2K regardless of version.
+
+There shouldn't be, all it ever does with the date is ask the system
+for it and then print it out.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-77
+@unnumberedsec unnamed-faq-77
+@example
+@verbatim
+To: "Hans Dermot Doran" <htd@ibhdoran.com>
+Subject: Re: flex problem
+In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT.
+Date: Tue, 21 Jul 1998 14:23:34 PDT
+From: Vern Paxson <vern>
+
+> To overcome this, I gets() the stdin into a string and lex the string. The
+> string is lexed OK except that the end of string isn't lexed properly
+> (yy_scan_string()), that is the lexer dosn't recognise the end of string.
+
+Flex doesn't contain mechanisms for recognizing buffer endpoints.  But if
+you use fgets instead (which you should anyway, to protect against buffer
+overflows), then the final \n will be preserved in the string, and you can
+scan that in order to find the end of the string.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-78
+@unnumberedsec unnamed-faq-78
+@example
+@verbatim
+To: soumen@almaden.ibm.com
+Subject: Re: Flex++ 2.5.3 instance member vs. static member
+In-reply-to: Your message of Mon, 27 Jul 1998 02:10:04 PDT.
+Date: Tue, 28 Jul 1998 01:10:34 PDT
+From: Vern Paxson <vern>
+
+> %{
+> int mylineno = 0;
+> %}
+> ws      [ \t]+
+> alpha   [A-Za-z]
+> dig     [0-9]
+> %%
+>
+> Now you'd expect mylineno to be a member of each instance of class
+> yyFlexLexer, but is this the case?  A look at the lex.yy.cc file seems to
+> indicate otherwise; unless I am missing something the declaration of
+> mylineno seems to be outside any class scope.
+>
+> How will this work if I want to run a multi-threaded application with each
+> thread creating a FlexLexer instance?
+
+Derive your own subclass and make mylineno a member variable of it.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-79
+@unnumberedsec unnamed-faq-79
+@example
+@verbatim
+To: Adoram Rogel <adoram@hybridge.com>
+Subject: Re: More than 32K states change hangs
+In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT.
+Date: Tue, 04 Aug 1998 22:28:45 PDT
+From: Vern Paxson <vern>
+
+> Vern Paxson,
+>
+> I followed your advice, posted on Usenet bu you, and emailed to me
+> personally by you, on how to overcome the 32K states limit. I'm running
+> on Linux machines.
+> I took the full source of version 2.5.4 and did the following changes in
+> flexdef.h:
+> #define JAMSTATE -327660
+> #define MAXIMUM_MNS 319990
+> #define BAD_SUBSCRIPT -327670
+> #define MAX_SHORT 327000
+>
+> and compiled.
+> All looked fine, including check and bigcheck, so I installed.
+
+Hmmm, you shouldn't increase MAX_SHORT, though looking through my email
+archives I see that I did indeed recommend doing so.  Try setting it back
+to 32700; that should suffice that you no longer need -Ca.  If it still
+hangs, then the interesting question is - where?
+
+> Compiling the same hanged program with a out-of-the-box (RedHat 4.2
+> distribution of Linux)
+> flex 2.5.4 binary works.
+
+Since Linux comes with source code, you should diff it against what
+you have to see what problems they missed.
+
+> Should I always compile with the -Ca option now ? even short and simple
+> filters ?
+
+No, definitely not.  It's meant to be for those situations where you
+absolutely must squeeze every last cycle out of your scanner.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-80
+@unnumberedsec unnamed-faq-80
+@example
+@verbatim
+To: "Schmackpfeffer, Craig" <Craig.Schmackpfeffer@usa.xerox.com>
+Subject: Re: flex output for static code portion
+In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT.
+Date: Mon, 17 Aug 1998 23:57:42 PDT
+From: Vern Paxson <vern>
+
+> I would like to use flex under the hood to generate a binary file
+> containing the data structures that control the parse.
+
+This has been on the wish-list for a long time.  In principle it's
+straight-forward - you redirect mkdata() et al's I/O to another file,
+and modify the skeleton to have a start-up function that slurps these
+into dynamic arrays.  The concerns are (1) the scanner generation code
+is hairy and full of corner cases, so it's easy to get surprised when
+going down this path :-( ; and (2) being careful about buffering so
+that when the tables change you make sure the scanner starts in the
+correct state and reading at the right point in the input file.
+
+> I was wondering if you know of anyone who has used flex in this way.
+
+I don't - but it seems like a reasonable project to undertake (unlike
+numerous other flex tweaks :-).
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-81
+@unnumberedsec unnamed-faq-81
+@example
+@verbatim
+Received: from 131.173.17.11 (131.173.17.11 [131.173.17.11])
+	by ee.lbl.gov (8.9.1/8.9.1) with ESMTP id AAA03838
+	for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 00:47:57 -0700 (PDT)
+Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2])
+	by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694
+	for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 09:47:55 +0200
+Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200
+From: Georg Rehm <georg@hal.cl-ki.uni-osnabrueck.de>
+Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de>
+Subject: "flex scanner push-back overflow"
+To: vern@ee.lbl.gov
+Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST)
+Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
+X-NoJunk: Do NOT send commercial mail, spam or ads to this address!
+X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/
+X-Mailer: ELM [version 2.4ME+ PL28 (25)]
+MIME-Version: 1.0
+Content-Type: text/plain; charset=US-ASCII
+Content-Transfer-Encoding: 7bit
+
+Hi Vern,
+
+Yesterday, I encountered a strange problem: I use the macro processor m4
+to include some lengthy lists into a .l file. Following is a flex macro
+definition that causes some serious pain in my neck:
+
+AUTHOR           ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...])
+
+The complete list contains about 10kB. When I try to "flex" this file
+(on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased
+some of the predefined values in flexdefs.h) I get the error:
+
+myflex/flex -8  sentag.tmp.l
+flex scanner push-back overflow
+
+When I remove the slashes in the macro definition everything works fine.
+As I understand it, the double quotes escape the slash-character so it
+really means "/" and not "trailing context". Furthermore, I tried to
+escape the slashes with backslashes, but with no use, the same error message
+appeared when flexing the code.
+
+Do you have an idea what's going on here?
+
+Greetings from Germany,
+	Georg
+--
+Georg Rehm                                     georg@cl-ki.uni-osnabrueck.de
+Institute for Semantic Information Processing, University of Osnabrueck, FRG
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-82
+@unnumberedsec unnamed-faq-82
+@example
+@verbatim
+To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
+Subject: Re: "flex scanner push-back overflow"
+In-reply-to: Your message of Thu, 20 Aug 1998 09:47:54 PDT.
+Date: Thu, 20 Aug 1998 07:05:35 PDT
+From: Vern Paxson <vern>
+
+> myflex/flex -8  sentag.tmp.l
+> flex scanner push-back overflow
+
+Flex itself uses a flex scanner.  That scanner is running out of buffer
+space when it tries to unput() the humongous macro you've defined.  When
+you remove the '/'s, you make it small enough so that it fits in the buffer;
+removing spaces would do the same thing.
+
+The fix is to either rethink how come you're using such a big macro and
+perhaps there's another/better way to do it; or to rebuild flex's own
+scan.c with a larger value for
+
+	#define YY_BUF_SIZE 16384
+
+- Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-83
+@unnumberedsec unnamed-faq-83
+@example
+@verbatim
+To: Jan Kort <jan@research.techforce.nl>
+Subject: Re: Flex
+In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200.
+Date: Sat, 05 Sep 1998 00:59:49 PDT
+From: Vern Paxson <vern>
+
+> %%
+>
+> "TEST1\n"       { fprintf(stderr, "TEST1\n"); yyless(5); }
+> ^\n             { fprintf(stderr, "empty line\n"); }
+> .               { }
+> \n              { fprintf(stderr, "new line\n"); }
+>
+> %%
+> -- input ---------------------------------------
+> TEST1
+> -- output --------------------------------------
+> TEST1
+> empty line
+> ------------------------------------------------
+
+IMHO, it's not clear whether or not this is in fact a bug.  It depends
+on whether you view yyless() as backing up in the input stream, or as
+pushing new characters onto the beginning of the input stream.  Flex
+interprets it as the latter (for implementation convenience, I'll admit),
+and so considers the newline as in fact matching at the beginning of a
+line, as after all the last token scanned an entire line and so the
+scanner is now at the beginning of a new line.
+
+I agree that this is counter-intuitive for yyless(), given its
+functional description (it's less so for unput(), depending on whether
+you're unput()'ing new text or scanned text).  But I don't plan to
+change it any time soon, as it's a pain to do so.  Consequently,
+you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak
+your scanner into the behavior you desire.
+
+Sorry for the less-than-completely-satisfactory answer.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-84
+@unnumberedsec unnamed-faq-84
+@example
+@verbatim
+To: Patrick Krusenotto <krusenot@mac-info-link.de>
+Subject: Re: Problems with restarting flex-2.5.2-generated scanner
+In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT.
+Date: Thu, 24 Sep 1998 23:28:43 PDT
+From: Vern Paxson <vern>
+
+> I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately
+> trying to make my scanner restart with a new file after my parser stops
+> with a parse error. When my compiler restarts, the parser always
+> receives the token after the token (in the old file!) that caused the
+> parser error.
+
+I suspect the problem is that your parser has read ahead in order
+to attempt to resolve an ambiguity, and when it's restarted it picks
+up with that token rather than reading a fresh one.  If you're using
+yacc, then the special "error" production can sometimes be used to
+consume tokens in an attempt to get the parser into a consistent state.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-85
+@unnumberedsec unnamed-faq-85
+@example
+@verbatim
+To: Henric Jungheim <junghelh@pe-nelson.com>
+Subject: Re: flex 2.5.4a
+In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST.
+Date: Tue, 27 Oct 1998 16:50:14 PST
+From: Vern Paxson <vern>
+
+> This brings up a feature request:  How about a command line
+> option to specify the filename when reading from stdin?  That way one
+> doesn't need to create a temporary file in order to get the "#line"
+> directives to make sense.
+
+Use -o combined with -t (per the man page description of -o).
+
+> P.S., Is there any simple way to use non-blocking IO to parse multiple
+> streams?
+
+Simple, no.
+
+One approach might be to return a magic character on EWOULDBLOCK and
+have a rule
+
+	.*<magic-character>	// put back .*, eat magic character
+
+This is off the top of my head, not sure it'll work.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-86
+@unnumberedsec unnamed-faq-86
+@example
+@verbatim
+To: "Repko, Billy D" <billy.d.repko@intel.com>
+Subject: Re: Compiling scanners
+In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST.
+Date: Thu, 14 Jan 1999 00:25:30 PST
+From: Vern Paxson <vern>
+
+> It appears that maybe it cannot find the lfl library.
+
+The Makefile in the distribution builds it, so you should have it.
+It's exceedingly trivial, just a main() that calls yylex() and
+a yyrap() that always returns 1.
+
+> %%
+>       \n      ++num_lines; ++num_chars;
+>       .       ++num_chars;
+
+You can't indent your rules like this - that's where the errors are coming
+from.  Flex copies indented text to the output file, it's how you do things
+like
+
+	int num_lines_seen = 0;
+
+to declare local variables.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-87
+@unnumberedsec unnamed-faq-87
+@example
+@verbatim
+To: Erick Branderhorst <Erick.Branderhorst@asml.nl>
+Subject: Re: flex input buffer
+In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST.
+Date: Tue, 09 Feb 1999 21:03:37 PST
+From: Vern Paxson <vern>
+
+> In the flex.skl file the size of the default input buffers is set.  Can you
+> explain why this size is set and why it is such a high number.
+
+It's large to optimize performance when scanning large files.  You can
+safely make it a lot lower if needed.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-88
+@unnumberedsec unnamed-faq-88
+@example
+@verbatim
+To: "Guido Minnen" <guidomi@cogs.susx.ac.uk>
+Subject: Re: Flex error message
+In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST.
+Date: Thu, 25 Feb 1999 00:11:31 PST
+From: Vern Paxson <vern>
+
+> I'm extending a larger scanner written in Flex and I keep running into
+> problems. More specifically, I get the error message:
+> "flex: input rules are too complicated (>= 32000 NFA states)"
+
+Increase the definitions in flexdef.h for:
+
+#define JAMSTATE -32766 /* marks a reference to the state that always j
+ams */
+#define MAXIMUM_MNS 31999
+#define BAD_SUBSCRIPT -32767
+
+recompile everything, and it should all work.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-90
+@unnumberedsec unnamed-faq-90
+@example
+@verbatim
+To: "Dmitriy Goldobin" <gold@ems.chel.su>
+Subject: Re: FLEX trouble
+In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT.
+Date: Tue, 01 Jun 1999 00:15:07 PDT
+From: Vern Paxson <vern>
+
+>   I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20
+> but rule "/*"(.|\n)*"*/" don't work ?
+
+The second of these will have to scan the entire input stream (because
+"(.|\n)*" matches an arbitrary amount of any text) in order to see if
+it ends with "*/", terminating the comment.  That potentially will overflow
+the input buffer.
+
+>   More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error
+> 'unrecognized rule'.
+
+You can't use the '/' operator inside parentheses.  It's not clear
+what "(a/b)*" actually means.
+
+>   I now use workaround with state <comment>, but single-rule is
+> better, i think.
+
+Single-rule is nice but will always have the problem of either setting
+restrictions on comments (like not allowing multi-line comments) and/or
+running the risk of consuming the entire input stream, as noted above.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-91
+@unnumberedsec unnamed-faq-91
+@example
+@verbatim
+Received: from mc-qout4.whowhere.com (mc-qout4.whowhere.com [209.185.123.18])
+	by ee.lbl.gov (8.9.3/8.9.3) with SMTP id IAA05100
+	for <vern@ee.lbl.gov>; Tue, 15 Jun 1999 08:56:06 -0700 (PDT)
+Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999
+To: vern@ee.lbl.gov
+Date: Tue, 15 Jun 1999 08:55:43 -0700
+From: "Aki Niimura" <neko@my-deja.com>
+Message-ID: <KNONDOHDOBGAEAAA@my-deja.com>
+Mime-Version: 1.0
+Cc:
+X-Sent-Mail: on
+Reply-To:
+X-Mailer: MailCity Service
+Subject: A question on flex C++ scanner
+X-Sender-Ip: 12.72.207.61
+Organization: My Deja Email  (http://www.my-deja.com:80)
+Content-Type: text/plain; charset=us-ascii
+Content-Transfer-Encoding: 7bit
+
+Dear Dr. Paxon,
+
+I have been using flex for years.
+It works very well on many projects.
+Most case, I used it to generate a scanner on C language.
+However, one project I needed to generate  a scanner
+on C++ lanuage. Thanks to your enhancement, flex did
+the job.
+
+Currently, I'm working on enhancing my previous project.
+I need to deal with multiple input streams (recursive
+inclusion) in this scanner (C++).
+I did similar thing for another scanner (C) as you
+explained in your documentation.
+
+The generated scanner (C++) has necessary methods:
+- switch_to_buffer(struct yy_buffer_state *b)
+- yy_create_buffer(istream *is, int sz)
+- yy_delete_buffer(struct yy_buffer_state *b)
+
+However, I couldn't figure out how to access current
+buffer (yy_current_buffer).
+
+yy_current_buffer is a protected member of yyFlexLexer.
+I can't access it directly.
+Then, I thought yy_create_buffer() with is = 0 might
+return current stream buffer. But it seems not as far
+as I checked the source. (flex 2.5.4)
+
+I went through the Web in addition to Flex documentation.
+However, it hasn't been successful, so far.
+
+It is not my intention to bother you, but, can you
+comment about how to obtain the current stream buffer?
+
+Your response would be highly appreciated.
+
+Best regards,
+Aki Niimura
+
+--== Sent via Deja.com http://www.deja.com/ ==--
+Share what you know. Learn what you don't.
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-92
+@unnumberedsec unnamed-faq-92
+@example
+@verbatim
+To: neko@my-deja.com
+Subject: Re: A question on flex C++ scanner
+In-reply-to: Your message of Tue, 15 Jun 1999 08:55:43 PDT.
+Date: Tue, 15 Jun 1999 09:04:24 PDT
+From: Vern Paxson <vern>
+
+> However, I couldn't figure out how to access current
+> buffer (yy_current_buffer).
+
+Derive your own subclass from yyFlexLexer.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-93
+@unnumberedsec unnamed-faq-93
+@example
+@verbatim
+To: "Stones, Darren" <Darren.Stones@nectech.co.uk>
+Subject: Re: You're the man to see?
+In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT.
+Date: Wed, 23 Jun 1999 09:01:40 PDT
+From: Vern Paxson <vern>
+
+> I hope you can help me.  I am using Flex and Bison to produce an interpreted
+> language.  However all goes well until I try to implement an IF statement or
+> a WHILE.  I cannot get this to work as the parser parses all the conditions
+> eg. the TRUE and FALSE conditons to check for a rule match.  So I cannot
+> make a decision!!
+
+You need to use the parser to build a parse tree (= abstract syntax trwee),
+and when that's all done you recursively evaluate the tree, binding variables
+to values at that time.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-94
+@unnumberedsec unnamed-faq-94
+@example
+@verbatim
+To: Petr Danecek <petr@ics.cas.cz>
+Subject: Re: flex - question
+In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT.
+Date: Fri, 02 Jul 1999 16:52:13 PDT
+From: Vern Paxson <vern>
+
+> file, it takes an enormous amount of time. It is funny, because the
+> source code has only 12 rules!!! I think it looks like an exponencial
+> growth.
+
+Right, that's the problem - some patterns (those with a lot of
+ambiguity, where yours has because at any given time the scanner can
+be in the middle of all sorts of combinations of the different
+rules) blow up exponentially.
+
+For your rules, there is an easy fix.  Change the ".*" that comes fater
+the directory name to "[^ ]*".  With that in place, the rules are no
+longer nearly so ambiguous, because then once one of the directories
+has been matched, no other can be matched (since they all require a
+leading blank).
+
+If that's not an acceptable solution, then you can enter a start state
+to pick up the .*\n after each directory is matched.
+
+Also note that for speed, you'll want to add a ".*" rule at the end,
+otherwise rules that don't match any of the patterns will be matched
+very slowly, a character at a time.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-95
+@unnumberedsec unnamed-faq-95
+@example
+@verbatim
+To: Tielman Koekemoer <tielman@spi.co.za>
+Subject: Re: Please help.
+In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT.
+Date: Thu, 08 Jul 1999 08:20:39 PDT
+From: Vern Paxson <vern>
+
+> I was hoping you could help me with my problem.
+>
+> I tried compiling (gnu)flex on a Solaris 2.4 machine
+> but when I ran make (after configure) I got an error.
+>
+> --------------------------------------------------------------
+> gcc -c -I. -I. -g -O parse.c
+> ./flex -t -p  ./scan.l >scan.c
+> sh: ./flex: not found
+> *** Error code 1
+> make: Fatal error: Command failed for target `scan.c'
+> -------------------------------------------------------------
+>
+> What's strange to me is that I'm only
+> trying to install flex now. I then edited the Makefile to
+> and changed where it says "FLEX = flex" to "FLEX = lex"
+> ( lex: the native Solaris one ) but then it complains about
+> the "-p" option. Is there any way I can compile flex without
+> using flex or lex?
+>
+> Thanks so much for your time.
+
+You managed to step on the bootstrap sequence, which first copies
+initscan.c to scan.c in order to build flex.  Try fetching a fresh
+distribution from ftp.ee.lbl.gov.  (Or you can first try removing
+".bootstrap" and doing a make again.)
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-96
+@unnumberedsec unnamed-faq-96
+@example
+@verbatim
+To: Tielman Koekemoer <tielman@spi.co.za>
+Subject: Re: Please help.
+In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT.
+Date: Fri, 09 Jul 1999 00:27:20 PDT
+From: Vern Paxson <vern>
+
+> First I removed .bootstrap (and ran make) - no luck. I downloaded the
+> software but I still have the same problem. Is there anything else I
+> could try.
+
+Try:
+
+	cp initscan.c scan.c
+	touch scan.c
+	make scan.o
+
+If this last tries to first build scan.c from scan.l using ./flex, then
+your "make" is broken, in which case compile scan.c to scan.o by hand.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-97
+@unnumberedsec unnamed-faq-97
+@example
+@verbatim
+To: Sumanth Kamenani <skamenan@crl.nmsu.edu>
+Subject: Re: Error
+In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT.
+Date: Tue, 20 Jul 1999 00:18:26 PDT
+From: Vern Paxson <vern>
+
+> I am getting a compilation error. The error is given as "unknown symbol- yylex".
+
+The parser relies on calling yylex(), but you're instead using the C++ scanning
+class, so you need to supply a yylex() "glue" function that calls an instance
+scanner of the scanner (e.g., "scanner->yylex()").
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-98
+@unnumberedsec unnamed-faq-98
+@example
+@verbatim
+To: daniel@synchrods.synchrods.COM (Daniel Senderowicz)
+Subject: Re: lex
+In-reply-to: Your message of Mon, 22 Nov 1999 11:19:04 PST.
+Date: Tue, 23 Nov 1999 15:54:30 PST
+From: Vern Paxson <vern>
+
+Well, your problem is the
+
+switch (yybgin-yysvec-1) {      /* witchcraft */
+
+at the beginning of lex rules.  "witchcraft" == "non-portable".  It's
+assuming knowledge of the AT&T lex's internal variables.
+
+For flex, you can probably do the equivalent using a switch on YYSTATE.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-99
+@unnumberedsec unnamed-faq-99
+@example
+@verbatim
+To: archow@hss.hns.com
+Subject: Re: Regarding distribution of flex and yacc based grammars
+In-reply-to: Your message of Sun, 19 Dec 1999 17:50:24 +0530.
+Date: Wed, 22 Dec 1999 01:56:24 PST
+From: Vern Paxson <vern>
+
+> When we provide the customer with an object code distribution, is it
+> necessary for us to provide source
+> for the generated C files from flex and bison since they are generated by
+> flex and bison ?
+
+For flex, no.  I don't know what the current state of this is for bison.
+
+> Also, is there any requrirement for us to neccessarily  provide source for
+> the grammar files which are fed into flex and bison ?
+
+Again, for flex, no.
+
+See the file "COPYING" in the flex distribution for the legalese.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-100
+@unnumberedsec unnamed-faq-100
+@example
+@verbatim
+To: Martin Gallwey <gallweym@hyperion.moe.ul.ie>
+Subject: Re: Flex, and self referencing rules
+In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST.
+Date: Sat, 19 Feb 2000 18:33:16 PST
+From: Vern Paxson <vern>
+
+> However, I do not use unput anywhere. I do use self-referencing
+> rules like this:
+>
+> UnaryExpr               ({UnionExpr})|("-"{UnaryExpr})
+
+You can't do this - flex is *not* a parser like yacc (which does indeed
+allow recursion), it is a scanner that's confined to regular expressions.
+
+		Vern
+@end verbatim
+@end example
+
+@c TODO: Evaluate this faq.
+@node unnamed-faq-101
+@unnumberedsec unnamed-faq-101
+@example
+@verbatim
+To: slg3@lehigh.edu (SAMUEL L. GULDEN)
+Subject: Re: Flex problem
+In-reply-to: Your message of Thu, 02 Mar 2000 12:29:04 PST.
+Date: Thu, 02 Mar 2000 23:00:46 PST
+From: Vern Paxson <vern>
+
+If this is exactly your program:
+
+> digit [0-9]
+> digits {digit}+
+> whitespace [ \t\n]+
+>
+> %%
+> "[" { printf("open_brac\n");}
+> "]" { printf("close_brac\n");}
+> "+" { printf("addop\n");}
+> "*" { printf("multop\n");}
+> {digits} { printf("NUMBER = %s\n", yytext);}
+> whitespace ;
+
+then the problem is that the last rule needs to be "{whitespace}" !
+
+		Vern
+@end verbatim
+@end example
+
+@node What is the difference between YYLEX_PARAM and YY_DECL?
+@unnumberedsec What is the difference between YYLEX_PARAM and YY_DECL?
+
+YYLEX_PARAM is not a flex symbol. It is for Bison. It tells Bison to pass extra
+params when it calls yylex() from the parser.
+
+YY_DECL is the Flex declaration of yylex. The default is similar to this:
+
+@example
+@verbatim
+#define int yy_lex ()
+@end verbatim
+@end example
+
+
+@node Why do I get "conflicting types for yylex" error?
+@unnumberedsec Why do I get "conflicting types for yylex" error?
+
+This is a compiler error regarding a generated Bison parser, not a Flex scanner.
+It means you need a prototype of yylex() in the top of the Bison file.
+Be sure the prototype matches YY_DECL.
+
+@node How do I access the values set in a Flex action from within a Bison action?
+@unnumberedsec How do I access the values set in a Flex action from within a Bison action?
+
+With $1, $2, $3, etc. These are called "Semantic Values" in the Bison manual.
+See @ref{Top, , , bison, the GNU Bison Manual}.
+
+@node Appendices, Indices, FAQ, Top
+@appendix Appendices
+
+@menu
+* Makefiles and Flex::          
+* Bison Bridge::                
+* M4 Dependency::               
+* Common Patterns::               
+@end menu
+
+@node Makefiles and Flex, Bison Bridge, Appendices, Appendices
+@appendixsec Makefiles and Flex
+
+@cindex Makefile, syntax
+
+In this appendix, we provide tips for writing Makefiles to build your scanners.
+
+In a traditional build environment, we say that the @file{.c} files are the
+sources, and the @file{.o} files are the intermediate files. When using
+@code{flex}, however, the @file{.l} files are the sources, and the generated
+@file{.c} files (along with the @file{.o} files) are the intermediate files.
+This requires you to carefully plan your Makefile.
+
+Modern @command{make} programs understand that @file{foo.l} is intended to
+generate @file{lex.yy.c} or @file{foo.c}, and will behave
+accordingly@footnote{GNU @command{make} and GNU @command{automake} are two such
+programs that provide implicit rules for flex-generated scanners.}@footnote{GNU @command{automake}
+may generate code to execute flex in lex-compatible mode, or to stdout. If this is not what you want,
+then you should provide an explicit rule in your Makefile.am}.  The
+following Makefile does not explicitly instruct @command{make} how to build
+@file{foo.c} from @file{foo.l}. Instead, it relies on the implicit rules of the
+@command{make} program to build the intermediate file, @file{scan.c}:
+
+@cindex Makefile, example of implicit rules
+@example
+@verbatim
+    # Basic Makefile -- relies on implicit rules
+    # Creates "myprogram" from "scan.l" and "myprogram.c"
+    #
+    LEX=flex
+    myprogram: scan.o myprogram.o
+    scan.o: scan.l
+
+@end verbatim
+@end example
+
+
+For simple cases, the above may be sufficient. For other cases,
+you may have to explicitly instruct @command{make} how to build your scanner.
+The following is an example of a Makefile containing explicit rules:
+
+@cindex Makefile, explicit example
+@example
+@verbatim
+    # Basic Makefile -- provides explicit rules
+    # Creates "myprogram" from "scan.l" and "myprogram.c"
+    #
+    LEX=flex
+    myprogram: scan.o myprogram.o
+            $(CC) -o $@  $(LDFLAGS) $^
+
+    myprogram.o: myprogram.c
+            $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^
+
+    scan.o: scan.c
+            $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ -c $^
+
+    scan.c: scan.l
+            $(LEX) $(LFLAGS) -o $@ $^
+
+    clean:
+            $(RM) *.o scan.c
+
+@end verbatim
+@end example
+
+Notice in the above example that @file{scan.c} is in the @code{clean} target.
+This is because we consider the file @file{scan.c} to be an intermediate file.
+
+Finally, we provide a realistic example of a @code{flex} scanner used with a
+@code{bison} parser@footnote{This example also applies to yacc parsers.}.
+There is a tricky problem we have to deal with. Since a @code{flex} scanner
+will typically include a header file (e.g., @file{y.tab.h}) generated by the
+parser, we need to be sure that the header file is generated BEFORE the scanner
+is compiled. We handle this case in the following example:
+
+@example
+@verbatim
+    # Makefile example -- scanner and parser.
+    # Creates "myprogram" from "scan.l", "parse.y", and "myprogram.c"
+    #
+    LEX     = flex
+    YACC    = bison -y
+    YFLAGS  = -d
+    objects = scan.o parse.o myprogram.o
+
+    myprogram: $(objects)
+    scan.o: scan.l parse.c
+    parse.o: parse.y
+    myprogram.o: myprogram.c
+
+@end verbatim
+@end example
+
+In the above example, notice the line,
+
+@example
+@verbatim
+    scan.o: scan.l parse.c
+@end verbatim
+@end example
+
+, which lists the file @file{parse.c} (the generated parser) as a dependency of
+@file{scan.o}. We want to ensure that the parser is created before the scanner
+is compiled, and the above line seems to do the trick. Feel free to experiment
+with your specific implementation of @command{make}.
+
+
+For more details on writing Makefiles, see @ref{Top, , , make, The
+GNU Make Manual}.
+
+@node Bison Bridge, M4 Dependency, Makefiles and Flex, Appendices
+@section C Scanners with Bison Parsers
+
+@cindex bison, bridging with flex
+@vindex yylval
+@vindex yylloc
+@tindex YYLTYPE
+@tindex YYSTYPE
+
+This section describes the @code{flex} features useful when integrating
+@code{flex} with @code{GNU bison}@footnote{The features described here are
+purely optional, and are by no means the only way to use flex with bison.
+We merely provide some glue to ease development of your parser-scanner pair.}.
+Skip this section if you are not using
+@code{bison} with your scanner.  Here we discuss only the @code{flex}
+half of the @code{flex} and @code{bison} pair.  We do not discuss
+@code{bison} in any detail.  For more information about generating
+@code{bison} parsers, see @ref{Top, , , bison, the GNU Bison Manual}.
+
+A compatible @code{bison} scanner is generated by declaring @samp{%option
+bison-bridge} or by supplying @samp{--bison-bridge} when invoking @code{flex}
+from the command line.  This instructs @code{flex} that the macro
+@code{yylval} may be used. The data type for
+@code{yylval}, @code{YYSTYPE},
+is typically defined in a header file, included in section 1 of the
+@code{flex} input file.  For a list of functions and macros
+available, @xref{bison-functions}.
+
+The declaration of yylex becomes,
+
+@findex yylex (reentrant version)
+@example
+@verbatim
+      int yylex ( YYSTYPE * lvalp, yyscan_t scanner );
+@end verbatim
+@end example
+
+If @code{%option bison-locations} is specified, then the declaration
+becomes,
+
+@findex yylex (reentrant version)
+@example
+@verbatim
+      int yylex ( YYSTYPE * lvalp, YYLTYPE * llocp, yyscan_t scanner );
+@end verbatim
+@end example
+
+Note that the macros @code{yylval} and @code{yylloc} evaluate to pointers.
+Support for @code{yylloc} is optional in @code{bison}, so it is optional in
+@code{flex} as well. The following is an example of a @code{flex} scanner that
+is compatible with @code{bison}.
+
+@cindex bison, scanner to be called from bison
+@example
+@verbatim
+    /* Scanner for "C" assignment statements... sort of. */
+    %{
+    #include "y.tab.h"  /* Generated by bison. */
+    %}
+
+    %option bison-bridge bison-locations
+    %
+
+    [[:digit:]]+  { yylval->num = atoi(yytext);   return NUMBER;}
+    [[:alnum:]]+  { yylval->str = strdup(yytext); return STRING;}
+    "="|";"       { return yytext[0];}
+    .  {}
+    %
+@end verbatim
+@end example
+
+As you can see, there really is no magic here. We just use
+@code{yylval} as we would any other variable. The data type of
+@code{yylval} is generated by @code{bison}, and included in the file
+@file{y.tab.h}. Here is the corresponding @code{bison} parser:
+
+@cindex bison, parser
+@example
+@verbatim
+    /* Parser to convert "C" assignments to lisp. */
+    %{
+    /* Pass the argument to yyparse through to yylex. */
+    #define YYPARSE_PARAM scanner
+    #define YYLEX_PARAM   scanner
+    %}
+    %locations
+    %pure_parser
+    %union {
+        int num;
+        char* str;
+    }
+    %token <str> STRING
+    %token <num> NUMBER
+    %%
+    assignment:
+        STRING '=' NUMBER ';' {
+            printf( "(setf %s %d)", $1, $3 );
+       }
+    ;
+@end verbatim
+@end example
+
+@node M4 Dependency, Common Patterns, Bison Bridge, Appendices
+@section M4 Dependency
+@cindex m4
+The macro processor @code{m4}@footnote{The use of m4 is subject to change in
+future revisions of flex. It is not part of the public API of flex. Do not depend on it.}
+must be installed wherever flex is installed.
+@code{flex} invokes @samp{m4}, found by searching the directories in the
+@code{PATH} environment variable. Any code you place in section 1 or in the
+actions will be sent through m4. Please follow these rules to protect your
+code from unwanted @code{m4} processing.
+
+@itemize
+
+@item Do not use symbols that begin with, @samp{m4_}, such as, @samp{m4_define},
+or @samp{m4_include}, since those are reserved for @code{m4} macro names. If for 
+some reason you need m4_ as a prefix, use a preprocessor #define to get your
+symbol past m4 unmangled.
+
+@item Do not use the strings @samp{[[} or @samp{]]} anywhere in your code. The
+former is not valid in C, except within comments and strings, but the latter is valid in
+code such as @code{x[y[z]]}. The solution is simple. To get the literal string 
+@code{"]]"}, use @code{"]""]"}. To get the array notation @code{x[y[z]]},
+use @code{x[y[z] ]}. Flex will attempt to detect these sequences in user code, and
+escape them. However, it's best to avoid this complexity where possible, by
+removing such sequences from your code.
+
+@end itemize
+
+@code{m4} is only required at the time you run @code{flex}. The generated
+scanner is ordinary C or C++, and does @emph{not} require @code{m4}.
+
+@node Common Patterns, ,M4 Dependency, Appendices
+@section Common Patterns
+@cindex patterns, common
+
+This appendix provides examples of common regular expressions you might use
+in your scanner.
+
+@menu
+* Numbers::         
+* Identifiers::         
+* Quoted Constructs::       
+* Addresses::       
+@end menu
+
+
+@node Numbers, Identifiers, ,Common Patterns
+@subsection Numbers
+
+@table @asis
+
+@item C99 decimal constant
+@code{([[:digit:]]@{-@}[0])[[:digit:]]*}
+
+@item C99 hexadecimal constant
+@code{0[xX][[:xdigit:]]+}
+
+@item C99 octal constant
+@code{0[01234567]*}
+
+@item C99 floating point constant
+@verbatim
+ {dseq}      ([[:digit:]]+)
+ {dseq_opt}  ([[:digit:]]*)
+ {frac}      (({dseq_opt}"."{dseq})|{dseq}".")
+ {exp}       ([eE][+-]?{dseq})
+ {exp_opt}   ({exp}?)
+ {fsuff}     [flFL]
+ {fsuff_opt} ({fsuff}?)
+ {hpref}     (0[xX])
+ {hdseq}     ([[:xdigit:]]+)
+ {hdseq_opt} ([[:xdigit:]]*)
+ {hfrac}     (({hdseq_opt}"."{hdseq})|({hdseq}"."))
+ {bexp}      ([pP][+-]?{dseq})
+ {dfc}       (({frac}{exp_opt}{fsuff_opt})|({dseq}{exp}{fsuff_opt}))
+ {hfc}       (({hpref}{hfrac}{bexp}{fsuff_opt})|({hpref}{hdseq}{bexp}{fsuff_opt}))
+
+ {c99_floating_point_constant}  ({dfc}|{hfc})
+@end verbatim
+
+See C99 section 6.4.4.2 for the gory details.
+
+@end table
+
+@node Identifiers, Quoted Constructs, Numbers, Common Patterns
+@subsection Identifiers
+
+@table @asis
+
+@item C99 Identifier
+@verbatim
+ucn        ((\\u([[:xdigit:]]{4}))|(\\U([[:xdigit:]]{8})))
+nondigit    [_[:alpha:]]
+c99_id     ([_[:alpha:]]|{ucn})([_[:alnum:]]|{ucn})*
+@end verbatim
+
+Technically, the above pattern does not encompass all possible C99 identifiers, since C99 allows for
+"implementation-defined" characters. In practice, C compilers follow the above pattern, with the
+addition of the @samp{$} character.
+
+@item UTF-8 Encoded Unicode Code Point
+@verbatim
+[\x09\x0A\x0D\x20-\x7E]|[\xC2-\xDF][\x80-\xBF]|\xE0[\xA0-\xBF][\x80-\xBF]|[\xE1-\xEC\xEE\xEF]([\x80-\xBF]{2})|\xED[\x80-\x9F][\x80-\xBF]|\xF0[\x90-\xBF]([\x80-\xBF]{2})|[\xF1-\xF3]([\x80-\xBF]{3})|\xF4[\x80-\x8F]([\x80-\xBF]{2})
+@end verbatim
+
+@end table
+
+@node Quoted Constructs, Addresses, Identifiers, Common Patterns
+@subsection Quoted Constructs
+
+@table @asis
+@item C99 String Literal
+@code{L?\"([^\"\\\n]|(\\['\"?\\abfnrtv])|(\\([0123456]@{1,3@}))|(\\x[[:xdigit:]]+)|(\\u([[:xdigit:]]@{4@}))|(\\U([[:xdigit:]]@{8@})))*\"}
+
+@item C99 Comment
+@code{("/*"([^*]|"*"[^/])*"*/")|("/"(\\\n)*"/"[^\n]*)}
+
+Note that in C99, a @samp{//}-style comment may be split across lines,  and, contrary to popular belief,
+does not include the trailing @samp{\n} character.
+
+A better way to scan @samp{/* */} comments is by line, rather than matching
+possibly huge comments all at once. This will allow you to scan comments of
+unlimited length, as long as line breaks appear at sane intervals. This is also
+more efficient when used with automatic line number processing. @xref{option-yylineno}.
+
+@verbatim
+<INITIAL>{
+    "/*"      BEGIN(COMMENT);
+}
+<COMMENT>{
+    "*/"      BEGIN(0);
+    [^*\n]+   ;
+    "*"[^/]   ;
+    \n        ;
+}
+@end verbatim
+
+@end table
+
+@node Addresses, ,Quoted Constructs, Common Patterns
+@subsection Addresses
+
+@table @asis
+
+@item IPv4 Address
+@verbatim
+dec-octet     [0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]
+IPv4address   {dec-octet}\.{dec-octet}\.{dec-octet}\.{dec-octet}
+@end verbatim
+
+@item IPv6 Address
+@verbatim
+h16           [0-9A-Fa-f]{1,4}
+ls32          {h16}:{h16}|{IPv4address}
+IPv6address   ({h16}:){6}{ls32}|
+              ::({h16}:){5}{ls32}|
+              ({h16})?::({h16}:){4}{ls32}|
+              (({h16}:){0,1}{h16})?::({h16}:){3}{ls32}|
+              (({h16}:){0,2}{h16})?::({h16}:){2}{ls32}|
+              (({h16}:){0,3}{h16})?::{h16}:{ls32}|
+              (({h16}:){0,4}{h16})?::{ls32}|
+              (({h16}:){0,5}{h16})?::{h16}|
+              (({h16}:){0,6}{h16})?::
+@end verbatim
+
+See @uref{http://www.ietf.org/rfc/rfc2373.txt, RFC 2373} for details.
+Note that you have to fold the definition of @code{IPv6address} into one
+line and that it also matches the ``unspecified address'' ``::''.
+
+@item URI
+@code{(([^:/?#]+):)?("//"([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?}
+
+This pattern is nearly useless, since it allows just about any character
+to appear in a URI, including spaces and control characters.  See
+@uref{http://www.ietf.org/rfc/rfc2396.txt, RFC 2396} for details.
+
+@end table
+
+
+@node Indices,  , Appendices, Top
+@unnumbered Indices
+
+@menu
+* Concept Index::               
+* Index of Functions and Macros::  
+* Index of Variables::          
+* Index of Data Types::         
+* Index of Hooks::              
+* Index of Scanner Options::    
+@end menu
+
+@node Concept Index, Index of Functions and Macros, Indices, Indices
+@unnumberedsec Concept Index
+
+@printindex cp
+
+@node Index of Functions and Macros, Index of Variables, Concept Index, Indices
+@unnumberedsec Index of Functions and Macros
+
+This is an index of functions and preprocessor macros that look like functions.
+For macros that expand to variables or constants, see @ref{Index of Variables}.
+
+@printindex fn
+
+@node Index of Variables, Index of Data Types, Index of Functions and Macros, Indices
+@unnumberedsec Index of Variables
+
+This is an index of variables, constants, and preprocessor macros
+that expand to variables or constants.
+
+@printindex vr
+
+@node Index of Data Types, Index of Hooks, Index of Variables, Indices
+@unnumberedsec Index of Data Types
+@printindex tp
+
+@node Index of Hooks, Index of Scanner Options, Index of Data Types, Indices
+@unnumberedsec Index of Hooks
+
+This is an index of "hooks" that the user may define. These hooks typically  correspond
+to specific locations in the generated scanner, and may be used to insert arbitrary code.
+
+@printindex hk
+
+@node Index of Scanner Options,  , Index of Hooks, Indices
+@unnumberedsec Index of Scanner Options
+
+@printindex op
+
+@c A vim script to name the faq entries. delete this when faqs are no longer
+@c named "unnamed-faq-XXX".
+@c
+@c fu! Faq2 () range abort
+@c     let @r=input("Rename to: ")
+@c     exe "%s/" . @w . "/" . @r . "/g"
+@c     normal 'f
+@c endf
+@c nnoremap <F5>  1G/@node\s\+unnamed-faq-\d\+<cr>mfww"wy5ezt:call Faq2()<cr>
+
+@bye
diff --git a/ecs.c b/ecs.c
new file mode 100644
index 0000000..e2abbe4
--- /dev/null
+++ b/ecs.c
@@ -0,0 +1,219 @@
+/* ecs - equivalence class routines */
+
+/*  Copyright (c) 1990 The Regents of the University of California. */
+/*  All rights reserved. */
+
+/*  This code is derived from software contributed to Berkeley by */
+/*  Vern Paxson. */
+
+/*  The United States Government has rights in this work pursuant */
+/*  to contract no. DE-AC03-76SF00098 between the United States */
+/*  Department of Energy and the University of California. */
+
+/* This file is part of flex */
+
+/*  Redistribution and use in source and binary forms, with or without */
+/*  modification, are permitted provided that the following conditions */
+/*  are met: */
+
+/*  1. Redistributions of source code must retain the above copyright */
+/*     notice, this list of conditions and the following disclaimer. */
+/*  2. Redistributions in binary form must reproduce the above copyright */
+/*     notice, this list of conditions and the following disclaimer in the */
+/*     documentation and/or other materials provided with the distribution. */
+
+/*  Neither the name of the University nor the names of its contributors */
+/*  may be used to endorse or promote products derived from this software */
+/*  without specific prior written permission. */
+
+/*  THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR */
+/*  IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED */
+/*  WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR */
+/*  PURPOSE. */
+
+
+#include "flexdef.h"
+
+/* ccl2ecl - convert character classes to set of equivalence classes */
+
+void    ccl2ecl ()
+{
+	int     i, ich, newlen, cclp, ccls, cclmec;
+
+	for (i = 1; i <= lastccl; ++i) {
+		/* We loop through each character class, and for each character
+		 * in the class, add the character's equivalence class to the
+		 * new "character" class we are creating.  Thus when we are all
+		 * done, character classes will really consist of collections
+		 * of equivalence classes
+		 */
+
+		newlen = 0;
+		cclp = cclmap[i];
+
+		for (ccls = 0; ccls < ccllen[i]; ++ccls) {
+			ich = ccltbl[cclp + ccls];
+			cclmec = ecgroup[ich];
+
+			if (cclmec > 0) {
+				ccltbl[cclp + newlen] = cclmec;
+				++newlen;
+			}
+		}
+
+		ccllen[i] = newlen;
+	}
+}
+
+
+/* cre8ecs - associate equivalence class numbers with class members
+ *
+ * fwd is the forward linked-list of equivalence class members.  bck
+ * is the backward linked-list, and num is the number of class members.
+ *
+ * Returned is the number of classes.
+ */
+
+int     cre8ecs (fwd, bck, num)
+     int     fwd[], bck[], num;
+{
+	int     i, j, numcl;
+
+	numcl = 0;
+
+	/* Create equivalence class numbers.  From now on, ABS( bck(x) )
+	 * is the equivalence class number for object x.  If bck(x)
+	 * is positive, then x is the representative of its equivalence
+	 * class.
+	 */
+	for (i = 1; i <= num; ++i)
+		if (bck[i] == NIL) {
+			bck[i] = ++numcl;
+			for (j = fwd[i]; j != NIL; j = fwd[j])
+				bck[j] = -numcl;
+		}
+
+	return numcl;
+}
+
+
+/* mkeccl - update equivalence classes based on character class xtions
+ *
+ * synopsis
+ *    Char ccls[];
+ *    int lenccl, fwd[llsiz], bck[llsiz], llsiz, NUL_mapping;
+ *    void mkeccl( Char ccls[], int lenccl, int fwd[llsiz], int bck[llsiz],
+ *			int llsiz, int NUL_mapping );
+ *
+ * ccls contains the elements of the character class, lenccl is the
+ * number of elements in the ccl, fwd is the forward link-list of equivalent
+ * characters, bck is the backward link-list, and llsiz size of the link-list.
+ *
+ * NUL_mapping is the value which NUL (0) should be mapped to.
+ */
+
+void    mkeccl (ccls, lenccl, fwd, bck, llsiz, NUL_mapping)
+     Char    ccls[];
+     int     lenccl, fwd[], bck[], llsiz, NUL_mapping;
+{
+	int     cclp, oldec, newec;
+	int     cclm, i, j;
+	static unsigned char cclflags[CSIZE];	/* initialized to all '\0' */
+
+	/* Note that it doesn't matter whether or not the character class is
+	 * negated.  The same results will be obtained in either case.
+	 */
+
+	cclp = 0;
+
+	while (cclp < lenccl) {
+		cclm = ccls[cclp];
+
+		if (NUL_mapping && cclm == 0)
+			cclm = NUL_mapping;
+
+		oldec = bck[cclm];
+		newec = cclm;
+
+		j = cclp + 1;
+
+		for (i = fwd[cclm]; i != NIL && i <= llsiz; i = fwd[i]) {	/* look for the symbol in the character class */
+			for (; j < lenccl; ++j) {
+				register int ccl_char;
+
+				if (NUL_mapping && ccls[j] == 0)
+					ccl_char = NUL_mapping;
+				else
+					ccl_char = ccls[j];
+
+				if (ccl_char > i)
+					break;
+
+				if (ccl_char == i && !cclflags[j]) {
+					/* We found an old companion of cclm
+					 * in the ccl.  Link it into the new
+					 * equivalence class and flag it as
+					 * having been processed.
+					 */
+
+					bck[i] = newec;
+					fwd[newec] = i;
+					newec = i;
+					/* Set flag so we don't reprocess. */
+					cclflags[j] = 1;
+
+					/* Get next equivalence class member. */
+					/* continue 2 */
+					goto next_pt;
+				}
+			}
+
+			/* Symbol isn't in character class.  Put it in the old
+			 * equivalence class.
+			 */
+
+			bck[i] = oldec;
+
+			if (oldec != NIL)
+				fwd[oldec] = i;
+
+			oldec = i;
+
+		      next_pt:;
+		}
+
+		if (bck[cclm] != NIL || oldec != bck[cclm]) {
+			bck[cclm] = NIL;
+			fwd[oldec] = NIL;
+		}
+
+		fwd[newec] = NIL;
+
+		/* Find next ccl member to process. */
+
+		for (++cclp; cclflags[cclp] && cclp < lenccl; ++cclp) {
+			/* Reset "doesn't need processing" flag. */
+			cclflags[cclp] = 0;
+		}
+	}
+}
+
+
+/* mkechar - create equivalence class for single character */
+
+void    mkechar (tch, fwd, bck)
+     int     tch, fwd[], bck[];
+{
+	/* If until now the character has been a proper subset of
+	 * an equivalence class, break it away to create a new ec
+	 */
+
+	if (fwd[tch] != NIL)
+		bck[fwd[tch]] = bck[tch];
+
+	if (bck[tch] != NIL)
+		fwd[bck[tch]] = fwd[tch];
+
+	fwd[tch] = NIL;
+	bck[tch] = NIL;
+}
diff --git a/examples/Makefile.am b/examples/Makefile.am
new file mode 100644
index 0000000..62688ba
--- /dev/null
+++ b/examples/Makefile.am
@@ -0,0 +1,29 @@
+# This file is part of flex.
+
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+
+# Neither the name of the University nor the names of its contributors
+# may be used to endorse or promote products derived from this software
+# without specific prior written permission.
+
+# THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE.
+
+EXTRA_DIST = \
+	testxxLexer.l \
+	debflex.awk \
+	README
+
+SUBDIRS = \
+	manual \
+	fastwc
diff --git a/examples/README b/examples/README
new file mode 100644
index 0000000..46f542b
--- /dev/null
+++ b/examples/README
@@ -0,0 +1,15 @@
+This directory contains some examples of what you can do with
+flex. These files are not tested regularly so you might have to tinker
+a bit before they work for you. Updates, new files and patches are welcome.
+
+	- debflex.awk, an awk script for anotating flex debug output.
+	  It presently only works with gawk and mawk, not with "old"
+	  or "new" awk.
+
+	- testxxLexer.l, a sample C++ program that uses flex's scanner
+	  class option ("-+").
+
+	- fastwc/, a subdirectory containing examples of how to use flex
+	  to write progressively higher-performance versions of the Unix
+	  "wc" utility.  This certainly should work with 2.5, but hasn't
+	  been tested.
diff --git a/examples/debflex.awk b/examples/debflex.awk
new file mode 100644
index 0000000..b1eda49
--- /dev/null
+++ b/examples/debflex.awk
@@ -0,0 +1,119 @@
+# Clarify the flex debug trace by substituting first line of each rule.
+# Francois Pinard <pinard@iro.umontreal.ca>, July 1990.
+#
+# Rewritten to process correctly \n's in scanner input.
+# BEGIN section modified to correct a collection of rules.
+# Michal Jaegermann <michal@phys.ualberta.ca>, December 1993
+#
+# Sample usage:
+#	flex -d PROGRAM.l
+#	gcc -o PROGRAM PROGRAM.c -lfl
+#	PROGRAM 2>&1 | gawk -f debflex.awk PROGRAM.l
+#
+# (VP's note: this script presently does not work with either "old" or
+#  "new" awk; fixes so it does will be welcome)
+
+BEGIN {
+    # Insure proper usage.
+
+    if (ARGC != 2) {
+	print "usage: gawk -f debflex.awk FLEX_SOURCE <DEBUG_OUTPUT";
+	exit (1);
+    }
+
+    # Remove and save the name of flex source.
+
+    source = ARGV[1];
+    ARGC--;
+
+    # Swallow the flex source file.
+
+    line = 0;
+    section = 1;
+    while (getline <source) {
+
+	# Count the lines.
+
+	line++;
+
+	# Count the sections.  When encountering section 3,
+	# break out of the awk BEGIN block.
+
+	if (match ($0, /^%%/)) {
+	    section++;
+	    if (section == 3) {
+		break;
+	    }
+	}
+	else {
+	    # Only the lines in section 2 which do not begin in a
+	    # tab or space might be referred to by the flex debug
+	    # trace.  Save only those lines.
+
+	    if (section == 2 && match ($0, /^[^ \t]/)) {
+		rules[line] = $0;
+	    }
+	}
+    }
+    dashes = "-----------------------------------------------------------";
+    collect = "";
+    line = 0;
+}
+
+# collect complete rule output from a scanner
+$0 !~ /^--/ {
+    collect = collect "\n" $0;
+    next;
+}
+# otherwise we have a new rule - process what we got so far
+{
+    process();
+}
+# and the same thing if we hit EOF
+END {
+    process();
+}
+
+function process() {
+
+    # splitting this way we loose some double dashes and
+    # left parentheses from echoed input - a small price to pay
+    n = split(collect, field, "\n--|[(]");
+
+    # this loop kicks in only when we already collected something
+    for (i = 1; i <= n; i++) {
+	if (0 != line) {
+	    # we do not care for traces of newlines.
+	    if (0 == match(field[i], /\"\n+\"[)]/)) {
+		if (rules[line]) {
+		    text = field[i];
+		    while ( ++i <= n) {
+			text = text field[i];
+		    }
+		    printf("%s:%d: %-8s -- %s\n",
+			   source, line, text, rules[line]);
+		}
+		else {
+		    print;
+		    printf "%s:%d: *** No such rule.\n", source, line;
+		}
+	    }
+	    line = 0;
+	    break;
+	}
+	if ("" != field[i]) {
+	    if ("end of buffer or a NUL)" == field[i]) {
+		print dashes;  # Simplify trace of buffer reloads
+		continue;
+	    }
+	    if (match(field[i], /accepting rule at line /)) {
+		# force interpretation of line as a number
+		line = 0 + substr(field[i], RLENGTH);
+		continue;
+	    }
+	    # echo everything else
+	    printf("--%s\n", field[i]);
+	}
+    }
+    collect = "\n" $0;  # ... and start next trace
+}
diff --git a/examples/fastwc/Makefile.am b/examples/fastwc/Makefile.am
new file mode 100644
index 0000000..b2831d9
--- /dev/null
+++ b/examples/fastwc/Makefile.am
@@ -0,0 +1,29 @@
+# This file is part of flex.
+
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+
+# Neither the name of the University nor the names of its contributors
+# may be used to endorse or promote products derived from this software
+# without specific prior written permission.
+
+# THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE.
+
+EXTRA_DIST = \
+	wc5.l \
+	wc4.l \
+	wc3.l \
+	wc2.l \
+	wc1.l \
+	mywc.c \
+	README
diff --git a/examples/fastwc/README b/examples/fastwc/README
new file mode 100644
index 0000000..0dd3afe
--- /dev/null
+++ b/examples/fastwc/README
@@ -0,0 +1,56 @@
+This directory contains some examples illustrating techniques for extracting
+high-performance from flex scanners.  Each program implements a simplified
+version of the Unix "wc" tool: read text from stdin and print the number of
+characters, words, and lines present in the text.  All programs were compiled
+using gcc (version unavailable, sorry) with the -O flag, and run on a
+SPARCstation 1+.  The input used was a PostScript file, mainly containing
+figures, with the following "wc" counts:
+
+	lines  words  characters
+	214217 635954 2592172
+
+
+The basic principles illustrated by these programs are:
+
+	- match as much text with each rule as possible
+	- adding rules does not slow you down!
+	- avoid backing up
+
+and the big caveat that comes with them is:
+
+	- you buy performance with decreased maintainability; make
+	  sure you really need it before applying the above techniques.
+
+See the "Performance Considerations" section of flexdoc for more
+details regarding these principles.
+
+
+The different versions of "wc":
+
+	mywc.c
+		a simple but fairly efficient C version
+
+	wc1.l	a naive flex "wc" implementation
+
+	wc2.l	somewhat faster; adds rules to match multiple tokens at once
+
+	wc3.l	faster still; adds more rules to match longer runs of tokens
+
+	wc4.l	fastest; still more rules added; hard to do much better
+		using flex (or, I suspect, hand-coding)
+
+	wc5.l	identical to wc3.l except one rule has been slightly
+		shortened, introducing backing-up
+
+Timing results (all times in user CPU seconds):
+
+	program	  time 	 notes
+	-------   ----   -----
+	wc1       16.4   default flex table compression (= -Cem)
+	wc1        6.7   -Cf compression option
+	/bin/wc	   5.8	 Sun's standard "wc" tool
+	mywc	   4.6   simple but better C implementation!
+	wc2	   4.6   as good as C implementation; built using -Cf
+	wc3	   3.8   -Cf
+	wc4	   3.3   -Cf
+	wc5	   5.7   -Cf; ouch, backing up is expensive
diff --git a/examples/fastwc/mywc.c b/examples/fastwc/mywc.c
new file mode 100644
index 0000000..92e5a36
--- /dev/null
+++ b/examples/fastwc/mywc.c
@@ -0,0 +1,26 @@
+/* A simple but fairly efficient C version of the Unix "wc" tool */
+
+#include <stdio.h>
+#include <ctype.h>
+
+main()
+{
+	register int c, cc = 0, wc = 0, lc = 0;
+	FILE *f = stdin;
+
+	while ((c = getc(f)) != EOF) {
+		++cc;
+		if (isgraph(c)) {
+			++wc;
+			do {
+				c = getc(f);
+				if (c == EOF)
+					goto done;
+				++cc;
+			} while (isgraph(c));
+		}
+		if (c == '\n')
+			++lc;
+	}
+done:	printf( "%8d%8d%8d\n", lc, wc, cc );
+}
diff --git a/examples/fastwc/wc1.l b/examples/fastwc/wc1.l
new file mode 100644
index 0000000..0d4fcf2
--- /dev/null
+++ b/examples/fastwc/wc1.l
@@ -0,0 +1,19 @@
+/* First cut at a flex-based "wc" tool. */
+
+ws    [ \t]
+nonws [^ \t\n]
+
+%option main noyywrap
+%%
+	int cc = 0, wc = 0, lc = 0;
+
+{nonws}+	cc += yyleng; ++wc;
+
+{ws}+		cc += yyleng;
+
+\n		++lc; ++cc;
+
+<<EOF>>		{
+		printf( "%8d %8d %8d\n", lc, wc, cc );
+		yyterminate();
+		}
diff --git a/examples/fastwc/wc2.l b/examples/fastwc/wc2.l
new file mode 100644
index 0000000..0da9953
--- /dev/null
+++ b/examples/fastwc/wc2.l
@@ -0,0 +1,21 @@
+/* Somewhat faster "wc" tool: match more text with each rule */
+
+ws    [ \t]
+nonws [^ \t\n]
+word  {ws}*{nonws}+
+
+%option main noyywrap
+%%
+	int cc = 0, wc = 0, lc = 0;
+
+{word}{ws}*	cc += yyleng; ++wc;
+{word}{ws}*\n	cc += yyleng; ++wc; ++lc;
+
+{ws}+		cc += yyleng;
+
+\n+		cc += yyleng; lc += yyleng;
+
+<<EOF>>		{
+		printf( "%8d %8d %8d\n", lc, wc, cc );
+		yyterminate();
+		}
diff --git a/examples/fastwc/wc3.l b/examples/fastwc/wc3.l
new file mode 100644
index 0000000..3cc5d57
--- /dev/null
+++ b/examples/fastwc/wc3.l
@@ -0,0 +1,25 @@
+/* Somewhat faster still: potentially match a lot of text with each rule */
+
+ws    [ \t]
+nonws [^ \t\n]
+word  {ws}*{nonws}+
+words {word}{ws}+
+
+%option main noyywrap
+%%
+	int cc = 0, wc = 0, lc = 0;
+
+{word}{ws}*		cc += yyleng; ++wc;
+{word}{ws}*\n		cc += yyleng; ++wc; ++lc;
+{words}{word}{ws}*	cc += yyleng; wc += 2;
+{words}{2}{word}{ws}*	cc += yyleng; wc += 3;
+{words}{3}{word}{ws}*	cc += yyleng; wc += 4;
+
+{ws}+			cc += yyleng;
+
+\n+			cc += yyleng; lc += yyleng;
+
+<<EOF>>		{
+		printf( "%8d %8d %8d\n", lc, wc, cc );
+		yyterminate();
+		}
diff --git a/examples/fastwc/wc4.l b/examples/fastwc/wc4.l
new file mode 100644
index 0000000..90c36ee
--- /dev/null
+++ b/examples/fastwc/wc4.l
@@ -0,0 +1,28 @@
+/* Fastest version of wc: add rules to pick up newlines, too */
+
+ws    [ \t]
+nonws [^ \t\n]
+word  {ws}*{nonws}+
+words {word}{ws}+
+
+%option main noyywrap
+%%
+	int cc = 0, wc = 0, lc = 0;
+
+{word}{ws}*		++wc; cc += yyleng;
+{word}{ws}*\n		++wc; cc += yyleng; ++lc;
+{words}{word}{ws}*	wc += 2; cc += yyleng;
+{words}{word}{ws}*\n	wc += 2; cc += yyleng; ++lc;
+{words}{2}{word}{ws}*	wc += 3; cc += yyleng;
+{words}{2}{word}{ws}*\n	wc += 3; cc += yyleng; ++lc;
+{words}{3}{word}{ws}*	wc += 4; cc += yyleng;
+{words}{3}{word}{ws}*\n	wc += 4; cc += yyleng; ++lc;
+
+{ws}+			cc += yyleng;
+
+\n+			cc += yyleng; lc += yyleng;
+
+<<EOF>>		{
+		printf( "%8d %8d %8d\n", lc, wc, cc );
+		yyterminate();
+		}
diff --git a/examples/fastwc/wc5.l b/examples/fastwc/wc5.l
new file mode 100644
index 0000000..c479480
--- /dev/null
+++ b/examples/fastwc/wc5.l
@@ -0,0 +1,25 @@
+/* Oops; slight change from wc3.l introduces backtracking */
+
+ws    [ \t]
+nonws [^ \t\n]
+word  {ws}*{nonws}+
+words {word}{ws}+
+
+%option main noyywrap
+%%
+	int cc = 0, wc = 0, lc = 0;
+
+{word}{ws}*		cc += yyleng; ++wc;
+{word}{ws}*\n		cc += yyleng; ++wc; ++lc;
+{words}{word}		cc += yyleng; wc += 2;	/* oops */
+{words}{2}{word}{ws}*	cc += yyleng; wc += 3;
+{words}{3}{word}{ws}*	cc += yyleng; wc += 4;
+
+{ws}+			cc += yyleng;
+
+\n+			cc += yyleng; lc += yyleng;
+
+<<EOF>>		{
+		printf( "%8d %8d %8d\n", lc, wc, cc );
+		yyterminate();
+		}
diff --git a/examples/manual/ChangeLog b/examples/manual/ChangeLog
new file mode 100644
index 0000000..fb5e4a2
--- /dev/null
+++ b/examples/manual/ChangeLog
@@ -0,0 +1,24 @@
+Tue Oct  5 21:51:59 1993  Vern Paxson
+
+	* Removed FILTER/ subdirectory.
+
+	* Removed alloca.c.
+
+	* Changed Makefile definition of CC to just "gcc -g", removed
+	  assumption of alloca being present.
+
+	* Added pointer to MISC/fastwc/ to wc.lex.
+
+Tue Jun  8 15:47:39 1993  Gavin Thomas Nicol  (nick at sillybugs)
+
+	* Changed main() in wc.lex extensively. The old version would not
+	  work correctly without the YY_NEW_FILE. (lex handles the older
+	  version OK though).
+
+	* Added a rule to expr.lex to handle whitespace. The old version
+ 	  reported an illegal character.
+
+	* Added -traditional to the gcc flags because the flex definition
+	  for free() clashes with some systems that have old header files.
+	  
+
diff --git a/examples/manual/Makefile.am b/examples/manual/Makefile.am
new file mode 100644
index 0000000..9ab3004
--- /dev/null
+++ b/examples/manual/Makefile.am
@@ -0,0 +1,55 @@
+# This file is part of flex.
+
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+
+# Neither the name of the University nor the names of its contributors
+# may be used to endorse or promote products derived from this software
+# without specific prior written permission.
+
+# THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE.
+
+EXTRA_DIST = \
+	ChangeLog \
+	Makefile.examples \
+	README \
+	cat.lex   \
+	dates.lex \
+	datetest.dat \
+	eof_rules.lex \
+	eof_test01.txt \
+	eof_test02.txt \
+	eof_test03.txt \
+	expr.lex \
+	expr.y \
+	front.lex \
+	front.y \
+	j2t.lex \
+	myname.lex \
+	myname.txt \
+	myname2.lex \
+	numbers.lex \
+	pas_include.lex \
+	pascal.lex \
+	reject.lex \
+	replace.lex \
+	string1.lex \
+	string2.lex \
+	strtest.dat \
+	unput.lex \
+	user_act.lex \
+	userinit.lex \
+	wc.lex \
+	yymore.lex \
+	yymore2.lex \
+	yymoretest.dat
diff --git a/examples/manual/Makefile.examples b/examples/manual/Makefile.examples
new file mode 100644
index 0000000..f4d8297
--- /dev/null
+++ b/examples/manual/Makefile.examples
@@ -0,0 +1,88 @@
+#############################################################
+#
+# Makefile     : Makefile for Flex examples.
+# Author       : G.T.Nicol
+# Last Updated : 1993/10/05
+#
+# If you use bison, you may have to supply an alloca
+#
+#############################################################
+
+CC       = gcc -g
+LEX      = flex -i -I 
+YACC     = bison -d -y
+ALLOCA   =
+
+############################################################
+#
+# DO NOT CHANGE ANYTHING FROM HERE ON !!!!!!!!! 
+#
+############################################################
+
+all: expr front myname eof wc replace user_act string1\
+     string2 yymore numbers dates cat
+
+expr:   expr.y expr.lex
+	$(YACC) expr.y
+	$(LEX)  expr.lex
+	$(CC) -o expr lex.yy.c y.tab.c $(ALLOCA) -ll -lm
+
+front:  front.y front.lex
+	$(YACC) front.y
+	$(LEX)  front.lex
+	$(CC) -o front lex.yy.c y.tab.c $(ALLOCA) -ll -lm
+
+numbers: numbers.lex
+	$(LEX)  numbers.lex
+	$(CC)   lex.yy.c -o numbers
+
+dates: dates.lex
+	$(LEX)  dates.lex
+	$(CC)   lex.yy.c -o dates -ll
+
+yymore: yymore.lex
+	$(LEX)  yymore.lex
+	$(CC)   lex.yy.c -o yymore -ll
+
+string1: string1.lex
+	$(LEX)  string1.lex
+	$(CC)   lex.yy.c -o string1 -ll
+
+string2: string2.lex
+	$(LEX)  string2.lex
+	$(CC)   lex.yy.c -o string2 -ll
+
+myname: myname.lex
+	$(LEX)  myname.lex
+	$(CC)   lex.yy.c -o myname -ll
+
+myname2: myname2.lex
+	$(LEX)  myname2.lex
+	$(CC)   lex.yy.c -o myname2 -ll
+
+eof:    eof_rules.lex
+	$(LEX)  eof_rules.lex
+	$(CC)   lex.yy.c -o eof -ll
+
+wc:     wc.lex
+	$(LEX)  wc.lex
+	$(CC)   lex.yy.c -o wc -ll
+
+cat:    cat.lex
+	$(LEX)  cat.lex
+	$(CC)   lex.yy.c -o cat -ll
+
+replace: replace.lex
+	$(LEX)  replace.lex
+	$(CC)   lex.yy.c -o replace -ll
+
+user_act: expr.y expr.lex
+	$(LEX)  user_act.lex
+	$(CC) -o user_act lex.yy.c -ll 
+
+clean:
+	rm -f *.BAK *.o core *~* *.a 
+	rm -f *.tab.h *.tab.c
+	rm -f myname expr lex.yy.c *.out eof wc yymore
+	rm -f replace front user_act string1 string2
+	rm -f dates numbers cat
diff --git a/examples/manual/README b/examples/manual/README
new file mode 100644
index 0000000..e11569a
--- /dev/null
+++ b/examples/manual/README
@@ -0,0 +1,12 @@
+This directory contains the example programs from the manual, and a
+few other things as well. To make all the programs, simply type "make
+-f Makefile.examples", and assuming you have flex and gcc, all will be
+well.
+
+To build the programs individually, type
+
+   make -f Makefile.examples program_name
+
+For example:
+
+   make -f Makefile.examples expr
diff --git a/examples/manual/cat.lex b/examples/manual/cat.lex
new file mode 100644
index 0000000..7890aa2
--- /dev/null
+++ b/examples/manual/cat.lex
@@ -0,0 +1,45 @@
+/*
+ * cat.lex: A demonstration of YY_NEW_FILE.
+ */
+
+%{
+#include <stdio.h>
+
+char **names = NULL;
+int  current = 1;
+%}
+
+%%
+<<EOF>> {
+           current += 1;
+           if(names[current] != NULL){
+              yyin = fopen(names[current],"r");
+              if(yyin == NULL){
+                fprintf(stderr,"cat: unable to open %s\n",
+                        names[current]);
+                yyterminate();
+              }
+              YY_NEW_FILE;
+           } else {
+             yyterminate();
+           }
+        }
+%%
+
+int main(int argc, char **argv)
+{
+    if(argc < 2){
+       fprintf(stderr,"Usage: cat files....\n");
+       exit(1);
+    }
+    names = argv;
+
+    yyin = fopen(names[current],"r");
+    if(yyin == NULL){
+      fprintf(stderr,"cat: unable to open %s\n",
+              names[current]);
+      yyterminate();
+    }
+
+    yylex();
+}
diff --git a/examples/manual/dates.lex b/examples/manual/dates.lex
new file mode 100644
index 0000000..9429e1d
--- /dev/null
+++ b/examples/manual/dates.lex
@@ -0,0 +1,106 @@
+/*
+ * dates.lex: An example of using start states to
+ *            distinguish between different date formats.
+ */
+
+%{
+#include <ctype.h>
+
+char month[20],dow[20],day[20],year[20];
+
+%}
+
+skip        of|the|[ \t,]*
+
+mon	    (mon(day)?)
+tue	    (tue(sday)?)
+wed	    (wed(nesday)?)
+thu	    (thu(rsday)?)
+fri	    (fri(day)?)
+sat	    (sat(urday)?)
+sun	    (sun(day)?)
+
+day_of_the_week	({mon}|{tue}|{wed}|{thu}|{fri}|{sat}|{sun})
+
+jan	    (jan(uary)?)
+feb	    (feb(ruary)?)
+mar	    (mar(ch)?)
+apr	    (apr(il)?)
+may	    (may)
+jun	    (jun(e)?)
+jul	    (jul(y)?)
+aug	    (aug(ust)?)
+sep	    (sep(tember)?)
+oct	    (oct(ober)?)
+nov	    (nov(ember)?)
+dec	    (dec(ember)?)
+
+first_half  ({jan}|{feb}|{mar}|{apr}|{may}|{jun})
+second_half ({jul}|{aug}|{sep}|{oct}|{nov}|{dec})
+month 	    {first_half}|{second_half}
+
+nday         [1-9]|[1-2][0-9]|3[0-1]
+nmonth       [1-9]|1[0-2]
+nyear        [0-9]{1,4}
+
+year_ext    (ad|AD|bc|BC)?
+day_ext     (st|nd|rd|th)?
+
+%s LONG SHORT
+%s DAY DAY_FIRST YEAR_FIRST YEAR_LAST YFMONTH YLMONTH
+
+%%
+
+  /* the default is month-day-year */
+
+<LONG>{day_of_the_week}    strcpy(dow,yytext); 
+<LONG>{month}              strcpy(month,yytext); BEGIN(DAY);
+
+  /* handle the form: day-month-year */
+
+<LONG>{nday}{day_ext}      strcpy(day,yytext);   BEGIN(DAY_FIRST);
+<DAY_FIRST>{month}         strcpy(month,yytext); BEGIN(LONG);
+<DAY>{nday}{day_ext}       strcpy(day,yytext);   BEGIN(LONG);  
+
+<LONG>{nyear}{year_ext}  {
+                           printf("Long:\n");
+                           printf("  DOW   : %s \n",dow);
+                           printf("  Day   : %s \n",day);
+                           printf("  Month : %s \n",month);
+                           printf("  Year  : %s \n",yytext);
+                           strcpy(dow,"");
+                           strcpy(day,"");
+                           strcpy(month,"");
+                         }
+
+  /* handle dates of the form: day-month-year */
+
+<SHORT>{nday}              strcpy(day,yytext);  BEGIN(YEAR_LAST);
+<YEAR_LAST>{nmonth}        strcpy(month,yytext);BEGIN(YLMONTH);
+<YLMONTH>{nyear}           strcpy(year,yytext); BEGIN(SHORT);
+
+  /* handle dates of the form: year-month-day */
+
+<SHORT>{nyear}             strcpy(year,yytext); BEGIN(YEAR_FIRST);
+<YEAR_FIRST>{nmonth}       strcpy(month,yytext);BEGIN(YFMONTH);
+<YFMONTH>{nday}            strcpy(day,yytext);  BEGIN(SHORT);
+
+
+<SHORT>\n                {
+                           printf("Short:\n");
+                           printf("  Day   : %s \n",day);
+                           printf("  Month : %s \n",month);
+                           printf("  Year  : %s \n",year);
+                           strcpy(year,""); 
+                           strcpy(day,"");
+                           strcpy(month,"");
+                         }
+
+long\n                      BEGIN(LONG);
+short\n                     BEGIN(SHORT);
+
+{skip}*
+\n
+.
+
+
diff --git a/examples/manual/datetest.dat b/examples/manual/datetest.dat
new file mode 100644
index 0000000..427f407
--- /dev/null
+++ b/examples/manual/datetest.dat
@@ -0,0 +1,28 @@
+short
+1989:12:23
+1989:11:12
+23:12:1989
+11:12:1989
+1989/12/23
+1989/11/12
+23/12/1989
+11/12/1989
+1989-12-23
+1989-11-12
+23-12-1989
+11-12-1989
+long
+Friday the 5th of January, 1989
+Friday, 5th of January, 1989
+Friday, January 5th, 1989
+Fri, January 5th, 1989
+Fri, Jan 5th, 1989
+Fri, Jan 5, 1989
+FriJan 5, 1989
+FriJan5, 1989
+FriJan51989
+Jan51989
+
+
+
+
diff --git a/examples/manual/eof_rules.lex b/examples/manual/eof_rules.lex
new file mode 100644
index 0000000..b575f2c
--- /dev/null
+++ b/examples/manual/eof_rules.lex
@@ -0,0 +1,65 @@
+/*
+ * eof_rules.lex : An example of using multiple buffers
+ *                 EOF rules, and start states
+ */
+
+%{
+                              
+#define MAX_NEST 10                   
+
+YY_BUFFER_STATE include_stack[MAX_NEST];
+int             include_count = -1;
+
+%}
+
+
+%x INCLUDE
+
+%%
+
+^"#include"[ \t]*\"  BEGIN(INCLUDE);
+<INCLUDE>\"          BEGIN(INITIAL); 
+<INCLUDE>[^\"]+ {      /* get the include file name */
+          if ( include_count >= MAX_NEST){
+             fprintf( stderr, "Too many include files" );
+             exit( 1 );
+          }
+
+          include_stack[++include_count] = YY_CURRENT_BUFFER;
+
+          yyin = fopen( yytext, "r" );
+          if ( ! yyin ){
+             fprintf( stderr, "Unable to open \"%s\"\n",yytext);
+             exit( 1 );
+          }
+
+          yy_switch_to_buffer(yy_create_buffer(yyin,YY_BUF_SIZE));
+
+          BEGIN(INITIAL);
+        }
+<INCLUDE><<EOF>> 
+        {
+            fprintf( stderr, "EOF in include" );
+            yyterminate();
+        }
+<<EOF>> {
+          if ( include_count <= 0 ){
+            yyterminate();
+          } else {
+            yy_delete_buffer(include_stack[include_count--] );
+            yy_switch_to_buffer(include_stack[include_count] );
+            BEGIN(INCLUDE);
+          }
+        }
+[a-z]+               ECHO;
+.|\n                 ECHO;
+
+
+
+
+
+
+
+
+
+
diff --git a/examples/manual/eof_test01.txt b/examples/manual/eof_test01.txt
new file mode 100644
index 0000000..ec5f083
--- /dev/null
+++ b/examples/manual/eof_test01.txt
@@ -0,0 +1,17 @@
+This is test file #1
+
+-------------------------------------------------
+
+We will now include test #2 in a standard way.
+
+#include "eof_test02.txt"
+
+-------------------------------------------------
+
+And now we will include test # 2 with a different
+format
+
+#include "eof_test02.txt"
+-------------------------------------------------
+
+and this is the end of the test.
diff --git a/examples/manual/eof_test02.txt b/examples/manual/eof_test02.txt
new file mode 100644
index 0000000..19f52cf
--- /dev/null
+++ b/examples/manual/eof_test02.txt
@@ -0,0 +1,8 @@
+INCLUDE #2
+
+This is the second file that will
+be included.
+
+>>> Foo are GNU?
+
+#include "eof_test03.txt"
diff --git a/examples/manual/eof_test03.txt b/examples/manual/eof_test03.txt
new file mode 100644
index 0000000..e737879
--- /dev/null
+++ b/examples/manual/eof_test03.txt
@@ -0,0 +1,7 @@
+INCLUDE #3
+
+This is the third file that will
+be included.
+
+>>> echo "I am `whoami`!!"
+
diff --git a/examples/manual/expr.lex b/examples/manual/expr.lex
new file mode 100644
index 0000000..9adfcaa
--- /dev/null
+++ b/examples/manual/expr.lex
@@ -0,0 +1,35 @@
+/*
+ * expr.lex : Scanner for a simple
+ *            expression parser.
+ */
+
+%{
+#include "y.tab.h"
+
+%}
+
+%%
+
+[0-9]+     { yylval.val = atol(yytext);
+             return(NUMBER);
+           }
+[0-9]+\.[0-9]+ { 
+             sscanf(yytext,"%f",&yylval.val);
+             return(NUMBER);
+           }
+"+"        return(PLUS);
+"-"        return(MINUS);
+"*"        return(MULT);
+"/"        return(DIV);
+"^"        return(EXPON);
+"("        return(LB);
+")"        return(RB);
+\n         return(EOL);
+[\t ]*     /* throw away whitespace */
+.          { yyerror("Illegal character"); 
+             return(EOL);
+           }
+%%
+
+
+
diff --git a/examples/manual/expr.y b/examples/manual/expr.y
new file mode 100644
index 0000000..1ac3092
--- /dev/null
+++ b/examples/manual/expr.y
@@ -0,0 +1,64 @@
+/*
+ * expr.y : A simple yacc expression parser
+ *          Based on the Bison manual example. 
+ */
+
+%{
+#include <stdio.h>
+#include <math.h>
+
+%}
+
+%union {
+   float val;
+}
+
+%token NUMBER
+%token PLUS MINUS MULT DIV EXPON
+%token EOL
+%token LB RB
+
+%left  MINUS PLUS
+%left  MULT DIV
+%right EXPON
+
+%type  <val> exp NUMBER
+
+%%
+input   :
+        | input line
+        ;
+
+line    : EOL
+        | exp EOL { printf("%g\n",$1);}
+
+exp     : NUMBER                 { $$ = $1;        }
+        | exp PLUS  exp          { $$ = $1 + $3;   }
+        | exp MINUS exp          { $$ = $1 - $3;   }
+        | exp MULT  exp          { $$ = $1 * $3;   }
+        | exp DIV   exp          { $$ = $1 / $3;   }
+        | MINUS  exp %prec MINUS { $$ = -$2;       }
+        | exp EXPON exp          { $$ = pow($1,$3);}
+        | LB exp RB                      { $$ = $2;        }
+        ;
+
+%%
+
+yyerror(char *message)
+{
+  printf("%s\n",message);
+}
+
+int main(int argc, char *argv[])
+{
+  yyparse();
+  return(0);
+}
+
+
+
+
+
+
+
+
diff --git a/examples/manual/front.lex b/examples/manual/front.lex
new file mode 100644
index 0000000..449cb00
--- /dev/null
+++ b/examples/manual/front.lex
@@ -0,0 +1,40 @@
+%{
+#include <stdio.h>
+#include <string.h>
+#include "y.tab.h"                 /* this comes from bison        */
+
+#define TRUE  1
+#define FALSE 0
+
+#define copy_and_return(token_type) { strcpy(yylval.name,yytext); \
+                                      return(token_type); }
+
+int             yylexlinenum = 0;  /* so we can count lines        */
+%}
+
+%%
+                            /* Lexical scanning rules begin from here.  */
+
+MEN|WOMEN|STOCKS|TREES      copy_and_return(NOUN)
+MISTAKES|GNUS|EMPLOYEES     copy_and_return(NOUN)
+LOSERS|USERS|CARS|WINDOWS   copy_and_return(NOUN)
+
+DATABASE|NETWORK|FSF|GNU    copy_and_return(PROPER_NOUN)
+COMPANY|HOUSE|OFFICE|LPF    copy_and_return(PROPER_NOUN)
+
+THE|THIS|THAT|THOSE         copy_and_return(DECLARATIVE)
+
+ALL|FIRST|LAST              copy_and_return(CONDITIONAL)
+
+FIND|SEARCH|SORT|ERASE|KILL copy_and_return(VERB)
+ADD|REMOVE|DELETE|PRINT     copy_and_return(VERB)
+
+QUICKLY|SLOWLY|CAREFULLY    copy_and_return(ADVERB)
+
+IN|AT|ON|AROUND|INSIDE|ON   copy_and_return(POSITIONAL)
+
+"."                         return(PERIOD);                             
+"\n"                        yylexlinenum++; return(NEWLINE);            
+.                                                                       
+%%
+
diff --git a/examples/manual/front.y b/examples/manual/front.y
new file mode 100644
index 0000000..77b5ca1
--- /dev/null
+++ b/examples/manual/front.y
@@ -0,0 +1,118 @@
+/* C code supplied at the beginning of the file.  */
+
+%{
+
+#include <stdio.h>                          
+#include <string.h>
+
+extern int  yylexlinenum;                   /* these are in YYlex      */
+extern char *yytext;                        /* current token           */
+
+
+%}
+
+/* Keywords and reserved words begin here.  */
+
+%union{                                     /* this is the data union  */
+    char   name[128];                       /* names                   */
+}
+
+/*-------------------- the reserved words -----------------------------*/
+
+%token PERIOD
+%token NEWLINE
+%token POSITIONAL
+
+%token VERB
+%token ADVERB
+
+%token PROPER_NOUN
+%token NOUN
+
+%token DECLARATIVE
+%token CONDITIONAL
+
+
+%type  <name> declarative
+%type  <name> verb_phrase
+%type  <name> noun_phrase
+%type  <name> position_phrase
+%type  <name> adverb
+
+%type  <name> POSITIONAL VERB ADVERB PROPER_NOUN 
+%type  <name> NOUN DECLARATIVE CONDITIONAL
+
+%%
+
+sentence_list : sentence
+              | sentence_list NEWLINE sentence
+              ;
+
+
+sentence : verb_phrase noun_phrase position_phrase adverb period 
+           {
+             printf("I understand that sentence.\n");
+             printf("VP = %s \n",$1);
+             printf("NP = %s \n",$2);
+             printf("PP = %s \n",$3);
+             printf("AD = %s \n",$4);
+           }
+         | { yyerror("That's a strange sentence !!");  }
+         ;
+
+position_phrase : POSITIONAL  declarative PROPER_NOUN 
+                  {
+                    sprintf($$,"%s %s %s",$1,$2,$3);
+                  }   
+                | /* empty */ { strcpy($$,""); }
+                ;
+               
+
+verb_phrase : VERB { strcpy($$,$1); strcat($$," "); }
+            | adverb VERB  
+              {
+                sprintf($$,"%s %s",$1,$2);
+              }
+            ;
+
+adverb : ADVERB      { strcpy($$,$1); }
+       | /* empty */ { strcpy($$,""); }
+       ;
+
+noun_phrase : DECLARATIVE NOUN 
+              {
+                sprintf($$,"%s %s",$1,$2);
+              }
+            | CONDITIONAL declarative NOUN 
+                  {
+                    sprintf($$,"%s %s %s",$1,$2,$3);
+                  }   
+            | NOUN { strcpy($$,$1); strcat($$," "); }
+            ;
+
+declarative : DECLARATIVE { strcpy($$,$1); }
+            | /* empty */ { strcpy($$,""); }
+            ;
+
+period : /* empty */
+       | PERIOD
+       ;
+
+
+%%
+
+/* Supplied main() and yyerror() functions.  */
+
+int main(int argc, char *argv[])
+{
+  yyparse();   /* parse the file          */
+  return(0);
+}
+
+int yyerror(char *message)
+{
+  extern FILE *yyout;
+
+  fprintf(yyout,"\nError at line %5d. (%s) \n",
+                     yylexlinenum,message);
+}
diff --git a/examples/manual/j2t.lex b/examples/manual/j2t.lex
new file mode 100644
index 0000000..08fbd21
--- /dev/null
+++ b/examples/manual/j2t.lex
@@ -0,0 +1,442 @@
+/*
+ * j2t.lex : An example of the use (possibly abuse!)
+ *           of start states.
+ */
+
+%{
+#define MAX_STATES 1024
+#define TRUE  1
+#define FALSE 0
+
+#define CHAPTER   "@chapter"
+#define SECTION   "@section"
+#define SSECTION  "@subsection"
+#define SSSECTION "@subsubsection"
+
+int  states[MAX_STATES];
+int  statep = 0;
+
+int  need_closing = FALSE;
+
+char buffer[YY_BUF_SIZE];
+
+extern char *yytext;
+
+/*
+ * set up the head of the *.texinfo file the program
+ * will produce. This is a standard texinfo header.
+ */
+
+void print_header(void)
+{
+   printf("\\input texinfo @c -*-texinfo-*-\n");
+   printf("@c           %c**start of header\n",'%');
+   printf("@setfilename       jargon.info\n");
+   printf("@settitle          The New Hackers Dictionary\n");
+   printf("@synindex          fn cp\n");
+   printf("@synindex          vr cp\n");
+   printf("@c           %c**end of header\n",'%');
+   printf("@setchapternewpage odd\n");
+   printf("@finalout\n");
+   printf("@c @smallbook\n");
+   printf("\n");
+   printf("@c ==========================================================\n\n");
+   printf("@c  This file was produced by j2t. Any mistakes are *not* the\n");
+   printf("@c  fault of the jargon file editors.                        \n");
+   printf("@c ==========================================================\n\n");
+   printf("@titlepage\n");
+   printf("@title     The New Hackers Dictionary\n");
+   printf("@subtitle  Version 2.9.10\n");
+   printf("@subtitle  Generated by j2t\n");
+   printf("@author    Eric S. Raymond, Guy L. Steel, Mark Crispin et al.\n"); 
+   printf("@end titlepage\n");
+   printf("@page\n");
+   printf("\n@c ==========================================================\n");
+   printf("\n\n");
+   printf("@unnumbered Preface\n");
+   printf("@c          *******\n");
+}
+
+/*
+ * create the tail of the texinfo file produced.
+ */
+
+void print_trailer(void)
+{
+   printf("\n@c ==========================================================\n");
+   printf("@contents\n");   /* print the table of contents */
+   printf("@bye\n\n");     
+}
+
+/*
+ * write an underline under a section
+ * or chapter so we can find it later. 
+ */
+
+void write_underline(int len, int space, char ch)
+{
+  int loop;
+
+  printf("@c ");
+
+  for(loop=3; loop<space; loop++){
+    printf(" ");
+  }
+
+  while(len--){
+    printf("%c",ch);
+  }
+  printf("\n\n");
+}
+
+/*
+ * check for texinfo special characters
+ * and escape them
+ */
+
+char *check_and_convert(char *string)
+{
+  int  buffpos = 0;
+  int  len,loop;
+
+  len = strlen(string);  
+  for(loop=0; loop<len; loop++){
+    if(string[loop] == '@' || string[loop] == '{' || string[loop] == '}'){
+      buffer[buffpos++] = '@';
+      buffer[buffpos++] = string[loop];
+    } else {
+      buffer[buffpos++] = string[loop];
+    }
+  }
+  buffer[buffpos] = '\0';
+  return(buffer);
+}
+
+/*
+ * write out a chapter,section, or subsection
+ * header
+ */
+
+void write_block_header(char *type)
+{
+  int loop;
+  int len;
+
+  (void)check_and_convert(yytext);
+  len = strlen(buffer);
+  for(loop=0; buffer[loop] != '\n';loop++)
+         ;
+  buffer[loop] = '\0';
+  printf("%s %s\n",type,buffer);
+  write_underline(strlen(buffer),strlen(type)+1,'*');
+}
+
+%}
+
+/*
+ * the flex description starts here 
+ */
+
+%x HEADING EXAMPLE ENUM EXAMPLE2
+%x BITEM BITEM_ITEM 
+%s LITEM LITEM2 
+
+%%
+
+^#[^#]*"#"  /* skip the header & trailer */
+                     /* chapters have asterisks under them 
+                      * and are terminated by a colon
+                      */
+^[^\n:]+\n[*]+\n      write_block_header(CHAPTER); 
+
+^"= "[A-Z]" ="\n"="*  { /* we create a seciton for each category */
+                        if(need_closing == TRUE){
+                          printf("@end table\n\n\n");
+                        }
+                        need_closing = TRUE;
+                        write_block_header(SECTION);
+                        printf("\n\n@table @b\n");
+                      }
+
+"Examples:"[^\.]+     ECHO;
+
+"*"[^*\n]+"*"         { /* @emph{}(emphasized) text */
+                        yytext[yyleng-1] = '\0';
+                        (void)check_and_convert(&yytext[1]);
+                        printf("@i{%s}",buffer);
+                      }
+
+"{{"[^}]+"}}"         { /* special emphasis */
+                        yytext[yyleng-2] = '\0';
+                        (void)check_and_convert(&yytext[2]);
+                        printf("@b{%s}",buffer);
+                      }
+
+"{"[^}]+"}"           { /* special emphasis */
+                        yytext[yyleng-1] = '\0';
+                        (void)check_and_convert(&yytext[1]);
+                        printf("@b{%s}",buffer);
+                      }
+  
+ /* escape some special texinfo characters */
+<INITIAL,LITEM,LITEM2,BITEM,ENUM,EXAMPLE,EXAMPLE2>"@"  printf("@@");
+<INITIAL,LITEM,LITEM2,BITEM,ENUM,EXAMPLE,EXAMPLE2>"{"  printf("@{");
+<INITIAL,LITEM,LITEM2,BITEM,ENUM,EXAMPLE,EXAMPLE2>"}"  printf("@}");
+
+ /*
+  * reproduce @example code
+  */
+
+":"\n+[^\n0-9*]+\n"     "[^ ]   {
+                        int loop;
+                        int len;
+                        int cnt;
+
+                        printf(":\n\n@example \n");
+                        strcpy(buffer,yytext);
+                        len = strlen(buffer);
+                        cnt = 0;
+                        for(loop=len; loop > 0;loop--){
+                          if(buffer[loop] == '\n')
+                             cnt++;
+                          if(cnt == 2)
+                              break;
+                        }
+                        yyless(loop+1);
+                        statep++;
+                        states[statep] = EXAMPLE2;
+                        BEGIN(EXAMPLE2);
+                      }
+<EXAMPLE,EXAMPLE2>^\n  {
+                      printf("@end example\n\n");
+                      statep--; 
+                      BEGIN(states[statep]);
+                    }
+
+ /*
+  * repoduce @enumerate lists
+  */
+
+":"\n+[ \t]*[0-9]+"."   {
+                      int loop;
+                      int len;
+
+                      printf(":\n\n@enumerate \n");
+                      strcpy(buffer,yytext);
+                      len = strlen(buffer);
+                      for(loop=len; loop > 0;loop--){
+                        if(buffer[loop] == '\n')
+                           break;
+                      }
+                      yyless(loop);
+                      statep++;
+                      states[statep] = ENUM;
+                      BEGIN(ENUM);
+                    }
+
+<ENUM>"@"           printf("@@");
+<ENUM>":"\n+"     "[^0-9]    {
+                    printf(":\n\n@example\n");
+                    statep++;
+                    states[statep] = EXAMPLE;
+                    BEGIN(EXAMPLE); 
+                  }
+
+
+<ENUM>\n[ \t]+[0-9]+"." {
+                    printf("\n\n@item ");
+                   }
+<ENUM>^[^ ] | 
+<ENUM>\n\n\n[ \t]+[^0-9] {
+                    printf("\n\n@end enumerate\n\n");
+                    statep--;
+                    BEGIN(states[statep]);
+                  }
+ 
+ /* 
+  * reproduce one kind of @itemize list
+  */
+
+":"\n+":"         {
+                    int loop;
+                    int len;
+
+                    printf(":\n\n@itemize @bullet \n");
+                    yyless(2);
+                    statep++;
+                    states[statep] = LITEM2;
+                    BEGIN(LITEM2);
+                  }
+<LITEM2>^":".+":" {
+                    (void)check_and_convert(&yytext[1]);
+                    buffer[strlen(buffer)-1]='\0';
+                    printf("@item @b{%s:}\n",buffer);
+                  }
+ 
+<LITEM2>\n\n\n+[^:\n] {
+                    printf("\n\n@end itemize\n\n");
+                    ECHO;
+                    statep--;
+                    BEGIN(states[statep]);
+                  }
+ 
+ /*
+  * create a list out of the revision history part.
+  * We need the "Version" for this because it 
+  * clashes with other rules otherwise.
+  */
+
+:[\n]+"Version"[^:\n*]+":" {
+                    int loop;
+                    int len;
+
+                    printf(":\n\n@itemize @bullet \n");
+                    strcpy(buffer,yytext);
+                    len = strlen(buffer);
+                    for(loop=len; loop > 0;loop--){
+                      if(buffer[loop] == '\n')
+                         break;
+                    }
+                    yyless(loop);
+                    statep++;
+                    states[statep] = LITEM;
+                    BEGIN(LITEM);
+                  }
+<LITEM>^.+":"     {
+                    (void)check_and_convert(yytext);
+                    buffer[strlen(buffer)-1]='\0';
+                    printf("@item @b{%s}\n\n",buffer);
+                  }
+ 
+<LITEM>^[^:\n]+\n\n[^:\n]+\n  {
+                    int loop;
+
+                    strcpy(buffer,yytext);
+                    for(loop=0; buffer[loop] != '\n'; loop++);
+                    buffer[loop] = '\0';
+                    printf("%s\n",buffer);
+                    printf("@end itemize\n\n");
+                    printf("%s",&buffer[loop+1]);
+                    statep--;
+                    BEGIN(states[statep]);
+                  }
+ 
+ /*
+  * reproduce @itemize @bullet lists
+  */
+
+":"\n[ ]*"*"      {
+                    int loop;
+                    int len;
+
+                    printf(":\n\n@itemize @bullet \n");
+                    len = strlen(buffer);
+                    for(loop=0; loop < len;loop++){
+                      if(buffer[loop] == '\n')
+                         break;
+                    }
+                    yyless((len-loop)+2);
+                    statep++;
+                    states[statep] = BITEM;
+                    BEGIN(BITEM);
+                  }
+
+<BITEM>^" "*"*"   {
+                    printf("@item");
+                    statep++;
+                    states[statep] = BITEM_ITEM;
+                    BEGIN(BITEM_ITEM);
+                  }
+<BITEM>"@"          printf("@@");
+<BITEM>^\n        { 
+                    printf("@end itemize\n\n");
+                    statep--;
+                    BEGIN(states[statep]);
+                  } 
+<BITEM_ITEM>[^\:]* {
+                     printf(" @b{%s}\n\n",check_and_convert(yytext));
+                   }
+<BITEM_ITEM>":"   { 
+                    statep--; 
+                    BEGIN(states[statep]);
+                  }
+
+ /* 
+  * recreate @chapter, @section etc.
+  */
+
+^:[^:]*           { 
+                    (void)check_and_convert(&yytext[1]); 
+                    statep++;
+                    states[statep] = HEADING;
+                    BEGIN(HEADING); 
+                  }
+<HEADING>:[^\n]   {
+                    printf("@item @b{%s}\n",buffer); 
+                    write_underline(strlen(buffer),6,'~');
+                    statep--; 
+                    BEGIN(states[statep]);
+                  }
+<HEADING>:\n"*"*  { 
+                    if(need_closing == TRUE){
+                      printf("@end table\n\n\n");
+                      need_closing = FALSE;
+                    }
+                    printf("@chapter %s\n",buffer); 
+                    write_underline(strlen(buffer),9,'*');
+                    statep--; 
+                    BEGIN(states[statep]);
+                  }
+<HEADING>:\n"="*  { 
+                    if(need_closing == TRUE){
+                     printf("@end table\n\n\n");
+                      need_closing = FALSE;
+                    }
+                    printf("@section %s\n",buffer); 
+                    write_underline(strlen(buffer),9,'=');
+                    statep--; 
+                    BEGIN(states[statep]);
+                  }
+<HEADING>"@"        printf("@@");
+<HEADING>:\n"-"*  { 
+                    if(need_closing == TRUE){
+                      printf("@end table\n\n\n");
+                      need_closing = FALSE;
+                    }
+                    printf("@subsection %s\n",buffer); 
+                    write_underline(strlen(buffer),12,'-');
+                    statep--; 
+                    BEGIN(states[statep]);
+                  }
+
+ /*
+  * recreate @example text
+  */
+
+^"     "          {
+                    printf("@example\n");
+                    statep++;
+                    states[statep] = EXAMPLE;
+                    BEGIN(EXAMPLE); 
+                  }
+<EXAMPLE>^"     "
+.                 ECHO;
+
+%%
+
+/*
+ * initialise and go.
+ */
+
+int main(int argc, char *argv[])
+{
+  states[0] = INITIAL;
+  statep    = 0;
+  print_header();
+  yylex();
+  print_trailer();
+  return(0);
+}
+
+
+
diff --git a/examples/manual/myname.lex b/examples/manual/myname.lex
new file mode 100644
index 0000000..2e36095
--- /dev/null
+++ b/examples/manual/myname.lex
@@ -0,0 +1,15 @@
+/*
+ *
+ * myname.lex : A sample Flex program
+ *              that does token replacement.
+ */
+
+%%
+
+%NAME     { printf("%s",getenv("LOGNAME")); }
+%HOST     { printf("%s",getenv("HOST"));    }
+%HOSTTYPE { printf("%s",getenv("HOSTTYPE"));}
+%HOME     { printf("%s",getenv("HOME"));    }
+
+%%
+
diff --git a/examples/manual/myname.txt b/examples/manual/myname.txt
new file mode 100644
index 0000000..0631840
--- /dev/null
+++ b/examples/manual/myname.txt
@@ -0,0 +1,6 @@
+Hello, my name name is %NAME. Actually 
+"%NAME" isn't my real name, it is the
+alias I use when I'm on %HOST, which
+is the %HOSTTYPE I use. My HOME 
+directory is %HOME. 
+
diff --git a/examples/manual/myname2.lex b/examples/manual/myname2.lex
new file mode 100644
index 0000000..cef55d8
--- /dev/null
+++ b/examples/manual/myname2.lex
@@ -0,0 +1,19 @@
+/*
+ * myname2.lex : A sample Flex program
+ *               that does token replacement.                      
+ */
+
+%{
+#include <stdio.h>
+%}
+
+%x STRING
+%%
+\"                ECHO; BEGIN(STRING); 
+<STRING>[^\"\n]*  ECHO;
+<STRING>\"        ECHO; BEGIN(INITIAL);
+
+%NAME     { printf("%s",getenv("LOGNAME")); }
+%HOST     { printf("%s",getenv("HOST"));    }
+%HOSTTYPE { printf("%s",getenv("HOSTTYPE"));}
+%HOME     { printf("%s",getenv("HOME"));    }
diff --git a/examples/manual/numbers.lex b/examples/manual/numbers.lex
new file mode 100644
index 0000000..6484d6e
--- /dev/null
+++ b/examples/manual/numbers.lex
@@ -0,0 +1,145 @@
+/*
+ * numbers.lex : An example of the definitions and techniques
+ *               for scanning numbers
+ */
+
+%{
+#include <stdio.h>
+
+#define UNSIGNED_LONG_SYM   1
+#define SIGNED_LONG_SYM     2
+#define UNSIGNED_SYM        3
+#define SIGNED_SYM          4
+#define LONG_DOUBLE_SYM     5
+#define FLOAT_SYM           6
+
+union _yylval {
+  long double    ylong_double;
+  float          yfloat; 
+  unsigned long  yunsigned_long;
+  unsigned       yunsigned;
+  long           ysigned_long;
+  int            ysigned;
+} yylval;
+
+%}
+
+digit             [0-9]
+hex_digit         [0-9a-fA-F]
+oct_digit         [0-7]
+
+exponent          [eE][+-]?{digit}+
+i                 {digit}+
+float_constant    ({i}\.{i}?|{i}?\.{i}){exponent}?
+hex_constant      0[xX]{hex_digit}+
+oct_constant      0{oct_digit}*
+int_constant      {digit}+
+long_ext          [lL]
+unsigned_ext      [uU]
+float_ext         [fF]
+ulong_ext         {long_ext}{unsigned_ext}|{unsigned_ext}{long_ext}
+
+%%
+
+{hex_constant}{ulong_ext} {  /* we need to skip the "0x" part */
+                             sscanf(&yytext[2],"%lx",&yylval.yunsigned_long); 
+                             return(UNSIGNED_LONG_SYM);
+                          }
+{hex_constant}{long_ext}  {  
+                             sscanf(&yytext[2],"%lx",&yylval.ysigned_long); 
+                             return(SIGNED_LONG_SYM);
+                          }
+{hex_constant}{unsigned_ext}  { 
+                             sscanf(&yytext[2],"%x",&yylval.yunsigned); 
+                             return(UNSIGNED_SYM);
+                          }
+{hex_constant}            { /* use %lx to protect against overflow */
+                             sscanf(&yytext[2],"%lx",&yylval.ysigned_long); 
+                             return(SIGNED_LONG_SYM);
+                          }
+{oct_constant}{ulong_ext} {
+                             sscanf(yytext,"%lo",&yylval.yunsigned_long); 
+                             return(UNSIGNED_LONG_SYM);
+                          }
+{oct_constant}{long_ext}  {
+                             sscanf(yytext,"%lo",&yylval.ysigned_long); 
+                             return(SIGNED_LONG_SYM);
+                          }
+{oct_constant}{unsigned_ext}  {
+                             sscanf(yytext,"%o",&yylval.yunsigned); 
+                             return(UNSIGNED_SYM);
+                          }
+{oct_constant}            { /* use %lo to protect against overflow */
+                             sscanf(yytext,"%lo",&yylval.ysigned_long); 
+                             return(SIGNED_LONG_SYM);
+                          }
+{int_constant}{ulong_ext} {