\documentclass{article} \usepackage[T1]{fontenc} \usepackage{xcolor,tokcycle,verbatimbox,tabstackengine,lipsum,lmodern} \colorlet{bred}{red!90!blue} \usepackage{caption} \captionsetup{font={sf, footnotesize},labelfont={sf},textfont={footnotesize}} \fboxsep=1.5pt\relax \makeatletter \let\svlt< \let\svgt> \catcode`>=\active % \catcode`<=\active % \def\openesc#1{\if\svlt#1\itshape\rmfamily\let\xsvgt\@empty\else \svlt\itshape\rmfamily#1\let\xsvgt\svgt\fi} \def\closeesc{\upshape\ttfamily\xsvgt} \def\vbdelim{\small\catcode`<=\active\catcode`>=\active \def<{\openesc} \def>{\closeesc}} \catcode`>=12 % \catcode`<=12 % \newcommand\TokCycle{\textsf{tokcycle}} \newcommand\Character{\textit{Character}} \newcommand\Group{\textit{Group}} \newcommand\Macro{\textit{Macro}} \newcommand\Space{\textit{Space}} \newcommand\CGMS{\textit{C-G-M-S}} \newcommand\CGMnS{\textit{C}, \textit{G}, \textit{M}, and \textit{S}} \begingroup \catcode`|=\active \gdef|{\textbackslash} \gdef\xmacaux#1{\ttfamily\char92#1\endgroup} \endgroup \gdef\macname{\begingroup\catcode`|=\active \xmacaux} \newcommand\thevbox{\medskip\theverbbox\par\medskip\noindent} \newenvironment{specfig} {\par\bigskip\vbox\bgroup\small\sffamily\centering} {\egroup\medskip\par} \newtoks\gtoks \newcommand\svgtoks[1]{\global\gtoks\expandafter{\the#1}} \frenchspacing \begin{document} %%% AGAIN %%% \begin{myverbbox}{\argone}#1\end{myverbbox} \begin{myverbbox}[\vbdelim\normalsize]{\vbnewcytoks} \newcytoks{} \end{myverbbox} {\centering{\LARGE Package \TokCycle{} \Large (v\tcver)\par} Steven B Segletes {\footnotesize $<$SSegletes@verizon.net$>$} \\\small{contributor: Christian Tellechea}\footnote{ {I am extremely grateful to Christian $<$unbonpetit@netc.fr$>$} for his assistance in the development of this package. The \macname{addcytoks} macro was provided by him. He gave constant reminders on what the parser should be able to achieve, thus motivating me to spend the extra time striving for a generality of application that would not come naturally to me. I value highly his collegiality and hold his expertise in the highest regard. }% \\\today\par} \begin{quote} The \TokCycle{} package helps one to build tools to process tokens from an input stream. If a macro to process an arbitrary single (non-macro, non-space) token can be built, then \TokCycle{} can provide a wrapper for cycling through an input stream (including macros, spaces, and groups) on a token-by-token basis, using the provided macro on each successive character. \TokCycle{} characterizes each successive token in the input stream as a \Character, a \Group, a \Macro, or a \Space. Each of these token categories are processed with a unique directive, to bring about the desired effect of the token cycle. \textit{If}-condition flags are provided to identify active, implicit, and catcode-6 tokens as they are digested. The package provides a number of options for handling groups. \end{quote} \enlargethispage{1.4\baselineskip} \tableofcontents \makeatother \section{The \TokCycle{} macros and environments}\label{s:intro} The purpose of the \TokCycle{} package is to provide a tool to assist the user in developing token-processing macros and environments. The types of processing are limited only by the creativity of the user, but examples might include letter-case-operations, letter spacing, dimensional manipulation, simple-ciphering, \{group\} manipulation, macro removal, etc. In one sense, it can be thought of as a streaming editor that operates on \LaTeX{} input streams. The package can be loaded into both plain \TeX{}, by way of the invocation \verb|\input tokcycle.tex| as well as \LaTeX, via \verb|\usepackage{tokcycle}|. It provides a total of 6 macros/pseudo environments, based on three criteria: \begin{itemize} \item Two pseudo-environments with the phrase ``\texttt{tokencycle}'' in the name, and four macros containing the phrase ``\texttt{tokcycle}''. The pseudo-\hspace{0pt}environments operate within a group and typeset their result upon completion. The macros operate within the document's current scope, but do not typeset the result automatically. In the case of both macros and pseudo-environments, the transformed result is available for later use, being stored in the package token list named \macname{cytoks}. \item Two macros and one pseudo-environment containing the phrase ``\texttt{xpress}''. Without the phrase, the macro/environment requires four \textit{processing directives} to be \textit{explicitly} specified, followed by the input stream. With the phrase present, only the input stream is to be provided. In the \texttt{xpress} case, the \textit{processing directives} are to have been separately specified via external macro and/or are taken from the most recent \TokCycle{} macro invocation (failing that, are taken from the package initialization). \item Two macros containing the phrase ``\texttt{expanded}''. When present, the input stream of the token cycle is subject to the new \TeX{} primitive, \macname{expanded}, prior to processing by the \TokCycle{} macro. Expansion of specific macros in the input stream can be inhibited in the input stream with the use of \macname{noexpand}. Note that there are no \texttt{expanded} environments in \TokCycle, as \TokCycle{} environments do not pre-tokenize their input stream. \end{itemize} The basic approach of all \TokCycle{} macros/environments is to grab each successive token (or group) from the input stream, decide what category it is, and use the currently active processing directives to make any desired transformation of the token (or group). Generally, with rare exception, the processed tokens should be stored in the token register \macname{cytoks}, using the tools provided by the package. The cycle continues until the input stream is terminated. As tokens/groups are read from the input stream, they are categorized according to four type classifications, and are subjected to the user-specified processing directive associated with that category of token/group. The \TokCycle{} categories by which the input stream is dissected include \Character{}, \Group{}, \Macro{}, and \Space{} (alphabetized for easy recall). Catcode-0 tokens are directed to the \Macro{} directive for processing. Catcode-10 tokens are directed to the \Space{} directive. When an explicit catcode-1 token\footnote{ Throughout this document the terms \textit{catcode-} and \textit{cat-} are used interchangeably. } is encountered in the \TokCycle{} input stream, the contents of the entire group (sans the grouping) are directed to the \Group{} directive for further processing, which may in turn, redirect the individual tokens to the other categories. The handling options of implicit cat-1 and 2 tokens are described later in this document (section~\ref{s:impgrp}). Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted, are directed to the \Character{} directive for processing. \subsection{Provided (built-in) \TokCycle{} macros and environments} The syntax of the non-\texttt{xpress} macros/environments is\medskip \begin{verbbox}[\vbdelim] \tokcycle <> \expandedtokcycle {} {} {} {} {} \end{verbbox} \theverbbox\medskip\par\noindent or, alternately, for the pseudo-environment, \begin{verbbox}[\vbdelim] \tokencycle {} {} {} {}\endtokencycle \end{verbbox} \thevbox For the \texttt{xpress} macros, the syntax is \begin{verbbox}[\vbdelim] \tokcyclexpress <> \expandedtokcyclexpress {} \end{verbbox} \thevbox or, alternately, for the \texttt{xpress}-pseudo-environment, \begin{verbbox}[\vbdelim] \tokencyclexpress\endtokencyclexpress \end{verbbox} \thevbox \begin{sloppypar} \subsection{Create your own \TokCycle{} environments} In addition to the above macros/environments, the means is provided to define new \TokCycle{} environments: \begin{verbbox}[\vbdelim] \tokcycleenvironment\environment_name {} {} {} {} \end{verbbox} \thevbox This will then permit simplified invocations of the form \begin{verbbox}[\vbdelim] \environment_name\endenvironment_name \end{verbbox} \thevbox More recently, an even more versatile \textit{extended} version has been made available: \begin{verbbox}[\vbdelim] \xtokcycleenvironment\environment_name {} {} {} {} {} {} \end{verbbox} \thevbox This will also permit simplified invocations of the form \begin{verbbox}[\vbdelim] \environment_name\endenvironment_name \end{verbbox} \thevbox However, with this extended environment definition, additional set-up and close-out code will be executed prior to and following the token cycle, within the scope of the environment. The close-out code, while executed after the token cycle, occurs before the \macname{cytoks} token list is actually typeset; therefore the close-out code can both execute macros and/or add tokens to the imminently-typeset token list. Any definitions or counters that change as a result of executing the \CGMS{} directives are, in fact, \textit{past tense} when the close-out code is executed. Finally, the set-up and close-out code, as well as the \CGMS{} directives can invoke \macname{tcafterenv}, whose argument is executed upon the exit from the scope of \textit{this} defined environment.\footnote{ The \macname{tcafterenv} macro is recognized \textit{only} in the context of an extended \macname{xtokcycleenvironment} definition. It is not available for direct use in the package's \macname{tokencycle} pseudo-environment, nor in \macname{tokcycleenvironment} definitions. } As with all \TokCycle{} environments, those created by way of \macname{tokcycleenvironment} and \macname{xtokcycleenvironment}, while operating in the scope of a group, will preserve the contents of \macname{cytoks} when exiting their scope. \subsection{Externally specified directives and directive resets} For use in \texttt{xpress} mode, the directives for the \CGMS{} categories may be externally pre-specified, respectively, via the four macros \macname{Characterdirective}, \macname{Groupdirective}, \macname{Macrodirective}, and \macname{Spacedirective}, each taking an argument containing the particulars of the directive specification. Each of these directives may be individually reset to the package default with the following argument-free invocations: \macname{resetCharacterdirective}, \macname{resetGroupdirective}, \macname{resetMacrodirective}, or \macname{resetSpacedirective}. In addition, \macname{resettokcycle} is also provided, which not only resets all four directives collectively, but it also resets, to the default configuration, the manner in which explicit and implicit group tokens are processed. Finally, it resets the \macname{aftertokcycle} macro to empty. \end{sloppypar} The default directives at package outset and upon reset are \begin{verbbox}[\vbdelim] \Characterdirective{\addcytoks{#1}} \Groupdirective{\processtoks{#1}} \Macrodirective{\addcytoks{#1}} \Spacedirective{\addcytoks{#1}} \aftertokcycle{} \stripgroupingfalse \stripimplicitgroupingcase{0} \end{verbbox} \thevbox The interpretation of these directives will be explained in the remainder of this document. Let it suffice for now to say that the default directive settings pass through the input stream to the output stream, without change.\footnote{ Except, possibly, in the case of catcode-6 tokens, which will be later addressed in sections~\ref{s:cat6} and \ref{s:whennot}. } \section{Commands in the \TokCycle{} directives}\label{s:coms} The document-level token cycling tools provided in the package are listed in section~\ref{s:intro}. For each of those commands and/or pseudo-environments, the user must (explicitly or implicitly) detail a set of directives to specify the manner in which the \Character{}, \Group{}, \Macro{}, and \Space{} tokens found in the input stream are to be processed. The \CGMS{} processing directives consist of normal \TeX/\LaTeX{} commands and user-defined macros to accomplish the desired effect. There are, however, several macros provided by the package to assist the user in this endeavor. The recommended way to apply this package is to collect the \TokCycle-transformed results of the input stream in a token register provided by the package, named \macname{cytoks}. Its contents can then be typeset via \verb|\the\cytoks|.\footnote{% If a token-cycle input stream contains no macros, or is to be detokenized, or if the input-stream tokens are not to be typeset, it may be possible (though not required) to bypass \macname{cytoks} and typeset the output directly. } The macro for appending things to \macname{cytoks}, to be used in the package directives, is \macname{addcytoks}. \subsection{Adding tokens to the output stream: \macname{addcytoks}} The macro provided to append tokens to the \macname{cytoks} token register is named \macname{addcytoks[]\{\}}. Its mandatory argument consists of tokens denoting \textit{what} you would like to append to the \macname{cytoks} register, while the optional argument denotes \textit{how} you would like them appended (valid options include positive integers \verb|[<|\textit{n}\verb|>]| and the letter \texttt{[x]}). When the optional argument is omitted, the specified tokens are appended \textit{literally} to the register (without expansion). An integer option, call it \textit{n}, takes the the mandatory argument, and expands it $n$-times, and appends the result to \macname{cytoks}. The \texttt{[x]} option employs the \macname{expanded} primitive to maximally expand the argument before appending it to \macname{cytoks}. The \texttt{[x]} option will prove useful when the \Character{} or other directives involve a transformation that is fully expandable. Its use will allow the expanded result of the transformation to be placed in the token list, rather than the unexpanded transformation instructions. \subsubsection{\argone} In the context of the \CGMnS{} processing directives, one may refer to \argone{} (for example, in the argument to \macname{addcytoks} or \macname{processtoks}). \TeX{} users know that the first parameter to a \TeX{} macro is denoted as \argone. The specification of all \TokCycle{} processing directives is structured in such a way that ``\argone{}'' may be directly employed to indicate the current token (or group) under consideration in the input stream. \subsubsection{Transforming the input stream} Within the \TokCycle{} \CGMS{} directives, the command \verb|\addcytoks{#1}| is used to echo the token being processed to the output stream that is being collected in the \macname{cytoks} token list. However, one may do more than merely echo the input---one may transform it. If the user creates a macro, let's call it \macname{xform}, to take a single character of input and convert it in some way to something else, then this macro can be used inside the \Character{} directive as \verb|\addcytoks{\xform{#1}}|. In this case, if an `\texttt{e}' were the token under consideration by the \Character{} directive, `\verb|\xform{e}|' would be added to the \macname{cytoks} token list. Since \TokCycle{} operates upon each successive token in the input stream, this \macname{xform} macro that operates on a single token can be used to completely transform the \TokCycle{} input stream. If the \macname{xform} macro is expandable, reaching its termination in a known number, $n$, of expansions, \macname{addcytoks} can take advantage of that with its optional argument, performing the said number of expansions of \macname{xform} \textit{before} adding the result to \macname{cytoks}. If the expandability of \macname{xform} is known, but the number $n$ is not, the optional argument \verb|[x]| to \macname{addcytoks} may be used to fully expand the argument (by way of the \macname{expanded} primitive). In that case, the resulting \macname{cytoks} result will show no evidence of the \macname{xform} macro, but only the results of its transformation. The accompanying examples document, \texttt{tokcycle-examples.pdf}, is full of examples showing specific \TokCycle-induced transformations. \subsection{\Group{} directive: \macname{ifstripgrouping}, and \macname{processtoks}} The \Group{} directive is unique, in that it is the only directive whose argument (\argone) may consist of more than a single token. There are two issues to consider when handling the tokens comprising a group: do I retain or remove the grouping (cat-1,2 tokens)? Do I process the group's token content individually through the token cycle, or collectively as a unit? Grouping in the output stream is determined by the externally set condition \macname{ifstripgrouping}. The package default is \macname{stripgroupingfalse}, such that any explicit grouping that appears in the input stream will be echoed in the output stream. The alternative, \macname{stripgroupingtrue}, is dangerous in the sense that it will strip the cat-1,2 grouping from the group's tokens, thereby affecting or even breaking code that utilizes such grouping for macro arguments. Apply \macname{stripgroupingtrue} with care. The issue of treating the tokens comprising the content of a group individually or collectively is determined by the choice of macro inside the \Group{} directive. Within the \Group{} directive, the argument \argone{} contains the collective tokens of the group (absent the outer cat-1,2 grouping). The default directive \macname{processtoks\{\argone\}} will recommit the group's tokens individually to be processed through the token cycle. In contrast, the command \macname{addcytoks\{\argone\}} in the \Group{} directive would directly add the collective group of tokens to the \macname{cytoks} register, without being processed individually by \TokCycle. \subsubsection{The \macname{groupedcytoks} macro for more sophisticated group processing} \label{s:grpcy} \begin{sloppypar} To this point, there have been presented two options for handling grouped material in the token cycle: process each of the grouped tokens (via \macname{processtoks}) or echo the grouped tokens (via \macname{addcytoks}). Obviously, one may also discard the grouped tokens altogether by employing neither of the above choices in the context of the \Group{} directive. \end{sloppypar} However, there are times where you might wish to perform a macro task (for example, one that might add additional tokens to \macname{cytoks}) immediately \textit{outside} of the group, before or after. If you were to add the task macro before or after the \macname{processtoks} invocation (anywhere in the \Group{} directive), it will still be performed, by default, \textit{inside} the \macname{cytoks} group. The macro \macname{groupedcytoks} allows one to manually specify the grouping duties of the \Group{} directive. Thus, the following two group-processing configurations are functionally identical: the default \begin{verbbox}[\vbdelim] \stripgroupingfalse \Groupdirective{\processtoks{#1}} \end{verbbox} \thevbox and \begin{verbbox}[\vbdelim] \stripgroupingtrue \Groupdirective{\groupedcytoks{\processtoks{#1}}} \end{verbbox} \thevbox Therefore, unless the goal is to introduce additional nesting levels into the input stream, the first rule of using \macname{groupedcytoks} is to \macname{stripgroupingtrue} before entering the token cycle. The argument of \macname{groupedcytoks} specifies the tasks that are to occur inside a \macname{cytoks} grouping. So, in the following example \begin{verbbox}[\vbdelim] \stripgroupingtrue \Groupdirective{\taskA \groupedcytoks{\taskB\processtoks{#1}\taskC}% \taskD} \end{verbbox} \thevbox the tasks A and D will occur \textit{outside} of the \macname{cytoks} group. Tasks B and C occur inside the group, immediately before and after processing the tokens of the group, respectively. Thus, the use of \macname{groupedcytoks} in this way permits \TokCycle{} taskings to occur outside of the grouping applied to the output. If \macname{stripgroupingtrue} had been omitted while still using \macname{groupedcytoks}, tasks A and D would have been in the explicit group, and tasks B and C, along with the grouped tokens \argone{}, would have been nested within an additional group. \subsubsection{Implicit grouping tokens: \macname{stripimplicitgroupingcase}} \label{s:impgrp} Implicit grouping tokens (e.g., \macname{bgroup} \& \macname{egroup}) can be handled in one of \textit{three} separate ways. Therefore, rather than using an \textit{if}-condition, the external declaration \macname{stripimplicitgroupingcase\{\}} is provided, which takes one of 3 integers as its argument (0, 1, or $-$1). The package-default case of ``0'' indicates that the implicit-group tokens will not be stripped, but rather echoed directly into the output stream. The case of ``1'' indicates that the implicit-group tokens will be stripped and not placed into the output stream (as with explicit grouping, this is a dangerous case and should be specified with care). Finally, the special case of $-$1 indicates that the implicit-group tokens should instead be passed to the \Character{} directive for further handling (note that the \macname{implicittoktrue} condition will be set\footnote{as well as the internal condition \macname{tc@implicitgrptrue}}). Such a special treatment has limited application---for example, when the intent is to detokenize these tokens. \subsection{Looking ahead at the input stream} In the normal mode of \TokCycle{} operation, each token of the input stream is successively digested and sent to one of four user-defined directives for processing. Such an approach works well if there is no interdependency of adjacent tokens in the input stream. When such an interdepency exists (for example, the argument associated with an invoked macro), one might typically use flags that are set or cleared in one directive that can then be status-checked when the subsequent token is processed. While such an approach is wholly valid, it can create a complex web of flag setting/checking that can span across multiple directives. With the release of v1.4 of the package, an alternative approach to handle such dependencies has been developed: look-ahead features, so that the occurrence of a particular token or condition can provoke an immediate examination of the input stream to handle a possible dependency, without waiting for the next iteration of the token cycle. It should be noted that the use of look-ahead is perhaps more \textit{dangerous} than the traditional use of flags---for if it is not performed with care, tampering with the input stream can destroy its integrity. The commands developed for these situations will now be described. They are designed to be employed as part of \TokCycle's \Character, \Macro, and \Space{} directives for checking and/or handling successively linked tokens in the input stream. They cannot be used inside the \Group{} directive. \subsubsection{\macname{tcpeek}} When issued witin a directive as, for example \macname{tcpeek|z}, the next token from the input stream is \macname{futurelet} into \macname{z}. This has several implications. The input stream remains undisturbed, so that the peeked-at token will still be the token to be digested in the next iteration of the token cycle. Because it has been \macname{let}, \macname{z} does not \textit{contain} the token from the input stream in the manner of a \macname{def}, but rather it is a separate token \textit{possessing the same properties} as the input-stream token. As such, it is ideal to be used for \macname{ifx} comparisons. For example, \macname{ifx|bgroup|z} will detect whether a cat-1 group-opening token is at the head of the input stream, e.g., an opening brace, `\{'; the comparison \macname{ifx|tcsptoken|z} will detect whether a cat-10 space is at the head of the input stream. Such comparisons can be useful in directing the logic of the \TokCycle{} directive. \subsubsection{\macname{tcpop}} When issued as, for example \macname{tcpop|z}, the next token(s) from the input stream is \textit{immediately} digested as an argument and the tokens are placed into the macro \macname{z}. This token (or tokens) will no longer be digested as part of the next iteration within the token cycle, unless the tokens are subsequently replaced at the head of the input stream (see \macname{tcpush} below). Beware that when \TeX{} absorbs an argument, leading white space is lost. Further, if the argument was enclosed in cat-1,2 braces, the braces are stripped and the whole group is absorbed as the argument. In some cases, if the argument needs manipulation, this brace-stripping behavior may be desired. However, if the retention of the leading white space and the braces are desired, one can use manual techniques (possibly involving macros such as \macname{tcpeek}, \macname{tcpopwhitespace} and \macname{tcpushgroup}) or one can instead use \macname{tcpopliteral} (see below) in lieu of \macname{tcpop}. If \macname{tcpop} is used nonetheless, to aid in such matters when cat-10 space is at the head of the input stream, the flag \macname{spacepoppedtrue} will be set when a \macname{tcpop} is issued. However, even \macname{ifspacepopped} will be unable to differentiate explicit ``white'' space that is lost during argument absorption versus implicit space (e.g., \macname{@sptoken} or active-implicit space) that is not discarded. Another option for dealing with leading white space is \macname{tcpopwhitespace} (see below). \subsubsection{\macname{tcpopliteral}} This macro is an alternative to \macname{tcpop}, if the retention of leading white space and possibly braces are required when popping an argument from the input stream. This can be particularly useful if the popped tokens must later be placed back into the input stream in their original state. The syntax, \macname{tcpopliteral|z}, pops an argument into \macname{z}, while retaining possible leading whitespace and brace groups. \subsubsection{\macname{tcpopappto}\ and \macname{tcpopliteralappto}} When issued as, for example \macname{tcpopappto|z}, the next token(s) from the input stream is absorbed as an argument and \textit{appended} to the replacement text of \macname{z}. The same group/space provisos affecting \macname{tcpop} apply here as well. The control sequence to be appended, here \macname{z}, may not be undefined when \macname{tcpopappto} is invoked. This macro is just a convenient joining of two macros: \macname{tcpop} and \macname{tcappto\#1from\#2} (see below). The same applies for the macro \macname{tcpopliteralappto}, which conveniently joins \macname{tcpopliteral} and \macname{tcappto\#1from\#2}. \subsubsection{\macname{tcpopuntil}} This command pops one or more tokens from the input stream in the manner of a delimited argument. Thus, when issued as, for example \macname{tcpopuntil~0|z}, tokens from the input stream are absorbed into \macname{z} until an \texttt{0} token is reached. Unlike delimited-argument absorption, however, the delimiter (\texttt{0} in this example) is also added to the specified control sequence, in this case \macname{z}. This construct is very useful for obtaining optional-argument tokens from the input stream. Consider the following code, with an input stream of \texttt{[1ex]\{2ex\}}: \begin{verbatim} \tcpeek\z \ifx[\z\tcpopuntil ]\q\else\def\q{}\fi \end{verbatim} Since an optional argument is next in the input stream, \macname{q} will obtain the replacement text \texttt{[1ex]}, and the input stream will now begin with \texttt{\{2ex\}}. If an optional argument had not been next in the input stream, then \macname{q} would be empty and the input stream would still begin with \texttt{\{2ex\}}. \subsubsection{\macname{tcpopwhitespace}} When issued as, for example \macname{tcpopwhitespace|z}, leading white space from the input stream will be absorbed and set in \macname{z}, in the manner of a \macname{def}. Unlike the flag \macname{ifspacepopped}, leading implicit space will not be indicated by this macro---only leading explicit white space. If no leading white space is present, \macname{z} will be empty following its invocation and no tokens will have been removed from the input stream. \subsubsection{\macname{tcpush}} To this point, several commands have been described for reading and/or extracting tokens from the input stream. There are also a corresponding commands for placing material at the head of the input stream, should that need arise. With an invocation of \macname{tcpush|z}, the replacement text of \macname{z} will be pushed onto the input stream. Multiple pushes will be handled in a last-in-first-out fashion. Beware that, because argument absorption in \TeX{} will strip the braces of an absorbed group, care must be taken. If the input stream leads with a grouped quantity, such as \texttt{\{abc\}}, then \macname{tcpop|z|tcpush|z} will end up with \texttt{abc} in the input stream, with the braces missing. \textit{The commands \macname{tcpop} and \macname{tcpush} are not strictly inverse operations} (see \macname{tcpopliteral} as an alternative). For this reason, the command \macname{tcpushgroup} is also provided (see below). The command \macname{tcpush} supports an optional argument that functions in the same manner as that of \macname{addcytoks}, to provide additional levels of expansion beyond the mere replacement text. Thus, \macname{tcpush[2]|z} will take the replacement text of \macname{z}, expand it twice, and push the result onto the input stream. In a similar way, \macname{tcpush[x]|z} will fully expand the replacement text of \macname{z} and then place those tokens onto the input stream. Whereas \macname{addytoks} places tokens into the output stream (the token list \macname{cytoks}), \macname{tcpush} places tokens at the head of the input stream. \subsubsection{\macname{tcpushgroup}} The command \macname{tcpushgroup} functions in a manner similar to \macname{tcpush}, except that the replacement text of the argument is placed onto the input stream within a braced (cat-1,2) group. So if the replacement text of \macname{z} is \texttt{abc}, then the invocation \macname{tcpushgroup|z} will place \texttt{\{abc\}} onto the input stream. As with \macname{tcpush}, and with the identical syntax, the command \macname{tcpushgroup} supports an optional argument that directs addition levels of expansion beyond the mere replacement text. \subsubsection{\macname{tcappto\#1from\#2}} This macro does not touch the input stream, per se. However, as tokens are popped from the input stream and placed into macros, this command conveniently allows for their aggregation Thus, if \macname{q} contains \texttt{abc} and the newly popped \macname{z} contains \texttt{d}, then \macname{tcappto |q from |z} will append the contents of \macname{z} to \macname{q}, so that, upon conclusion, \macname{q} will contain the tokens \texttt{abcd}. Unexpandable tokens can also be appended directly using this macro, using the syntax of \macname{tcappto |q from\{123\}}. Expandable tokens can also be added directly with the use of this macro, by using a leading \macname{noexpand} or \macname{empty}, as in \macname{tcappto |q from\{|noexpand|today\}}. \subsection{Truncating the input stream} The basic process of \TokCycle{} is one in which an input stream (or argument) of tokens is examined and processed based on directives set up by the user. Typically, the input stream is processed token-by-token to exhaustion, which occurs when a defined terminating token is reached. In section~\ref{s:escape}, we will see how a sequence of tokens can be exempted from any processing that would otherwise occur in the directive, and be passed instead directly to the output stream. However, in this section, we will examine how \TokCycle{} can be directed to \textit{dynamically} truncate, that is, discard the remaining input stream based on the tokens found therein. Truncation commands may be issued within the \Character, \Macro, and \Space{} directives. More will be said about the \Group{} directive in section~\ref{s:truncingrp}. Truncation can be made to apply either to the remainder of the current \TokCycle-nesting level (corresponding to an explicit catcode-1,2 group in the input stream)\footnote{ Note that implicit groups ({\ttfamily\macname{bgroup}...\macname{egroup}}) do not create a new \TokCycle{} nesting level, nor do {\ttfamily\macname{begingroup}...\macname{endgroup}}. Only explicit catcode-1,2 groups, in the nature of {\ttfamily\{...\}}, create a nested \TokCycle{} level. The current \TokCycle{} nesting level may be obtained from the \TeX{} count \macname{tcdepth}, which starts at zero upon \TokCycle{} entry. This count is incremented and decremented as groups are entered and exited in the input stream. It may be examined within the \CGMS{} directives in order to guide decisions. } or for the remainder of the total \TokCycle{} input stream. \subsubsection{\macname{truncategroup}\ and \macname{truncategroupiftokis}} \label{s:truncgrp} As to truncating the input stream for the remainder of the \TokCycle{} nesting level, the command is simply \macname{truncategroup}. So, for example, the directive \begin{verbbox}[\vbdelim] \Spacedirective{\truncategroup} \end{verbbox} \thevbox would direct \TokCycle{}, if it ever finds a (catcode-10) space token in the input stream, to discard the remainder of the tokens within the group in which that space was initially found. More often, however, one would desire to issue the truncation \textit{conditionally}. One could use one of the conditional commands of section~\ref{s:misc}, for example, \begin{verbbox}[\vbdelim] \tctestifcon{}{\truncategroup}{} \end{verbbox} \thevbox Alternatively, if the \textit{condition} is the occurrence of a particular single token in the input stream, one may use an abbreviated syntax: \begin{verbbox}[\vbdelim] \truncategroupiftokis{}{} \end{verbbox} \thevbox Thus, an example to echo macro tokens to the output stream, unless a \macname{relax} token is found, in which case terminate the group, would be \begin{verbbox}[\vbdelim] \Macrodirective{\truncategroupiftokis{\relax}{\addcytoks{#1}}} \end{verbbox} \thevbox Obviously, checks for particular character tokens would be placed in the \Character{} directive, rather than the \Macro{} directive. \subsubsection{\macname{truncatecycle}\ and \macname{truncatecycleiftokis}} There are two commands that are in every way analogous to the group-\hspace{0pt}terminating macros of section~\ref{s:truncgrp}. These commands are \macname{truncatecycle} and \macname{truncatecycleiftokis}. But in this case, all tokens are discarded to the end of the input stream, not just the current group (i.e., \TokCycle-nesting level). One point to note, however, is that open group levels will be closed by the \macname{truncatecycle} command, so that the tokens, both those being executed as well as those retained in \macname{cytoks}, will be group-balanced. \subsubsection{Truncating from within the \Group{} Directive} \label{s:truncingrp} The truncate commands described in prior sections may \textit{not} be executed from the \Group{} directive, as has already been mentioned. What does termination even mean in the \macname{Groupdirective} context? The \Group{} directive is executed whenever an explicit catcode-1,2 group is encountered in the input stream. The argument, \argone, to that directive will be the complete contents of the group (sans the catcode-1,2 delimiters). So, to truncate the group from the outset, not doing anything with \argone{} will accomplish the desired result. For example, to truncate all group-level-3 tokens, you could say, using the conditional commands of section~\ref{s:misc}, \begin{verbbox}[\vbdelim] \Groupdirective{\tctestifnum{\tcdepth=3}{}{\processtoks{#1}}} \end{verbbox} \thevbox There is no need to use an explicit \macname{truncategroup} macro to accomplish it. On the other hand, to truncate the complete token cycle if, for example, group-level-3 tokens are encountered, you have to set some sort of flag that is immediately picked up by one of the other directives. Here is an example of how that can be done: \begin{verbbox}[\vbdelim] \Groupdirective{\tctestifnum{\tcdepth=3}{\processtoks{\truncatenow}}% {\processtoks{#1}}} \Macrodirective{\truncatecycleiftokis{\truncatenow}{\addcytoks{#1}}} \end{verbbox} \thevbox Note that, as long as \macname{stripgroupingfalse} is active, the above examples will result in empty, rather than absent, level-3 groups, since the group open occurs before the truncation is processed. If the presence of even an empty group is to be avoided in such a case, one may employ the \macname{groupedcytoks} techniques described in section~\ref{s:grpcy}. \subsection{Escaping content from \TokCycle{} processing\label{s:escape}} There are times you may wish to prevent tokens in the \TokCycle{} input stream from being operated on by \TokCycle. Rather, you just want the content passed through unchanged to the output; that is, with the intent to have multi-token content bypass the \TokCycle{} directives altogether. \begin{sloppypar} The method developed by the package is to enclose the escaped content in the input stream between a set of \TokCycle{} escape characters, initially set to a vertical rule character found on the keyboard: ``$|\mkern1mu$''. The main proviso is that the escaped content cannot straddle a group boundary (the complete group may, however, be escaped). The escape character can be changed with \begin{verbbox}[\vbdelim] \settcEscapechar{}\end{verbbox} \theverbbox $\!\!\!$. \end{sloppypar} \subsection{Flagged tokens} Certain token types are trapped and flagged via true/false \textit{if}-conditions. These \textit{if}-conditions can be examined within the appropriate directive (generally the \Character{} directive), to direct the course of action within the directive. \subsubsection{Active characters} \paragraph{\macname{ifactivetok}:} Active (cat-13) tokens that occur in the input stream result in the flag \macname{ifactivetok} being set \macname{activetoktrue}. Note that the expansion of the token's active \macname{def} occurs \textit{after} \TokCycle{} processing. With active \macname{let}'s, there is no text substitution; however, the assignment is already active at the time of \TokCycle{} processing. % The only exception to this rule is with pre-expanded input, \macname{expandedtokcycle[xpress]}. If an active token's substitution is governed by a \macname{def}, the text substitution will have occurred before reaching the token cycle. \paragraph{\macname{ifactivetokunexpandable}:} This flag is similar to \macname{ifactivetok}, in that a token must be active for this to be set true. However, in addition, it is only true if the active token is \macname{let} to a character or a primitive, neither of which can be expanded. Active characters assigned via \macname{def} or else \macname{let} to a macro will \textit{not} qualify as \macname{activetokunexpandabletrue}. \paragraph{\macname{ifactivechar}:} This flag, rather than testing the token, tests the character code of the token, to see if it is set active. Generally, the token and its character code will be syncronized in their \textit{activeness}. However, if a token is tokenized when active, but the corresponding character code is made non-active in the meantime, prior to the token reaching \TokCycle{} processing, this flag will be set \macname{activecharfalse}. A similar discrepency will arise if a token is not active when tokenized, but the character code is made active in the interim, prior to \TokCycle{} processing. \subsubsection{Implicit tokens: \macname{ifimplicittok}} Implicit tokens, those assigned via \macname{let} to characters\footnote{% Some clarification may be needed. Control sequences and active characters that are \macname{let} to something other than a cat-0 control sequence will be flagged as implicit. If implicit, a token will be processed through the \Character{} directive (exceptions noted). On the other hand, if a control sequence or active character is \macname{let} to a cat-0 control sequence, it will be directed to the \Macro{} directive for processing, without the implicit flag. }% , are flagged with a \texttt{true} setting for \macname{ifimplicittok}. Generally, implicit tokens will be directed to the \Character{} directive for processing. There are, however, two exceptions: i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup}) will not appear in any directive unless the \macname{stripimplicitgroupingcase} has been set to $-$1; and ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed by the \Space{} directive. \subsubsection{Active-implicit tokens, including spaces\label{s:acimsp}} One may occasionally run across a token that is both active and implicit. For example, in the following code, \texttt{Q} is made both active and implicit: \begin{verbbox}[\vbdelim] \catcode`Q=\active \let Qx \end{verbbox} \thevbox In general, both \macname{ifactivetok} and \macname{ifimplicittok} tests can be performed together to determine such cases. This is true even in the case of active-implicit catcode-10 spaces, which are always processed through the \Space{} directive. As of \TokCycle{} v1.4, the actual active-implicit space, if encountered in the input stream, will be passed as \verb|#1| to the \Space{} directive, \textit{as long as the character code of the token is still \macname{active}.} If the character code of that token is no longer active, a generic implicit space token named \macname{tcsptoken} is instead passed to the the \Space{} directive as \verb|#1|. In either case, the catcode-12 version of the active character that was digested will be, for that moment, retained in a definition named \macname{theactivespace}. This can be useful if detokenization is required of the spaces. Such an example is described in the \texttt{tokcycle-examples} adjunct document. \subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}} \label{s:cat6} Typically, category-code 6 tokens (like \texttt{\#}) are used to designate a parameter (e.g., \argone--\texttt{\#9}). Since they are unlikely to be used in that capacity inside a tokcycle input stream, the package behavior is to convert them into something cat-12 and set the \textit{if}-condition \macname{catSIXtrue}. In this manner, \macname{ifcatSIX} can be used inside the \Character{} directive to trap and convert cat-6 tokens into something of the user's choosing. \begin{sloppypar} As to the default nature of this conversion (if no special actions are taken), explicit cat-6 characters are converted into the identical character with category code of 12. On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) are converted into a fixed-name macro, \macname{implicitsixtok}, whose cat-12 substitution text is a \macname{string} of the original implicit-macro name. \end{sloppypar} \subsubsection{Parameters (\argone--\texttt{\#9}): \macname{whennotprocessingparameter}} \label{s:whennot} While, generally, one would not intend to use parameters in the \TokCycle{} input stream, the package provides, not a flag, but a macro to allow it. The macro is to be used \textit{within} the \Character{} directive with the syntax:\smallskip \begin{verbbox}[\vbdelim] \whennotprocessingparameter#1{}\end{verbbox} \theverbbox\smallskip \noindent Here, the \argone{} doesn't refer to \argone{} as it might appear in the input stream, but to the sole parameter of the \Character{} directive, referring to the current token being processed. With this syntax, when the token under consideration is a parameter (e.g., \argone--\texttt{\#9}), that parameter is added to the \macname{cytoks} register. If the token under consideration is not a parameter, the code in the final argument is executed. \subsection{Misc: general \textit{if}-condition tools} \label{s:misc} \TeX{} comes equipped with a variety of \macname{if...} condition primitives. When dealing with macros for which the order of expansion is important, the \macname{else} and \macname{fi} can sometimes get in the way of proper expansion and execution. These four restructured \textit{if} macros came in handy writing the package, and may be of use in creating your directives, preventing \macname{else} and \macname{fi} from getting in the way:\medskip \begin{verbbox}[\vbdelim] \tctestifcon{}{}{} \tctestifx{<\ifx-comparison-toks>}{}{} \tctestifnum{<\ifnum-condition>}{}{} \tctestifcatnx{}{} \end{verbbox} \theverbbox \medskip\noindent Note that unlike those macros requiring three arguments, the lone exception is \macname{tctestifcatnx}, which requires four, comparing the catcodes of the two input toks \textbf{without expansion} (\texttt{nx} denotes \macname{noexpand}). \subsection{\TokCycle{} completion: \macname{aftertokcycle}\ and \macname{tcendgroup}} The \TokCycle{} \textit{macros}, upon completion, do nothing. Unlike \macname{tokencycle} \textit{environments}, they don't even output \macname{the}\macname{cytoks} token register. A command has been provided, \macname{aftertokcycle}, which takes an argument to denote the actions to be executed following completion of \textit{all} subsequent \TokCycle{} \textit{macro} invocations within the scope of the current group. It might be specified as simply as \verb|\aftertokcycle{\the\cytoks}|, so as to have the output stream automatically typeset by future invocations of the \macname{tokcycle} macro. The meaning of \macname{aftertokcycle} can be reset with \macname{aftertokcycle\{\}}, but is also reset as part of \macname{resettokcycle}. Unlike macros, the \TokCycle{} environments are unaffected by \macname{aftertokcycle}, as they actually set it locally (within their grouped scope) to accomplish their own purposes. Speaking of the \TokCycle{} environments, there is a command employed by the environments \macname{tokencycle} and \macname{tokencyclexpress} that is likewise available to the user: \macname{tcendgroup}. This is a version of \macname{endgroup} that additionally transmits the current value of the \macname{cytoks} token list \textit{outside of the group}. So if a user wishes to confine certain \TokCycle{} activity inside a group, but wishes to have access to the \macname{cytoks} result afterward, he may simply wrap the activity in \verb|\begingroup...\tcendgroup|. \subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}% \label{s:grouping}} In order to avoid making the \TokCycle{} parser overly complex, requiring multiple passes of the input stream, the package defaults to using catcode-1,2 braces \texttt{\{~\}} to bring about grouping in the output stream, regardless of what the actual cat-1,2 tokens are in the input stream. As long as their sole purpose in the token cycle is for grouping and scoping, this arrangement will produce the expected output. However, if the actual character-code of these tokens is important to the result (e.g., when detokenized), there is one other option. The package allows the external specification of which cat-1,2 tokens should be used in the \TokCycle{} output stream. The syntax is \verb|\settcGrouping{{#1}}|, to use the standard braces for this purpose (default). If angle-brackets \verb|< >| were to be the \textit{new} grouping tokens, then, after their catcodes were changed, and \macname{bgroup} and \macname{egroup} were reassigned, one would invoke \verb|\settcGrouping<<#1>>|. These will then be the grouping tokens in the \TokCycle{} output stream until set to something else. \section{Usage Examples} See the adjunct file, \texttt{tokcycle-examples.pdf}, for an array of \TokCycle{} examples. \section{Summary of known package limitations} The goal of this package is not to build the perfect token-stream parser. It is, rather, to provide the means for users to build useful token-processing tools for their \TeX/\LaTeX{} documents. What follows are the known limitations of the package. which arise, in part, from the single-pass parsing algorithm embedded in the package. Surely, there are more cases associated with arcane catcode-changing syntax that are not accounted for; I encourage you to bring them to my attention. If I can't fix them, I can at least disclaim and declaim them below: \begin{itemize} \item One must inform the package (via \macname{settcGrouping}) of changes to the cat-1,2 tokens \textit{if} there is a need to detokenize the output with the specified bracing group; however, grouping will still be handled properly (i.e., cat-1,2 tokens will be detected), even if the package is not notified. See section~\ref{s:grouping}. \item Should one need to keep track of the \textit{names} of implicit cat-6 tokens (e.g., \verb|\svhash| following \verb|\let\svhash#|), care must be exercised. When encountered in the input stream, a cat-12 \macname{string} of the implicit-cat-6 token \textit{name} is stored in the \macname{implicitsixtok} macro. It is the \macname{implicitsixtok} token which is passed to the \Character{} directive as \verb|#1|, in lieu of the actual cat-6 token. If there is at most a single implicit-cat-6 named token in the input stream, then \macname{implicitsixtok} will reflect its name. However, if there is more than one, \macname{implicitsixtok} will be redefined on the fly with each occurrence. Therefore, should one intend to add more than one such token \textit{name} to the token list by way of \macname{addcytoks}, \textit{it must be expanded first} (e.g., \macname{addcytoks[1]\{\argone\}}), in order to capture the \textit{name} of that particular cat-6 implicit token, rather than the name of the last one in the input stream. There is no similar problem for explicit cat-6 tokens. In any event, all cat-6 tokens are trapped and flag as \texttt{true} the \macname{ifcatSIX} condition, which is how the user should detect their presence. \item A similar warning is to be made for active-implicit spaces as was made for implicit-cat-6 tokens. In the case of active-implicit spaces, the name (a cat-12 string) of the active character is momentarily stored in the \macname{theactivespace} macro. If one wishes to output the active \textit{name} to the \macname{cytoks} token list, one must make sure that \macname{theactivespace} is expanded once before being added to the token list. One difference here, however, is that, unlike a cat-6 token, an active-implicit space (and not a string of it) is actually passed to the \Space{} directive as \texttt{\#1}. In all but one special case, the active-implicit \texttt{\#1} will match that which occurred in the input stream (see section~\ref{s:acimsp}). \end{itemize} \section*{Acknowledgments} In addition to Christian Tellechea, a contributor to this package, the author would like to thank Dr.\@ David Carlisle for his assistance in understanding some of the nuances of token registers. Likewise, his explanation about how a space token is defined in \TeX{} (see \verb|https://tex.stackexchange.com|\allowbreak \verb|/questions/64197/|\allowbreak\verb|pgfparser-module-and-blank-spaces/64200#64200|) proved to be useful here. The \texttt{tex.stackexchange} site provides a wonderful opportunity to interact with the leading developers and practitioners of \TeX{} and \LaTeX{}. \vspace{.6in} \section*{Source Code} %\vspace{-0.1in} \Large\texttt{tokcycle.sty} \verbfilenobox[\footnotesize]{tokcycle.sty}% \vspace{1.5in} \noindent\texttt{tokcycle.tex} \verbfilenobox[\footnotesize]{tokcycle.tex} \end{document}