What is setext
----------------

  The following is extracted from text written by Ian Feldman.
  
  As originally explained in TidBITS#100 and mentioned there from 
  now on, that publication now comes "wrapped as a setext." The noun
  itself stands for both a method to wrap (format) texts according 
  to specific layout rules and for a single _structure_enhanced_
  text.  The latter is a text which has been formatted in such a
  fashion that it contains clues as to the typographical and logical
  structure of its source (word-processed) document(s), if any.
  Those clues, which are called "typotags," facilitate later automatic
  detection of that structure so it can be validated, extracted,
  processed, transformed, enhanced as needed, if needed.

  It follows that setexts, being nothing but pure text (albeit with a
  special layout), are eminently readable using ANY editor or word
  processor in existence today or tommorrow, on any computer with a
  computer program that is capable of opening and reading text files. 
  By default all properly setext-ized files will have an ".etx" or
  ".ETX" suffix.  This stands for an "emailable/ enhanced text", the
  ExtraTerrestrial overtones nothwistanding ;-))

  Unlike other forms of text encoding that use explicit, visible tag
  elements such as <this> and <\that>, the setext format relies
  solely on the presence of _implicit_ typotags, carefully chosen
  to be as visually unobtrusive as possible.  The underlined word
  above is one such instance of the defacto "invisible" coding. 
  Inserted typotags will at worst appear as mere "typos" in the text.

  [Extensions made to the original set of typotags have muddied this
  clarity a little bit, but they were necessary for NEdit development.]

  Similarly, just to give an example, here is a short description
  of the four types of word emphasis typotags that setexts MAY
  contain, limited to one emphasis type ONLY per word or word group:

 -------------------  ----------------------------  --------------
!      **aBoldWord**  **multiple bold words**       ; bold-tt
!_anUnderlinedWord_    _multiple_underlined_words_  ; underline-tt
!    ~anItalicWord~    ~multiple italicized words~  ; italic-tt
!         aHotWord_     multiple_hot_words_         ; hot-tt 
 ----------------------------------------------------------------- 

 What makes a setext?
---------------------

  Before any decoding can take place a text has first to be
  verified whether it is a setext and not some arbitrarily-wrapped
  stream of characters.  Although there are more ways than one to
  achieve that goal there is one _primary_ test that has to be
  passed with colors or else the text being tested cannot be a
  setext. 

  Chief among the typotags are two that signal presence of setext
  titles and subheads inside the text.  A setext document can be
  formatted more or less properly, may contain or lack any other of
  its "native" elements but it has to have at least one proper
  subhead or a title in order to be declared as "a certified
  setext."

    Column 1 of text line
    |
    V
    Here are a few demo setext subheads:     
    ------------------------------------

    _ _ _ _ Which Share Just One _ _ _ _
    ------------------------------------

            ----------> UnifyinG FeaturE
    ------------------------------------

    of EQUAL RIGHTMOST VISIBLE character 
    ------------------------------------

      length as that of its subhead-tt's    
    ------------------------------------

    [this line is called subhead-string]
    ------------------------------------

    [the one below is called subhead-tt]
    ------------------------------------

    [together they make a valid subhead]
    ------------------------------------

       (!) and of course, subheads do not have to be of the same length ;-)
    -----------------------------------------------------------------------

      (nor have to begin in column 1)    
    ---------------------------------

     although it is recommended that they stay below 40 characters
    --------------------------------------------------------------

        Second Setext In This File
    ==============================

     ((end of examples))
     -------------------
     ((_not_ a subhead))
    ^
    |
    Column 1 of text line

  Note, the last 3 lines of the examples do not constitute a valid
  subhead because they do not start in column 1.

  Chief among the reasons why one should first look for presence of
  subheads rather than titles is that it is fully conceivable that a
  setext might have been created without an explicit title-tt in
  order to allow decoder programs to distinguish between part one
  and any subsequent ones in a possible multi-part mailing. This
  absence of a title-tt could be enough of a signal to start looking
  for possible "part x of y" message in either the subject line,
  filename or anywhere "above" the first detected subhead of the
  current text. 

  Therefore, here's a formal definition of what makes a setext:

 +-------------------------------------------------------------+
 |  a text that contains at least one verified setext subhead  |
 |  or setext title                                            |
 +-------------------------------------------------------------+

 Other considerations
---------------------

  A possibility arises to keep the paragraph text unwrapped, rather
  than folded uniformly at say the 66th character mark.  After all,
  if the setext is primarily to be displayed inside an editor,
  rather than on an 80 character terminal screen, then there is not
  much sense in prior folding of the lines to a specific
  guaranteed-to-fit-on-a-TTY-screen length.  The editor/word
  processor program will fit the unwrapped text to the available
  display area, and might actually prefer to have to deal with
  whole unwrapped paragraphs rather than with otherwise relatively
  short lines. 

  Most text-processing programs with native word-wrap capabilities
  actually consider return-terminated lines to be paragraphs in
  their own right.  Thus, if a setext is not to travel via email
  anyway (because of it being distributed differently or making use
  of accented characters) then it might as well arrive in unfolded
  state so that no extra time need be spent on making the
  paragraphs "whole again." [This is not the choice that is taken
  with NEdit help because it is easier to visualize the final text
  for those who do not use text wrapping.] 
  
  Observe that it is not the state of the paragraph text that makes
  or breaks a setext.  No, the sole criterion of whether a text is
  a setext is the presence of at least one verified subhead, as
  described above. Thus even texts with unfolded paragraphs are
  setexts if they contain at least one subhead-tt.

  The sole mechanism used in setext to encode which of such lines
  are in reality paragraphs (as opposed to those that shouldn't be
  folded mechanically) is the character indent.  In fact, after the
  subhead-tt the second most important typotag is the indent-tt,
  made up of exactly two space characters, which denotes any such
  indented lines as ready-candidates for reflowing by so inclined
  front-ends (either on their own or as part of like-indented lines
  above and below it).  So any potentially long line of a setext
  that has been indent-tted will be understood (by any validated
  setext front-end) as to be ready for wrapping-to-length if so
  required. 

.. All the following document by Steven Haehn

Typotags Available
------------------

  The following table contains typotags recognized by the setext
  utility. The "setext form" column in the table is formatted such
  that the left most character of the column represents the first
  character in a line of setext. The circumflex character (^) means
  that the characters of the typotag are significant only when they
  are anchored to the front of the setext line. Typotags marked
  with an asterisk (*) are extensions added for NEdit help
  generation.

!! ============   ===================  ==================
!!      name of   setext form          acted upon or
!!  the typotag   of typotag           displayed as
!! ============   ===================  ==================
!!     title-tt  "Title                a title
!!                ====="               in chosen style
!! ------------   -------------------  ------------------
!!   subhead-tt  "Subhead              a subhead
!!                -------"             in chosen style
!! ------------   -------------------  ------------------
!!   section-tt  ^#> section-text      a section heading
!!                                     with '#' from 1..9
!!                                     in chosen style
!! ------------   -------------------  ------------------
!!    indent-tt  ^  lines indented     lines undented
!!               ^  by 2 spaces        and unfolded
!! ------------   -------------------  ------------------
!!      bold-tt       **[multi]word**  1+ bold word(s)
!!    italic-tt          ~multi word~  1+ italic word(s)
!! underline-tt        [_multi]_word_  underlined text
!!       hot-tt         [multi_]word_  1+ hot word(s)
!!     quote-tt  ^>[space][text]       > [mono-spaced]
!!    bullet-tt  ^*[space][text]       [bullet] [text]
!!   untouch-tt   `_quoted typotag!_`  `_left alone!_`
!!   notouch-tt* ^!followed by text    text-left-alone
!!     field-tt*     |>name[=value]<|  value of name
!!      line-tt* ^   ---               horizontal rule
!! ------------   -------------------  ------------------
!!      href-tt* ^.. _word URL         jump to address
!!      note-tt  ^.. _word Note:("*")  ("cause error")
!!    target-tt*     _[multi_]word     [multi ]word
!! ------------   -------------------  ------------------
!!   twobuck-tt   $$ [last on a line]  [parse another]
!!  suppress-tt  ^..[space][not dot]   [line hidden]
!!    twodot-tt  ^..[alone on a line]  [taken note of]
!! ------------   -------------------  ------------------
!!     maybe-tt* ^.. ? name[~] text    show text when
!!                                     name defined
!!  maybenot-tt* ^.. ! name[~] text    show text when
!!                                     name NOT defined
!!  endmaybe-tt* ^.. ~ name            end of a multi-
!!                                     line maybe[not]-tt
!! ------------   -------------------  ------------------
!!  passthru-tt* ^!![text]             text emitted
!!                                     without processing
!! ------------   -------------------  ------------------
!!    escape-tt*  @x where 'x'  is     x is what remains
!!                escaped character    @@ needed for 1 @
!! ============   ===================  ==================
!!

  The title-tt, subhead-tt and indent-tt have already been
  discussed in length in the previous sections. All typotag
  elements, but the subhead-tt, are optional, that is, not
  necessary for a setext to be declared as such. The simple
  character marking typotags, bold-tt, italic-tt, and underline-tt
  have been used throughout the document and are used to mark text
  with their obvious meanings.

3>Section-tt (document divisions)

  The section-tt allows subdividing of the setext into further
  subsections for greater nesting capability. Typical usage starts
  the numbering level at 3 because the title-tt and subhead-tt
  basically represent sections 1 and 2, respectively.

3>Bullet-tt (list marker)

  The bullet-tt typotag is use to create a list of items. Note that
  it can only be used to create single line entries, like the
  following:
  
    Column 1 of text line
    |
    V
    * This is the first bullet.
    * This is the second bullet.
  
  Remember that you have to insert empty lines immediately before
  and after the bullet list.

3>Untouch-tt, Notouch-tt, Passthru-tt, Escape-tt (quoting text)

  Each one of these leave-my-text-alone typotags offer varying
  degrees of operation. The untouch-tt surrounds the text that
  is not to be interpreted. The accent grave (`) character is
  used to start and finish the untouchable text. (An extension
  to this has been allowed in the setext utility. An untouch-tt
  may be terminated by an apostrophe (').) The following are
  all valid untouch-tt typotags.
  
    `this is the _original_ version of the untouch-tt`
    `this is the _extended_ form of the untouch-tt'
    `This couldn't _be_ a problem could it?'

  Note that the third example has used the contraction "couldn't"
  which did not terminate the untouch-tt because the apostrophe was
  not followed by whitespace or punctuation.
  
  The notouch-tt typotag is used to take care of entire lines of
  text. The difference between this and the untouch-tt is that there
  is no visual residual typotag mark left in the output. It is
  replaced by a space. For example,
  
    Column 1 of text line
    |
    V
    ! This line of text will look like this sans the ! in column 1.

  becomes,
  
      This line of text will look like this sans the ! in column 1.

  The difference between the passthru-tt and the notouch-tt is
  the subtle difference of not replacing the markers with space, but
  totally removing them. (The original usage was to try to emit
  special 'C' compiler directives directly into the help code
  product). Thus,
  
    Column 1 of text line
    |
    V
    !!#ifdef VMS

  would turn into
  
    #ifdef VMS
  
  The escape-tt (@) is used to escape the special markers of 
  the other typotags and itself. Here is an example of escaping
  itself.
  
    develop@@nedit.org

  This will become "develop@nedit.org" in resulting documents.
  

3>Suppress-tt, Twodot-tt (author annotations or comments)

  The suppress-tt typotag allows an author to place annotations in a
  setext document which will not appear in a generated product. Most
  of the extensions to the original setext definition were placed
  inside this form of typotag.
  
    Column 1 of text line
    |
    V
    .. This is a document comment that would normally disappear
    .. from generated text, html, or the like. These lines are
    .. what constitute a suppress-tt. The following line is the
    .. twodot-tt.
    ..
    
3>Hot-tt, Href-tt, Target-tt (hyperlinking text)

  These three typotags are used in conjunction to create
  hypertext reference mechanism used int HTML and NEdit
  help code generation. The hot-tt is an original typotag which
  needed the additional two tags to be able create actual hyperlinks
  to other sections of the document, or to external references that
  could be exploited. These tags are ignored (stripped) when
  generating simple text documents.
  
  The hot-tt typotag is used to mark the text which would be used as
  the doorway to accessing other parts of the document. It either
  references a title or subhead string directly, or an href-tt. An
  href-tt (hypertext reference typotag) is used as an intermediary
  for the hyperlink destination. Its value either specifies an
  external document reference, or an internal document reference.
  The target-tt is used to mark the internal document references
  mentioned in a href-tt.
  
  Now for some examples. All the marked text will be inside
  parenthesis so it will stand out as to what explicitly is being
  marked. 
  
  This hot-tt directly references the (Typotags_Available_)
  subheading above. Whereas, the following hot-tt (references_) 
  the href-tt marked by this target-tt (_typotag).

  Here is what the href-tt would look like:
  
    Column 1 of text line
    |
    V
!   .. _references #typotsg

.. The following line is the actual hypertext reference in this
.. document. This annotation is an example of supress-tt usage. 
.. _references #typotag

3>Maybe[not]-tt, Endmaybe-tt (conditional text regions)

  Multiple line maybe-tt or maybenot-tt (conditional text regions)
  are introduced as follows:

    Column 1 of text line
    |
    V
    .. ? name~   (this is the maybe-tt)
    .. ! name~   (this is the maybenot-tt)
    
  Both are terminated with an endmaybe-tt on a separate line.

    Column 1 of text line
    |
    V
    .. ~ name

  The name* of the conditional region is left up to the text
  author.  Single line maybe[not]-tt typotags do not use the '~'
  character at the end of the name and are terminated at the end
  of the line. 

    Column 1 of text line
    |
    V
    .. ? oneLine (This is a one line maybe-tt)
    .. ! oneLine (This is a one line maybenot-tt)
  
  * There are some predefined conditional region names that are
  already known to the setext parser: html, text, and (NEdit) help.
  The special conditional text region named "html" allows a mixture
  of setext and HTML tags.

  Nesting of conditional text regions is allowed. For instance, if
  there are three conditional regions, A, B, and C, C can be nested
  inside B, which can be nested inside A. For example,
  A-B-C...C-B-A. 

      Column 1 of text line
    |
    V
    .. ? A~    Example of legally nested conditional text regions
    .. ? B~
    .. ? C~
    .. ~ C
    .. ~ B
    .. ~ A
  
  Note that a surrounding region cannot end before one of its inner
  regions is terminated (eg. of illegal nesting A-B-C...C-A-B,
  where A terminated prior to B).

3>Field-tt (variable definition and substitution)

  Field-tt typotags are used to define variables and reference
  their values. Field definitions can only occur within a
  suppress-tt.

  For example, to define the variable 'author' and fill it
  with the value "Steven Haehn":
 
      Column 1 of text line
    |
    V
    .. |>author=Steven Haehn<|

  To use the value of the defined variable, place the field-tt,
  |>author<|, in any printable text region. If there is no known
  value for the  field, it will remain unchanged and appear as
  written in the setext.

  The following are predefined for use in a field-tt
  for any setext document translated by the setext utility.

    Date = <MonthName day, year>         (eg. December 6, 2001)
    date = <MonthAbbreviation day, year> (eg. Dec 6, 2001)
    year = <year>                        (eg. 2001)

3>Line-tt (horizontal rule demarcation)

  This typotag is used to place horizontal markers into generated
  text documents. Like the following.

   Column 4 of text line
   |
   V
   -------------------------------------------------------------  

3>Twobuck-tt (setext termination marker)

  This typotag is used to mark the end of document parsing.
  
 $$

$Id: setext-info.txt,v 1.3 2002/09/26 12:37:38 ajhood Exp $