ePublisher Engine
| Version: | 1.1.0 |
|---|---|
| Date: | 2005-11-06 |
| Changes: | Initial version. |
Abstract
Overview of the design and implementation of ePublisher's engine.
Table of Contents
1 Introduction
The ePublisher engine was designed to provide the minimum framework necessary to solve an unlimited number of user problems. One of the inherent flaws in Publisher 2003 was the strict processing model it enforced upon developers and users. ePublisher's engine eliminates those design limitations and was architected with the goal of enabling third-party developers ready access into the ePublisher workflow. Much work still remains to fully realize this goal, but you can see elements of it in ePublisher today.
1.1 Changes
Initial version.
2 Engine Foundations
ePublisher leverages XSL to transform source documents to target formats. But XSL alone cannot extract XML from Word and FrameMaker documents nor can it render FrameMaker vector graphics. It requires native access to source documents and a standard starting point for XSL transformation to begin. Next, there must be a standard method for coordinating XSL transforms on files.
2.1 Adapters
To extract content from source documents, ePublisher defines an adapter interface. At present, this interface is not available to third-party developers, though there are plans to publish it in the future. This adapter interface allows ePublisher to transparently support a variety of source documents. At present, ePublisher has built-in support for Microsoft Word and Adobe FrameMaker.
When processing, the engine is not aware of the type of adapter it is working with. It simply asks for one that can process a given source document and makes requests to it.
Adapters are responsible for:
- Reporting book files.
- Scanning for styles, conditions, cross-references, and variables.
- Applying specified conditions, cross-reference formats, and variables to conversion documents.
- Extracting native drawings and images.
- Exporting conversion documents to WIF.
This final step is where the source document is effectively brought into the world of XSL processing.
2.2 WIF
WIF is an acronym for "WebWorks Intermediate Format". It is an internal standard at Quadralay Corporation. The goals for WIF are:
- Represent source documents in a standard schema for XSL processing.
- Preserve individual word processor features with high fidelity.
- Strongly favor XSL processing over ease of authoring.
The most frequently asked question regarding WIF can be paraphrased as:
Why didn't you guys use DocBook? Why didn't you guys use OpenOffice? Why didn't you guys use <insert your favorite XML format here>?
The answer is can be summed up as:
Because every XML format has a goal. And that goal is not to sit in the middle and try to be like everything else.
DocBook, for instance, is great for authoring structured content by hand. It isn't so great if the source document lacks structure, i.e. nearly every Word document ever written. Additionally, DocBook just records structure. Formatting is delibrately excluded. As an intermediate format, this model will fail miserably for Word documents defined with style overrides and the single style "Normal".
OpenOffice attempts to mimic Word's features, but that can be detrimental to FrameMaker documents. One case, conditional text, OpenOffice allows users to define just that, conditional text. No formatting, no tables, no graphics, just text. This is certainly an improvement over Word, which has no native conditional text feature, but is not sufficient for FrameMaker documents.
Finally, what about the vernable MIF format from FrameMaker and Publisher 2003? First, as a non-XML file format, XSL processing is not even possible. Second, even if MIF was restructured into a working XML form, it was not designed to handle cases such as nested tables without resorting to anchored frames, etc. It was built for FrameMaker documents only.
Ultimately, the most flexible way to resolve these issues was to create WIF. It is a stable XML schema. However, it will change and grow as needed to support future requirements such as Structured FrameMaker documents, structured Word documents, etc.
2.3 Processing Files by Type
Once source documents are transformed into WIF, XSL processing can commence. What is the best way to organize this XSL processing?
DocBook has a method for organizing XSL processing. It is called a Makefile. Or it can be a Perl script, shell script, VB Script, SConscript, Nmake file, Nant, Ant, etc. In the end, it is something a user writes to handcraft a workflow.
For ePublisher, the goal was to enable flexibility without drugery. DocBook requires intimate knowledge of an XSL tranform's input/output file names. ePublisher requires intimate knowledge of an XSL tranform's input/output file types. This is a subtle but powerful distinction.
Consider processing XSL with exact file names:
xslt doctoc.xsl alpha.xml > alpha_toc.xml xslt doctoc.xsl delta.xml > delta_toc.xml xslt grouptoc.xsl alpha_toc.xml delta_toc.xml > group_toc.xml
Now consider using file types in place of file names:
xslt doctoc.xsl Source > Doc_TOC xslt grouptoc.xsl Doc_TOC > Group_TOC
where file types are defined as follows:
Type Filename Source alpha.xml Source delta.xml Doc_TOC alpha_toc.xml Doc_TOC delta_toc.xml Group_TOC group_toc.xml
The key difference is that there is no need to specify xslt doctoc.xsl Source > Doc_TOC twice. It will fire for every "Source" type document found and generate the corresponding output "Doc_TOC" files.
2.4 Finding Files
The idea of processing files based on type rather than individual file names allows developers to define workflows for any number of input/output files. But how does each XSL transform know what file(s) to process or emit?
ePublisher addresses this issue by defining an XML schema to store file information. A simplified form looks like:
<Files> <File name="alpha.xml" type="Source" /> <File name="delta.xml" type="Source" /> </Files>
This XML document is fed as input to every single XSL transform. And every XSL transform is required to emit the list of generated files using this same schema. So, invoking doctoc.xsl with the above file list will return:
<Files> <File name="alpha_toc.xml" type="Doc_TOC" /> <File name="delta_toc.xml" type="Doc_TOC" /> </Files>
When the final transform is invoked, grouptoc.xsl, the input file list is:
<Files> <File name="alpha.xml" type="Source" /> <File name="delta.xml" type="Source" /> <File name="alpha_toc.xml" type="Doc_TOC" /> <File name="delta_toc.xml" type="Doc_TOC" /> </Files>
and the result file list will be:
<Files> <File name="group_toc.xml" type="Group_TOC" /> </Files>
Why were the "Source" documents feed into the grouptoc.xsl transform? Because all previous output files are available to all downstream XSL transforms.
This is a major change from Publisher 2003. Publisher 2003 defined a Scan stage, a Pre-convert stage, and a Convert stage. Three strikes and you're out. ePublisher allows developers to define as many XSL stages as they like and each stage can process, reprocess, examine, poke, prod, any preceding file generated.
2.5 Format
A format is where developers glue individual XSl transforms together to process documents. A format is broken down into Pipelines and Stages.
2.5.1 Pipeline
A pipeline defines a set of Stages that will be executed in sequence. Further, pipelines can have dependencies on other pipelines. Therefore, developers can insure that required input files will be created before the pipeline executes.
2.5.2 Stage
The smallest descrete action possible in a format. Usually indicates an XSL transform to be executed. In the future, non-XSL actions may be possible. All stages define an output file type and zero or more input file types to operate on.
2.5.3 TOC Example
For the TOC processing example listed above, the corresponding format.wwfmt XML file would be:
<Format> <Pipeline name="TOC"> <Depends pipeline="Locale" /> <Stage type="xsl" action="doc_toc.xsl"> <Parameter name="ParameterDependsType" value="Source" /> <Parameter name="ParameterType" value="Doc_TOC" /> </Stage> <Stage type="xsl" action="group_toc.xsl"> <Parameter name="ParameterDependsType" value="Doc_TOC" /> <Parameter name="ParameterType" value="Group_TOC" /> </Stage> </Pipeline> </Format>
This representation succintly defines the XSL transforms and file types to operate on.
3 Engine Workflow
The ePublisher engine operates on files as follows:
- Apply conditions, cross-reference formats, and variables to conversion documents.
- Save conversion documents to WIF.
- Determine execution order for Format Pipelines.
- Select the next Pipeline available for processing and execute all Stages defined.
A couple of things to keep in mind:
- A Pipeline Stage is invoked exactly once. The Stage must itself determine the number of files to process and the number of files to emit.
- A Stage may pass any message it likes to another Stage provided it is emitted to a file first and then the file's type and path are emitted in the XML result.
A Stage's XSL transform is feed the following global parameters:
GlobalPipelineName:
The name of the pipeline in which this Stage is defined.
GlobalProject:The ePublisher project file as XML.
GlobalInput:All previously generated files derived from the "Generate Selected" command in the Files XML schema.
GlobalFiles:All previously generated files in the Files XML schema.
3.1 Extension Objects
To fill in the blanks where XSL fears to tread, XSL extension objects exist to handle actions such as processing image files, modifying the file system, and writing multiple documents from a single XSL transform.
Note
TBD Document all XSL extension objects.