-
Notifications
You must be signed in to change notification settings - Fork 9
/
Copy pathsgml.doc
1534 lines (1262 loc) · 59.2 KB
/
sgml.doc
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\documentclass[11pt]{article}
\usepackage{times}
\usepackage{pl}
\usepackage{html}
\sloppy
\makeindex
\onefile
\htmloutput{.} % Output directory
\htmlmainfile{sgml} % Main document file
\bodycolor{white} % Page colour
\begin{document}
\title{SWI-Prolog SGML/XML parser}
\author{Jan Wielemaker \\
VU University Amsterdam \\
University of Amsterdam \\
The Netherlands \\
E-mail: \email{[email protected]}}
\maketitle
\begin{abstract}
Markup languages are an increasingly important method for
data-representation and exchange. This article documents the package
\pllib{sgml}, a foreign library for SWI-Prolog to parse SGML
and XML documents, returning information on both the document and the
document's DTD. The parser is designed to be small, fast and flexible.
\end{abstract}
\pagebreak
\tableofcontents
\vfill
\vfill
\newpage
\section{Introduction}
\label{sec:sgml-intro}
Markup languages have recently regained popularity for two reasons. One
is document exchange, which is largely based on HTML, an instance of
SGML, and the other is for data exchange between programs, which is
often based on XML, which can be considered a simplified and
rationalised version of SGML.
James Clark's SP parser is a flexible SGML and XML parser. Unfortunately
it has some drawbacks. It is very big, not very fast, cannot work under
event-driven input and is generally hard to program beyond the scope of
the well designed generic interface. The generic interface however does
not provide access to the DTD, does not allow for flexible handling of
input or parsing the DTD independently of a document instance.
The parser described in this document is small (less than 100 kBytes
executable on a Pentium), fast (between 2 and 5 times faster than SP),
provides access to the DTD, and provides flexible input handling.
The document output is equal to the output produced by \jargon{xml2pl},
an SP interface to SWI-Prolog written by Anjo Anjewierden.
\section{Bluffer's Guide}
\label{sec:sgml-bluffers-guide}
This package allows you to parse SGML, XML and HTML data into a Prolog
data structure. The high-level interface defined in \pllib{sgml}
provides access at the file-level, while the low-level interface defined
in the foreign module works with Prolog streams. Please use the source
of \file{sgml.pl} as a starting point for dealing with data from
other sources than files, such as SWI-Prolog resources, network-sockets,
character strings, \emph{etc.} The first example below loads an HTML file.
\begin{code}
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<title>Demo</title>
</head>
<body>
<h1 align=center>This is a demo</title>
Paragraphs in HTML need not be closed.
This is called `omitted-tag' handling.
</body>
</html>
\end{code}
\begin{code}
?- load_html('test.html', Term, []),
print_term(Term, []).
[ element(html,
[],
[ element(head,
[],
[ element(title,
[],
[ 'Demo'
])
]),
element(body,
[],
[ '\n',
element(h1,
[ align = center
],
[ 'This is a demo'
]),
'\n\n',
element(p,
[],
[ 'Paragraphs in HTML need not be closed.\n'
]),
element(p,
[],
[ 'This is called `omitted-tag\' handling.'
])
])
])
].
\end{code}
The document is represented as a list, each element being an atom to
represent \const{CDATA} or a term \term{element}{Name, Attributes, Content}.
Entities (e.g. \verb$<$) are expanded and included in the
atom representing the element content or attribute value.%
\footnote{Up to SWI-Prolog 5.4.x, Prolog could not represent
\jargon{wide} characters and entities that did not fit in
the Prolog characters set were emitted as a term
\term{number}{+Code}. With the introduction of wide
characters in the 5.5 branch this is no longer needed.}
\subsection{`Goodies' Predicates}
\label{sec:sgml-goodies}
These predicates are for basic use of the library, converting entire and
self-contained files in SGML, HTML, or XML into a structured term. They
are based on load_structure/3.
\begin{description}
\predicate{load_sgml}{3}{+Source, -ListOfContent, :Options}
Calls load_structure/3 with the given \arg{Options}, using the
default option \term{dialect}{sgml}
\predicate{load_xml}{3}{+Source, -ListOfContent, :Options}
Calls load_structure/3 with the given \arg{Options}, using the
default option \term{dialect}{xml}
\predicate{load_html}{3}{+Source, -ListOfContent, :Options}
Calls load_structure/3 with the given \arg{Options}, using the
default options \term{dialect}{HTMLDialect}, where \arg{HTMLDialect}
is \const{html4} or \const{html5} (default), depending on the Prolog
flag \const{html_dialect}. Both imply the option \term{shorttag}{false}.
The option \term{dtd}{DTD} is passed, where \arg{DTD} is the HTML DTD as
obtained using \term{dtd}{html, DTD}. See dtd/2.
\end{description}
\section{Predicate Reference}
\label{sec:sgml-predicates}
\subsection{Loading Structured Documents}
\label{sec:sgml-load-structure}
SGML or XML files are loaded through the common predicate
load_structure/3. This is a predicate with many options. For
simplicity a number of commonly used shorthands are provided:
load_sgml_file/2, load_xml_file/2, and
load_html_file/2.
\begin{description}
\predicate{load_structure}{3}{+Source, -ListOfContent, +Options}
Parse \arg{Source} and return the resulting structure in
\arg{ListOfContent}. \arg{Source} is either a term of the format
\term{stream}{StreamHandle} or a file-name. \arg{Options} is a list of
options controlling the conversion process.
A proper XML document contains only a single toplevel element whose name
matches the document type. Nevertheless, a list is returned for
consistency with the representation of element content. The
\arg{ListOfContent} consists of the following types:
\begin{description}
\termitem{\arg{Atom}}{}
Atoms are used to represent \const{CDATA}. Note
this is possible in SWI-Prolog, as there is no length-limit on atoms and
atom garbage collection is provided.
\termitem{element}{Name, ListAttributes, ListOfContent}
\arg{Name} is the name of the element. Using SGML, which is
case-insensitive, all element names are returned as lowercase atoms.
\arg{ListOfAttributes} is a list of \arg{Name}=\arg{Value} pairs for
attributes. Attributes of type \const{CDATA} are returned literal. Multi-valued
attributes (\const{NAMES}, \emph{etc.}) are returned as a list of atoms.
Handling attributes of the types \const{NUMBER} and \const{NUMBERS} depends on
the setting of the \term{number}{+NumberMode} attribute through
set_sgml_parser/2 or load_structure/3. By
default they are returned as atoms, but automatic conversion to Prolog
integers is supported. \arg{ListOfContent} defines the content for the
element.
\termitem{sdata}{Text}
If an entity with declared content-type \const{SDATA} is encountered, this
term is returned holding the data in \arg{Text}.
\termitem{ndata}{Text}
If an entity with declared content-type \const{NDATA} is encountered, this
term is returned holding the data in \arg{Text}.
\termitem{pi}{Text}
If a processing instruction is encountered (\verb$<?...?>$), \arg{Text}
holds the text of the processing instruction. Please note that the
\verb$<?xml ...?>$ instruction is handled internally.
\end{description}
The \arg{Options} list controls the conversion process.
Currently defined options are below.
Other options are passed to sgml_parse/2.
\begin{description}
\termitem{dtd}{?DTD}
Reference to a DTD object. If specified, the \verb$<!DOCTYPE ...>$
declaration is ignored and the document is parsed and validated against
the provided DTD. If provided as a variable, the created DTD is
returned. See \secref{implicitdtd}.
\termitem{dialect}{+Dialect}
Specify the parsing dialect. Supported are \const{sgml} (default),
\const{html4}, \const{html5}, \const{html} (same as \const{html4},
\const{xhtml}, \const{xhtml5}, \const{xml} and \const{xmlns}. See the
option \const{dialect} of set_sgml_parser/2 for details.
\termitem{shorttag}{+Bool}
Define whether SHORTTAG abbreviation is accepted. The default is true
for SGML mode and false for the XML modes. Without SHORTTAG, a
\chr{/} is accepted with warning as part of an unquoted
attribute-value, though \verb$/>$ still closes the element-tag in
XML mode. It may be set to false for parsing HTML documents to allow for
unquoted URLs containing \chr{/}.
\termitem{space}{+SpaceMode}
Sets the `space-handling-mode' for the initial environment. This mode is
inherited by the other environments, which can override the inherited
value using the XML reserved attribute \const{xml:space}. See
\secref{xml-whitespace}.
\termitem{number}{+NumberMode}
Determines how attributes of type \const{NUMBER} and \const{NUMBERS} are
handled. If \const{token} (default) they are passed as an atom. If
\const{integer} the parser attempts to convert the value to an integer.
If successful, the attribute is passed as a Prolog integer. Otherwise it
is still passed as an atom. Note that SGML defines a numeric attribute
to be a sequence of digits. The \const{-} sign is not allowed and
\exam{1} is different from \exam{01}. For this reason the default is to
handle numeric attributes as tokens. If conversion to integer is
enabled, negative values are silently accepted.
\termitem{case_sensitive_attributes}{+Boolean}
Treat attribute values as case sensitive. The default is \const{true}
for XML and \const{false} for SGML and HTML dialects.
\termitem{case_preserving_attributes}{+Boolean}
Treat attribute values as case insensitive but do not alter their
case. The default is \const{false}. Setting this option sets the
\const{case_sensitive_attributes} to the same value. This option
was added to support HTML quasi quotations and most likely has
little value in other contexts.
\termitem{system_entities}{+Boolean}
Define whether SYSTEM entities are expanded. The default is \const{false}.
\termitem{defaults}{+Bool}
Determines how default and fixed values from the DTD are used. By
default, defaults are included in the output if they do not appear in
the source. If \const{false}, only the attributes occurring in the source
are emitted.
\termitem{entity}{+Name, +Value}
Defines (overwrites) an entity definition. At the moment, only
\const{CDATA} entities can be specified with this construct. Multiple
entity options are allowed.
\termitem{file}{+Name}
Sets the name of the file on which errors are reported. Sets the
linenumber to 1.
\termitem{line}{+Line}
Sets the starting line-number for reporting errors.
\termitem{max_memory}{+Max}
Sets the maximum buffer size in bytes available for input data and CDATA output. If this limit is reached a resource error is raised. Using \term{max_memory}{0} (the default) means no resource limit will be enforced.
\termitem{ignore_doctype}{+Bool}
If set, doctype declarations in the document will be ignored. This can help prevent XXE attacks
\termitem{cdata}{+Representation}
Specify the representation of \jargon{cdata} elements. Supported are
\const{atom} (default), and \const{string}. The choice is not obvious.
Strings are allocated on the Prolog stacks and subject to normal stack
garbage collection. They are quicker to create and avoid memory
fragmentation. But, multiple copies of the same string are stored
multiple times, while the text is shared if atoms are used. Strings are
also useful for security sensitive information as they are invisible to
other threads and cannot be enumerated using, e.g., current_atom/1.
Finally, using strings allows for resource usage limits using the global
stack limit (see set_prolog_stack/2).
\termitem{attribute_value}{+Representation}
Specify the representation of attribute values. Supported are
\const{atom} (default), and \const{string}. See above for the
advantages and disadvantages of using strings.
\termitem{keep_prefix}{+Boolean}
If \const{true}, xmlns namespaces with prefixes are returned as
\term{ns}{Prefix, URI} terms. If \const{false} (default), the prefix is
ignored and the xmlns namespace is returned as just the URI.
\end{description}
\end{description}
\subsection{Handling white-space} \label{sec:xml-whitespace}
SGML2PL has four modes for handling white-space. The initial mode can be
switched using the \term{space}{SpaceMode} option to
load_structure/3 and set_sgml_parser/2. In XML
mode, the mode is further controlled by the \const{xml:space}
attribute, which may be specified both in the DTD and in the document.
The defined modes are:
\begin{description}
\termitem{space}{sgml}
In SGML, newlines at the start and end of an element are
removed.\footnote{In addition, newlines at the end of lines
containing only markup should be deleted. This is not yet
implemented.} This is the default mode for the SGML dialect.
\termitem{space}{preserve}
White space is passed literally to the application. This mode leaves most
white space handling to the application. This is the default mode for
the XML dialect. Note that \verb$\r\n$ is still translated to \verb$\n$.
To preserve whitespace \emph{exactly}, use \term{space}{strict} (see
below)
\termitem{space}{strict}
White space is passed strictly to the application. This mode leaves all
white space handling to the application. This is useful for producing
and verifying XML signatures.
\termitem{space}{default}
In addition to \const{sgml} space-mode, all consequtive white-space is
reduced to a single space-character. This mode canonicalises all white
space.
\termitem{space}{remove}
In addition to \const{default}, all leading and trailing white-space is
removed from \const{CDATA} objects. If, as a result, the \const{CDATA}
becomes empty, nothing is passed to the application. This mode is
especially handy for processing `data-oriented' documents, such as RDF.
It is not suitable for normal text documents. Consider the HTML
fragment below. When processed in this mode, the spaces between the
three modified words are lost. This mode is not part of any standard;
XML 1.0 allows only \const{default} and \const{preserve}.
\begin{code}
Consider adjacent <b>bold</b> <ul>and</ul> <it>italic</it> words.
\end{code}
\end{description}
\subsection{XML documents} \label{sec:xml}
The parser can operate in two modes: \const{sgml} mode and \const{xml} mode, as
defined by the \term{dialect}{Dialect} option. Regardless of this
option, if the first line of the document reads as below, the parser is
switched automatically into XML mode.
\begin{code}
<?xml ... ?>
\end{code}
Currently switching to XML mode implies:
\begin{itemlist}
\item [XML empty elements]
The construct \verb$<element [attribute...] />$ is recognised as
an empty element.
\item [Predefined entities]
The following entitities are predefined: \const{lt} (\verb$<$), \const{gt}
(\verb$>$), \const{amp} (\verb$&$), \const{apos} (\verb$'$)
and \const{quot} (\verb$"$).
\item [Case sensitivity]
In XML mode, names are treated case-sensitive, except for the DTD
reserved names (i.e. \exam{ELEMENT}, \emph{etc.}).
\item [Character classes]
In XML mode, underscores (\verb$_$) and colon (\verb$:$) are
allowed in names.
\item [White-space handling]
White space mode is set to \const{preserve}. In addition to setting
white-space handling at the toplevel the XML reserved attribute
\const{xml:space} is honoured. It may appear both in the document
and the DTD. The \const{remove} extension is honoured as
\const{xml:space} value. For example, the DTD statement below
ensures that the \const{pre} element preserves space, regardless
of the default processing mode.
\begin{code}
<!ATTLIST pre xml:space nmtoken #fixed preserve>
\end{code}
\end{itemlist}
\subsubsection{XML Namespaces} \label{sec:xmlns}
Using the \jargon{dialect} \const{xmlns}, the parser will interpret XML
namespaces. In this case, the names of elements are returned as a term
of the format
\begin{quote}
\arg{URL}\const{:}\arg{LocalName}
\end{quote}
If an identifier has no namespace and there is no default namespace it
is returned as a simple atom. If an identifier has a namespace but this
namespace is undeclared, the namespace name rather than the related URL
is returned.
Attributes declaring namespaces ({\tt xmlns:<ns>=<url>}) are reported
as if \const{xmlns} were not a defined resource.
In many cases, getting attribute-names as \arg{url}:\arg{name}
is not desirable. Such terms are hard to unify and sometimes multiple
URLs may be mapped to the same identifier. This may happen due to poor
version management, poor standardisation or because the the application
doesn't care too much about versions. This package defines two
call-backs that can be set using set_sgml_parser/2 to deal
with this problem.
The call-back \const{xmlns} is called as XML namespaces are noticed.
It can be used to extend a canonical mapping for later use
by the \const{urlns} call-back. The following illustrates this behaviour.
Any namespace containing \const{rdf-syntax} in its URL or that is used as
\const{rdf} namespace is canonicalised to \const{rdf}. This implies that any
attribute and element name from the RDF namespace appears as
\verb$rdf:<name>$
\begin{code}
:- dynamic
xmlns/3.
on_xmlns(rdf, URL, _Parser) :- !,
asserta(xmlns(URL, rdf, _)).
on_xmlns(_, URL, _Parser) :-
sub_atom(URL, _, _, _, 'rdf-syntax'), !,
asserta(xmlns(URL, rdf, _)).
load_rdf_xml(File, Term) :-
load_structure(File, Term,
[ dialect(xmlns),
call(xmlns, on_xmlns),
call(urlns, xmlns)
]).
\end{code}
The library provides iri_xml_namespace/3 to break down an IRI into
its namespace and localname:
\begin{description}
\predicate[det]{iri_xml_namespace}{3}{+IRI, -Namespace, -Localname}
Split an IRI (Unicode URI) into its \arg{Namespace} (an IRI) and
\arg{Localname} (a Unicode XML name, see xml_name/2). The
\arg{Localname} is defined as the longest last part of the IRI that
satisfies the syntax of an XML name. With IRI schemas that are designed
to work with XML namespaces, this will typically break the IRI on the
last \chr{\#} or \chr{/}. Note however that this can produce unexpected
results. E.g., in the example below, one might expect the namespace to
be \url{http://example.com/images\#}, but an XML name cannot start with
a digit.
\begin{code}
?- iri_xml_namespace('http://example.com/images#12345', NS, L).
NS = 'http://example.com/images#12345',
L = ''.
\end{code}
As we see from the example above, the \arg{Localname} can be the empty
atom. Similarly, \arg{Namespace} can be the empty atom if \arg{IRI} is
an XML name. Applications will often have to check for either or both
these conditions. We decided against failing in these conditions because
the application typically wants to know which of the two conditions
(empty namespace or empty localname) holds. This predicate is often used
for generating RDF/XML from an RDF graph.
\predicate[det]{iri_xml_namespace}{2}{+IRI, -Namespace}
Same as iri_xml_namespace/3, but avoids creating an atom for the
\arg{Localname}.
\end{description}
\subsection{DTD-Handling}
\label{sec:sgml-dtd}
The DTD (\textbf{D}ocument \textbf{T}ype \textbf{D}efinition) is a
separate entity in sgml2pl, that can be created, freed, defined and
inspected. Like the parser itself, it is filled by opening it as a
Prolog output stream and sending data to it. This section summarises the
predicates for handling the DTD.
\begin{description}
\predicate{new_dtd}{2}{+DocType, -DTD}
Creates an empty DTD for the named \arg{DocType}. The returned
DTD-reference is an opaque term that can be used in the other predicates
of this package.
\predicate{free_dtd}{1}{+DTD}
Deallocate all resources associated to the DTD. Further use of \arg{DTD}
is invalid.
\predicate{load_dtd}{2}{+DTD, +File}
Define the DTD by loading the SGML-DTD file \arg{File}. Same
as load_dtd/3 with empty option list.
\predicate{load_dtd}{3}{+DTD, +File, +Options}
Define the DTD by loading \arg{File}. Defined options are the
\const{dialect} option from open_dtd/3 and the \const{encoding}
option from open/4. Notably the \const{dialect} option must
match the dialect used for subsequent parsing using this DTD.
\predicate{open_dtd}{3}{+DTD, +Options, -OutStream}
Open a DTD as an output stream. See load_dtd/2 for an example.
Defined options are:
\begin{description}
\termitem{dialect}{Dialect}
Define the DTD dialect. Default is \const{sgml}. Using \const{xml} or
\const{xmlns} processes the DTD case-sensitive.
\end{description}
\predicate{dtd}{2}{+DocType, -DTD}
Find the DTD representing the indicated \jargon{doctype}. This predicate
uses a cache of DTD objects. If a doctype has no associated dtd, it
searches for a file using the file search path \exam{dtd} using the call:
\begin{code}
...,
absolute_file_name(dtd(Type),
[ extensions([dtd]),
access(read)
], DtdFile),
...
\end{code}
Note that DTD objects may be modified while processing errornous
documents. For example, loading an SGML document starting with
\verb$<?xml ...?>$ switches the DTD to XML mode and encountering unknown
elements adds these elements to the DTD object. Re-using a DTD object to
parse multiple documents should be restricted to situations where the
documents processed are known to be error-free.
The DTD \const{html} is handled separately. The Prolog flag
\const{html_dialect} specifies the default html dialect, which is either
\const{html4} or \const{html5} (default).\footnote{Note that HTML5 has
no DTD. The loaded DTD is an informal DTD that includes most of the
HTML5 extensions (\url{http://www.cs.tut.fi/~jkorpela/html5-dtd.html}).
In addition, the parser sets the \const{dialect} flag of the DTD object.
This is used by the parser to accept HTML extensions.} Next, the
corresponding DTD is loaded.
\predicate{dtd_property}{2}{+DTD, ?Property}
This predicate is used to examine the content of a DTD. Property is one
of:
\begin{description}
\termitem{doctype}{DocType}
An atom representing the document-type defined by this DTD.
\termitem{elements}{ListOfElements}
A list of atoms representing the names of the elements in this DTD.
\termitem{element}{Name, Omit, Content}
The DTD contains an element with the given name. \arg{Omit} is a term of
the format \term{omit}{OmitOpen, OmitClose}, where both arguments are
booleans (\const{true} or \const{false} representing whether the open-
or close-tag may be omitted. \arg{Content} is the content-model of the
element represented as a Prolog term. This term takes the following
form:
\begin{description}
\termitem{empty}{}
The element has no content.
\termitem{cdata}{}
The element contains non-parsed character data. All data up to the
matching end-tag is included in the data (\jargon{declared content}).
\termitem{rcdata}{}
As \const{cdata}, but entity-references are expanded.
\termitem{any}{}
The element may contain any number of any element from the DTD in
any order.
\termitem{\#pcdata}{}
The element contains parsed character data .
\termitem{\arg{element}} An element with this name.
\termitem{*}{SubModel}
0 or more appearances.
\termitem{?}{SubModel}
0 or one appearance.
\termitem{+}{SubModel}
1 or more appearances.
\termitem{,}{SubModel1, SubModel2}
\arg{SubModel1} followed by \arg{SubModel2}.
\termitem{\&}{SubModel1, SubModel2}
\arg{SubModel1} and \arg{SubModel2} in any order.
\termitem{\chr{|}}{SubModel1, SubModel2}
\arg{SubModel1} or \arg{SubModel2}.
\end{description}
\termitem{attributes}{Element, ListOfAttributes}
\arg{ListOfAttributes} is a list of atoms representing the attributes
of the element \arg{Element}.
\termitem{attribute}{Element, Attribute, Type, Default}
Query an element. \arg{Type} is one of \const{cdata}, \const{entity},
\const{id}, \const{idref}, \const{name}, \const{nmtoken},
\const{notation}, \const{number} or \const{nutoken}. For DTD types that
allow for a list, the notation \term{list}{Type} is used. Finally, the
DTD construct \verb$(a|b|...)$ is mapped to the term
\term{nameof}{ListOfValues}.
\arg{Default} describes the sgml default. It is one \const{required},
\const{current}, \const{conref} or \const{implied}. If a real default is
present, it is one of \term{default}{Value} or \term{fixed}{Value}.
\termitem{entities}{ListOfEntities}
\arg{ListOfEntities} is a list of atoms representing the names of the
defined entities.
\termitem{entity}{Name, Value}
\arg{Name} is the name of an entity with given value. Value is one of
\begin{description}
\termitem{\arg{Atom}}{}
If the value is atomic, it represents the literal value of the entity.
\termitem{system}{Url}
\arg{Url} is the URL of the system external entity.
\termitem{public}{Id, Url}
For external public entities, \arg{Id} is the identifier. If an URL is
provided this is returned in \arg{Url}. Otherwise this argument is
unbound.
\end{description}
\termitem{notations}{ListOfNotations}
Returns a list holding the names of all \const{NOTATION} declarations.
\termitem{notation}{Name, Decl}
Unify \arg{Decl} with a list if \term{system}{+File} and/or
\term{public}{+PublicId}.
\end{description}
\end{description}
\subsubsection{The DOCTYPE declaration}
\label{sec:sgml-doctype}
As this parser allows for processing partial documents and process the
DTD separately, the DOCTYPE declaration plays a special role.
If a document has no DOCTYPE declaraction, the parser returns a list
holding all elements and CDATA found. If the document has a DOCTYPE
declaraction, the parser will open the element defined in the DOCTYPE as
soon as the first real data is encountered.
\subsection{Extracting a DTD} \label{sec:implicitdtd}
Some documents have no DTD. One of the neat facilities of this library
is that it builds a DTD while parsing a document with an
\jargon{implicit} DTD. The resulting DTD contains all elements
encountered in the document. For each element the content model is a
disjunction of elements and possibly \verb$#PCDATA$ that can be
repeated. Thus, if we found element \const{y} and CDATA in element
\const{x}, the model is:
\begin{code}
<!ELEMENT x - - (y|#PCDATA)*>
\end{code}
Any encountered attribute is added to the attribute list with the type
\const{CDATA} and default \const{\#IMPLIED}.
The example below extracts the elements used in an unknown XML document.
\begin{code}
elements_in_xml_document(File, Elements) :-
load_structure(File, _,
[ dialect(xml),
dtd(DTD)
]),
dtd_property(DTD, elements(Elements)),
free_dtd(DTD).
\end{code}
\subsection{Parsing Primitives}
\label{sec:sgml-primitives}
\begin{description}
\predicate{new_sgml_parser}{2}{-Parser, +Options}
Creates a new parser. A parser can be used one or multiple times for
parsing documents or parts thereof. It may be bound to a DTD or the DTD
may be left implicit, in which case it is created from the document
prologue or parsing is performed without a DTD. Options:
\begin{description}
\termitem{dtd}{?DTD}
If specified with an initialised DTD, this DTD is used for parsing the
document, regardless of the document prologue. If specified using as a
variable, a reference to the created DTD is returned. This DTD may be
created from the document prologue or build implicitely from the
document's content.
\end{description}
\predicate{free_sgml_parser}{1}{+Parser}
Destroy all resources related to the parser. This does not destroy the
DTD if the parser was created using the \term{dtd}{DTD} option.
\predicate{set_sgml_parser}{2}{+Parser, +Option}
Sets attributes to the parser. Currently defined attributes:
\begin{description}
\termitem{file}{File}
Sets the file for reporting errors and warnings. Sets the line to 1.
\termitem{line}{Line}
Sets the current line. Useful if the stream is not at the start of the
(file) object for generating proper line-numbers.
\termitem{linepos}{LinePos}
Sets notion of the current column in the source line.
\termitem{charpos}{Offset}
Sets the current character location. See also the \term{file}{File}
option.
\termitem{position}{Position}
Set source location from a stream position term as obtained using
\term{stream_property}{Stream, position(Position)}.
\termitem{dialect}{Dialect}
Set the markup dialect. Known dialects:
\begin{description}
\termitem{sgml}{}
The default dialect is to process as SGML. This implies markup is
case-insensitive and standard SGML abbreviation is allowed (abreviated
attributes and omitted tags).
\termitem{html}{}
\nodescription
\termitem{html4}{}
This is the same as \const{sgml}, but implies \term{shorttag}{false}
and accepts XML empty element declarations (e.g.,
\verb$<img src="..."/>$).
\termitem{html5}{}
In addition to \const{html}, accept attributes named \verb$data-$
without warning. This value initialises the charset to UTF-8.
\termitem{xhtml}{}
\nodescription
\termitem{xhtml5}{}
These document types are processed as \const{xml}. Dialect
\const{xhtml5} accepts attributes named \verb$data-$ without warning.
\termitem{xml}{}
This dialect is selected automatically if the processing instruction
\verb$<?xml ...>$ is encountered. See \secref{xml} for details.
\termitem{xmlns}{}
Process file as XML file with namespace support. See \secref{xmlns} for
details. See also the \verb$qualify_attributes$ option below.
\end{description}
\termitem{xmlns}{+URI}
Set the default namespace of the outer environment. This option is
provided to process partial XML content with proper namespace
resolution.
\termitem{xmlns}{+NS, +URI}
Specify a namespace for the outer environment. This option is
provided to process partial XML content with proper namespace
resolution.
\termitem{qualify_attributes}{Boolean}
How to handle unqualified attribute (i.e. without an explicit namespace)
in XML namespace (\const{xmlns}) mode. Default and standard compliant is
not to qualify such elements. If \const{true}, such attributes are
qualified with the namespace of the element they appear in. This option
is for backward compatibility as this is the behaviour of older
versions. In addition, the namespace document suggests unqualified
attributes are often interpreted in the namespace of their element.
\termitem{space}{SpaceMode}
Define the initial handling of white-space in PCDATA. This attribute is
described in \secref{xml-whitespace}.
\termitem{number}{NumberMode}
If \const{token} (default), attributes of type number are passed as a
Prolog atom. If \const{integer}, such attributes are translated into
Prolog integers. If the conversion fails (e.g. due to overflow) a
warning is issued and the value is passed as an atom.
\termitem{encoding}{Encoding}
Set the initial encoding. The default initial encoding for XML documents is
UTF-8 and for SGML documents ISO-8859-1. XML documents may change the
encoding using the \exam{encoding=} attribute in the header.
Explicit use of this option is only required to parse non-conforming
documents. Currently accepted values are \const{iso-8859-1} and
\const{utf-8}.
\termitem{doctype}{Element}
Defines the toplevel element expected. If a \verb$<!DOCTYPE$
declaration has been parsed, the default is the defined doctype. The
parser can be instructed to accept the first element encountered as the
toplevel using \exam{doctype(_)}. This feature is especially
useful when parsing part of a document (see the \const{parse} option to
sgml_parse/2.
\end{description}
\predicate{get_sgml_parser}{2}{+Parser, -Option}
Retrieve infomation on the current status of the parser. Notably useful
if the parser is used in the call-back mode. Currently defined options:
\begin{description}
\termitem{file}{-File}
Current file-name. Note that this may be different from the provided
file if an external entity is being loaded.
\termitem{line}{-Line}
Line-offset from where the parser started its processing in the file-object.
\termitem{charpos}{-CharPos}
Offset from where the parser started its processing in the file-object.
See \secref{indexaccess}.
\termitem{charpos}{-Start, -End}
Character offsets of the start and end of the source processed causing the
current call-back. Used in \program{PceEmacs} to for colouring
text in SGML and XML modes.
\termitem{source}{-Stream}
Prolog stream being processed. May be used in the \const{on_begin}, \emph{etc.}
callbacks from sgml_parse/2.
\termitem{dialect}{-Dialect}
Return the current dialect used by the parser (\const{sgml},
\const{html}, \const{html5}, \const{xhtml}, \const{xhtml5}, \const{xml}
or \const{xmlns}).
\termitem{event_class}{-Class}
The \jargon{event class} can be requested in call-back events. It
denotes the cause of the event, providing useful information for syntax
highlighting. Defined values are:
\begin{description}
\termitem{explicit}{}
The code generating this event is explicitely present in the
document.
\termitem{omitted}{}
The current event is caused by the insertion of an omitted tag.
This may be a normal event in SGML mode or an error in XML mode.
\termitem{shorttag}{}
The current event (\const{begin} or \const{end}) is caused by an
element written down using the \jargon{shorttag} notation
(\verb$<tag/value/>$.
\termitem{shortref}{}
The current event is caused by the expansion of a
\jargon{shortref}. This allows for highlighting shortref strings
in the source-text.
\end{description}
\termitem{doctype}{-Element}
Return the defined document-type (= toplevel element). See also
set_sgml_parser/2.
\termitem{dtd}{-DTD}
Return the currently used DTD. See dtd_property/2 for obtaining information
on the DTD such as element and attribute properties.
\termitem{context}{-StackOfElements}
Returns the stack of currently open elements as a list. The head of this
list is the current element. This can be used to determine the context
of, for example, CDATA events in call-back mode. The elements
are passed as atoms. Currently no access to the attributes is provided.
\termitem{allowed}{-Elements}
Determines which elements may be inserted at the current location. This
information is returned as a list of element-names. If character data is
allowed in the current location, \const{\#pcdata} is part of
\arg{Elements}. If no element is open, the \jargon{doctype} is returned.
This option is intended to support syntax-sensitive editors. Such an
editor should load the DTD, find an appropriate starting point and then
feed all data between the starting point and the caret into the parser.
Next it can use this option to determine the elements allowed at this
point. Below is a code fragment illustrating this use given a parser
with loaded DTD, an input stream and a start-location.
\begin{code}
...,
seek(In, Start, bof, _),
set_sgml_parser(Parser, charpos(Start)),
set_sgml_parser(Parser, doctype(_)),
Len is Caret - Start,
sgml_parse(Parser,
[ source(In),
content_length(Len),
parse(input) % do not complete document
]),
get_sgml_parser(Parser, allowed(Allowed)),
...
\end{code}
\end{description}
\predicate{sgml_parse}{2}{+Parser, +Options}
Parse an XML file. The parser can operate in two input and two output
modes. Output is either a structured term as described with
load_structure/2 or call-backs on predefined events. The
first is especially suitable for manipulating not-too-large documents,
while the latter provides a primitive means for handling very large
documents.
Input is a stream. A full description of the option-list is below.
\begin{description}
\termitem{document}{-Term}
A variable that will be unified with a list describing the content of
the document (see load_structure/2).
\termitem{source}{+Stream}
An input stream that is read. This option \emph{must} be given.
\termitem{content_length}{+Characters}
Stop parsing after \arg{Characters}. This option is useful to parse
input embedded in \jargon{envelopes}, such as the HTTP protocol.
\termitem{cdata}{+Representation}
Specify the representation of \jargon{cdata} elements. Supported are
\const{atom} (default), and \const{string}. See load_structure/3 for
details.
\termitem{parse}{+Unit}
Defines how much of the input is parsed. This option is used to parse
only parts of a file.
\begin{description}
\termitem{file}{}
Default. Parse everything upto the end of the input.
\termitem{element}{}
The parser stops after reading the first element. Using
\term{source}{Stream}, this implies reading is stopped as soon
as the element is complete, and another call may be issued on the same
stream to read the next element.
\termitem{content}{}
The value \const{content} is like \const{element} but assumes the
element has already been opened. It may be used in a call-back from
\term{call}{\const{on_begin}, Pred} to parse individual elements after
validating their headers.
\termitem{declaration}{}
This may be used to stop the parser after reading the first
declaration. This is especially useful to parse only the \exam{doctype}
declaration.
\termitem{input}{}
This option is intended to be used in conjunction with the
\term{allowed}{Elements} option of get_sgml_parser/2.
It disables the parser's default to complete the parse-tree by closing
all open elements.
\end{description}
\termitem{max_errors}{+MaxErrors}
Set the maximum number of errors. If this number is exceeded further
writes to the stream will yield an I/O error exception. Printing of
errors is suppressed after reaching this value. The default is 50.
Using \term{max_errors}{-1} makes the parser continue, no matter how
many errors it encounters.
\begin{quote}
\term{error}{limit_exceeded(max_errors, Max), _}