Linking Expat from another package

Who this is for

Read this if your package has C code that is written against Expat itself — it calls XML_ParserCreate(), XML_GetBuffer(), XML_StopParser() — and you want it to build without a system libexpat. Typically the code is vendored and cannot be rewritten: xlsxio, which zuxlsx vendors, is that case.

If instead you are writing new C code that you control, stop here and read vignette("streaming") first. The registered function table described there is the better path for new code: it needs no configure, no linker flags, and picks up a zuxml fix when zuxml is upgraded. This vignette is for the case where that is not an option.

The two modes

The archive exists because the table cannot express a pull-style reader. A reader that returns a row to its caller and resumes where it left off is built on XML_StopParser()/XML_ResumeParser(), and there is no equivalent in zuxml.h.

What an installed zuxml gives you

# system.file() returns "" outside an installed layout, and dir("") would then
# list the working directory rather than nothing.
installed <- function(what) {
  path <- system.file(what, package = "zuxml")
  if (nzchar(path)) dir(path) else "(not an installed layout)"
}

installed("include")
#> [1] "expat_external.h" "expat.h"          "zuxml.h"
arch <- .Platform$r_arch
installed(if (nzchar(arch)) file.path("lib", arch) else "lib")
#> [1] "libzuxml.a"

The archive installs under lib${R_ARCH}, as the shared object does under libs${R_ARCH}. R_ARCH is empty on most platforms, which makes that plain lib.

LinkingTo: zuxml puts that include directory on your compiler’s path (R appends it to CLINK_CPPFLAGS), so #include <expat.h> resolves with no further work.

The headers are not kept under inst/include/ in zuxml’s sources. They are copied out of the vendored tree at install time by src/install.libs.R, so the header you compile against is by construction the one the objects in the archive were compiled from.

There is no LinkingTo for a library

LinkingTo handles headers and nothing else. For the archive you have to find its directory yourself, and the only portable place to do that is a configure script that writes src/Makevars.

The two shortcuts are both worse. $(shell ...) in Makevars forces SystemRequirements: GNU make. An Imports: zuxml entry gets you nothing: Imports is about loading a namespace at run time, which a statically linked consumer never does.

#!/bin/sh
set -eu

: "${R_HOME:?configure must be run by R CMD INSTALL, which sets R_HOME}"
RSCRIPT="${R_HOME}/bin/Rscript"

# lib/<arch> first, then lib: right whether R sets an architecture or not,
# and whichever layout the installed zuxml uses.
lib_dir() {
  "${RSCRIPT}" --vanilla -e "arch <- .Platform\$r_arch; d <- if (nzchar(arch)) system.file('lib', arch, package = '$1') else ''; if (!nzchar(d)) d <- system.file('lib', package = '$1'); cat(d)"
}

ZUXML_LIB=$(lib_dir zuxml)

if [ -z "${ZUXML_LIB}" ] || [ ! -f "${ZUXML_LIB}/libzuxml.a" ]; then
  echo "configure: libzuxml.a was found in neither zuxml/lib/<arch> nor zuxml/lib." >&2
  exit 1
fi

sed -e "s|@ZUXML_LIB@|${ZUXML_LIB}|g" src/Makevars.in > src/Makevars

#!/bin/sh
rm -f src/Makevars src/Makevars.win

PKG_CPPFLAGS = -DXML_STATIC
PKG_LIBS = '@ZUXML_LIB@/libzuxml.a'

LinkingTo: zuxml

Which Expat you are getting

The archive holds the Expat objects zuxml’s own shared object was linked from, and nothing else — no zux_* wrapper, no R glue, since either would be dead weight or a duplicate symbol inside your shared object.

It is compiled with zuxml’s feature policy, which is stricter than a distro build. Your code inherits the compiled-in half of that policy:

• XML_GE 0, and XML_DTD never defined. General entities, parameter entities, external subsets and the external-entity machinery are compiled out, so XXE and entity amplification are impossible rather than disabled. In a document with no DOCTYPE, any entity reference other than the five built-ins and numeric character references is a parse error. With one, it may not be: see the next section.
• Do not define XML_GE=1 on your own command line. expat.h would then declare XML_SetBillionLaughsAttackProtection*(), which the archive does not define, and you would get a link error. Those limiters are pointless here anyway — the attack they bound is already compiled out.
• XML_NS 1. Namespace-aware parsing via XML_ParserCreateNS() works.
• XML_CONTEXT_BYTES 1024, so XML_GetInputContext() is available.
• XML_UNICODE is not defined, so XML_Char is plain char and input is UTF-8. XML_LARGE_SIZE and XML_ATTR_INFO are not defined either.

What you do not inherit

The rest of zuxml’s policy is not compiled into Expat. It lives in zuxml’s own parser, above Expat, which is exactly the code an archive consumer does not call. So XML_ParserCreate() on the archive gives you none of it:

• No DOCTYPE rejection. zuxml refuses a DOCTYPE by default, and an internal subset always. That matters more than it looks. With XML_GE 0, Expat does not record entity declarations, so in a document that declares <!ENTITY e "x"> in an internal subset, a reference &e; does not fail: it arrives in your character-data handler as the four characters &e;. The document is silently wrong. A DOCTYPE that names an external subset makes Expat skip undeclared references instead of failing on them.
• No resource limits. zuxml’s max_depth, max_nodes, max_attrs and max_text are counted in its own handlers. Expat bounds none of them.

If your input can be hostile, install the same guards yourself. Rejecting a DOCTYPE is one handler, which is what zuxml does in src/zux_parser.c:

static void XMLCALL
reject_doctype(void *user, const XML_Char *name, const XML_Char *sysid,
               const XML_Char *pubid, int has_internal_subset) {
  XML_Parser p = (XML_Parser)user;
  (void)name; (void)sysid; (void)pubid; (void)has_internal_subset;
  /* Record why, then stop: XML_Parse() returns XML_STATUS_ERROR. */
  XML_StopParser(p, XML_FALSE);
}

/* after XML_ParserCreate(): */
XML_SetUserData(p, p);   /* or your own struct holding p */
XML_SetStartDoctypeDeclHandler(p, reject_doctype);

To accept a DOCTYPE but refuse an internal subset, as zuxml’s allow_doctype = TRUE does, stop only when has_internal_subset is non-zero. Bound depth by counting in your start- and end-element handlers, and text by counting in your character-data handler, stopping the parser the same way.

Security fixes reach you only when you rebuild. Your copy of Expat is linked into your shared object. When zuxml updates its bundled Expat, for a security fix or otherwise, your users get the fix only once your package is rebuilt and re-released against the new zuxml. R does not rebuild LinkingTo dependents on upgrade.

Check that you actually linked it

On Linux and Windows a wrong PKG_LIBS fails at link time. On macOS it does not. R links a package shared object with -undefined dynamic_lookup, so a consumer that never linked the archive still builds, still loads, and still parses — against whatever Expat happens to be in the process, usually the system one. Every behavioural test passes. The only reliable check is the symbol table:

# Nothing may be left for the loader to resolve.
nm -u path/to/yourpkg.so | grep XML_

# And Expat has to actually be in there.
nm path/to/yourpkg.so | grep ' [TtDd] _*XML_' | wc -l

A worked example

tools/zuxmltest/ in zuxml’s development sources is a complete, minimal package in exactly this shape — configure, configure.win, cleanup, src/Makevars.in, and a src/consume.c that drives XML_GetBuffer/XML_ParseBuffer with a suspend/resume round trip. tools/run-downstream-check installs it against a freshly built zuxml and asserts all of the above, including that it still parses after zuxml has been removed from the library path.

Both live in the git repository rather than in the installed package, since tools/ is not shipped.