The questions below are loosly based on those raised by particpants in the introduction to XML workshop presented at the Association of Moving Image Archivists 2009 conference in St. Louis, MO on 3 November.

In general, tangible examples are crucial to the teaching and understanding of PBCore. At present, the PBCore examples are hap-hazard and follow little logical progression. An improvement in this area would be beneficial to the adoption of PBCore. In addition, tools should be created to support new PBCore-based applications which would make distiguishing between well-formed XML, valid PBCore, and PBCore that conforms to a community of practice easier.

- Where are the XML attributes?
After an introduction to XML, which taught the partipants about the basic building blocks of XML (elements, entities, and attributes), the lack of attributes in PBCore was confusing. Rather than:

<title type="Program">Jimmy Carter</title>

PBCore requires:

<pbcoreTitle>
<title>Jimmy Carter</title>
<titleType>Program</titleType>
</pbcoreTitle>

As a developer, the additional mechanics to parse each type, each authority, or each role are annoying copy+paste jobs, but it is clear that even those new to XML develop the same expectations. With some of the recent developments from DCMI to make Dublin Core more relevant to the changing metadata landscape, it seems like PBCore has failed to evolve.

The reason, as best I can determine, is the PBCore 1.x schema was developed based on existing XML exports from a relational database where that convention is born out of the need for a semantically agnostic schema rather than proper schema creation.

- What is PBCore’s relation to Dublin Core?
PBCore is introduced as being a derivative or extension of Dublin Core, but for some shared element names, there is no obvious relationship. This should either be clarified in future development or dropped.

- What is the difference between the formatPhysical, formatMediaType, and formatGenerations?
These three instantiation-level metadata elements all describe similar problems slightly differently

* formatPhysical (or formatDigital, perhaps) describes the carrier format, which may be independent of the content on the carrier
* formatMediaType describes the content present on the carrier
* formatGenerations describes the type of content on the carrier

The PBCore value lists could be clarified to remove some of the current (seemingly) redundant information

- Why are formatPhysical and formatDigital formatted different? Or, why wouldn’t one use multiple instantiations to express the different formats for which an item is available?

The value list for formatDigital is based on the IANA MIME type registry, while the formatPhysical list is the aggregate of the source elements, which is reflected in the inconsistency of formatting. Could the formatPhysical list become more cohesive and resemble MIME types?

The relation between current instantiations is, at best, unclear and not systematic. The biggest flaw in the current approach is that it is difficult to express the provenance of an instantiation and it’s relation to the intellectual work. The current situation also breaks the 1:1 correspondance between an instantiation and a carrier/file/etc. Some major restructuring, possibly breaking backwards-compatibility is necessary to correct these issues. In the meantime, I would recommend creating a new instantiation for each instance and using the pbcoreAnnotation field to supply basic provenance information.

- The PBCore outline graph is confusing.
As is, the outline graph mixes XML elements with conceptual groupings which makes it confusing to someone new to XML or to PBCore. The graphic could be easily revised to use shaded groups to communicate the content classes, rather than tree nodes.

- The PBCore metadata dictionary picklists provide no definitions or best practices
The metadata dictionary, which may be the most important part of PBCore 1.x, is marginalized on the website. The picklists are offered only as lists and fail to provide appropriate definitions for titleType, descriptionType, etc. Without this guidance, each implementor is forced to make determinations without respect to a community of practice. Taking descriptionType as an example, guidance is needed to describe when to use the format-specific types (program, series, etc) vs the generic type labels (abstract, summary).

- The PBCore website conflates schema rules with best practices
The PBCore website recommends best practices and guidelines for usage closely integrated with the schema requirements. This placement is confusing; while the best practices are very important and are essential resources for understanding, it adds difficulty to the understanding of PBCore.

- A schema-validating XML editor complains when the XML document lacks recommended or optional fields
In particular, oXygen indicates to the user that fields like pbcoreGenre are REQUIRED for conformance to PBCore, while the website leads one to believe this is not the case. In fact, this should not be the case because genre is very specific to broadcasting/traffic needs and will likely be missing in general usage.

This leads me to believe that PBCore should examine the approach the TEI community took with regard to modulization. Proper modulization would provide implementors with a relevant set of metadata elements necessary for use, and perhaps make it easier to integrate PBCore with other metadata schemas (for example, a rights schema or technical metadata standard), leaving PBCore responsible for description and rules for aggregation.

- How do you exchange records? Or, how can I put multiple description documents in the same file?

A PBCoreDescriptionDocument, according to the PBCore schema, should have only one document per file, which is common XML practice, but unknown to those new to XML. Participants were attracted to aggregations as a way to deliver contextually complete documents containing metadata records for relations, etc. Other standards have explored aggregations independent of standards (say, Atom or OAI-PMH), which is probably a more-sound approach.

- Extensions are hard, confusing.
Yep.

Like

Unlike

This is a post describing shortcomings and potential improvements for PBCore, an XML markup for media material interchange. These suggestions try to work within the current confines of PBCore, rather than introducing radical changes (which could bring PBCore more in line with the rest of the XML and linked data worlds). Further, we recognize the strength of PBCore is in descriptive metadata, and these suggestions are primarily to strengthen those components, rather than trying to compete on technical metadata.

1. Define what all the data dictionary elements mean — “clip”, “element”, “actuality”, “version of”, etc. These need to be defined in order for the community to better apply consistently. Other communities have come up with these already – we just need to determine which ones apply to which elements.  See for example, the European Broadcasting Union does a nice job of distributing machine-readable XML definitions for their data dictionary.
2. Enhance semantics of relation types by creating an ontology (using rdfs or similar, like the Fedora RELS-EXT ontology) – eg. instead of simply “version of” allow “derivation of”, “copy of” “identical to” etc.
3. PBCore only has contextual date on individual instantiations, but we want an overall date with types for created/issued/etc (e.g. the date an interview was conducted). A similar issue exists for locations. Both of these are different from pbcoreCoverage — coverage is about the content, rather than the context.
4. Format of the content — whether it is an interview, a panel discussion, a live event, b-roll, beauty shots, etc. formatGenerations provides a piece of this puzzle, but this is ultimately descriptive metadata, which probably don’t belong in an instantiation. EBUCore provides for part of this with a controlled vocabulary for editorial formats, but it’s not granular enough (e.g. Discussion/Interview/Debate/Talkshow). Our suggestion is to explore enhancing the genre data dictionary to include archival descriptors like “interview” “b-roll”, which would solve this in a backwards-compatible way.
5. Machine parseable rights language; we’re embedding the Open Digital Rights Language (ODRL) as a member of pbcoreRightsSummary, but it would be nice to have a common way to express rights (both rights the publisher has, and rights granted by the publisher to the user). An alternate (and perhaps desirable and necessary) solution would be to at least investigate better ways to combine PBCore with established schemas like ODRL, MODS, etc.
6.  A way to identify the primary title and description of an asset, for use in a discovery interface. Existing solutions, like picking titles based on hierarchy, or using a separate metadata document, are flawed.
7. A formal way to order, prioritize, and relate instantiations within a record (e.g. programs within a series, provenance/hierarchy of digital instances).
8.  A way to label the type for a pbcoreSubject is (e.g. person, organization, place, date, etc), in addition to the existing authority reference.
9. Authority references should be available in most (if not all) PBCore containers, which could help enable linked data applications. This could be accomplished through new xml attributes, which would be ignored by legacy applications, and perhaps better in line with other standards.
10. Better handling of “element” level materials, for archival raw footage and similar. Finished programs are handled decently in the existing PBCore, but the data dictionaries aren’t prepared for this level.
11. Adopt proper RDF relationships for PBCore relations.
12. Consider adding educational levels and standards. PBCore currently addresses this tangentially with audienceLevel and audienceRating.
13. Better way to handle metadata about people, whether by enhancing the existing structure, supporting an hCard microformat, or otherwise.
14. Semantics to deal with thumbnails for discovery interfaces, or how to attach visual representations/facsimiles of a PBCore media instantiation. This is probably a low priority, nice to have change.
15. Content flags, which include advisory messages about sensitive content, are regularly created for broadcast programs, but PBCore doesn’t provide a way to capture these. Perhaps the best way here is to add time-based metadata to the descriptive material (but, then, what do you base the timecode against? See next.)
16. BONUS: Add timecode information to instantiations and relationships to identify sections of content, in order to support time-based metadata, content flags, etc.

Like

Unlike

Adapted from an email I just wrote, but I think there is some good resources here, so I thought I’d share more widely.

I’ve toyed around with the BagIt standard, and have a demonstrator for a very homogenous use-case (using Ruby, Ruote, and ruby-bagit) but it doesn’t factor into our DAM -> Fedora workflow yet. From my limited implementation, it would certainly be nice to see DAMS beginning to adopt it , if a few issues can be addressed, either by the standard or by convention.

The biggest issue with the BagIt standard at this point is that it is exclusively a framework for transferring a collection of files, but doesn’t yet provide a way to create complex/compound objects out of the contents. The Library of Congress has been using BagIt for their Chronicaling America newspaper project ( tech notes) , but the reconstruction of objects and relationships has been implicit (based on a file naming convention) or manually done. This probably works in the simplest cases, where each BagIt item can be mapped into a compound object with either limited or embedded metadata, but I’m not sure if this could be easily applied to the problem of creating and relating multiple (heterogeneous) complex objects. Ben O’Steen at Oxford has proposed an extension to add an RDF manifest to the BagIt package to provide this sort of relationships , but I haven’t pursued that further. There has also been some recent development around combining BagIt and OAI-ORE, which might be a better way of approaching the problem using existing standards.

A further wrinkle, at our end, is that our Fedora repository is holding compressed access copies of the content, which cannot be stored in the DAM (because the DAM content model fails to account for proxy objects or similar). I imagine this is going to be a problem with almost all large datastreams, and something infrastructure will have to adapt to.

Like

Unlike

This week, I had the opportunity to write a data import handler (DIH) for the Solr search server, which elegantly mapped a mySQL database to the Solr schema. Before this, I had been writing small scripts with an XML output, because the scope of the underlying data wasn’t neatly contained in a single document or database. This is a new feature in Solr 1.3, and it really seems to make integrating search almost trivial, to the point where anyone who can write an SQL query can begin replacing the in-built fulltext engines with a Solr service, offering more flexibility, efficient faceting, and a document-centric view appropriate for search.

The basic skeleton looked something like this:

<dataConfig>
        <dataSource driver="com.mysql.jdbc.Driver" batchSize="-1" url="jdbc:mysql://localhost:3306/cms?zeroDateTimeBehavior=convertToNull" user="root" />
<document name="doc">
        <entity transformer="RegexTransformer" name="page" query="SELECT ... FROM ... JOIN ... JOIN ... JOIN ..">
<field column="title" name="dc.title" />
[...]
<field column="names" splitBy="," name="dc.contributor" />
        </entity>
    </document>
</dataConfig>

A couple things to note:
In the dataSource configuration, I’ve set the batchSize=”-1″, which lowers the number of rows kept in memory and prevents solr (and the servlet engine) from running out of memory

Second, in the jdbc configuration, I’m using zeroDateTimeBehavior=convertToNull, which is a very easy way of dealing with those pesky “0000-00-00 00:00:00″ dates that normally come out of the database, and allows solr to gracefully skip that field.

In some multivalued field declarations (like the names -> dc.contributor), I’m using the regex transformer, and its helper splitBy, to reverse a mySQL GROUP_CONCAT() field, which at least saves a query (and forces more of the data marshaling logic into the SQL query, leaving the Solr mapping fairly straightforward).

The Solr transformers look incredibly powerful and almost certainly worth pursuing further in the future. One update I eagerly await is the integration of the DIH with Solr Cell, a text+metadata extraction service, under [#SOLR-1358], which would let you merge previously extracted (or entered) metadata with the fulltext of documents. When this feature is added, I think I can pretty much give up on my transforming scripts and switch to the DIH for all purposes.

Like

Unlike