The Silvered Screen

Locksmith's TKR Implementation Mk II

2012-02-13T15:35:00.003Z

Locksmith's first pass on implementing the TKR did not exactly hit the mark and, after a round or two of discussion with those kind enough to try out and comment on the service, the API has been revised to better support the TKR and its goals.

Before we get into the changes, it is worth reiterating two salient points of the TKR:

A CPL will contain the base URL for retrieving KDMs for that CPL
The base URL will be combined with the screen server's thumbprint to gather all KDMs for that server

See section 3.1 of the TKR specification for further info.

With those in mind, let's look at why Locksmith's first pass at implementing the TKR needed some revision.

The Perils of URL Construction

Here's how the TKR URL looked in the first pass:


https://locksmith.artsalliancemedia.com/api/kdm/thumbprint/<server thumbprint>/cpl/<cpl uuid>

See the problem? As the base URL should be embedded in the CPL and the thumbprint appended, we managed to stick the thumbprint right in the middle of the URL – simply appending the thumbprint wasn't an option. Systems would have to rip the URL apart and embed the thumbprint right in the middle.

This is bad; Locksmith's new URL implementation is much simpler and more elegant:


https://locksmith.artsalliancemedia.com/api/tkr/<cpl uuid>/<server thumbprint>

The base URL is now neatly encapsulated and the server thumbprint (in hex) just needs to be appended to it.

Tar-balls and TKRs

Locksmith TKR Mk I required you to end the URL with /kdmbundle.tar in order to get your KDMs in the SMPTE 430-2 bundled format – again not really in the spirit of TKR. So now, by default, the new TKR URL above will return a KDM bundle, or a 404 if no KDMs are matched to the CPL uuid and thumbprint. The bundle is now also nicely named with CPL UUID.

One URL to Rule Them All

TKR support boils down to getting a single base URL right:


https://locksmith.artsalliancemedia.com/api/tkr/<cpl uuid>/<server thumbprint>

As Locksmith can determine both the CPL UUID and server thumbprint from the KDM, all KDMs uploaded into Locksmith will be automatically available for retrieval through the TKR method.

KDM Delivery with Locksmith

2012-01-25T16:28:00.001Z

Delivering KDMs

KDM delivery for digital cinema is, like much of this fledgling technology, still in its infancy and is probably one of the most immature areas of digital cinema management. Getting KDMs to site is currently haphazard and fraught with error; typically KDMs are emailed to site – or sent on consumable media – and there is little-to-no feedback as to whether the KDM is actually valid for a server's certificate on its intended screen and has been successfully ingested, or that playback has successfully occurred for its targeted CPL.

Recently Mike Radford put forward a proposal at ISDCF for a method of distributing KDMs that could easily be integrated into on-site systems such as a TMS or directly into the SMS. The methodology, called Theater Key Retrieval (TKR), would allow KDM distributors to set up their own TKR servers and from which keys could be drawn down based on the certificate thumbprints of the on-site screen servers and the UUID of the encrypted CPL. The TKR servers would provide a RESTful API and the URL for gathering keys would be generated from an embedded link in the encrypted CPL's XML (in the language attribute of the Issuer tag), to which the certificate thumbprint of the screen server would be appended. That URL would link to a KDM bundle (ISO 26430-9:2009) containing all valid keys for the combination of CPL and thumbprint.

It's an elegant and simple approach to KDM delivery: the target server thumbprints and CPL id can be easily determined from the KDM XML and on-site systems can interrogate screen server details to gather the correct thumbprints.

Locksmith

Locksmith is an implementation of the ISDCF proposal: KDMs can be uploaded into the cloud-hosted service and then requested based on CPL ids and certificate thumbprints. Locksmith provides a RESTful JSON API that implements an extended set of functionality required by TKR.

Locksmith also extends TKR's functionality in several ways; for example it provides methods to pull all valid and uncollected KDMs for a certificate thumbprint, mark KDMs as delivered, failed or successfully played back, and third parties can upload their own KDMs into Locksmith for delivery.

Locksmith can additionally understand FLM-x feeds and gather site and certificate information that can be accessed through the API and used to match sites to certificate thumbprints (more on that in a future blog post).

An early access version of Locksmith is available now. To try it out, you can request a username and password for the system by emailing us.

Locksmith's API

Locksmith provides a RESTful API that returns JSON-formatted data (though both KDM XML and KDM tar bundles can be directly requested). Currently Locksmith requires a username and password to access and is served exclusively over HTTPS. We'll outline a few of the API calls available in the current build of Locksmith here and expand on these in a later post.

The easiest way to play with the API is through a web browser, ideally with a plugin that pretty-prints the JSON output. I typically use Chrome with the JSONView extension; Firefox with the JSONovich add-on also works well. cURL can be easily used from the command line and the API documentation below provides examples using this.

There's a Python snippet here that shows you how to access the system programmatically.

Any problems, issues, comments or rants about the service and API, we'd love to get feedback.

Retrieving KDMs TKR Style

KDMs can be retrieved by specifying both their target CPL UUID and the server certificate thumbprint:


curl -u <user>:<password> https://locksmith.artsalliancemedia.com/api/kdm/thumbprint/98a48a64c18f8e7f5df4c4036a188c1e5a8f59e4/cpl/65ce635c-b937-4035-8d6d-f1d5fddaea66

The thumbprint (as per the TKR) should be hex-encoded; if you want to use the base64 encoded string, the API supports that by appending the URL with ?encoding=base64, but be sure to ensure that the string is correctly escaped to keep it browser-friendly:


curl -u <user>:<password> https://locksmith.artsalliancemedia.com/api/kdm/thumbprint/mKSKZMGPjn9d9MQDahiMHlqPWeQ%3D/cpl/65ce635c-b937-4035-8d6d-f1d5fddaea66?encoding=base64

Unexpired, pending KDMs are returned in a JSON-formatted list which contains the KDM XML along with meta-data both parsed out of the XML for easy access and status information regarding each KDM. A KDM bundle can also be requested by terminating the URL with /kdmbundle.tar:


curl -u <user>:<password> https://locksmith.artsalliancemedia.com/api/kdm/thumbprint/98a48a64c18f8e7f5df4c4036a188c1e5a8f59e4/cpl/65ce635c-b937-4035-8d6d-f1d5fddaea66/kdmbundle.tar

Retrieving KDMs by Thumbprint

Locksmith can also serve up all valid and pending KDMs for a single device, irrespective of CPL. It provides a simple means to download all pending KDMs for a server in a single call:


curl -u <user>:<password> https://locksmith.artsalliancemedia.com/api/kdm/thumbprint/98a48a64c18f8e7f5df4c4036a188c1e5a8f59e4

Marking KDM Status

Locksmith tracks the status of KDMs and only serves up those KDMs which are pending delivery. A KDM is automatically marked as pending when it is uploaded to the system. Once a KDM is retrieved from Locksmith, the retriever can mark the KDM as delivered, failed or played. This is designed to both hide KDMs that are no longer pending and to provide feedback to the KDM distributors as to whether the KDMs have been delivered/ingested and any problems discovered.

To facilitate batch reporting of KDM statuses, the URL can be given either a single KDM UUID, or a JSON-formatted list of UUIDs:


https://locksmith.artsalliancemedia.com/api/kdm/status/<status>/<UUID or JSON list of UUIDs>

status can be one of pending, delivered, failed or played.

Uploading KDMs

KDMs can be individually uploaded to Locksmith by POSTing XML to:


https://locksmith.artsalliancemedia.com/api/kdm/save

An example Python snippet for uploading KDMs into Locksmith is available here.

No other information is needed; Locksmith can parse the KDM XML to determine the target CPL and server thumbprint.

We'll add the ability to batch upload KDMs in a future Locksmith build.

More To Come

The API calls above will give you the basic tools to upload, distribute and mark KDMs through Locksmith. The API has a number of other reporting and FLM-related calls that extend Locksmith's functionality further, but we'll cover that in a later post.

Why you need a TMS ...

2011-11-11T13:53:00.001Z

Scaling Digital Cinema Software

2011-10-22T13:01:00.000+01:00

Ready or not, digital cinema systems are fast becoming the norm; incumbent 35mm systems are being ripped out daily and digital screen playback servers and projectors installed to replace them. These new systems promise an unprecedented level of flexibility in theatre management, such as large libraries of local content, easy creation and modification of pre-show playlists and automated scheduling.

The current crop of screen servers and integrated media boards offer APIs that can be remotely accessed and theatre management systems (TMS) exist to provide control of all screens within a single cinema, AAM’s TMS included.

The application of these APIs go further: the potential to centrally manage screens across a cinema circuit. For example, AAM’s VPF software gathers logs and CPL meta-data from over 2,000 screens on a daily basis to a central location and Director provides centralised control of content, pre-show and scheduling for an exhibitor’s complete circuit.

Scaling Problems

The promise of automated cinema brings its own challenges, the chief amongst these being scale.

The scale problem is two-fold: how to scale a centralised system that communicates remotely with a thousand-plus screens while dealing with network vagaries, servers that are busily playing content, powered-down equipment and a host of other issues; and when there are multiple systems competing for the attention of a screen server, how do you ensure that they don’t impede or interrupt playback?

For central control of a large circuit, scaling communication to each and every play-out server – not to mention the projection, point of sale and library systems – is a difficult task; server availability, time-outs, network issues all play a part in making the cycle of invoking a call to every server – say to collect logs or CPL data – a lengthy and error-prone task.

Further compounding the problem, each model of screen server is a distinct beast; their APIs have very different implementations, communication protocols and response characteristics in terms of speed and reliability; each behaves differently when the API is under heavy load. While this may be manageable for an external system in isolation, when multiple systems require access to the screen server it is impossible to police.

Theatre Core

The answer we’ve come up with to both problems of scale is to create a small service, called Theatre Core, that resides at each cinema theatre and proxies all interaction with digital cinema equipment.

It acts as a gateway to digital equipment within a theatre, most critically the screen servers, and wraps the different APIs provided by those systems.

Theatre Core offers a number of advantages:

Homogeneous API Access

Theatre Core provides a unified API for communication with different classes of digital cinema equipment. It has a set of standardised APIs for screen servers, projectors, library servers and POS systems, with more planned in the future. These hide the details of each manufacturer’s API. Any application that uses Theatre Core implements a single API and then gets access to all supported equipment.

Managed Access

The API access load placed on a screen server can be precisely controlled by Theatre Core and the throughput throttled as necessary. Theatre Core knows the capabilities of the equipment with which it integrates and so can ensure that playback and other critical functionality is not impacted by too many external systems demanding attention.

But what does Theatre Core do when it needs to respond to a request for data, but doesn’t want to overload a system’s API?

System State Caching

Theatre Core maintains an internal state representation of screen servers, projectors and other equipment and manages the frequency of updating this state. Whenever a request for data about a server is received – for example CPL data, the validity of its KDMs, its playback status – Theatre Core can respond immediately using its internal state model.

This significantly reduces the load on a screen server’s API and buffers external systems from potentially slow responses a play-out server may provide. Theatre Core is designed to provide low latency responses to requests, even under significant load. Some screen servers have response times in the range of seconds for certain calls; Theatre Core hides these failings by caching.

Caching state does have its trade-offs. Theatre Core may not be fully synchronised when a screen server’s state changes and small lag-time may occur between the server changing and Theatre Core updating.

That said, Theatre Core typically updates state every few seconds (the precise timings are dependent on the equipment and a lot of testing on our part to see what works best); in practice discrepancies are rarely seen.

That trade-off gives blisteringly fast responses for log retrieval, KDMS, CPLs, playback state, etc. and lightens the load considerably on digital equipment.

Theatre and not Screen Level Scaling

Theatre Core not only abstracts away the specifics of communication with equipment, it also provides aggregation functionality to collect data across multiple systems within a theatre in a single API call. For example you can get a listing of all CPLs in a theatre, gathered from every screen and library server; retrieve the playback state of all screen servers; list what equipment is contactable and gather the secure logs from all servers.

This removes the necessity for central systems to have to request information from each and every piece of equipment at a complex; for many tasks, just a single call is made and the aggregated data returned. It provides far better scaling for systems: instead of making hundreds or thousands of calls to gather circuit-wide data, a call per complex can be made. Factoring in Theatre Core’s caching capabilities, the speed with which circuit-wide operations can occur is massively increased.

AAM Software Ecosystem

We use Theatre Core as the basis for all communication to digital cinema equipment, both for systems local to the cinema (TMS and LMS) and centralised systems (Director and VPF). All of these systems access on-site equipment through Theatre Core and, in practice, have been deployed in production without scaling issues to over 2,000 screens and rising.

Digital cinema promises a level of automation that is impossible with legacy 35mm systems; however to achieve these promises requires some careful thought and planning for cinema circuits of any great size.

Theatre Core is designed to make building these systems – in a world of non-standard APIs, performance and capability – straightforward, removing the complexity of multi-equipment support while at the same time providing an architecture that will scale into the many thousands of screens.