IMS Global TECH TALK
Contributed by Dr. Colin Smythe, IMS Chief Architect
Compliance, Conformance, and Certification: Why Getting IMS Certified is Important
One of the benefits of IMS membership is having your certified products listed in the IMS Product Directory—the official list of all learning apps and tools that have passed IMS interoperability certification. A product must demonstrate support of one or more IMS specifications through conformance testing to appear in the directory. IMS awards each certification for 12 months, so every product must undergo successful recertification to maintain its listing. This 12-month cycle allows vendors to use agile development processes without requiring recertification for every product release. New major versions of a product must be certified. It is not unusual for several versions of a product to show up in the product directory. It is important to note that the product receives certification and not a deployment of the product.
The IMS Product Directory also includes products vetted for student data privacy using the IMS TrustEd Apps process. Many, but not all vetted products, have also achieved IMS standards’ certification. This blog focuses on the products that go through conformance testing for IMS certification.
Defining the conformance requirements and providing the associated conformance test capabilities are essential to the IMS specification development process. Each IMS specification must have a Conformance & Certification document. These documents describe the certification process and define the conformance criteria that a product must achieve for each available certification. Most specifications have more than one certification. An example of this is a service-based specification with certifications as a Service Provider and a Service Consumer (a product must be either or both). The conformance and certification aspects are addressed once the project group responsible for developing the specification publishes the Member Candidate Final documents. These documents are available to IMS members only. A minimum number of products must be certified before the Final Release of an IMS specification can be published. This means the document set is publicly available to everyone.
An IMS specification cannot have a Final Release until IMS members define conformance and certification and create and use the conformance test.
One objective of the IMS certification is to demonstrate the level of adoption by vendors committed to open solutions to the market. When a new version 1.0 specification is first published, the conformance requirements are defined to encourage broad adoption. Over time the level of adoption will change, and the specification itself will evolve. Therefore, the IMS specification maintenance process allows for the certification requirements to be changed to fit the changing needs of a market—even when there are no changes to the functionality supported by the specification. Also, it is not unusual for the conformance testing to be continually improved. This flexible approach to certification is another reason why products must undergo annual recertification.
It is becoming more common for organizations to require IMS specifications as part of the procurement process. It is natural for vendors to claim compliance. From an IMS perspective, compliance is claimed by vendors who are not IMS Certified. IMS members can provide their IMS product registration numbers, and users can easily confirm their certification by a quick inspection of the IMS Product Directory. If support of an IMS specification is required, the tendering process should include checking the product’s IMS Registration Number. In most cases, the claim for compliance is wishful thinking and based on unjustified confidence in the in-house interoperability testing. Sometimes, it is a cynical misrepresentation. Buyer beware.
When products claim compliance but are not certified, there are two implications. First, the product has not been through IMS conformance testing. Usage of the IMS conformance test systems is essential in producing a correct implementation of the specification. In most cases, a solution goes through several iterations of conformance testing before being certified. There is no restriction on the usage of our conformance test system for IMS members, meaning they are not just for certification. Secondly, if there is a failure of interoperability when using the IMS specifications, the IMS certification requires the vendors to work together, and if appropriate with IMS, to resolve the problem. In some cases, IMS may have to: improve the implementation guidance, correct the specification, and improve the conformance test systems to avoid such incompatibilities in the future. Experience has shown that products that are not certified do not implement the corresponding IMS specification correctly. Superficially, they appear to work, but there will likely be many significant errors in the implementation.
Certification requires IMS membership. Is access to certification sufficient justification for IMS membership? Undoubtedly, YES!
IMS has invested millions of dollars in developing and supporting our extensive test and conformance systems and related artifacts. Five to ten full-time software developers are working on the various IMS test and conformance systems at any one time. Even the largest organizations see significant benefits in using the IMS test and conformance systems. A further benefit is that the IMS technical team provides a wide range of support to help IMS members adopt and adapt IMS specifications. As part of our specification development process, IMS creates the following testing and conformance artifacts:
-
Service Provider and Consumer conformance test systems (used for OneRoster, LTI, CASE, etc.)
-
Reference implementations of the full specification
-
Online validation of content instances (used for Common Cartridge, QTI, etc.)
-
Reference test sets for testing data import capabilities (used for Common Cartridge, OneRoster, QTI, etc.)
It is important to stress that all of these artifacts are available, for unlimited use, to IMS members. All of these artifacts are being continually improved.
While Certification is very important, it is product-focused and not deployment-specific. There is a limit to the degree of interoperability guaranteed through certification only. It is possible for two products certified for the same specification not to interoperate. For example, in OneRoster, there are both REST-based and CSV-based bindings, and interoperability between these is not possible. Therefore being certified alone is insufficient; the right type of certification is required for interoperability. The IMS Product Directory provides sufficient details to ensure the right type of certification is available. In the case of deployed systems, there are many different ways in which a certified system can be configured (this may also depend on the business model used by the vendor).
As the next step beyond certification, IMS has created the Compatibility Check (CCx), which provides the Characterization of a deployment. Furthermore, CCx enables characterizations to be compared. This means that we can compare the characterizations of certified Service Providers and Consumers to show all of the interoperable and non-interoperable features (including the usage of extensions). At present, CCx supports OneRoster and Common Cartridge, but it will be extended to cover many of the IMS specifications over time. I will go into more details about characterization and CCx in a later blog but understand that the characterization of products is the way forward. It is a far better measure of interoperability than certification alone.
