Building a Large Chart Ecosystem with AnyChart and Native XML Databases

September 22nd, 2011 by Timothy Loginov

Today we are glad to present you a detailed case study: Building a Large Chart Ecosystem with AnyChart and Native XML Databases by Dan McCreary.

Dan McCreary is an enterprise metadata strategy architect who has worked in the technology field for over 17 years. His career has taken him through the fields of solid state physics, computer chip design, supercomputing, working with Steve Jobs and NeXT Computer, and founding his own consulting firm — Integrity Solutions, which is now part of the WAM!NET Professional Services division. In the fall of 2000, Dan McCreary in collaboration with several other technology and marketing professionals formed Dan McCreary and Associates — a technology strategy development consulting firm specializing in helping firms align their business and technology strategies by effectively managing their metadata.

Executive Summary

The AnyChart system, when used within a native XML database, is ideal for building and maintaining a very large number of charts that can be quickly managed by non-programmers. Central to the AnyChart product is the declarative nature of XML files and the focus on “what” should be displayed, but not “how” the charts are rendered. This allows the target platform to change from Flash to HTML5 (currently in Beta) without any changes by the chart authoring system. AnyChart’s domain specific language for chart specification can also be easily extended for business specific tasks. This approach has saved our customers tens of thousands of dollars building interactive dashboards as well as much higher quality and consistency of the final products.

This document shows how functions such as validation, search, quality, consistency, extensibility, and agility can be easily maintained using standardized, cross-platform XQuery libraries. This is just one example of how NoSQL (not only SQL) systems are ideal for managing semi-structured data.

Background

There are many charting software packages that are designed to create a simple bar charts or line charts on the web. But most of these systems break down once you need to create multiple charts for multiple users in multiple business units and yet still retain consistency and quality across hundreds or even thousands of charts. Our experience is that AnyChart, when combined with native XML databases, can allow large complex charting systems to be easily maintained by non-programmers.

By using AnyChart system and native XML databases, we have created a large complex ecosystem where even very large numbers of charts can be quickly and easily managed. The secret is to combine the declarative nature of AnyChart XML files with a native XML database that can easily query all the charts in use based on a variety of search parameters.

Unlike many charting software systems that provide Java, .Net, JSON or JavaScript interfaces, the core chart format in AnyChart is an XML file. Unlike imperative or procedural code, XML can be queried with XQuery just like a relational database can be queried with SQL. XML files have the benefit of being standardized as well as having a very large set of tools to test compliance to business rules. For example, unlike JSON files, XML files can be validated against XPath expression tests, XML Schemas as well as be check for business rule compliance using tools such as Schematron and XQuery reports.

This case study will show that when combined with a native XML database, AnyChart systems can be very easy to manage by non-programmers.

We now discuss several aspects of managing a large number of charting systems:

  1. Validation
  2. Search
  3. Quality
  4. Consistency
  5. Extensibility
  6. Reusability
  7. Agility
  8. Versioning and Differences
  9. Auditing

Our focus in each of these topics is to demonstrate how each feature saves organizations time and funding and makes the chart development process more flexible and agile.

Validation

There are two steps we use to check the format of our charts. We use XML editing tools such as the oXygen XML editor to enter chart data and the native XML databases to store our data.

Most XML database systems come with a single line function to validate an XML file against a set of rules specified in an XML Schema file. By creating internal standards for charts based around an XML Schema file, we can quickly validate the structure of all elements within a chart.

We have also used XQuery scripts to extract just subsections of all charts of specific types for validation. For example, we can extract all axes for time-series charts for consistency across many different chart authors and many lines of business.

Chart Search

Most native XML databases have integrated search functions. Some also make it easy to integrate fulltext keyword search for any text within an XML document. These features allow us to build highly specific search functions that have high quality search features that can be easily customized by non-programmers. Because XML description tags can be embedded directly into AnyChart files, this means that even full-text keyword searches can be used with AnyChart.

Central to the measurements of search quality is the improvement of both search precision (getting the documents you want) and relevancy (avoiding hits on the documents you are not interested in finding). By using customized search forms that allow us to do keyword search as well as adding element-level specification, we have found that our users can find the right sample chart usually in the first 10 search results. This means that they do not quickly give up and start creating a new chart from scratch that has already been created. This feature is central to our high-reuse strategy and lower cost per new chart development.

AnyChart provides a large amount of high-quality on-line documentation and example charts on its own web site that can be searched with search engines like Google. However, this documentation is not integrated into our local XML content management system, and the search tools frequently do not find the appropriate documents. By integrating documentation on requirements, testing, quality, and policy into a single search engine, all the related documentation can be found using a single search function.

Chart Quality

There are many aspects to chart quality that should be reviewed by a human factors and user experience team. In the past many of these checks were manual and could only be done after charts were being used in a production setting. For example, many charts are used to find specific patterns in large data sets. Studies show that extra “chart ink” prevents this pattern recognition process. Disabling chart borders and grids should be automatically checked by the system. These checks can now be done automatically by the system as the chart is being developed.

Chart Consistency

Large organizations may have a large number of business rules to help users see consistent charts in consistent ways. For example, an organization may decide that all time-series charts should display dates across the horizontal axis using a 90 degree rotation for the date text.

To perform this check, a simple XQuery can be quickly created using a template XQuery application. The user only needs to have a sample consistency check template and have an XPath expression to extract the correct data.

Extensibility

Adding additional user-defined value columns is very complex in many relational database systems. But with AnyChart XML format we can add custom XML tags directly inside of AnyChart XML files without disrupting chart behavior. So, for example, we can add additional chart metadata that describes who requested the chart, who approved the design, when the chart went into production etc. directly in the Chart XML format. This allows us to store all of our chart data in a single document in our version control systems.

Reusability

As we created advanced chart libraries, we found that many charts share common elements. These static elements can be placed in small XML files and then “assembled” using an XQuery based include process. In many cases the XML fragments also needed to be parameterized with elements such as custom titles and labels. XQuery functions allow the user to pass these parameters in using a simple inheritance process, much like object oriented systems. This allows each chart to be “contextualized” based on how it is being used by a project and supports high-variability charting systems as well as reuse of existing charts.

Agility

Many articles have been written about the need for business to quickly respond to changing business requirements. We have found that the combination of AnyChart and XQuery allows our team to quickly add new functionality or change functionally without re-building our core framework of tools. Our approach also makes it easier for new staff to quickly become productive without having to learn the details of how these systems work. Our business analyst’s focus much more on what the users want and leave the details of how the charts are rendered to the AnyChart application.

Many XML systems also integrate XForms as a key component of updating XML data stores. Using XForms and native XML systems allows us to quickly build forms for gathering new chart requests and updating chart development status. The combination of XForms, Rest interfaces, and XQuery back-end databases is called XRX. This simple clean architecture avoids many of the complexities of using objects and relational databases and is considered one of the key factors that allow a higher participation of non-programmers. We are still early in our use of XForms, but we feel that in the future many chart “specifications” will be generated directly from end-users using XForms applications.

Versioning and Differences

Because XML files can be “versioned” directly within XML databases, we can also easily store the AnyChart specification as well as the chart metadata directly in a single file and then compare the versions of these files using visual “difference” tools that are part of our XML development environment. We can also run detailed reports with XQuery that show the time span between a chart request and when this chart went into production to measure our responsiveness.

Auditing

Many industries have strict reporting requirements about what events occurred in the software development lifecycle. For example, an auditor might want to know who required a specific feature, when the feature was first requested, when it was approved, who approved it, and what charts used these features. In the past these audits would take many weeks of research to complete and were very expensive. Our current system automates all these reports using XQuery reports on AnyChart files as well as joins to our software requirements management system that also is stored in the same native XML database. These features have already saved our customers many hours and thousands of dollars in audit expenses.

Summary

Using AnyChart with native XML database and XQuery saves organizations thousands of dollars and allows them to be more agile in development of new charting applications. Central to this is the process of enabling non-programmers to play a larger role in the management of chart creation.

Please contact Dan McCreary if you have any questions: http://www.danmccreary.com/

No Comments Yet

CAPTCHA

*

Categories
Most Recent