A jewel in the crown of structural biology, the Protein Data Bank is an unparalleled archive of protein and nucleic acid structures. Its operation depends on information-rich relational databases, the schemas for which are efficiently expressed in the macromolecular CIF (mmCIF) formalism. Discrete entities that may be stored in the databases are characterised in CIF-format data dictionaries, which express ontologies widely used in areas as diverse as structural genomics, NMR, cryo-electron microscopy and protein production.
 Okay. So I'll be giving you a brief description of how MMCIF has been used in structural bioinformatics and structural biology. As Brian said, I'm representing the RCSB PDB and the WWPDB, which is the international organization that is now sort of looking over taking responsibility for deposition and distribution of macromolecular structure data. By way of overview, I'll try to extend a little bit the history of how MMCIF was developed and how it's been implemented, give you an idea of how the PDB is archiving structure experimental and reference data, and how MMCIF is in some ways come to the rescue when addressing some of the current challenges that we face in data archiving, and then give you a brief description of some recent developments on the data deposition and delivery front. Samir Belankar will be talking about this in more detail when he discusses our new deposition and annotation and validation system later in the meeting. So this is another view of the timeline, kind of specific to MMCIF, but it gives a picture of the Brian's sort of milestone of the original development of the small molecule CIF dictionary. There was a convening of a working group by the IUCR, who's originally chaired by Paula Fitzgerald, to develop an extension to small molecule CIF dictionary for macromolecular structure. That process took a little more than a decade. It involved a great deal of community participation. And if one thing that I would like to emphasize is that this is the CIF effort overall has been very much a community grassroots effort and has benefited greatly from the participation of people within the crystallographic community, but other communities as well. These workshops that have contributed to the development of CIF have involved participation from not only people in the crystallographic domain, but also in computer science and informatics in general. And a lot of very valuable advice was obtained and has impacted the ultimate evolution of MMCIF and I think the whole CIF technology framework in general. So from the perspective of the PDB and PDB delivery, Brian already indicated that somewhere in the late 1990s, the management of the underlying infrastructure for managing macromolecular data at the PDB adopted MMCIF as a vehicle for its data management. And that was based primarily not on format considerations, but on the dictionary. And most of what I'll be saying today is very dictionary-centric. The dictionaries have a very rich semantic and computer accessible content that we've tried to exploit at the PDB in our data management. Because we've extended the original CIF content to address methods in macromolecular structural biology other than crystallography, we've used the acronym PDB exchange dictionary, but in general this technology framework is essentially what was developed for MMCIF. Now as Brian indicated, the WWPDB is using this technology and most recently the WWPDB has redeveloped its entire deposition and annotation data processing pipeline with the dictionary technology at its underpinning. So the dictionary framework that's currently in use now is used to represent all of the data that's collected and distributed by the PDB for describing experiment and structural models. It includes descriptions of X-ray, neutron, NMR, electron microscopy, and SAS methods. We're actively working on developing extensions to describe various hybrids of these methods in addition to sort of experimental and methodological experiment descriptions. There's inclusion of information describing the preceding steps of protein production, a great deal of the biological description of the sample, and additional structural and functional details associated with the resulting structural model. The overall dictionary content that we, the documents, all of this can be browsed at our dictionary resource site, mmsif.pdb.org. So that's scientific content. From the point of view of data management, it's the metadata content of the dictionary which is centrally important. And that includes semantic information, very detailed definitions and examples. It includes software accessible information that's used to standardize and validate the information such as data types, allowed boundaries, boundary values and control vocabularies, organizational or schema elements. The model which is embodied in mmsif can probably most easily be thought of as a sort of a tabular data model. It maps very well to describing tables of data. Thus we deal with structures which have rather complex identifiers starting from models to chains to residues to residue names, residue number, atoms and atom names. We deal with rather detailed natural keys or natural identifiers, keeping track of that and keeping track of perhaps multiple versions of those that may occur at different points in the data processing process is a referential integrity problem and the data dictionary provides parent-child relationships that allow us to manage that kind of complexity in software by documenting all of the related identifiers. In addition to that, Brian discussed the implementation of methods which are embraced in the dictionary framework as well as the ability to define relationships of interdependencies between data items such as defining the association between a target value and its associated precision. As I said, the dictionary is kind of central to our data management strategy and it's used as a tool to document and create schemas of different types for different applications, either database or format descriptions. It's also used underlying all of our tools for validation for accepting data through our deposition systems or developing data harvesting tools, pulling data out of other applications. It's also central in our ability to produce data in different concrete file formats on the PDB archive and that includes the traditional PDB format file, SIF files, files in XML and in RDF. Talk about those a little bit. PDB format was sort of the original format that was used by the archive dating back to the early 70s. It's a record-oriented format, the SIF format began to be used in the archive around 1997. These dates represent the data which all the data in the archive, all the structure data was distributed in this format. The XML version which we call PDBML became available in 2005 and the RDF implementation in 2011. Then the way these files are produced is basically using the MM SIF file as a master format, generating from that the other formats. PDB format, which is by far the most popular format, is a record-oriented format which includes textural remarks which are marked up kind of like this. The documentation for this format is basically by example in a textural format description document. It's very widely used and very widely supported in software. Brian showed you some examples of SIF format. That can either be key value with sort of well-defined names which are tied to the dictionary or tabular where the column names are basically precede a regular layout of rows and columns. We believe it's a simple syntax. Most people are familiar with it, agree. All of the semantics are defined in a data dictionary and currently there's pretty good support for this in almost all popular programming languages. The XML version adopts the same naming conventions as SIF with the introduction of a namespace. Key identifiers are represented as attributes in the XML style. Other items within categories are represented as elements. The XML file brings a variety of overhead with markup. XML PDB files are typically 10 to 12 times larger than any of the other representations. It's particularly a problem for Atom records which are the records which have the tables which have the longest length. In addition to providing fully marked up files, the PDB provides files where the Atoms are represented as, the Atom records are represented in a stringified form very much like they are in the record-oriented file and a regular expression is provided to help people break that up into individual data elements. The RDF example follows the same kind of protocol as the PDB-ML in terms of naming conventions. These names follow the names in the data dictionary. This is just a different schema to accommodate the RDF model. This also allows well-defined URLs to be associated with particular data items within the PDB. Again, this has been used to provide a unification of PDB data primarily with other biological data resources that have similarly implemented this approach such as Uniprode for instance. In addition to the experimental and structure data, PDB provides chemical reference data in SIF format. All of the individual chemical components, distinguishable molecules, either monomers or ligands in the PDB are represented in a chemical dictionary. Currently about 18,000 unique molecules in the PDB. It's the same approach that's used by CCP4 in developing their monomer library which is shared by most of the major refinement packages in macromolecular crystallography. The refinement libraries include a richer information as required for their packages, the essential chemical descriptions are contained in the PDB dictionary. That includes nomenclature, stereochemistry, aromatic annotation, an example set of model coordinates taken from the PDB archive and a set of ideal coordinates. In addition to the atom list, there's a comparable list of information describing the details of bonding. We don't embed this information currently in individual data files but provided as a reference file, people can access to describe any of the monomers in the file. We've recently released information in SIF format again describing the small polymer molecules in the PDB in an effort to provide both a full chemical representation as well as a sequence level representation of these molecules. This is accompanied with a significant amount of functional annotation associated with the roles of these molecules and the different structures in which they appear. That's the state of where we are. One of the challenges that we face is that structural biology is moving very quickly and dealing with larger and more complex molecules. The most popular format in the archive has not been able to really keep pace with the size and the complexity of molecules that we need to express. While we depend on a format in MMSIF, which is extremely flexible in that respect, we have used it in a way that we've only adopted nomenclature conventions in MMSIF that were supportable within the PDB framework. That includes restricting the scope of identifiers to the character limitations in the record-oriented format. That limits us to the representation of, say, 62 chains or a maximum of 100,000 atoms. The result of this is that large molecules, for instance, this ribosomal structure, is represented in 10 different PDB entries. It's not possible to represent this in a single PDB format entry. It's cumbersome both for the depositor. It's cumbersome for us in managing the data, and it's very cumbersome for the end user that has to try to assemble that final model. Size isn't the only issue. The textural remarks that are present in the PDB format file are not easy to document and come to us in a relatively non-standard way, so extracting that information and standardizing it over time has been difficult to achieve. The SIF dictionary, on the other hand, provides us with a mechanism to keep that information, keep every element of that information very, very well-defined. What we want to achieve in the deposition and annotation pipeline at the PDB is the ability to capture the information in a form from the data generation point, from the laboratory in such a way that when it goes into the PDB, it can come out of the PDB in a form that can be reused with fidelity to largely reproduce the experiment that's represented in the deposited entry and capture all of the associated annotation along with that. Now that's, we don't expect to be able to capture things at perhaps the ultimate level of detail that one might like, but we can certainly do a better job than we do with the format tools that we have at present. So about three years ago, we began circulating some format proposals to the community that were suggestions or starting points for how we might go about addressing this problem of both the metadata description, doing a better job with the metadata description and addressing size that led to a workshop in 2011 of the software refinement package developers, including CCP4, Phoenix, and Global Phasing. The outcome of that meeting was the recommendation that these groups in the community adopt SIF as a working format and as in particular a deposition format rather than using the conventional PDB format. And in May of this year, the working group released recommendations as well as implementations to address that. So the, this is a picture smiling group of developers after having made this recommendation in Hinkston and what's really, really key about this is that this group has, this group has delivered within all of the major packages now an option that allows them to use an MSIF as a deposition. To address the large structures, some minimal extensions have been proposed in the size of identifiers for chain IDs, for coordinate format, precision occupancies, and B factors. It's been recommended that these be used only as needed to represent large structures shortly after those recommendations became available. We had some particularly large depositions which would have required as many as 25 PDB entries. These structures have more than two and a half million ATOM records in each entry. These are now distributed on the PDB by the PDB in individual SIF files rather than only as a collection of divided files. So we're very mindful of trying to provide backward compatibility. So SIF provides for this and allows for us to deliver data which while having all the features that we want in terms of well-defined semantics taking advantage of our data dictionary can be presented in essentially the same flat column format as the original PDB. This helps to preserve people's ability to access this information in a line-based manner. We've also tried to, we will be adopting a style which will push method-specific details off to the right in the file so that people who are coming to this and working with different experimental methods either from 3DEM or NMR will be able to capture the sort of the non-method specific data in say the first 11 columns and anything method-specific in the latter columns. So all of that technology has gone into developing our new deposition and annotation system which is end-to-end built on the dictionary framework. As I said, Samira will talk about that a little bit later. There's a wide range of software tools including the major refinement packages that are now providing tools to deal with MM-SIF format for both visualization, parsing and checking. The work that I've done has basically been part of a global effort including our CSB and the US BMRB for the NMR component in Madison, PDBE in Hinkston and PDBJ in Osaka. This is part of the WWPDB group that our last get-together and thank you very much. Have a good day. Thank you.
 Okay.  So I'll be giving you a brief description of how MMCIF has been used in structural bioinformatics  and structural biology.  As Brian said, I'm representing the RCSB PDB and the WWPDB, which is the international  organization that is now sort of looking over taking responsibility for deposition and distribution  of macromolecular structure data.  By way of overview, I'll try to extend a little bit the history of how MMCIF was developed  and how it's been implemented, give you an idea of how the PDB is archiving structure  experimental and reference data, and how MMCIF is in some ways come to the rescue when addressing  some of the current challenges that we face in data archiving, and then give you a brief  description of some recent developments on the data deposition and delivery front.  Samir Belankar will be talking about this in more detail when he discusses our new deposition  and annotation and validation system later in the meeting.  So this is another view of the timeline, kind of specific to MMCIF, but it gives a picture  of the Brian's sort of milestone of the original development of the small molecule CIF dictionary.  There was a convening of a working group by the IUCR, who's originally chaired by Paula  Fitzgerald, to develop an extension to small molecule CIF dictionary for macromolecular  structure.  That process took a little more than a decade.  It involved a great deal of community participation.  And if one thing that I would like to emphasize is that this is the CIF effort overall has  been very much a community grassroots effort and has benefited greatly from the participation  of people within the crystallographic community, but other communities as well.  These workshops that have contributed to the development of CIF have involved participation  from not only people in the crystallographic domain, but also in computer science and informatics  in general.  And a lot of very valuable advice was obtained and has impacted the ultimate evolution of  MMCIF and I think the whole CIF technology framework in general.  So from the perspective of the PDB and PDB delivery, Brian already indicated that somewhere  in the late 1990s, the management of the underlying infrastructure for managing macromolecular  data at the PDB adopted MMCIF as a vehicle for its data management.  And that was based primarily not on format considerations, but on the dictionary.  And most of what I'll be saying today is very dictionary-centric.  The dictionaries have a very rich semantic and computer accessible content that we've  tried to exploit at the PDB in our data management.  Because we've extended the original CIF content to address methods in macromolecular structural  biology other than crystallography, we've used the acronym PDB exchange dictionary,  but in general this technology framework is essentially what was developed for MMCIF.  Now as Brian indicated, the WWPDB is using this technology and most recently the WWPDB  has redeveloped its entire deposition and annotation data processing pipeline with the  dictionary technology at its underpinning.  So the dictionary framework that's currently in use now is used to represent all of the  data that's collected and distributed by the PDB for describing experiment and structural  models.  It includes descriptions of X-ray, neutron, NMR, electron microscopy, and SAS methods.  We're actively working on developing extensions to describe various hybrids of these methods  in addition to sort of experimental and methodological experiment descriptions.  There's inclusion of information describing the preceding steps of protein production,  a great deal of the biological description of the sample, and additional structural and  functional details associated with the resulting structural model.  The overall dictionary content that we, the documents, all of this can be browsed at our  dictionary resource site, mmsif.pdb.org.  So that's scientific content.  From the point of view of data management, it's the metadata content of the dictionary  which is centrally important.  And that includes semantic information, very detailed definitions and examples.  It includes software accessible information that's used to standardize and validate the  information such as data types, allowed boundaries, boundary values and control vocabularies,  organizational or schema elements.  The model which is embodied in mmsif can probably most easily be thought of as a sort of a tabular  data model.  It maps very well to describing tables of data.  Thus we deal with structures which have rather complex identifiers starting from models to  chains to residues to residue names, residue number, atoms and atom names.  We deal with rather detailed natural keys or natural identifiers, keeping track of that  and keeping track of perhaps multiple versions of those that may occur at different points  in the data processing process is a referential integrity problem and the data dictionary  provides parent-child relationships that allow us to manage that kind of complexity in software  by documenting all of the related identifiers.  In addition to that, Brian discussed the implementation of methods which are embraced  in the dictionary framework as well as the ability to define relationships of interdependencies  between data items such as defining the association between a target value and its associated  precision.  As I said, the dictionary is kind of central to our data management strategy and it's used  as a tool to document and create schemas of different types for different applications,  either database or format descriptions.  It's also used underlying all of our tools for validation for accepting data through  our deposition systems or developing data harvesting tools, pulling data out of other  applications.  It's also central in our ability to produce data in different concrete file formats on  the PDB archive and that includes the traditional PDB format file, SIF files, files in XML  and in RDF.  Talk about those a little bit.  PDB format was sort of the original format that was used by the archive dating back  to the early 70s.  It's a record-oriented format, the SIF format began to be used in the archive around 1997.  These dates represent the data which all the data in the archive, all the structure data  was distributed in this format.  The XML version which we call PDBML became available in 2005 and the RDF implementation  in 2011.  Then the way these files are produced is basically using the MM SIF file as a master format,  generating from that the other formats.  PDB format, which is by far the most popular format, is a record-oriented format which includes  textural remarks which are marked up kind of like this.  The documentation for this format is basically by example in a textural format description  document.  It's very widely used and very widely supported in software.  Brian showed you some examples of SIF format.  That can either be key value with sort of well-defined names which are tied to the dictionary  or tabular where the column names are basically precede a regular layout of rows and columns.  We believe it's a simple syntax.  Most people are familiar with it, agree.  All of the semantics are defined in a data dictionary and currently there's pretty good  support for this in almost all popular programming languages.  The XML version adopts the same naming conventions as SIF with the introduction of a namespace.  Key identifiers are represented as attributes in the XML style.  Other items within categories are represented as elements.  The XML file brings a variety of overhead with markup.  XML PDB files are typically 10 to 12 times larger than any of the other representations.  It's particularly a problem for Atom records which are the records which have the tables  which have the longest length.  In addition to providing fully marked up files, the PDB provides files where the Atoms are  represented as, the Atom records are represented in a stringified form very much like they  are in the record-oriented file and a regular expression is provided to help people break  that up into individual data elements.  The RDF example follows the same kind of protocol as the PDB-ML in terms of naming conventions.  These names follow the names in the data dictionary.  This is just a different schema to accommodate the RDF model.  This also allows well-defined URLs to be associated with particular data items within the PDB.  Again, this has been used to provide a unification of PDB data primarily with other biological  data resources that have similarly implemented this approach such as Uniprode for instance.  In addition to the experimental and structure data, PDB provides chemical reference data  in SIF format.  All of the individual chemical components, distinguishable molecules, either monomers  or ligands in the PDB are represented in a chemical dictionary.  Currently about 18,000 unique molecules in the PDB.  It's the same approach that's used by CCP4 in developing their monomer library which  is shared by most of the major refinement packages in macromolecular crystallography.  The refinement libraries include a richer information as required for their packages,  the essential chemical descriptions are contained in the PDB dictionary.  That includes nomenclature, stereochemistry, aromatic annotation, an example set of model  coordinates taken from the PDB archive and a set of ideal coordinates.  In addition to the atom list, there's a comparable list of information describing the details  of bonding.  We don't embed this information currently in individual data files but provided as a  reference file, people can access to describe any of the monomers in the file.  We've recently released information in SIF format again describing the small polymer  molecules in the PDB in an effort to provide both a full chemical representation as well  as a sequence level representation of these molecules.  This is accompanied with a significant amount of functional annotation associated with the  roles of these molecules and the different structures in which they appear.  That's the state of where we are.  One of the challenges that we face is that structural biology is moving very quickly  and dealing with larger and more complex molecules.  The most popular format in the archive has not been able to really keep pace with the  size and the complexity of molecules that we need to express.  While we depend on a format in MMSIF, which is extremely flexible in that respect, we  have used it in a way that we've only adopted nomenclature conventions in MMSIF that were  supportable within the PDB framework.  That includes restricting the scope of identifiers to the character limitations in the record-oriented  format.  That limits us to the representation of, say, 62 chains or a maximum of 100,000 atoms.  The result of this is that large molecules, for instance, this ribosomal structure, is  represented in 10 different PDB entries.  It's not possible to represent this in a single PDB format entry.  It's cumbersome both for the depositor.  It's cumbersome for us in managing the data, and it's very cumbersome for the end user  that has to try to assemble that final model.  Size isn't the only issue.  The textural remarks that are present in the PDB format file are not easy to document and  come to us in a relatively non-standard way, so extracting that information and standardizing  it over time has been difficult to achieve.  The SIF dictionary, on the other hand, provides us with a mechanism to keep that information,  keep every element of that information very, very well-defined.  What we want to achieve in the deposition and annotation pipeline at the PDB is the  ability to capture the information in a form from the data generation point, from the laboratory  in such a way that when it goes into the PDB, it can come out of the PDB in a form that  can be reused with fidelity to largely reproduce the experiment that's represented in the deposited  entry and capture all of the associated annotation along with that.  Now that's, we don't expect to be able to capture things at perhaps the ultimate level  of detail that one might like, but we can certainly do a better job than we do with  the format tools that we have at present.  So about three years ago, we began circulating some format proposals to the community that  were suggestions or starting points for how we might go about addressing this problem of  both the metadata description, doing a better job with the metadata description and addressing  size that led to a workshop in 2011 of the software refinement package developers, including  CCP4, Phoenix, and Global Phasing.  The outcome of that meeting was the recommendation that these groups in the community adopt SIF  as a working format and as in particular a deposition format rather than using the conventional  PDB format.  And in May of this year, the working group released recommendations as well as implementations  to address that.  So the, this is a picture smiling group of developers after having made this recommendation  in Hinkston and what's really, really key about this is that this group has, this group  has delivered within all of the major packages now an option that allows them to use an MSIF  as a deposition.  To address the large structures, some minimal extensions have been proposed in the size  of identifiers for chain IDs, for coordinate format, precision occupancies, and B factors.  It's been recommended that these be used only as needed to represent large structures shortly  after those recommendations became available.  We had some particularly large depositions which would have required as many as 25 PDB  entries.  These structures have more than two and a half million ATOM records in each entry.  These are now distributed on the PDB by the PDB in individual SIF files rather than only  as a collection of divided files.  So we're very mindful of trying to provide backward compatibility.  So SIF provides for this and allows for us to deliver data which while having all the  features that we want in terms of well-defined semantics taking advantage of our data dictionary  can be presented in essentially the same flat column format as the original PDB.  This helps to preserve people's ability to access this information in a line-based manner.  We've also tried to, we will be adopting a style which will push method-specific details  off to the right in the file so that people who are coming to this and working with different  experimental methods either from 3DEM or NMR will be able to capture the sort of the non-method  specific data in say the first 11 columns and anything method-specific in the latter  columns.  So all of that technology has gone into developing our new deposition and annotation system which  is end-to-end built on the dictionary framework.  As I said, Samira will talk about that a little bit later.  There's a wide range of software tools including the major refinement packages that are now  providing tools to deal with MM-SIF format for both visualization, parsing and checking.  The work that I've done has basically been part of a global effort including our CSB  and the US BMRB for the NMR component in Madison, PDBE in Hinkston and PDBJ in Osaka.  This is part of the WWPDB group that our last get-together and thank you very much.  Have a good day.  Thank you.
mmCIFandstructuralbioinformatics_72_455.png mmCIFandstructuralbioinformatics_3252_12887.png mmCIFandstructuralbioinformatics_12888_12935.png mmCIFandstructuralbioinformatics_19248_20615.png mmCIFandstructuralbioinformatics_34572_36749.png