In December of last year we spoke with Adam Harmetz, the Lead Program Manager for the SharePoint Document and Records Management engineering team at Microsoft, about some of the new records management features coming out in SharePoint 2010. Response to that post has been overwhelming.
Adam has graciously agreed to sit down with us one more time and answer a few more questions on topics we did not address during our first conversation.
SPRM: Adam, we know you’re very busy preparing for the release of SharePoint 2010, so thanks for taking the time to speak with us again.
Adam: Thanks for setting this up, Don. Indeed, it has been a busy time, but the response to the 2010 product has been amazing, with tons of people downloading the beta and visiting sharepoint.microsoft.com to learn about the product.
SPRM: So speaking of releasing SharePoint 2010, do we have a firm date yet as to when that will happen?
Adam: May 12th is the SharePoint 2010 launch day, and you can read more about it on the SharePoint Team Blog. That same blog just posted some notes on what the analysts are saying about SharePoint 2010 that are worth checking out, too.
SPRM: Great, we’ll check that out.
As you know, Adam, email records management always seems to be on the front of our readers’ mind. And a number of our readers pointed out that we failed to ask about new email records management functionality in our original discussion.
Adam: E-mail is a critical part of business communication in every organization today – so it’s a really important piece of the records management puzzle to tackle. To help with this, there are a couple different options to choose from that your readers should be aware of.
For customers where only a small amount of the total mail constitutes an official record, submitting e-mail records to a SharePoint Records Center makes total sense. You can manage all official records together and e-mail sits alongside other electronic records; they are managed as a unit. Our partners like Colligo offer great drag and drop functionality from Outlook to SharePoint for this scenario. This approach really focuses on using one central and powerful store for achieving consistent records management regardless of medium or format of the record.
I’d also suggest your readers look at the new features that Exchange 2010 provides to solve many of the same problems for e-mail that SharePoint 2010 solves for collaborative documents and list data. Ian also runs a great blog on Exchange Archiving that your readers should definitely check out.
The work that the Exchange team did release is really impressive and aligns with the end user focus that SharePoint takes to the archiving, retention, and discovery process. Their flexible retention policies and personal archive allow you to think about records management without disrupting how the end user or IT pro works.
SPRM: OK, but just to be clear, will these new Exchange 2010 features provide true records management functionality that allows users to classify emails into a file plan based on the email’s content rather than the record’s format?
Adam: Absolutely! Your question reminds me of customer feedback from records managers that we hear all the time about how e-mail archiving solutions aren’t records management tools. Many just don’t have enough fine grained classification to really implement a nuanced RM practice. Don’t just think about Exchange 2010 as “the release Microsoft built in e-mail archiving,” but rather as the release we added a set of tools to start doing records management in situ. For instance, Exchange 2010 and Outlook 2010 offer end-user facing retention policy classifications on a per item or per folder basis, including Quick Step UI in Outlook that makes it one click to file and classify content. A picture can really help crystalize what’s going on here, so let me show you the UI for choosing a policy on a message or default policy for a folder in Outlook 2010:
Of course, there are a certain set of records management practices that require a dedicated records management application like SharePoint (e.g. hierarchical file plans), so it’s really a tradeoff discussion. I think we (as a company and as an industry) have a long way to go before there is true medium-independent records management that allows completely consistent classification no matter what format the content is in, how it’s delivered, or where it lives, but I do think the 2010 wave will be seen as a pivotal step in the right direction.
SPRM: The ability to permanently (and verifiably) destroy a record is a big concern for many of our readers, particularly those folks working in the government and the military. This wasn’t impossible in MOSS 07, but it was difficult.
Does SharePoint 2010 provide any new functionality that would help us ensure that when we destroy a record pursuant to its retention requirements, it is not recoverable?
Adam: If you think about this, this is really a requirement of the storage system underneath a content management repository. Just like backup/restore SLA, WORM storage, high availability – this is a feature where I as a record management application designer want to get out of the way and allow the storage subsystem to do this as cost effectively as possible.
To that end, SharePoint 2010 has been designed to take advantage of the Remote Blob Storage (RBS) functionality of SQL Server 2008 R2. RBS essentially allows choices on where to store the files (sometimes called blobs in database parlance) for SharePoint content. And we’ve baked into this design the notion that an RBS provider can provide expunge capabilities and SharePoint won’t do anything to prevent that (e.g. by storing a copy of the blob itself somewhere).
RBS is a new technology for SharePoint 2010 so we are working with our key partners in the Beta program to get pluggable storage providers to market. You can already download beta providers from Metalogix and AvePoint and there are many other storage and archiving vendors lined up to bring solutions to market at the same time or shortly after the release of SharePoint 2010. You can expect these vendors to offer a variety of different differentiators and I expect that expunge will be one of those key value adds.
SPRM: As you know, records management can no longer be limited to managing the information restricted to an organization’s official records repository. The Information Lifecycle Model must be applied throughout the enterprise in a holistic manner.
In our last conversation, you mentioned SharePoint 2010’s new taxonomy and metadata structure. Will you tell us more about them and how they will help us manage information across our enterprise?
Adam: As a records manager, have you ever had any of the followings problems/desires?
- I need to apply something consistently across a bunch of different sites in my enterprise.
- I want one version of the truth, although I need to be able to customize and localize where appropriate.
- I want my users to classify content, but it’s got to be dead-simple easy for them.
I’m sure the answer is yes; these are fundamental information governance problems. Turns out these are requirements that span well beyond just records management – having consistency across sites and easy tagging are important force multipliers that help people find the content they need and get their work done. It just so happens that records management in SharePoint gets to benefit from these enterprise investments in metadata.
Enough head in the clouds thinking, though – let me show you exactly what I mean. SharePoint 2010 offers a Managed Metadata Services that allow the definition of hierarchal and re-usable taxonomies. You can think of this new type of metadata column as a Choice field that has gone to the gym. Here’s a shot of a user entering managed metadata in the Office 2010 client application:
In this shot, we see a company’s product catalog implemented as a managed metadata column. There are various methods for entering this metadata (e.g. tree picker, type-ahead auto complete), all focused around making it as simple and easy as possible. Once tagged, of course, you can imagine a variety of very cool ways this metadata can be used to help people find the document again: a tag cloud that rolls up all items tagged like this in the enterprise, ability to refine enterprise searches only to particular products, rollups of the number of reviews tagged with a certain product, et cetera….
But metadata like this is just as valuable for records management scenarios as well. As we talked about in the previous post, end users can fill out this metadata and SharePoint can automatically route the document to the correct place in the file plan, for instance. And then there is the inherent value to information architects of having well classified content.
What holds all this together is that these managed metadata columns are defined once, centrally, in what we call an Enterprise Term Store. As you can see, this term store allows you to geek out with a whole bunch of operations to keep taxonomies in tune:
Once defined, these classifications/metadata definitions can be shared with the right sites in the enterprise through a process called Content Type Syndication. This ensures the classifications are available where they need to be (i.e. either everywhere, or on specific sites/types of documents that need particular pieces of metadata).
So that was a whirlwind tour of Managed Metadata in SharePoint 2010. There are a lot more details that will be hitting the ECM blog soon, so I encourage your readers to subscribe to that blog to get up to date.
SPRM: Finally, Adam, the question everybody’s been asking. A lot of our readers tell us they want to use SharePoint to manage their records, but they can’t even consider it without DoD 5015.2 certification. What can you tell us about Microsoft’s plans for DoD certification for SharePoint 2010?
Adam: Sure, we certainly hear that a lot, too – DoD 5015 has become a bit of a de facto standard for the industry. There are some really good reasons for that and some reasons why it’s not a good litmus test for a records management application. The silos within the ECM market (like document management, records management, web content management) are breaking down in a way that the standard just can’t keep up with.
The DoD 5015 standard has some really core records management features that truly help with information governance – and we’ve made great strides to include them out of the box in SharePoint 2010. Our auditing subsystem meets DoD 5015 specs, and DoD 5015 features like document identifier, multi-stage retention policy, holds, folder based permissions, hierarchical file plans, closing file plan elements, folder metadata defaults, and record/container metadata infrastructure are all part of the SharePoint 2010 release because they are valuable to our entire worldwide customer base.
On the other hand, there are features in the DoD 5015 standard that just will never be generically applicable. Does every records management system need to implement the specific XML schema required to submit records to the United States National Archives? What about the human workflow processes for declassifying secret United States Government data? Must cutoff (developed when there were only boxes of paper records) be the only way for processing batches of records in bulk?
I don’t think those are requirements that are going to truly determine whether a records management program is going to be successful. A focus on usability, integration into existing workflows and applications users are familiar with, ability for the application to be used in languages throughout the world, a scalable database/distributed architecture, one IT infrastructure that hosts a variety of ECM workloads and reduces cost, a support pipeline that will be around for as long as you have records in the system – these are all requirements that are so critical to a successful records management implementation and yet the DoD 5015 standard and testing procedures don’t stress them.
That’s not to say standards aren’t important. Indeed, I think we’ll look back five years from now and say that was the timeline that standards really started breaking down silos in this industry and delivered on that vision of inter-operating content management systems. The Content Management Interoperability Services will be a driving force here.
For United States Government customers who are required to run in a certified state, we are working with a range of partners right now to bring a variety of different SharePoint/partner solutions to market. Gimmal Group* is one of those partners and I expect a great product to come out of that that uses the SharePoint 2010 records management features and adds the specific features required for United States Government customers. The SharePoint Records Management engineering team didn’t pursue DoD 5015 compliance ourselves this release, but we really spent a ton of engineering time to make sure our partners will be successful at achieving certification.
Of course, no conversation about DoD 5015 is complete without thinking about the global customer base that is leveraging SharePoint. We constantly get questions from around the world about how to deal with local government and industry standards for information management. Let me throw just a few at you… MOREQ2, VERS, ISO 15489, DOMEA, TNA, ERKS, the list goes on. Some of these standards are loosely based on one another and some have contradictory elements. Rather than focus our engineering efforts on addressing each of these standards in turn, we made the choice to deliver the usability and innovation required to make records management deployments successful and allow our partner ecosystem to build out the SharePoint platform to deal with specific requirements for those customers that are mandated to adhere to a specific standard.
SPRM: Great, Adam, thanks. There seems to have been a lot of confusion out about Microsoft’s intentions with respect to the Standard. Hopefully, this will clear things up.
And thank you, once again, for your time, Adam. Please come back and speak with us sometime after you release SharePoint 2010. In the meantime, can we assume the Microsoft ECM Team blog is the best place to continue to get further information on SharePoint 2010 Records Management (other than this site, of course)?
Adam: Sure is. For instance, check out this post by Quentin Christensen on eDiscovery in SharePoint 2010.
[*In the spirit of full disclosure (and as many of you already know), I work for Gimmal. In order to prevent even the appearance of conflicting interests, I’ve tried to keep this blog and my professional life separate. This, however, is getting increasingly difficult as this particular post attests. Just to be clear, I am not working directly on this project, but I have been providing guidance to the development team from time to time and I’m sure that will continue. I have no plans to discuss the certified solution until after it is released, if at all. – Don]
).
