Migration Concepts and Strategies

migration content from a source to docuflow target with docmigrator is a combination of docmanager business rules (sap business objects and doctypes), rfcs and their connections, and sap background jobs all of these, plus consideration of day to day docuflow processing and how quickly you want to migrate your content determines the constellation of these components the diagram below provides a high level overview typically there is a timeline for a migration, and together with the volume of content to be migrated, a seconds/per document or document/per second target is determined the below sections outline the components, consideration, and best practices involved in order to achieve a successful migration along with overall best practices, each content repository may have an additional notes or best practices section, please ensure you reference if applicable sharing of docmanager rules and rfcs the first determination is whether content to be migrated will be sharing docmanager rules (which are at the sap business object and/or doctype level) with day to day docuflow processing things that will force separate docmanager rules if there is a need for separate folder locations in box if there is a need for separate sap security if there is a need for different meta data in short, rfc destinations (and their number of rfc connections per destination) are how docuflow can be load balanced to allow multiple docuflow migrator sap background jobs to be ran in parallel sharing or separation of rfc processing between migration processing and day to day processing in typical day to day docuflow processing, the docmanager rules determine the rfc destination, however docmigrator can override this to force use of dedicated rfc destinations at runtime (more on this below) parallel vs single stream processing the second determination is whether your volume vs timeline/milestones require multiple migration processes to run in parallel, or whether a single processing stream will suffice, in other words, how fast do you want or need your content to be migrated single stream additional consideration into carving and chunking migration is not required the only consideration is whether to share existing rfcs with day to day processing using a higher number of rfc connections (more on this below) parallel processing (most common and recommended) ideally is executed using separate rfc destinations than day to day processing, and further ideal over separate docuflow middleware connectors the number of dedicated sap background tasks available for migration dictates how many parallel processes can be executed for parallel processing, there are 2 ways to carve and chunk a migration within the docuflow docmigrator, both leverage the input/selection criteria to the docmigrator archivelink link parameters (most common and recommended) all fields in toa01 are available, both for “source” and “target” link entries in the migrator table once loaded/staged typical and easiest levels of carving are sap business object and/or doctype the advantage of using these two elements is it typically prevents from job running in parallel for the same migration load table entry at the exact same time the other fields in a link entry can be used (such as sap record/object id range) but extra caution must be given to not processing a load table entry at the same time over multiple jobs loading with custom abap to assign unique tags (requires development) this method typically has the chunks set by tag name set at time of the load process, and thus migration of the load table is done at the tag level note it is imperative that parallel jobs do not have input criteria that allow it to process the same rule at the same time they must be “unique” jobs to avoid runtime conflict running a migration for a document that is already migrated is ok, docuflow will not migrate or process a document twice rfc destinations and connections regardless of the processing style chosen able, but imperative to consider for parallel processing, is how to load balance the migration process over rfc destinations and rfc connections sharing docmanager rules if sharing the same docmanager rule as day to day processing is not concern, and load is not a concern, docmigrator can be run to leverage the docmanager rule in it’s entirety, including which rfc to use in this scenario, it is recommended to increase the number of rfc connections at the middleware level for each rfc to be shared, thus reducing the chance of migration activity and day to day processing waiting for an available rfc destination connection forcing rfc destination(s) to take full control over which rfc destination is to be used, docmigrator can be run explicitly specifying the rfc destination to use for that background job by using the "force rfc" input parameter (most common and recommended) in this scenario, ideally use 1 rfc destination per background job if background jobs are to share an rfc destination, increase the number of rfc connections to match the number of sap jobs running against that rfc at the same time (assuming no day to day processing is using that same rfc) ideally each background job uses 1 rfc destination with 1 rfc connection it is recommended to have a maximum of 10 rfc destinations per docuflow middleware connectors it is recommended to have a maximum of 10 rfc connections per rfc destination in theory, that is 100 rfc destination/connection combinations registered in your sap gateway, therefore mind your maximum allowed users as logged in clients in smgw additional best practises sap background jobs never have migration table entries shared between background jobs for parallel processing a potential collision could occur to process the same entry at the same time across more than 1 background job unique filenames from source content if your source files may contain non unique filenames, and there is a chance they may be migrated to the same repository location for a cms that does not allow same name files, versioning must be prevented this can be handled in two ways 1) in the docmigrator, choose the "force unique filename" this will append a unique uuid stamp to the source filename 2) in the docuflow middleware, at the rfc destination level enable "error on duplicate" this will result in an erorr for documents attempted to be migrated to the same folder if there is an file with that existing name these erorrs can then be triaged after the fact to either modify the filename in the source system, or re run the docuflow migration for those errors and choose "force unique filename" above rfc’s and docuflow middleware ideally avoid sharing docuflow migration rfc destinations with production day to day docuflow processing rfc destinations ideally avoid sharing docuflow migration with production day to day docuflow processing across the same middleware instance ideally have dedicated 1 or more docuflow middleware instances for migration (unless shared doctypes and docmanager rules for migration and day to day) consider total number of rfc destinations/rfc connections in your sap gateway maximum user count for production day to day docuflow processing and migration processing, servers hosting docuflow middleware should be dedicated to docuflow only often 1 non prod docuflow middleware instance/server is enough, ideally 1 per sap instance (dev/qual/prod/training) but that is optional/nice to have this only affects inbound to sap processing/testing as when you add a document to box, how does the connector know which sap system to go to (hint for the fancy pants out there yes you can add more than one metadata field at the saplink rule level in the mdfilters string) if your source data is also from a docuflow endpoint ensure the cms use tag in the docmigrator configuration is seperate from docmanager business rules and/or is configured to a dedicated rfc as a docuflow read will be performed on that rfc also important to ensure there are multiple rfc connections for a read rfc destination to ensure maximum throughput and performance repository id’s and doctype’s mapping to new repository id’s is almost always required and highly recommended mapping to new document types is optional remember that new document types typically requires updated sap security roles/profiles theory recap archivelink uses s wfar obj authorization object which has repository id, sap business object, and doctype thus if repository id is already part of your security model, you will have to accommodate the new repository id at a minimum recap section “sharing of docmanager rules and rfcs” above to determine if you have to have new doctypes (in addition to reasons you may have as well) deleting links / attachments list during migration of sap business objects/doctypes buckets, if the source content is to remain intact, and if using then docuflow docmanager attachment list only, consider “hiding” those from the attachment list until migration for that combination is complete (docmanager config) after migration buckets of sap business objects/doctypes are completed, and if using then docuflow docmanager attachment list only, consider “hiding” those from the attachment list until final verification (docmanager config) deletion of source links is only recommended after a verification window post migration this is also optional if using the docmanager attachment list, as you can also simply hide the orinal links as well performance notes set logging in docuflow middleware to error only less logging, less writing to disk increase the jvm size for docuflow middleware server to 2048 initial/4096 max have docuflow profile wizard closed during processing including metadata as part of the migration requires an additional api call as part of the “create”, but allows for the migration to process in one enchilada however, if migration milestones are short and if metadata isn’t required day 1, or if api max calls are a concern, consider transferring metadata as a second wave using the docuflow mdpro in sap to execute a mass push of metadata after the fact use box folder numbers instead of names, this reduces the number of api calls and increases performance significantly