Complex mainframe z/OS applications historically work with mixed data in databases and files of various formats linked through field relationships.

It is often necessary to copy current data from a production environment to test and to make it usable there. Also, with privacy concerns the data may need to be masked, scrambled or anonymized. Sometimes this can be done by simply copying a table or file. Usually though, ETL data extraction or creating test cases is a time-consuming business.

Creating consistent test cases for a heterogeneous environment?

There are indeed many reporting and extraction tools but they don�t really solve the problem. Either additional programming is needed to link queries, for instance a VSAM key with a DB2 column, or the usual infrastructure of a mainframe site with its separate environments for test, development and production is not adequately taken into account.

The job of providing suitable test cases for maintenance and enhancement work is therefore always an involved and expensive business.

- top of page -

D-SECT Solves The Problem

With D-SECT you can write inquiries in SQL format for:

• Relational databases (DB2)
  
• IMS, DL/1 databases
  
• VSAM datasets
  
• Sequential files
  
• Send output to mainframe or platforms running Microsoft SQL Server, Oracle, DB2, et.al.
  

The inquiries can be linked through any desired field relationship. Thus you could use the content of a field in a VSAM file as an argument for a WHERE condition which could be part of a SELECT clause for IMS segments.

In order to further refine processing you can use REXX language clauses besides the SQL statements. That way you can change selected fields or complete data structures.

The amount of data selected can be limited by way of a simple command, for instance, �select 1000 test cases� or �select every 100th case�. You can also specify that the test case should only be selected when the chain is complete, i.e. in linked files the case must be represented at least once in all files.

Output options allow for overwriting, partial replacement or additions to test data. The output may be made to the same computer or sent to another and can be flexibly controlled by parameters.

The selected subset retains its original format if desired. IMS remains IMS, VSAM remains VSAM. However, D-SECT is also able to convert, thus the selected data can be transferred to an IDE on a PC or to another platform. All necessary conversions are carried out automatically within D-SECT.

A D-SECT script contains the SQL statements required for the selection, the REXX statements for further modification and the D-SECT commands to control output. It is known by name within D-SECT and can be incorporated into other scripts.

D-SECT supports the optional division of processing levels such as development, test and production on computers or LPARs in a sysplex.

- top of page -

D-SECT's Object/Layer Concept

Usually development is already working on a new version of an application while the old version is still in production. This means that the production data no longer has the same layout as the test data required by development.

D-SECT Object/Layer concept looks after automatic adaptation of data structures. The various levels such as Development-1, TEST, Development-2, QA, PROD, are described as Layers as in the above diagram �Definition of Layers�. A D-SECT object contains among other things the description of the data layouts, the record formats, the table or segment descriptions.

An object exists once in a layer and can differ for each layer. If the data of an object is transferred from one processing level (layer) to another and the object descriptions differ, then D-SECT automatically takes care of the data conversion to the new format.

D-SECT as an extraction tool (ETL tool)

D-SECT is by no means limited to the creation and maintenance of test data. Using the same techniques you can select, convert and transport data for any purpose from mainframe applications.

For instance a data warehouse or any other application on another platform or a web application could be regularly supplied with data from z/OS or OS/390 files. The D-SECT runtime process consists of a stream of batch jobs which may be put under the control of a job scheduler.

- top of page -

Data propagation and Mainframe Change Data Capture with HotCopy

D-SECT extracts data; the sister product/component HotCopy propagates data. What�s the difference? D-SECT takes data from a source environment and moves it to a target environment. That is a batch process; the selection process takes data as-is from the source and transports it to the target environment. HotCopy�s Capture module recognizes changes in data �on the fly� and passes them continuously on to the target environment. A target file is thus updated with the source data near to real time.

For example: an IMS application has been in use for years. Now the data from the IMS application is to be regularly passed on to a new web application or to a data warehouse. This can be handled by HotCopy. In order to propagate data with HotCopy you do not need to make any changes on the host side, in this case in the IMS application. The target could be DB2 or any relational database on any desired platform. HotCopy transfers the data asynchronously to the target so that the source application is not in any way impeded or suffering loss of performance.

The following source applications are supported: CICS/VSAM, CICS/DL1, CICS/DB2, IMS and batch usage of VSAM, DL/1 and DB/2. The target platforms can be z/OS, OS/390, Unix, Windows. Targe databases can be DB2, Microsoft SQL Server, Oracle, etc.

D-SECT and HotCopy

While D-SECT has its own drivers and extracts data according to statements in a script, HotCopy is driven by the source application. When insert, delete or update modifications occur in the source application the HotCopy Capture mechanisms recognise these and initiate propagation.

To sum it up

Using D-SECT you can formulate inquiries in SQL to cause selection of appropriate portions of data in different organisational forms (IMS, VSAM, DB2) which result in a consistent subset.

The selected subset retains its original format. This means that for instance, IMS segments remain as such. They will be inserted into a database with the same GEN-parms as the original DB. Selected VSAM records appear in a newly defined VSAM cluster which will also have the same structure as the original. The same is true for DBs tables and rows.

- top of page -

Selection mechanisms

To limit the amount selected various mechanisms are available to enable a logical selection based on field contents or based on numeric amounts.

For the purpose of selection IMS/DL1 segments and VSAM datasets are treated as tables, that is, selection criteria are specified using SQL. Both key fields and non-key fields may be used in select statements as a basis for selection. AND/OR conditions are allowed as are relational operators such as GREATER, LESS THAN, FROM � TO, CONTAINS. Selections may also be based on lists of keys.

Limiting by amounts

• You may limit to every nth selected hit.
  
• You may specify a limit to the number of selected test cases. Only those records are counted which have hits in all source files based on the selection criteria � i.e complete chains.
  

Load options

It is possible to use the selected records to update a previously created file. When the reference chain leads to a DL1 segment having multiple selections the redundant segments can be suppressed. (Optional) This is also true for subsequent updates to selected files. If selection criteria result in a hit on a DL1 segment, it is optionally possible to automatically select all hierarchical parent or child segments and all other segments attached to the same root.

Transform, Anonymize, and Scrub features include:

• Transfer of the selected data into another form of organisation: EBCDIC/ASCII, IMS/SAM/VSAM, DB2/XYZ, etc.
  
• Transformation: add, change initialise fields to conform to differing layouts for various versions of an application.
  
• Modify data: rule-based changes to the content of fields, for instance to anonymise or mask data.
  

- top of page -