Syteview™ - Data Replication Toolset
   
   
Syteview DataGrabbers are small programs that you use to automate the replication of data from your Progress database to your Microsoft SQL Server.  Each DataGrabber handles the replication of one table from your Progress database to your MS-SQL Server database.

DataGrabbers always move an entire table from the Progress database server to the Syteview database in the SQL Server.   This replication process can take anywhere from several seconds to an hour or more depending on the number of records in the table that is being replicated and the processing power of the computers running the Progress database server and the Microsoft SQL Server.

Note: For times when you need to keep a table in your Progress database synchronized with a table in the Syteview database in real-time Syteview provides you with DataLink programs.

In order for a DataGrabber to retrieve data from your Progress database the database server must be running.   The Progress database server must also have been installed with the SQL-92 Server Engine.  When a DataGrabber runs the first thing it does is drop (delete) the previous copy of the table in the SQL Server database.  This assures you that the new copy of the table is an exact copy of the current table in your Progress database.  In addition to creating an exact copy of the table, the DataGrabbers will create indexes and grant permissions on the table created.

You define the indexes that each DataGrabber will create on its particular table with a text file named Indexes.txt.   This file must exist in the same sub-directory as the DataGrabber.  The format of the Indexes.txt files is quite simple.  See the Indexes.txt file for one of the pre-existing DataGrabbers in the \DataGrabbers sub-directory in the Syteview installation directory on your SQL Server for details.  Include an empty file named Indexes.txt in the Data Grabber’s sub-directory if you don’t want any indexes to be created for the table.

You define the permissions that each DataGrabber will grant on the table it is creating with a text file named Permissions.txt.  This file must exist in the same sub-directory as the DataGrabber.  The format of the Permissions.txt files is also quite simple.  See the Permissions.txt file for one of the pre-existing DataGrabbers in the \DataGrabbers sub-directory in the Syteview installation directory on your SQL Server for details.   Include an empty file named Permissions.txt in the DataGrabber’s sub-directory if you don’t want any permissions to be granted on the table.

Note: There is a batch file (Get.SQL.Tables.bat) in the \DataGrabbers directory of the Syteview installation directory that when run will startup all of the DataGrabber programs.  There is also an individual batch file in each DataGrabber’s home directory that can be used to run the DataGrabbers one at a time.  These batch files will work unchanged if you installed the Syteview SQL components to the D: drive, \Syteview directory.  If you installed Syteview to a different drive or directory you will need to edit these programs to reflect your situation.  By adding these batch files to the Windows Task Scheduler you can easily automate the replication of any of your Progress tables to the Syteview database in your SQL Server.

Syteview includes Data Grabbers for many of the tables in your Progress Syteline database.  The following table lists these Data Grabbers and their location in the Syteview installation directory on your SQL Server:

DataGrabber Table Replicated Location
GetAPTRXPs APTRXP \APTRXPS
GetARINVs ARINV \ARINVS
GetARPMTDs ARPMTD \ARPMTDs
GetARPMTs ARPMT \ARPMTs
GetARTRANs ARTRAN \ARTRANs
GetCHARTs CHART \CHARTs
GetCOs CO \COs
GetCOITEMs COITEM \COITEMs
GetCO-BLNs CO-BLN \CO-BLNs
GetCO-SHIPs CO-SHIP \CO-SHIPs
GetCUSTADDRs CUSTADDR \CUSTADDRs
GetCUSTOMERs CUSTOMER \CUSTOMERs
GetDISPCODEs DISPCODE \DISPCODEs
GetEMPLOYEEs EMPLOYEE \EMPLOYEEs
GetEVALCODEs EVALCODE \EVALCODEs
GetEXC-MESGs EXC-MESG \EXC-MESGs
GetFEATQTYs FEATQTY \FEATQTYs
GetFEATRANKs FEATRANK \FEATRANKs
GetFEATUREs FEATURE \FEATUREs
GetFRZCOSTs FRZCOST \FRZCOSTs
GetINV-HDRs INV-HDR \INV-HDRs
GetINV-ITEMs INV-ITEM \INV-ITEMs
GetITEMs ITEM \ITEMs
GetITEMLOCs ITEMLOC \ITEMLOCs
GetITEMPRICEs ITEMPRICE \ITEMPRICEs
GetITEMVENDs ITEMVEND \ITEMVENDs
GetITEMWHSEs ITEMWHSE \ITEMWHSEs
GetJOBs JOB \JOBs
GetJOBITEMs JOBITEM \JOBITEMs
GetJOBMATLs JOBMATL \JOBMATLs
GetJOBROUTEs JOBROUTE \JOBROUTEs
GetJOBTRANs JOBTRAN \JOBTRANs
GetJOB-REFs JOB-REF \JOB-REFs
GetJOB-SCHs JOB-SCH \JOB-SCHs
GetJOURNALs JOURNALs \JOURNALs
GetJRT-SCHs JRT-SCH \JRT-SCHs
GetLEDGERs LEDGER \LEDGERs
GetLOCATIONs LOCATION \LOCATIONs
GetLOTs LOT \LOTs
GetMATLTRACKs MATLTRACK \MATLTRACKs
GetMRP-EXCs MRP-EXC \MRP-EXCs
GetMRP-PARMs MRP-PARM \MRP-PARMs
GetMRPs MRP \MRPs
GetMRP-WBs MRP-WB \MRP-WBs
GetNOTEs NOTE \NOTEs
GetP-FUNCTs P-FUNCT \P-FUNCTs
GetPARMs PARM \PARMs
GetPERMITs PERMIT \PERMITs
GetPOs PO \POs
GetPOBLNCHGs POBLNCHG \POBLNCHGs
GetPOCHANGEs POCHANGE \POCHANGEs
GetPOCHGs POCHG \POCHGs
GetPOITEMs POITEM \POITEMs
GetPO-BLNs PO-BLN \PO-BLNs
GetPRICECODEs PRICECODE \PRICECODEs
GetPROBCODEs PROBCODE \PROBCODEs
GetPRTRXPs PRTRXP \PRTRXPs
GetRCPTs RCPT \RCPTs
GetRMAs RMA \RMAs
GetRMAITEMs RMAITEM \RMAITEMs
GetSERIALs SERIAL \SERIALs
GetSHIPCODEs SHIPCODE \SHIPCODEs
GetSLSMANs SLSMAN \SLSMANs
GetTERMs TERM \TERMs
GetUSER-LOCALs USER-LOCAL \USER-LOCALs
GetVCH-HDRs VCH-HDR \VCH-HDRs
GetVCH-ITEMs VCH-ITEM \VCH-ITEMs
GetVENDADDRs VENDADDR \VENDADDRs
GetVENDORs VENDOR \VENDORs
GetWCs WC \WCs
GetWHSEs WHSE \WHSEs


Each DataGrabber includes the Microsoft Visual Basic source code for the DataGrabber which allows you to modify the program if necessary.  This allows you to change your DataGrabbers when the structure of one of your Progress tables changes.  Having the source code will also allow you to add features to the DataGrabber programs such validating or modifying data as the table is replicated, or moving your Progress tables into programs or databases other than your Microsoft SQL Server.

Syteview also gives you the tools you need to create new DataGrabber programs allowing you to create DataGrabbers for your own user-defined Progress tables or for Progress tables used by other programs.  

Creating Syteview DataGrabbers

Microsoft SQL Server uses a Windows NT resource called Data Transformation Services (DTS) to work with data services and structures.  Using the Data Transformation Services you can create DTS Packages that will package together all the steps required to import data from your Progress database into the Microsoft SQL Server Syteline® database. Microsoft SQL Server provides you with an easy to use GUI program, the DTS Designer, that helps you design and run these DTS Packages.  To create a DTS Package that imports a table from your Progress database make sure your Progress Database Server is running and then open the SQL Server Enterprise Manager program, expand the SQL Server Group, expand your Server, then right click Data Transformation Services, select All Tasks and then select Import Data.  This will bring up the following screen.



Click the Next button to proceed.

On the next screen select the MERANT 3.6 32-BIT Progress SQL92 v9.1B ODBC driver which will connect to your Progress database and retrieve table structures and records.  You will need to select the System DSN: Progress-ODBC and enter your Username and Password before clicking the Next button to proceed. Note: Leave the Username and Password fields blank if you have not enabled security on your Progress database.

Note: See Creating the Progress-ODBC Connection for instructions on creating the ODBC connection to your Progress database.



On the next screen select the Microsoft OLE DB Provider for SQL Server driver which will connect to your Microsoft SQL Server and create table structures and import the records.

Put your SQL Server’s name, or (local), in the Server field.  Make sure Use Windows NT Authentication is selected and put Syteview in the Database field.



Click the Next button to continue.

One this next screen select Copy table(s) and views from the source database.



Click the Next button to continue.

On the next screen select the Progress database table you want to create the DataGrabber for.  In this example we will be creating a DataGrabber for the CO table.



Click the Next button to proceed.

Note: If you see an error message here backup and make sure your Progress Database Server is running.

On the next screen uncheck the Run Immediately checkbox and checkmark the Save DTS Package checkbox.



Click the Next Button to continue.

On this next screen give the DTS Package a name and description.  Put your SQL Server’s name in the Server name field and make sure Use Windows NT authentication is selected.



Click the Next Button to proceed.

The next screen just tells you that you have successfully created a DTS Package that will import the selected table from the Progress database into the SQL Server.



Congratulations. Click the Finish Button.

You just created a DTS Package.  To run the Package and import the table and records from the Progress database go back to the SQL Server Enterprise Manager and expand Data Transformation Services and click on Local Packages.   Double click the Package you just created to open it.



This is the DTS Package you just created.



Pull down the Package menu and select Execute to run the package and import the records.

Hopefully you will see something like this:



Unfortunately, there is a problem with this DTS Package.  The next time you execute it, the package will import the same records again right on top of the records you just imported.  To fix this problem we simply add one more step to the beginning of the package that drops the table before proceeding.

To do this select the Create Table icon and copy and paste it just to the left of the original Create Table icon.



Double click the new icon and change the Description from Create Table to Drop Table and change the SQL Statement to DROP TABLE [Syteline].[dbo].[your table name] and click OK.



Finally select the Drop Table icon you just created by clicking on it, then hold down the Ctrl key and click on the Create Table icon .  Make sure you selected the Drop Table icon BEFORE you select the Create Table icon.  Now pull down the Workflow menu and select On Completion.

You should now see an arrow going from the Drop Table icon to the Create Table icon.



Finally, pull down the Package menu, select Save and you are done.

Done creating the DTS Package that is.  Now if you want to be able to create a Visual Basic program (a Syteview DataGrabber) that executes the same steps you have just specified, run the DTS2VB.exe program in the DTS-to-VBCode sub-directory.

Question: Why would you want to create a Visual Basic program that grabs tables and records from the Progress database and puts them in the Microsoft SQL Server when you already have a nice DTS Package that does the same thing?

Well, for one thing, you can use that VB program to create an executable that you can call from a batch file that imports the table and records at a specified time using Windows Task Scheduler.  You could automate the DTS Package using SQL Server Agents, but by using the Task Scheduler you can change the schedule without opening the SQL Server Enterprise Manager and you might feel a lot more comfortable letting someone else change the schedule using the Task Scheduler than you would letting them mess around inside the SQL Server Enterprise Manager.

Another, and maybe a better reason is you can modify the source code of the VB program to do almost anything else you might want while the records are being imported.  Like maybe sending an email when it sees a new customer order, or… well that’s up to you.  When you have the source code you can do almost anything!  Or you can just schedule the DTS Packages and forget the next steps.

If you do want to create a new Data Grabber from the DTS Package then run the DTS2VB.exe program in the \DTS-to-VBCode sub-directory in the Syteview installation directory on your SQL Server.

The following window will pop up:



Fill in the name of the DTS Package you just created.  GetCOs in this case.



And then click OK.

On the next screen, fill in the Progress Username you will use when importing records from the Progress table if you have enabled security on your Progress database.  Leave the Username blank if you have not enabled security on your Progress database or if you are using a Password file.



On the next screen, fill in the Progress Password you will use when importing records from the Progress table if you have enabled security on your Progress database.  Leave the password blank if you have not enabled security on your Progress database or if you are using a Password file.



Note: The Password file mentioned above is located in the \DataGrabbers\Password sub-directory and is named ProgressDB.Password.txt.  If this file exists then each DataGrabber in the \DataGrabbers directory will use the username and password listed within when connecting to the Progress database to retrieve records.   Using this file allows you to change the Progress username and password without having to recompile the Visual Basic code that makes up your DataGrabbers.

Note: If you have not enabled security on your Progress database then you should rename or delete this file before creating your DataGrabber programs.



On the next screen, fill in the Microsoft SQL Server Username you will use when connecting to the SQL Server if you are not using NT Integrated Security.  Leave the username blank if you are using NT Integrated Security.

Note: You really SHOULD be using NT Integrated Security to connect to your SQL Server.



On the next screen, fill in the Microsoft SQL Server Password you will use when connecting to the SQL Server if you are not using NT Integrated Security.  Leave the password blank if you are using NT Integrated Security.



After clicking OK you will (hopefully) see the following screen:



Assuming you see this screen, you will find the source code for the new program that was just created (in this case it’s GetCOs.txt) in the \DTS-to-VBCode directory.  Highlight the program and right click it and select Open With.  When the Open With window pops up select Visual Basic 6 and click OK.

Note: you might want to move the GetCOs.txt file to its own sub-directory before using it to create the Visual Basic program.  We’ve put each of the GetXXXs programs that come with Syteview in their own sub-directory under the \DataGrabbers directory.

Visual Basic will open up with the GetCOs.txt in the editor.



Pull down the Project menu and select References.  In the References window checkmark the Microsoft DTSPackage Object Library



Click OK to close the window.

Pull down the File menu and select Save Project As.  Give the project a name similar to the DTS Package name.  For example: GetCO.vbp

Next, pull down the File menu again and select Make Exe.

You have just created a new Data Grabber that will import a table and records from your Progress database and put it in the Microsoft SQL Server.  Way to go.

To run the Data Grabber, put it in a sub-directory of its own and copy an INDEXES.TXT and a PERMISSIONS.TXT file from the sub-directory of a similar Data Grabber.  You could use the INDEXES.TXT and PERMISSIONS.TXT files from the \Syteview_SQL\DataGrabbers\GetCOs sub-directory on the Syteview Installation CD-ROM in this case.

Then edit the Permissions and Indexes files to grant the permissions and create the indexes you want on the table the new Data Grabber will be creating.

Finally, pop out to the DOS command line, change into the sub-directory with your new Data Grabber in it and run it.

Easy, wasn’t it?