EIM Delete process explained!

EIM delete is not as complex as merge process  but a proper understanding of its functioning is very important as one may happen to accidentally delete critical customer data in the process. Below are few things that one must understand for delete process to work as per expectation:

How does EIM delete process work?

1. EIM reads information from the EIM tables and the EIM configuration file to identify rows to delete from the Siebel base tables.
2. During its multiple passes through the EIM tables, EIM performs the following tasks:
• EIM initializes the EIM tables for deletion.
• It applies filter logic to do one of the following:
• Select rows for deleting
• Insert EIM tables rows that correspond to matching base table rows
• Select rows with matching user keys in the EIM tables
3. EIM updates other tables with rows containing foreign keys that point to newly deleted rows.

What delete methods are supported?

• EIM uses a combination of EIM table row contents and configuration file parameter values to determine the method for selecting rows to be deleted. The following methods are supported:

1. Delete rows in a Siebel base table with user key values specified in the corresponding EIM table. (DELETE EXACT)
2. Delete rows in the base table where the contents of a named column match those specified by a WHERE clause expression in the configuration file. (DELETE MATCHES)
3. Delete all rows in the base table (DELETE ALL ROWS)

How does the delete process flow?

• Preparing for an EIM delete process requires a thorough understanding of the parameter settings that specify delete criteria.
• To delete data, EIM performs the following steps:

1. EIM initializes EIM tables for delete.
• If CLEAR INTERFACE TABLE in the configuration file is TRUE, all rows with the specified batch number are deleted. CLEAR INTERFACE TABLE must be FALSE for a delete process that uses EIM table values to identify rows for deletion.
2. EIM deletes rows.
• If the DELETE EXACT parameter in the configuration file is set to TRUE, EIM deletes the rows from the table that match the user key defined in the EIM table.
• If the DELETE MATCHES parameter in the configuration file is set to a base table, EIM deletes the rows from the target base table that match the predicate specified in the parameter.
• If the DELETE ALL ROWS parameter in the configuration file is set to TRUE, EIM deletes all rows from the target base table.
3. EIM sets IF_ROW_STAT to DELETED for rows that are successfully processed.
• When a foreign key column that references the deleted record is a required one, the record with the foreign key is deleted. Otherwise, the foreign key column is cleared. 
• EIM deletion of a parent row causes cascade deletion of child rows only if the foreign key column in the child table is a mandatory column. Otherwise a cascade clear is performed.

DELETE EXACT Parameter:

• Specifies the base table rows to delete by using user key values specified in the EIM table.
• By default, DELETE EXACT = FALSE.
• If DELETE EXACT is set to TRUE, you must use the ONLY BASE TABLES parameter in conjunction with this parameter to identify the base tables.
• Although this parameter can be used to delete rows from both target and non-target base tables use the DELETE EXACT parameter to delete only non-target base tables containing user keys.
• Rows in non-target base tables that do not contain user keys will not be deleted. For example, you cannot use the DELETE EXACT parameter to update the S_ACTION_ARG table and the S_ESCL_ACTION table because there are no user keys defined for these tables.
• As another example, you can use DELETE EXACT to delete any of the non-target base tables such as S_ADDR_PER and S_ACCNT_POSTN using the EIM_ACCOUNT table. In this case, the EIM_ACCOUNT table would need to be loaded with records that would singularly identify the S_ACCNT_POSTN or the S_ADDR_PER record to be deleted.
• To use the DELETE EXACT parameter to delete data from base tables other than the target base table, specify the user key columns only for a single base table for each row in the EIM table.
• When specifying rows for exact deletion, make sure any columns not necessary to specify the row to be deleted are NULL to avoid problems with deleting from the wrong base table.
• EIM tries to enforce this behavior by requiring other user key columns to be NULL. If a row cannot be identified as clearly referring to a row in a single base table, that row will fail to be deleted.

DELETE MATCHES Parameter:

• Specifies a WHERE clause expression for filtering base table rows.
• The value is in two parts: the Siebel base table name and the filter expression that goes against the target base table.
• An example would be: DELETE MATCHES = S_ORG_EXT, (LAST_UPD > ‘2000-06-22’ AND LAST_UPD < ‘2000-06-23’)
• The expression is a self-contained WHERE clause expression (without the WHERE) and should use only literal values or column names (optionally prefixed with the base table name). There must also be a space separating the operator from the operand in this expression (a space must be added between > and ‘). When deleting rows for a specific date, you should use date ranges as shown in the example instead of setting the date equal to a specific date.
• By default, DELETE MATCHES expressions are not used.
• This parameter will only write the user keys values of the deleted target table rows to the EIM table columns. It will not write values of nonuser keys columns or non-target table rows column values to the EIM table.
• The deleted rows cannot be re-imported using the EIM table rows written by the EIM delete process, because they will not contain all the original information.
• Only use this parameter to delete rows from target base tables. Rows will be deleted from the target base table even if the DELETE ROWS parameter is set to FALSE for that table.

DELETE ALL ROWS Parameter:

• Specifies that all rows in the target base table are to be deleted.
• Default is FALSE.
• Existing values in the EIM table and DELETE MATCHES expressions are ignored.
• This parameter will only write the user keys values of the deleted target table rows to the EIM table columns. It will not write values of nonuser keys columns or non-target table rows column values to the EIM table.
• The deleted rows cannot be re-imported using the EIM table rows written by the EIM delete process, because they will not contain all the original information.

Common header and process section parameters for delete process:

PARAMETERS	DESCRIPTION
CASCADE DELETE ONLY	Default is FALSE, Set this parameter to TRUE to delete child records with null able foreign keys when the parent record is deleted. If FALSE, then when EIM deletes a parent record, it sets the foreign keys of the child records to NULL.
CLEAR INTERFACE TABLE	Specifies whether existing rows in the EIM table for the given batch number should be deleted, Valid values are true (the default unless DELETE EXACT = TRUE) and false (the default if DELETE EXACT = FALSE).
DELETE ALL ROWS	Used for deleting all rows in table, Default is FALSE.
DELETE EXACT	Delete using user key matching algorithm with rows in EIM table, Default is FALSE.
DELETE SKIP PRIMARY	Specifies whether EIM should perform a cascade update to the primary child column, Default is TRUE.
DELETE MATCHES	SQL WHERE fragment deletion criteria, Example: DELETE MATCHES = EIM_ACCOUNT, (NAME LIKE "TST_ACCT%")
DELETE ROWS	Specifies whether rows from the target base table can be deleted, Default is TRUE, Prevents deletions from one table while allowing them in others. For example, the following parameter setting prevents deletion of rows from the S_ADDR_ORG table: DELETE ROWS = S_ADDR_ORG, FALSE
IGNORE BASE COLUMNS	Specifies base table columns to be ignored by the import process, Use commas to separate column names, which can be qualified with base table names, Required and user key columns cannot be ignored, Improves performance when updating all but a few columns, The default is to not ignore any base table columns.
UPDATE ROWS	Specifies whether foreign key references can be updated, This parameter can be used to prevent the updating of foreign key references with a setting of FALSE, Default is TRUE, which affects all tables, To affect only specific tables; you can specify a table name. For example: UPDATE ROWS = S_CONTACT, TRUE. Also prevents updates in one table while allowing them in others, If this parameter is set to FALSE, EIM does not update rows in the specified base table, If you need to specify multiple tables, use one UPDATE ROWS statement for each table.

Image of Delete process parameters:

Creating Siebel ROW_ID through EIM

The unique identifier associated with every record in Siebel Enterprise databases is known as a Row ID. The column in which this value is found is ROW_ID and it is present on every table. The ROW_ID is unique for an entity. For example, the ROW_IDs for the same person in S_PARTY, S_CONTACT, and S_CONTACT_X are the same because they each refer to the same person.

Row IDs are used extensively throughout Siebel Enterprises to access specific records. Although users access records by a User Primary Key (such as Opportunity Name), it is more efficient for the Siebel Enterprise to store and access related data via the Row ID.  

The Row ID is a base-36 sequence number generated using a confidential, proprietary algorithm that ensures no duplication, thus protecting the referential integrity of the database. The ROW_ID column is VARCHAR(15), which may contain one or two non-numeric symbols (plus signs or hyphens, or both).

The format of the ROW_ID is one of the following:

CP-NS --> Records created by the user interface

CP+NP+NS --> Records created by Interface Manager (EIM)

CP-NP-NS --> Records created by EIM

(Starting in Siebel versions 6.2 and higher, and Siebel version 7)

where:

CP = Corporate Prefix, up to 2 alphanumeric characters

NP = Next Prefix, up to 6 alphanumeric characters

NS = Next Suffix, up to 7 alphanumeric characters

The maximum length of the ROW_ID is 15 alphanumeric characters. The corporate prefix will always be unique for any database (main or local). The server maintains its original value, and mobile databases created against that particular server database are always assigned a new, unique value.

The Siebel ROW_ID is a combination of the S_SEQUENCE_S and information from S_SSA_ID table. All connected users share the same prefix, which is obtained from the table S_SSA_ID on the server database. Remote Users are assigned a unique prefix when they are db-extracted. This value is also stored in the S_SSA_ID table on the local database.

Suffix is generated using an internal algorithm handled by the source code. When a new record is created through the user interface, the Siebel application reads the value of the current NS column from S_SSA_ID table and increments this value by a value more than 1 – for performance reasons, generally 50. The client caches these fifty potential ROW_IDs for future inserts. A new record entered from the user interface may result in many inserts to the underlying tables, depending on the business components used. When the client disconnects, cached ROW_IDs are lost.

The combination of the prefix and suffix generates a unique row_id. Since Siebel does not expose the algorithm behind generation of row ids, the ROW_ID generation is internal to the EIM process. Therefore Siebel provides user keys to map to a unique record in the base table.

During EIM, the EIM table columns, which map to the user keys of the base table, are populated with values so as to map to a unique record. If that record is to be updated by EIM later, the same user key values have to be populated in the corresponding EIM table columns. Otherwise, it will not resolve into the correct base table record.

For example, PERSON_ID and BU_ID constitute the user keys for S_CONTACT. The corresponding EIM_CONTACT columns are CON_PERSON_UID and CON_BU. When a new record is imported in S_CONTACT using EIM_CONTACT, the CON_PERSON_UID and CON_BU should uniquely identify a Contact record. Now, for updating some fields for this Contact using EIM_CONTACT, CON_PERSON_UID and CON_BU have to be populated with the same set of values that were used during the initial load in order to map to the same Contact record in the base table.

Similarly while loading a child table, which references the row id of the master table, the EIM table columns that map to the foreign-key column of the child table must be populated in the same way the user keys of the master table were populated.

For example, S_CONTACT_XM is a child table of S_CONTACT. The column PAR_ROW_ID of S_CONTACT_XM references the ROW_ID of S_CONTACT. S_CONTACT_XM is loaded by EIM_CON_DTL. The columns of EIM_CON_DTL that need to be populated to generate the PAR_ROW_ID of S_CONTACT_XM are CON_BU, CON_PERSON_UID and CON_PRIV_FLG. These EIM columns should be populated with the same set of values that were used to populate its parent S_CONTACT record through EIM_CONTACT. Note that though PRIV_FLG is not a part of the user key of S_CONTACT, but it is a required column and is part of the foreign-key mapping of its child table.

Database Tuning Tips for EIM (For Oracle Only)

1. Table Fragmentation:

Prior to running EIM it is important to clean up fragmented objects, especially those that will be used during the EIM processing. The following SQL statement can be used to identify objects with greater than 10 extents:

SELECT segment_name,segment_type,tablespace_name,extents
FROM dba_segments
WHERE owner = (Siebel table_owner)
and extents > 9;

To fix fragmentation, the objects will need to be rebuilt with appropriate storage parameters. Always be careful when rebuilding objects because of defaults, triggers, etc. on the objects.

2. Optimizer Mode:

Oracle optimization mode can also affect EIM performance. The Siebel application has been found to perform better under RULE based optimization under normal application usage. While there have been cases where it has been found that Cost based optimization has improved EIM performance this should only be attempted as a last resort and must be switched back to RULE for online usage. Be aware that Siebel only supports Rule-Based optimization. Optimization mode can be verified by running the following query:

SELECT NAME, VALUE FROM V$PARAMETER WHERE NAME = ‘OPTIMIZER_MODE’;

3. Purging an EIM table:

When purging data from the EIM table, use the TRUNCATE command as opposed to DELETE. The TRUNCATE command will release the data blocks and reset the high water mark while DELETE will not, thus causing additional blocks to be read during processing. Also, be sure to drop and re-create the indexes on the EIM table to release the empty blocks.

4. Index creation:

When working with large volumes of data in the EIM tables, index build time can be costly when refreshing the EIM table with a new data set. To improve the performance of the index build use the UNRECOVERABLE (Oracle 7.3) or NOLOGGING (Oracle 8) option. This will prevent Oracle from writing to the REDO LOG files. Users can also improve index build time by creating multiple SQL scripts to create the indexes and running these scripts in parallel through SQLPlus. Below is a sample SQL statement demonstrating the syntax for using the UNRECOVERABLE or NOLOGGING option:

CREATE INDEX S_SAMPLE_M1 ON
S_SAMPLE (SAMPLE_ID)
TABLESPACE TS_INDX
STORAGE (INITIAL 10M NEXT 5M PCTINCREASE 0)
UNRECOVERABLE/NOLOGGING; (choose one based on the Oracle version)

5. Disable archive logging:

It is recommended that Archive logging be disabled during the initial data loads. This can be enabled after the data loads are complete to provide for “point-in-time” recovery.

6. FREELIST parameter:

Multiple EIM processes can be executed against an EIM table provided they all use different batches or batch ranges. The concern is that contention for locks on common objects may be experienced. To run multiple jobs in parallel against the same EIM table, check that the ‘FREELIST’ parameter is set appropriately for the tables and indexes used in the EIM processing. This would include the EIM tables and indexes as well as the base tables and indexes. The value of this parameter specifies the number of block ids that will be stored in memory which are available for record insertion. As a rule of thumb, users should set this to at least half of the intended number of parallel jobs to be run against the same EIM table (example, a FREELIST setting of 10 should permit up to 20 parallel jobs against the same EIM table). This parameter is set at the time of object creation and the default for this parameter is 1. To check the value of this parameter for a particular object the following query can be used:

SELECT SEGMENT_NAME, SEGMENT_TYPE, FREELISTS
FROM DBA_SEGMENTS
WHERE SEGMENT_NAME=’<OBJECT NAME TO BE CHECKED>’;

To change this parameter the object must be rebuilt. Again, be careful when rebuilding objects because of defaults, triggers, etc. on the objects. To rebuild the object follow the steps below:

1. Export the data from the table with the grants.
2. Drop the table.
3. Re-create the table with the desired FREELIST parameter.
4. Import the data back into the table
5. Rebuild the indexes with the desired FREELIST parameter.

7. Caching tables:

One other measure that can help performance is to pin small tables that are frequently accessed in cache. The value of BUFFER_POOL_KEEP determines the 'keep pool', example, the portion of the buffer cache that will not be flushed by the LRU algorithm. The 'keep pool' allows one to 'pin' certain tables in memory, thus improving performance for accessing those tables. This will ensure that after the first time that the table is accessed it will always be in memory. Otherwise it is possible that the table will get pushed out of memory and will require disk access the next time used. Keep in mind that the amount of memory allocated to the ‘keep’ area is subtracted from the overall buffer cache memory (defined by DB_BLOCK_BUFFERS). A good candidate for this would be the S_LST_OF_VAL table. The syntax for pinning a table in cache is as follows:

ALTER TABLE S_LST_OF_VAL CACHE;

Using SQLPROFILE Parameter in IFB file

The inclusion of this parameter will greatly simplify the task of identifying the most expensive SQL statements. The insert of the following statement in the header section of the “IFB” file will place the most expensive SQL statements into the file:

SQLPROFILE = c:\temp\eimsql.sql

Below is an example of the file “eimsql.sql”:

<Start of the file – list of most expensive queries>

<…list of queries continues>

<Statistics by step and by pass>

<…list of statistics continues>

<SQL statements>

<…list of SQL statements continues>

How are INDEX HINTS used in EIM?

Enterprise Integration Manager (EIM) interacts with data objects (tables and columns) via SQL statements. How those queries are executed by the database in use is directly related to the EIM performance. Since optimization modes used by Siebel application differ among database platforms, index hints usages by EIM are different.

There are two EIM parameters related to index hints:

USE ESSENTIAL INDEX HINTS (TRUE by default)

USE INDEX HINTS (FALSE by default)

Below are suggested settings of the two parameters on different database platforms.

1. Oracle

USE ESSENTIAL INDEX HINTS = TRUE
USE INDEX HINTS = TRUE

2. MS SQL Server

USE ESSENTIAL INDEX HINTS = TRUE
USE INDEX HINTS = FALSE

3. DB2

Not applicable.

Perform testing with the IFB file parameter USE INDEX HINTS set to both settings (TRUE and FALSE). The default value for this parameter is TRUE. By setting this parameter to FALSE, EIM will not generate hints during processing. Performance gains can be realized if there are hints being generated that direct the database optimizer to use less than optimal indexes. EIM processing should be tested with both settings to determine which provides for better performance for each of the respective EIM jobs. Note that this parameter is only applicable for SQL Server and Oracle database platforms.