Issue: new BHOLD Access Management Connector doesn’t import all objects

Hi all,

Earlier this week, I had an issue with the new BHOLD Access Management connector (AMC), which imports users, groups and org.units from Microsoft BHOLD Suite (SP1). It appeared that not all groups that were present in BHOLD Core, were imported in FIM (even during a full import)!

To fully understand the problem (and solution), here’s some details about how the connector fetches the objects from BHOLD. The connector is built using the ECMA 2.0 API for building custom management agents. As most of you will probably know, this API enables you to specify a batch size. The idea is that the connector will retrieve the amount of records as specified in the batch size configuration value. After that, FIM Sync imports the records in the connector space. After importing, it calls the import-function again, until it returns that all records have been imported.

To get information from BHOLD, the AMC uses these stored procedures:

[GetFullImportKeysAndInitializeWatermark]
This stored procedure is executed during the initialization. Basically, it returns all the object types and the maximum id value of that object type.

[GetGroupBaseAttributes], [GetOrganizationalUnitBaseAttributes], [GetUserBaseAttributes]
This one retrieves the groups (called ‘permissions’ in BHOLD Core), org. units and users. It takes the following arguments:
@BatchSize; The size of each ‘page’ (as set in the FIM Sync run profile)
@ObjectKey; Only objects with an id higher than this value, will be imported. If the procedure is called the first time, the value ‘0’ is supplied.
@NextObjectKey; This is an ‘out’ parameter, that (unlike what the name suggests) contains the id of the last object that is part of the result.

So, the following workflow is followed:

1. Execute ‘GetFullImportKeysAndInitializeWatermark’
2. Execute ‘Get<object>BaseAttributes’, replacing ‘<object>’ with the retrieved object in step 1.
3. Reformat the retrieved values to a format that FIM Sync understands.
4. See if the value of @NextObjectKey (the out-parameter of the ‘Get<object>BaseAttributes’ procedure) is equal to the max id value as retrieved in step 1. If it is not, return ‘true’ to FIM Sync, indicating that there are more objects to be imported. If it is: return ‘false’, indicating that everything is imported.

According to how the AMC calls the stored procedures, this all looks implemented very well. Except: I’ve noticed that the AMC doesn’t retrieve all the groups. Furthermore: the number of imported objects is dependent on the batch size value.

Here’s the catch: to make all the above work, it is CRITICAL that the ‘Get<object>BaseAttributes’ procedure contains an ‘ORDER BY ‘ObjectIdentifier’ (id) clause! If not, it will miss objects.

Let’s say this is the content of your table that contains the groups:

PermissionId;PermissionName;ApplicationID
4;PermissionA;0
5;PermissionB;0
6;PermissionC;0
3;PermissionD;0
8;PermissionE;0

And, for this example, we use a batch size of ‘3’.

1. ‘GetFullImportKeysAndInitializeWatermark’ will be executed. The max id of the groups object type returned is ‘8’.
2. Next, ‘GetGroupBaseAttributes’ is executed with parameters:
   @BatchSize: 3
   @ObjectKey: 0
3. The result, without the ‘ORDER BY’ clause, will be:
   PermissionId;PermissionName;ApplicationID
   4;PermissionA;0
   5;PermissionB;0
   6;PermissionC;0
   The output parameter will contain: ‘6’, because this is the max id of the returned objects.
4. As the value ‘6’ is not equal to ‘8’ (step 1), FIM Sync gets the message that not all records have been retrieved yet.
5. ‘GetGroupBaseAttributes’ is executed again, but now with the following attributes:
   @BatchSize: 3
   @ObjectKey: 6
6. The result will be:
   8;PermissionE;0
   The output parameter will contain ‘8’.
7. This is equal to the max id, so import will stop.

As you will notice: PermissionD, with ID ‘3’, has been missed!!

The solution

Search for the following line in the ‘GetGroupBaseAttributes’ stored procedure…
WHERE [p].[PermissionID] > @ObjectKey
…and add the following to it:
    ORDER BY [p].PermissionId

Finally, I would like to say thanks to my colleague Kenny. We worked on this issue together, having a hard time finding out what the problem was. In the end, he was the one that noticed the ORDER BY clause. Thanks!

As always, if you have any questions or remarks, let me know!!