Home > Class Reference > ENSLIB namespace > %BigData.ShardingManager
Private  

%BigData.ShardingManager

Sharding Manager API

Inventory

Parameters Properties Methods Queries Indices ForeignKeys Triggers
11 53

Summary

Properties
addingToMirrors currentShardNumber masterExtRefPrefix messagingType
myIPAddress prevQryResultSet qryReplyLimit qryReplyStart
replyTimeout skipRebuildExtentIndex someShardAppServers
Methods
%ActivateNewShards %AddToSaveSet %AssignMirroredShard
%AssignShardAppServer %AssignShards %BrokerShardOp
%ClassIsLatestVersion %ClassName %CommandShardCount
%ConstructClone %DeassignShardAppServer %Disconnect
%DispatchClassMethod %DispatchGetModified %DispatchGetProperty
%DispatchMethod %DispatchSetModified %DispatchSetMultidimProperty
%DispatchSetProperty %EndParallelLoad %EntityShardCount
%ExecCommand %ExecLoc %ExecLocation
%Extends %FactorQuery %GetMasterExtendedReferencePrefix
%GetParameter %GetPrevQryResultSet %GetShardMapRanges
%GetShardServerNamesAndDirectories %GetShardingManager %GetSkipRebuildExtentIndex
%IdRangeSize %Init %InvalidateAllMappings
%IsA %IsEntitySharded %IsModified
%IsShardedEntity %ListJobs %ListShardAddresses
%ListShards %MapUnifiedDb %New
%NextAIRange %NextIdRange %NormalizeObject
%ObjectModified %OnNew %OriginalNamespace
%PackageName %ReassignShard %RemoveFromSaveSet
%RemoveShards %RunQueriesAsync %SendBack
%SerializeObject %SetCurrentNamespace %SetExecLoc
%SetIdSequenceToMaxId %SetModified %SetPrevQryResultSet
%SetShardBrokeredHost %SetSkipRebuildExtentIndex %ShardExtRefPrefix
%ShardNumber %SyncShards %TrueOnAllShards
%UpdateShardMapVersions %ValidateObject %VerifyShards
FastNextIdForShard GetShardingRole NextIdForShard
ShardNumber ShardNumberNS

Properties

property addingToMirrors as %Boolean;
property currentShardNumber as %Integer;
property masterExtRefPrefix as %String;
property messagingType ;
property myIPAddress as %String;
property prevQryResultSet  [ MultiDimensional ];
property qryReplyLimit as %Integer;
property qryReplyStart as %Integer;
property replyTimeout as %Integer;
property skipRebuildExtentIndex as %Boolean;
property someShardAppServers as %Boolean;

Methods

classmethod %ActivateNewShards(pNamespace As %String) as %Status
Propagate definitions of existing sharded and map-replicated tables to newly assigned shards, and add the number of newly assigned shards to the visible shard count, making them available so that any newly defined sharded entities will be sharded across the full set of shards including the new ones.
classmethod %AssignMirroredShard(pNamespace As %String, pMirrorName As %String, pPrimaryHost As %String, pPrimaryPort As %Integer, pBackupHost As %String, pBackupPort As %Integer, pShardNamespace As %String, pMirrorDBName As %String = "", pVIP As %String = "")
Assign a mirrored shard to the sharded namespace specified by pNamespace. The mirrored set specified by pMirrorName is expected to already exist, the the namespace specified by pShardNamespace is expected to already exist on both failover members of that mirror, defined over a mirrored database whose name is optionally specified by pMirrorDBName. If a VIP (Virtual IP Address) is configured for this mirror, specify it via optional argument pVIP.
classmethod %AssignShardAppServer(pMasterNamespace As %String, pShardNumber As %Integer, pAppServerHost As %String, pAppServerPort As %Integer, pAppServerNamespace As %String) as %Status
Assign a shard application server. A shard may be assigned zero or more application servers. The specified shard must have already been assigned, and the specified application server instance and namespace are expected to be configured separately from this call. pMasterNamespace: The master namespace owning the shard to which this application server is assigned. pShardNumber: The number of the shard to which this application server is assigned (this is a number between 1 and shardCount). pAppServerHost: The hostname of the app server (may be "localhost", or an IP address). pAppServerPort: The default (i.e. superserver) port of the app server. pAppServerNamespace: The namespace on the application server, whose default database is expected to be a remote database mapped to the database of the shard to which this application server is assigned.
classmethod %AssignShards(pNamespace As %String, pShardList As %List) as %Status
Assign a list of shards to the master namespace specified by pNamespace. Each shard is specified by host and port of the machine / InterSystems IRIS instance that hosts the shard, and the namespace on that instance that is the shard. Shard machines, InterSystems IRIS instances, and namespaces are assumed to be provisioned/created separately, but do not need to exist when this method is called. This method is expected to be called (directly or indirectly) from the provisioning/configuration interface, which is expected to verify that the specified instances and namespaces are available to assign as shards; that the specified namespaces are not already assigned as shards to a different logical application server namespace; and that none of the specified shard namespaces is the namespace specified by pNamespace (i.e. a namespace cannot be a shard of itself). The list of shards, specified by pShardList, is in the format $lb(host1,port1,namespace1,...hostN,portN,namespaceN).
classmethod %BrokerShardOp(pStorageModel As %String, pOp As %String, pClientVersion As %Integer = 0, pMaxRanges As %Integer = 32768) as %List
Broker a sharded operation for a specified entity, of a specified storage model, in the current namespace. Return a possibly transformed version of the operation text, suitable for sending to shards, and a list of the shards to which it can be sent. This may be multiple shards or a single shard. When it is a single shard, it may actually be the address of the master namespace (because the specified operation cannot be executed as a brokered sharded operation, or cannot be sharded at all, and therefore must be executed on the master). It is the caller's responsibility to notice if a single returned address matches the connection the caller is already using. When storage model is SQL: pOp name is a SQL DML statement (SELECT, INSERT, UPDATE, or DELETE). pClientVersion determines whether caller is new enough to support various capabilies. 0 (default) - Initial version 1 - Adds support for cosharding by foreign id pMaxRanges - Maximum number of shard map ranges to return, in initial reply. Default: 32768 On success, returns a $list consisting of: Status (1 for success) Operation text to send to specified shards, which may be transformed or may be identical to pOp. Number of shard key fields (0 if shard key is system-assigned, or if there is no shard key) For each key field: Field ordinal (SQL column number, adjusted to be 0-based) Field name Number of shards For each shard: Hostname Port Namespace Lower bound of hash map range (This is an unused placeholder if shard key is system assigned, or if there is no shard key) If shard key is is foreign id: Total number of shard map entries (i.e. number of already-assigned id ranges (>=0). Number of shard map entries returned: min(total number of entries, pMaxRanges) Range size. Extent name of shard map owner, to pass as pShardMapOwner argument to %GetShardMapRanges() Shard map entries, as a byte string. The ordinal positions of bytes in this string (starting from 1) represent id range numbers, and the value of each entry is the shard number to which that id range maps. range number = (id\range size)+1 NOTE: The case of shard key being foreign id is identified by the number of shard key columns being 1, and the returned $list containing elements beyond the shard addresses. On error, returns a $list consisting of: Status code Error message
classmethod %CommandShardCount(pCommand As %BigData.ShardableEntity) as %Integer
Returns the number of shards to which the specified command must be sent
classmethod %DeassignShardAppServer(pMasterNamespace As %String, pShardNumber As %Integer, pAppServerHost As %String, pAppServerPort As %Integer, pAppServerNamespace As %String) as %Status
De-assign a shard application server. An error is returned if the specified shard has not been assigned, if the specified app server has not been assigned to this shard, or if the specified master namespace is not a sharded namespace. pMasterNamespace: The master namespace owning the shard to which this application server is assigned. pShardNumber: The number of the shard to which this application server is assigned (this is a number between 1 and shardCount). pAppServerHost: The hostname of the app server (may be "localhost", or an IP address). pAppServerPort: The default (i.e. superserver) port of the app server. pAppServerNamespace: The namespace on the application server.
method %Disconnect() as %Status
Disconnect an app-server-side Sharding Manager connection. This closes any underlying messaging connections to shard servers.
classmethod %EndParallelLoad(pStorageModel As %String, pEntityName As %String, pSetIdSequence As %Boolean) as %Status
End parallel load for a specified entity, of a specified storage model, in the current namespace. Set entity's idsequence to the new highest id (TBD: this is currently SQL-specific). TBD: Release appropriate lock, or equivalent, to allow resumption of operations that are not supported during parallel load.
classmethod %EntityShardCount(pStorageModel As %String(MAXLEN=2), pEntityName As %String)
Get the shard count of the specified entity of the specified storage model. Returns 0 if there is no such sharded entity.
classmethod %ExecCommand(pCommand As %BigData.ShardableEntity, ByRef pInstanceCount As %Integer, ByRef pRowId As %String, ByRef pSQLCODE, pNewShardsOnly As %Boolean = 0, pShardMask="", pQueryId As %Integer = "") as %Status
Execute a command, encapsulated as a ShardableEntity, across the sharded cluster. Sets reference parameter pInstanceCount to the number of instances (e.g. rows, objects, documents) affected. Certain commands also set reference parameters pRowId and pSQLCODE. If pNewShardsOnly is non-zero, send this command only to newly-added, not-yet-activated shards (TBD: This may be obsolete)
method %ExecLoc(pMsgCode As %String) as %String
classmethod %ExecLocation(pMsgCode As %String) as %String
classmethod %FactorQuery(query As %String(MAXLEN=""), maxFactors As %Integer) as %List
classmethod %GetMasterExtendedReferencePrefix()
Get a string identifying the server and directory of the master database, that can be used as a prefix in extended global references to globals in that database, e.g. ^|%GetMasterExtendedReferencePrefix()|myGlobalName JMM968: With Unified Shard DBs, this is only needed to form lock names in the master database, based on shard-local globals which, because they are mapped to the shard database, require an explicit extended reference to refer to them in the master database. (This is only used for centralized locking to enfore unique constraints across shards.)
method %GetPrevQryResultSet(pNamespace As %String) as %Library.RegisteredObject
%GetPrevQryResult
classmethod %GetShardMapRanges(pStorageModel As %String, pShardMapOwner As %String, pFirstRange As %Integer = 1, pMaxRanges As %Integer = 32768)
Get shard map ranges for a table whose shard key is the system-generated rowid. Arguments: pStorageModel - "SQ", for SQL tables (nothing else supported currently) pShardMapOwner - extent name of the table owning this shard map pFirstRange - Start returning ranges from this range number (default 1) pMaxRanges - Maximum number of shard map ranges to return, in initial reply. Default: 32768 Returns %list whose elements are: Status Total number of shard map entries (i.e. number of already-assigned id ranges (>=0). Number of shard map entries returned: min(total number of entries, pMaxRanges) Range size. Shard map entries, as a byte string. The ordinal positions of bytes in this string (starting from 1) represent id range numbers, and the value of each entry is the shard number to which that id range maps. range number = (id\range size)+1
classmethod %GetShardServerNamesAndDirectories(pShardCount As %Integer, pSkipShard As %Integer = "", ByRef pShards, pFirstShard As %Integer = 1) as %Status
classmethod %GetShardingManager() as %BigData.ShardingManager
Get the singleton Sharding Manager instance.
method %GetSkipRebuildExtentIndex() as %Boolean
classmethod %IdRangeSize() as %Integer
method %Init() as %Status
Initialize the Sharding Manager.
classmethod %InvalidateAllMappings(pNamespace As %String) as %Status
EM calls this method when it has reset configured mappings on the shasrds, letting sharding manager know that it should reset all metadata indicating that mappings have been done, so that they will be re-done when next needed. Call this once per application namespace that owns shards.
classmethod %IsEntitySharded(pStorageModel As %String, pEntityName As %String) as %Boolean
Return 1 if specified entity exists, and is sharded, in the current namespace. Else return 0.
classmethod %IsShardedEntity(pStorageModel As %String, pEntityName As %String, pVerbose=0)
Return true if the specified entity is sharded and belongs to the specified storage model.
method %ListJobs() as %Status
List shard server and combining query jobs currently running on this instance.
classmethod %ListShardAddresses(pNamespace As %String = $namespace, pClientProtocolVersion As %Integer = 1, pIncludeBrokerToken As %Boolean = 1) as %List
method %ListShards(pMasterNamespace As %String = "") as %Status
List all shards configured in the current sharded cluster.
classmethod %MapUnifiedDb(pShardNumber As %Integer = 0, pQueryShardNumber=0) as %Status
On all shards, or a specified shard, map the globals and routines databases to the master's globals and routines databases, and wildcard-map all globals of the shard-local package, and ^IRIS.SM.Shard, to the shard's original local database, or, for query shards, to the original local database of the corresponding data shard. Each shard, in processing this message, will store its local directory and database name in $$$SSGlobal($$$SSShardsSubscript,,$$$SSDirectorySubscript) and $$$SSGlobal($$$SSShardsSubscript,,$$$SSDirectorySubscript), making it possible to use extended global references to remotely access data in that shard (e.g. when joining multiple sharded tables), and making it possible for query shards to map their globals database to the local database of the corresponding data shard. A shard's original globals and routines database names are stored in ^IRIS.SM so they can be restored if the shard is deassigned. For data shards, these are stored in $$$SSGlobal($$$SSShardsSubscript,,$$$SSDatabaseSubscript) and $$$SSGlobal($$$SSShardsSubscript,,$$$SSRoutinesDBSubscript) respectively. For query shards, these are stored in $$$SSGlobal($$$SSShardsSubscript,$$$SSAppServersSubscript,,,$$$SSDatabaseSubscript) and $$$SSGlobal($$$SSShardsSubscript,$$$SSAppServersSubscript,,,$$$SSRoutinesDBSubscript) respectively.
classmethod %NextAIRange(pStorageModel As %String(MAXLEN=2), pEntityName As %String, ByRef pRangeLow As %Integer, ByRef pRangeHigh As %Integer) as %Status
Get the next range of auto_increment values for a specified entity, where a chunk is a range of a fixed size that is a multiple of 64k. This is used to allocate ranges of auto increment values to shards of a sharded entity, to ensure that the sequences are locally dense within 64k chunks, while also ensuring that ids are unique across all shards, for this entity.
classmethod %NextIdRange(pStorageModel As %String(MAXLEN=2), pEntityName As %String, ByRef pRangeLow As %Integer, ByRef pRangeHigh As %Integer) as %Status
Get the next range of ids for a specified entity, where a chunk is a range of a fixed size that is a multiple of 64k. This is used to allocate ranges of system-assigned ids to shards of a sharded entity, to ensure that the sequences are locally dense within 64k chunks, for bitmap index efficiency, while also ensuring that ids are unique across all shards, for this entity. If the id is the shard key for this entity, add a node to this entity's shard map, to map this range to the shard which allocated it.
method %OnNew() as %Library.Status
This callback method is invoked by the %New method to provide notification that a new instance of an object is being created.

If this method returns an error then the object will not be created.

It is passed the arguments provided in the %New call. When customizing this method, override the arguments with whatever variables and types you expect to receive from %New(). For example, if you're going to call %New, passing 2 arguments, %OnNew's signature could be:

Method %OnNew(dob as %Date = "", name as %Name = "") as %Status If instead of returning a %Status code this returns an oref and this oref is a subclass of the current class then this oref will be the one returned to the caller of %New method.

classmethod %ReassignShard(pMasterNamespace As %String, pShardHost As %String, pShardPort As %Integer, pShardNamespace As %String, pShardNumber As %Integer, pShardMirrorName As %String, pShardBackupHost, pShardBackupPort, pShardVIP As %String) as %Status
%ReassignShard should only be called from the %SYSTEM.Sharding ReassignShard method
classmethod %RemoveShards(pNamespace As %String, pShardList As %List = "") as %Status
Remove one or more shards from the set of shards configured for a specified namespace. The shards to be removed are specified as a list in the format $lb(hostname, port, namespace [,...]). An error is returned if any sharded entities have been defined in these shards. This method does not delete the shard namespaces themselves, nor any data in them. If there are any shard app servers (aka query shards) assigned to the shard being removed, they are deassigned. THIS IS NOT SAFE TO USE IN PRODUCTION. This is provided only as a convenience during initial testing and development. 1/25/17: This may now be safely used via $SYSTEM.Sharding.DeassignShard, which performs appropriate checks.
classmethod %RunQueriesAsync() as %Boolean
classmethod %SendBack(pSendBackHandle As %String, pCommand As %BigData.ShardableEntity) as %Status
Send back to the originating node and instance a message to execute the command currently encapsulated in pCommand. pSendBackHandle was passed as a parameter to the ShardableEntity method that is calling this method, and should be passed to this method by reference.
method %SetCurrentNamespace()
method %SetExecLoc(pMsgCode As %String, pExecLoc As %String)
classmethod %SetIdSequenceToMaxId(pTableName, pJobbed=0)
method %SetPrevQryResultSet(pNamespace As %String, pValue As %String)
%SetPrevQryResult
classmethod %SetShardBrokeredHost(pNamespace As %String, pShardNumber As %Integer, pBrokeredHost As %String, pBackup As %Boolean = 0) as %Status
method %SetSkipRebuildExtentIndex(pValue As %Boolean)
method %ShardExtRefPrefix(pShardNumber As %Integer) as %String
method %ShardNumber() as %Integer
classmethod %SyncShards() as %Status
classmethod %TrueOnAllShards(pCommand As %String, ByRef pResult, pSkipCurrentShard As %Boolean = 0) as %Status
Execute on all shards an object script command which sets the variable SNOutput to a non-zero or zero (or "") value. If the command sets SNOutput to a non-zero value on all shards, set pResult to a non-zero value. If it sets SNOutput to a zero or "" value on any shard, set pResult to a zero or "" value. If any shard returns an error, set pResult to 0.
classmethod %UpdateShardMapVersions(pExecLoc As %String, pStart As %Integer, pLimit As %Integer)
classmethod %VerifyShards(pMasterNamespace="", pShardNumber As %Integer = "", pQueryShardNumber As %Integer = "", pReturnFirstError As %Boolean = 0, pVerifyDataShardFirst As %Boolean = 1, pRefreshSqmMetadata=1) as %Status
Verify accessibility and correct configuration of all or specified shards pMasterNamespace - Namespace in which to validate shards, defaults to current namespace pShardNumber - Number of shard to validate. If not specified, validate all shards. pQueryShardNumber - If specified, validate the specified query shard of the shard specified by pShardNumber. pShardNumber must be specified if pQueryShardNumber is specified. pReturnFirstError: If true, return status after first shard fails verification. Else (the default) attempt to verify all data shards, return nested status if any fail, else attempt to verify all query shards, return nested status if any fail. pVerifyDataShardFirst: If true, before verifying a query shard verify its data shard. This is true when verifying individual shards, false when verifying all shards (because in the latter case, all data shards are verified, recursively, before any query shards).
classmethod FastNextIdForShard(pNamespace As %String, pTableName As %String) as %Integer [ SQLProc = ]
classmethod GetShardingRole(namespace As %String, ByRef dataMaster As %String, ByRef shardNumber As %Integer) as %String
Given a namespace name, if this namespace plays a role in a sharded cluster, return a string indicating the role, as follows:

"DM" - Data Master

"AM" - Application Master

"DS" - Data Shard

"QS" - Query Shard

"" - Not part of a sharded cluster

If the namespace is part of a sharded cluster, the address of the Data Master of the cluster is returned by reference, in the form "::". If the namespace is a Data Shard or Query Shard, the shard number is returned by reference.

classmethod NextIdForShard(pNamespace As %String, pTableName As %String) as %Integer [ SQLProc = ]
Return the next id in sequence that maps to the current shard. This depends on the following variables already being set: $$$SSTempGlobal($$$SSParallelLoadSubscript,$$$SSStorageModelSQL,pNamespace,TableName,$$$SSHashRowsSubscript) $$$SSTempGlobal($$$SSParallelLoadSubscript,$$$SSStorageModelSQL,pNamespace,TableName,$$$SSHashColsSubscript) $$$SSTempGlobal($$$SSParallelLoadSubscript,$$$SSStorageModelSQL,pNamespace,pTableName,$$$SSRangeLowSubscript) $$$SSTempGlobal($$$SSParallelLoadSubscript,$$$SSStorageModelSQL,pNamespace,pTableName,$$$SSRangeHighSubscript) $$$SSTempGlobal($$$SSParallelLoadSubscript,$$$SSStorageModelSQL,pNamespace,pTableName,$$$SSLastIdSubscript) Important note: pNamespace is the namespace of the shard, not the app server namespace in which the table is logically defined. This is necessary in order to distinguish shards, if the same InterSystems IRIS instance hosts more than one shard namespace. NOTE: This implementation is SQL-specific, but could be made generic by adding a storageModel parameter.
final classmethod ShardNumber(pId As %Integer = 1) as %Integer [ SQLProc = ]
final classmethod ShardNumberNS(namespace As %String = "") as %Integer [ SQLProc = ]
Given a namespace name, if this is a Shard namespace, return the ShardNumber If not a shard namespace, return 0