%Embedding.TextSplitter
class %Embedding.TextSplitter extends %Library.RegisteredObject
Method Inventory
Methods
classmethod mergeSplits(splits As %List, chunkSize As %Integer = 4000, overlap As %Integer = 200, separator As %String = " ", stripWhitespace As %Boolean = 1) as %List
Method to merge splits to reach desired chunkSize and overlap
classmethod recursiveSplit(text As %String, separators As %DynamicArray, chunkSize As %Integer = 4000, overlap As %Integer = 200, stripWhitespace As %Boolean = 1, separatorIdx=0) as %List
Splits text into chunks recursively, based on a list of separators.
classmethod split(text As %String, separator As %String = " ", chunkSize As %Integer = 4000, overlap As %Integer = 200, stripWhitespace As %Boolean = 1) as %List
Splits text into chunks naively based on one separator.
Inherited Members
Inherited Methods
- %AddToSaveSet()
- %ClassIsLatestVersion()
- %ClassName()
- %ConstructClone()
- %DispatchClassMethod()
- %DispatchGetModified()
- %DispatchGetProperty()
- %DispatchMethod()
- %DispatchSetModified()
- %DispatchSetMultidimProperty()
- %DispatchSetProperty()
- %Extends()
- %GetParameter()
- %IsA()
- %IsModified()
- %New()
- %NormalizeObject()
- %ObjectModified()
- %OriginalNamespace()
- %PackageName()
- %RemoveFromSaveSet()
- %SerializeObject()
- %SetModified()
- %ValidateObject()