graphai.core.embedding.embedding module

graphai.core.embedding.embedding.embedding_to_json(v)

graphai.core.embedding.embedding.embedding_from_json(s)

graphai.core.embedding.embedding.generate_embedding_text_token(s, model_type)

Generates an md5-based token for a string :param s: The string :param model_type: Type of embedding model

Returns:

Token

graphai.core.embedding.embedding.get_text_token_count_using_model(model_tokenizer, text)

class graphai.core.embedding.embedding.EmbeddingModels

Bases: object

get_device()

load_models(load_heavies=True)

Loads sentence transformers model :returns: None

load_model(model_type)

model_loaded(model_type)

get_tokenizer(model_type)

set_tokenizer(model_type, tokenizer)

get_last_usage()

unload_model(unload_period=10800.0)

Unloads all models except the light, default model. :param unload_period: Minimum time that needs to have passed since last use of heavy model to qualify it for :param unloading. If set to 0: :param forces an unloading.:

Returns:

None if not enough time has passed since last use, a list of unloaded models otherwise.

get_token_count(text, model_type)

get_max_tokens(model_type)

embed(text, model_type='all-MiniLM-L12-v2')

graphai.core.embedding.embedding.copy_embedding_object(embedding_obj, model_type)

graphai.core.embedding.embedding.compute_embedding_text_fingerprint_callback(results, text, model_type)

graphai.core.embedding.embedding.token_based_embedding_lookup(token, model_type)

graphai.core.embedding.embedding.fingerprint_based_embedding_lookup(token, fp, model_type)

graphai.core.embedding.embedding.embed_text(models, text, model_type)

graphai.core.embedding.embedding.insert_embedding_into_db(results, token, text, model_type, force=False)

graphai.core.embedding.embedding.jsonify_embedding_results(results)

graphai.core.embedding.embedding.embedding_text_list_fingerprint_parallel(tokens, text_list, i, n)

graphai.core.embedding.embedding.embedding_text_list_dummy_fingerprint_parallel(tokens, text_list, i, n)

graphai.core.embedding.embedding.embedding_text_list_fingerprint_callback(results, model_type)

graphai.core.embedding.embedding.embedding_text_list_embed_parallel(input_list, embedding_obj, model_type, i, n, force=False)

graphai.core.embedding.embedding.embedding_text_list_embed_callback(results, model_type, force)

graphai.core.embedding.embedding.embedding_text_list_embed_jsonify_callback(results)