Evaluation

A structure that stores information about comprehension accuracy.

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Ideally the target words have highest correlations on the diagonal of the pertinent correlation matrices. Homophones support option is implemented.

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Count it as correct if one of the top k candidates is correct. Homophones support option is implemented.

accuracy_comprehension(S, Shat, data)

Evaluate comprehension accuracy for training data.

Obligatory Arguments

• S::Matrix: the (gold standard) S matrix
• Shat::Matrix: the (predicted) Shat matrix
• data::DataFrame: the dataset

Optional Arguments

• target_col::Union{String, Symbol}=:Words: the column name for target strings
• base::Vector=nothing: base features (typically a lexeme)
• inflections::Union{Nothing, Vector}=nothing: other features (typically in inflectional features)

Examples

accuracy_comprehension(
    S_train,
    Shat_train,
    latin_val,
    target_col=:Words,
    base=[:Lexeme],
    inflections=[:Person, :Number, :Tense, :Voice, :Mood]
    )

Note

In case of homophones/homographs in the dataset, the correct/incorrect values for base and inflections may be misleading! Consider the following example: The wordform "Äpfel" in German can be nominative plural, genitive plural and accusative plural. Let's assume we have a dataset in which "Äpfel" occurs in all three case/number combinations (i.e. there are homographs). If all these wordforms have the same semantic vectors (e.g. because they are derived from word2vec or fasttext which typically have a single vector per unique wordform), the predicted semantic vector of the wordform "Äpfel" will be equally correlated with all three case/number combinations in the dataset. In such cases, while the algorithm in this function can unambiguously conclude that the correct surface form "Äpfel" was comprehended, which of the three possible rows is the correct one will be picked somewhat non-deterministically (see https://docs.julialang.org/en/v1/base/collections/#Base.argmax). It is thus possible that the algorithm will then use the genitive plural instead of the intended nominative plural as the ground plural, and will report that "case" was comprehended incorrectly.

accuracy_comprehension(
    S_val,
    S_train,
    Shat_val,
    data_val,
    data_train;
    target_col = :Words,
    base = nothing,
    inflections = nothing,
)

Evaluate comprehension accuracy for validation data.

Obligatory Arguments

• S_val::Matrix: the (gold standard) S matrix of the validation data
• S_train::Matrix: the (gold standard) S matrix of the training data
• Shat_val::Matrix: the (predicted) Shat matrix of the validation data
• data_val::DataFrame: the validation dataset
• data_train::DataFrame: the training dataset

Optional Arguments

• target_col::Union{String, Symbol}=:Words: the column name for target strings
• base::Vector=nothing: base features (typically a lexeme)
• inflections::Union{Nothing, Vector}=nothing: other features (typically in inflectional features)

Examples

accuracy_comprehension(
    S_val,
    S_train,
    Shat_val,
    latin_val,
    latin_train,
    target_col=:Words,
    base=[:Lexeme],
    inflections=[:Person, :Number, :Tense, :Voice, :Mood]
    )

Note

In case of homophones/homographs in the dataset, the correct/incorrect values for base and inflections may be misleading! Consider the following example: The wordform "Äpfel" in German can be nominative plural, genitive plural and accusative plural. Let's assume we have a dataset in which "Äpfel" occurs in all three case/number combinations (i.e. there are homographs). If all these wordforms have the same semantic vectors (e.g. because they are derived from word2vec or fasttext which typically have a single vector per unique wordform), the predicted semantic vector of the wordform "Äpfel" will be equally correlated with all three case/number combinations in the dataset. In such cases, while the algorithm in this function can unambiguously conclude that the correct surface form "Äpfel" was comprehended, which of the three possible rows is the correct one will be picked somewhat non-deterministically (see https://docs.julialang.org/en/v1/base/collections/#Base.argmax). It is thus possible that the algorithm will then use the genitive plural instead of the intended nominative plural as the ground plural, and will report that "case" was comprehended incorrectly.

eval_SC(SChat::AbstractArray, SC::AbstractArray)

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Ideally the target words have highest correlations on the diagonal of the pertinent correlation matrices.

Obligatory Arguments

• SChat::Union{SparseMatrixCSC, Matrix}: the Chat or Shat matrix
• SC::Union{SparseMatrixCSC, Matrix}: the C or S matrix

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• R::Bool=false: if true, pairwise correlation matrix R is return

eval_SC(Chat_train, cue_obj_train.C)
eval_SC(Chat_val, cue_obj_val.C)
eval_SC(Shat_train, S_train)
eval_SC(Shat_val, S_val)

eval_SC(SChat::AbstractArray, SC::AbstractArray, SC_rest::AbstractArray)

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Ideally the target words have highest correlations on the diagonal of the pertinent correlation matrices.

Obligatory Arguments

• SChat::Union{SparseMatrixCSC, Matrix}: the Chat or Shat matrix
• SC::Union{SparseMatrixCSC, Matrix}: the training/validation C or S matrix
• SC_rest::Union{SparseMatrixCSC, Matrix}: the validation/training C or S matrix

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• R::Bool=false: if true, pairwise correlation matrix R is return

eval_SC(Chat_train, cue_obj_train.C, cue_obj_val.C)
eval_SC(Chat_val, cue_obj_val.C, cue_obj_train.C)
eval_SC(Shat_train, S_train, S_val)
eval_SC(Shat_val, S_val, S_train)

eval_SC(SChat::AbstractArray, SC::AbstractArray, data::DataFrame, target_col::Union{String, Symbol})

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Ideally the target words have highest correlations on the diagonal of the pertinent correlation matrices. Support for homophones.

Obligatory Arguments

• SChat::Union{SparseMatrixCSC, Matrix}: the Chat or Shat matrix
• SC::Union{SparseMatrixCSC, Matrix}: the C or S matrix
• data::DataFrame: datasets
• target_col::Union{String, Symbol}: target column name

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• R::Bool=false: if true, pairwise correlation matrix R is return

eval_SC(Chat_train, cue_obj_train.C, latin, :Word)
eval_SC(Chat_val, cue_obj_val.C, latin, :Word)
eval_SC(Shat_train, S_train, latin, :Word)
eval_SC(Shat_val, S_val, latin, :Word)

eval_SC(SChat::AbstractArray, SC::AbstractArray, SC_rest::AbstractArray, data::DataFrame, data_rest::DataFrame, target_col::Union{String, Symbol})

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Ideally the target words have highest correlations on the diagonal of the pertinent correlation matrices.

Obligatory Arguments

• SChat::Union{SparseMatrixCSC, Matrix}: the Chat or Shat matrix
• SC::Union{SparseMatrixCSC, Matrix}: the training/validation C or S matrix
• SC_rest::Union{SparseMatrixCSC, Matrix}: the validation/training C or S matrix
• data::DataFrame: the training/validation datasets
• data_rest::DataFrame: the validation/training datasets
• target_col::Union{String, Symbol}: target column name

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• R::Bool=false: if true, pairwise correlation matrix R is return

eval_SC(Chat_train, cue_obj_train.C, cue_obj_val.C, latin, :Word)
eval_SC(Chat_val, cue_obj_val.C, cue_obj_train.C, latin, :Word)
eval_SC(Shat_train, S_train, S_val, latin, :Word)
eval_SC(Shat_val, S_val, S_train, latin, :Word)

eval_SC(SChat::AbstractArray, SC::AbstractArray, batch_size::Int64)

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Ideally the target words have highest correlations on the diagonal of the pertinent correlation matrices. For large datasets, pass batch_size to process evaluation in chunks.

Obligatory Arguments

• SChat: the Chat or Shat matrix
• SC: the C or S matrix
• data: datasets
• target_col: target column name
• batch_size: batch size

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• verbose::Bool=false: if true, more information is printed

eval_SC(Chat_train, cue_obj_train.C, latin, :Word)
eval_SC(Chat_val, cue_obj_val.C, latin, :Word)
eval_SC(Shat_train, S_train, latin, :Word)
eval_SC(Shat_val, S_val, latin, :Word)

eval_SC(SChat::AbstractArray, SC::AbstractArray, data::DataFrame, target_col::Union{String, Symbol}, batch_size::Int64)

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Ideally the target words have highest correlations on the diagonal of the pertinent correlation matrices. For large datasets, pass batch_size to process evaluation in chunks. Support homophones.

Obligatory Arguments

• SChat::AbstractArray: the Chat or Shat matrix
• SC::AbstractArray: the C or S matrix
• data::DataFrame: datasets
• target_col::Union{String, Symbol}: target column name
• batch_size::Int64: batch size

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• verbose::Bool=false: if true, more information is printed

eval_SC(Chat_train, cue_obj_train.C, latin, :Word, 5000)
eval_SC(Chat_val, cue_obj_val.C, latin, :Word, 5000)
eval_SC(Shat_train, S_train, latin, :Word, 5000)
eval_SC(Shat_val, S_val, latin, :Word, 5000)

eval_SC_loose(SChat, SC, k)

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Count it as correct if one of the top k candidates is correct.

Obligatory Arguments

• SChat::Union{SparseMatrixCSC, Matrix}: the Chat or Shat matrix
• SC::Union{SparseMatrixCSC, Matrix}: the C or S matrix
• k: top k candidates

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)

eval_SC_loose(Chat, cue_obj.C, k)
eval_SC_loose(Shat, S, k)

eval_SC_loose(SChat, SC, k, data, target_col)

Assess model accuracy on the basis of the correlations of row vectors of Chat and C or Shat and S. Count it as correct if one of the top k candidates is correct. Support for homophones.

Obligatory Arguments

• SChat::Union{SparseMatrixCSC, Matrix}: the Chat or Shat matrix
• SC::Union{SparseMatrixCSC, Matrix}: the C or S matrix
• k: top k candidates
• data: datasets
• target_col: target column name

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)

eval_SC_loose(Chat, cue_obj.C, k, latin, :Word)
eval_SC_loose(Shat, S, k, latin, :Word)

eval_manual(res, data, i2f)

Create extensive reports for the outputs from build_paths and learn_paths.

eval_acc(res, gold_inds::Array)

Evaluate the accuracy of the results from learn_paths or build_paths.

Obligatory Arguments

• res::Array: the results from learn_paths or build_paths
• gold_inds::Array: the gold paths' indices

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• verbose::Bool=false: if true, more information is printed

Examples

# evaluation on training data
acc_train = JudiLing.eval_acc(
    res_train,
    cue_obj_train.gold_ind,
    verbose=false
)

# evaluation on validation data
acc_val = JudiLing.eval_acc(
    res_val,
    cue_obj_val.gold_ind,
    verbose=false
)

eval_acc(res, cue_obj::Cue_Matrix_Struct)

Evaluate the accuracy of the results from learn_paths or build_paths.

Obligatory Arguments

• res::Array: the results from learn_paths or build_paths
• cue_obj::Cue_Matrix_Struct: the C matrix object

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• verbose::Bool=false: if true, more information is printed

Examples

acc = JudiLing.eval_acc(res, cue_obj)

eval_acc_loose(res, gold_inds)

Lenient evaluation of the accuracy of the results from learn_paths or build_paths, counting a prediction as correct when the correlation of the predicted and gold standard semantic vectors is among the n top correlations, where n is equal to max_can in the 'learnpaths' or `buildpaths` function.

Obligatory Arguments

• res::Array: the results from learn_paths or build_paths
• gold_inds::Array: the gold paths' indices

Optional Arguments

• digits: the specified number of digits after the decimal place (or before if negative)
• verbose::Bool=false: if true, more information is printed

Examples

# evaluation on training data
acc_train_loose = JudiLing.eval_acc_loose(
    res_train,
    cue_obj_train.gold_ind,
    verbose=false
)

# evaluation on validation data
acc_val_loose = JudiLing.eval_acc_loose(
    res_val,
    cue_obj_val.gold_ind,
    verbose=false
)

extract_gpi(gpi, threshold=0.1, tolerance=(-1000.0))

Extract, using gold paths' information, how many n-grams for a gold path are below the threshold but above the tolerance.