Define Experiments

Experiment

class rupsycho.experiment.ExperimentDocument(*, id=None, metadata=None, name=None, description=None, parameters=None, prompt_template=None, models=None, demographic_profiles=None, questionnaire=None, type='ExperimentDocument', runnable_prompt=None, runnable_models=None, runnable_parser=None)[source]

Bases: BaseMedia, ExperimentProcessingMixin, ExperimentExportMixin, ModelManagementMixin, PromptTemplateMixin, PersonaManagementMixin

Class for storing an experiment and associated questionnaires along with metadata.

Extends LangChain’s BaseMedia and Pydantic’s BaseModel, integrating data validation and serialization capabilities with document handling features.

Example

from my_module import ExperimentDocument

experiment_doc = ExperimentDocument(
    name="Experiment 1",
    description="Test Experiment",
    demographic_profiles={...},
    models={...},
    questionnaire=...,
    metadata={"source": "Lab A"}
)

Parameters:

id (str | None)
metadata (Dict[str, Any])
name (str | None)
description (str | None)
parameters (Parameters)
prompt_template (NormalPromptTemplateConfig | ChatPromptTemplateConfig | LangchainPromptTemplateConfig)
models (Dict[str, LangChainModelConfig | LocalHuggingFaceModelConfig | RemoteHuggingFaceModelConfig | OllamaModelConfig | OpenAIModelConfig])
demographic_profiles (Dict[str, DemographicProfile])
questionnaire (Questionnaire | None)
type (Literal['ExperimentDocument'])
runnable_prompt (Any)
runnable_models (Dict[str, Any])
runnable_parser (Any)

name: str | None

description: str | None

parameters: Parameters

prompt_template: NormalPromptTemplateConfig | ChatPromptTemplateConfig | LangchainPromptTemplateConfig

models: Dict[str, LangChainModelConfig | LocalHuggingFaceModelConfig | RemoteHuggingFaceModelConfig | OllamaModelConfig | OpenAIModelConfig]

demographic_profiles: Dict[str, DemographicProfile]

questionnaire: Questionnaire | None

metadata: Dict[str, Any]: Arbitrary metadata associated with the content.

type: Literal['ExperimentDocument']

runnable_prompt: Any

runnable_models: Dict[str, Any]

runnable_parser: Any

class Config[source]

Bases: object

arbitrary_types_allowed = True

set_questionnaire(questionnaire)[source]

Sets the questionnaire for the experiment.

Parameters:: questionnaire (Questionnaire)
Return type:: None

set_parser(parser)[source]

Adds a parser to the experiment.

Parameters:: parser (Any)
Return type:: None

Experiment Processing Mixin

rupsycho.mixins.experiment_processing.get_default_model()[source]

Return type:: HuggingFacePipeline

class rupsycho.mixins.experiment_processing.ExperimentProcessingMixin[source]

Bases: object

A mixin that provides methods for processing experiments with various models, profiles, and instructions. This class is designed to facilitate the creation and execution of experiment chains, manage errors, track progress, and store results.

create_chain(model_key, seed=42, params={})[source]

Create a chain for running the experiment using the model identified by model_key.

Parameters:

model_key (str)
seed (int)
params (dict)

Return type:

Any

create_all_chains(seed=42, params={})[source]

Create chains for running the experiment with all models.

Parameters:

seed (int)
params (dict)

Return type:

Dict[str, Any]

initialize_answers_if_needed(instruction_item)[source]

Initialize the ‘answers’ attribute as a nested defaultdict if it doesn’t exist.

Parameters:: instruction_item (Any)
Return type:: None

create_input_dict(profile, instruction_item)[source]

Create the input dictionary required for invoking the chain.

Parameters:

profile (Any)
instruction_item (Any)

Return type:

Dict[str, Any]

process_single_experiment(pbar)[source]

Process the experiment, iterating through models, profiles, and instruction items.

Return type:: None

run()[source]

Run the experiment processing with a progress bar.

Return type:: None

Experiment Exporting Mixin

class rupsycho.mixins.experiment_exporting.ExperimentExportMixin[source]

Bases: object

Mixin providing methods to export the experiment results to a file or return the answers.

model_dump(include=None, exclude=None, exclude_unset=True, exclude_none=True)[source]

Convert the ExperimentDocument instance to a dictionary, allowing for inclusion or exclusion of specific fields.

Parameters:

include (Set[str] | None) – A set of field names to include in the output.
exclude (Set[str] | None) – A set of field names to exclude from the output.
exclude_unset (bool) – Exclude fields that were not explicitly set.
exclude_none (bool) – Exclude fields that are set to None.

Return type:

Dict[str, Any]

export_to_file(filename)[source]

Exports the experiment data to a file in JSON format.

Parameters:: filename (str)
Return type:: None

get_answers()[source]

Extracts and returns the answers from the experiment in a list holding the nested answer structure.

Returns:: A list of dictionaries representing the answers for each instruction item.
Return type:: List[Dict[str, Any]]

get_answers_as_dataframe()[source]

Returns a flat pandas DataFrame with the experiment’s answers.

Columns include: - “Instruction ID” - “Instruction Question” - “Model ID” - “Persona ID” - “Run Seed” - “Answer”

Returns:: A pandas DataFrame containing the flattened answers.
Return type:: DataFrame

Model Management Mixin

class rupsycho.mixins.model_managing.ModelManagementMixin[source]

Bases: object

Mixin providing methods to manage models in the experiment.

This includes adding models, setting models, and handling the conversion of model definitions into runnable models.

load_model(model_definition)[source]

Load the model from its serialized definition.

Parameters:: model_definition – Serialized model definition
Returns:: Loaded model, or None if an error occurs.

add_model(model, identifier=None)[source]

Adds a model to the experiment.

Parameters:

model (Any) – The model to be added.
identifier (str | None) – Optional identifier for the model.

Return type:

None

set_runnable_models()[source]

Converts model definitions into runnable models, updating the runnable_models dictionary.

Return type:: None

get_model(identifier)[source]

Retrieves a model by its identifier.

Parameters:: identifier (str) – The identifier of the model.
Returns:: The model if found, otherwise None.
Return type:: Any | None

remove_model(identifier)[source]

Removes a model from the experiment by its identifier.

Parameters:: identifier (str) – The identifier of the model to be removed.
Return type:: None

list_models()[source]

Lists all model identifiers in the experiment.

Returns:: A list of model identifiers.
Return type:: List[str]

has_model(identifier)[source]

Checks if a model exists in the experiment by its identifier.

Parameters:: identifier (str) – The identifier of the model to check.
Returns:: True if the model exists, False otherwise.
Return type:: bool

replace_model(identifier, new_model)[source]

Replaces an existing model in the experiment with a new one.

Parameters:

identifier (str) – The identifier of the model to replace.
new_model (Any) – The new model to replace the old one.

Return type:

None

clear_models()[source]

Removes all models from the experiment.

Return type:: None

count_models()[source]

Returns the number of models in the experiment.

Returns:: The number of models.
Return type:: int

get_all_runnable_models()[source]

Retrieves a dictionary of all runnable models.

Returns:: A dictionary of runnable models with identifiers as keys.
Return type:: Dict[str, Any]

Persona Management Mixin

class rupsycho.mixins.persona_managing.PersonaManagementMixin[source]

Bases: object

Mixin providing methods to manage personas in the experiment.

This includes adding personas (demographic profiles) and ensuring they are stored in a dictionary with unique identifiers.

add_persona(persona, identifier=None)[source]

Adds a demographic profile (persona) to the experiment.

Parameters:

persona (DemographicProfile) – The DemographicProfile instance to be added.
identifier (str | None) – Optional identifier for the persona. If not provided, a unique identifier is generated based on the persona’s id.

Return type:

None

get_persona(identifier)[source]

Retrieves a persona by its identifier.

Parameters:: identifier (str) – The identifier of the persona.
Returns:: The DemographicProfile instance if found, otherwise None.
Return type:: DemographicProfile | None

remove_persona(identifier)[source]

Removes a persona from the experiment by its identifier.

Parameters:: identifier (str) – The identifier of the persona to be removed.
Return type:: None

list_personas()[source]

Lists all persona identifiers in the experiment.

Returns:: A list of persona identifiers.
Return type:: list

clear_personas()[source]

Removes all personas from the experiment.

Return type:: None

Prompt Management Mixin

class rupsycho.mixins.prompt_managing.PromptTemplateMixin[source]

Bases: object

Mixin providing methods to manage the prompt template in the experiment.

This includes setting the prompt template, loading it, and converting it into a runnable form.

load_prompt(prompt_template)[source]

Load the prompt template from its serialized definition.

Parameters:: prompt_template (str) – Serialized prompt template.
Returns:: Loaded prompt, or None if an error occurs.
Return type:: Any | None

set_prompt(prompt)[source]

Adds a prompt to the experiment and converts it into its runnable form.

Parameters:: prompt (Any) – The prompt to be set.
Return type:: None

get_prompt()[source]

Retrieve the current runnable prompt template if available.

Returns:: The current runnable prompt, or None if not set.
Return type:: Any | None

get_prompt_config()[source]

Retrieve the serialized prompt template configuration.

Returns:: Serialized prompt template or None if not set.
Return type:: str | None

set_prompt_config(prompt_template)[source]

Set the prompt template configuration by loading its serialized form.

Parameters:: prompt_template (str) – Serialized prompt template configuration.
Return type:: None

reset_prompt()[source]

Resets the current prompt template and runnable form to None.

Return type:: None

has_prompt()[source]

Check whether a runnable prompt is set.

Returns:: True if a runnable prompt is set, False otherwise.
Return type:: bool

Experiment Collection

class rupsycho.experiment_collection.ExperimentCollection(experiments, *, id=None, metadata=None, type='ExperimentCollection')[source]

Bases: BaseMedia

Class for storing a collection of ExperimentDocument objects along with metadata.

Example

from my_module import ExperimentCollection, ExperimentDocument

experiment_doc1 = ExperimentDocument(
    name="Experiment 1",
    description="Test Experiment",
    demographic_profiles=[...],
    models=[...],
    questionnaire=...,
    metadata={"source": "Lab A"}
)

experiment_doc2 = ExperimentDocument(
    name="Experiment 2",
    description="Another Test",
    demographic_profiles=[...],
    models=[...],
    questionnaire=...,
    metadata={"source": "Lab B"}
)

experiment_collection = ExperimentCollection(
    experiments=[experiment_doc1, experiment_doc2],
    metadata={"source": "Lab Collection"}
)

Parameters:

experiments (List[ExperimentDocument])
id (str | None)
metadata (dict)
type (Literal['ExperimentCollection'])

experiments: List[ExperimentDocument]: List of ExperimentDocument objects representing the experiments.

type: Literal['ExperimentCollection']

class Config[source]

Bases: object

arbitrary_types_allowed = True

classmethod is_lc_serializable()[source]

Return whether this class is serializable.

Return type:: bool

classmethod get_lc_namespace()[source]

Get the namespace of the LangChain object.

Return type:: List[str]

run_all()[source]

Run the experiment processing for all ExperimentDocuments in the collection.

Return type:: None