Package 'MacBehaviour'

Title: Behavioural Studies of Large Language Models
Description: Efficient way to design and conduct psychological experiments for testing the performance of large language models. It simplifies the process of setting up experiments and data collection via language models’ API, facilitating a smooth workflow for researchers in the field of machine behaviour.
Authors: Xufeng Duan [aut, cre], Shixuan Li [aut], Zhenguang Cai [aut]
Maintainer: Xufeng Duan <[email protected]>
License: LGPL-3
Version: 1.2.7
Built: 2024-09-08 01:00:21 UTC
Source: CRAN

Help Index


Step3: Generate the experimental design matrix.

Description

Defines the experiment setup based on the stimuli loaded.

Usage

experimentDesign(data, session = 1, randomItem = FALSE, randomEvent = FALSE)

Arguments

data

A data frame that has been processed through the 'loadData' function, containing the experimental items and their attributes.

session

An integer indicating how many sessions (the whole set of trials) should be run. Default is 1, meaning no repetition.

randomItem

A logical indicating whether the Item should be randomized. Default is FALSE, meaning trials will occur in the order provided.

randomEvent

A logical indicating whether the Event should be randomized. Default is FALSE, meaning trials will occur in the order provided.

Value

A data.frame with the designed structure for the experiment, including any repetitions and randomizations as specified. Each row corresponds to a single trial or instance in the experiment.And it will display the type of experiment with the materials you provide.

Examples

df <- data.frame(
Run = c(1,2),
Item = c(1,2),
Event= c(1,1),
Condition = c(1,2),
TargetPrompt = c("1","2")
)

ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt)

Design=experimentDesign(ExperimentItem,session=1,randomItem=TRUE)

Step2: Load and format data

Description

Prepares the stimuli data for the experiment.

Usage

loadData(
  runList,
  itemList,
  conditionList,
  promptList,
  header = TRUE,
  eventList = NULL
)

Arguments

runList

A numeric vector of data representing the 'Run' column in the experiment.

itemList

A numeric vector of data representing the 'Item' column in the experiment.

conditionList

A numeric/character vector of data representing the 'Condition' column in the experiment.

promptList

A character vector of the main prompt (usually experiment items).

header

A logical value indicating if the output data.frame should include column headers (default is TRUE).

eventList

A numeric vector of data representing the 'Event' column in the experiment.

Value

A data frame with the processed columns 'Run', 'Event', 'Item', 'Condition', and 'Prompt', ready for use in experiments.

Examples

df <- data.frame(
Run = c(1,2),
Item = c(1,2),
Condition = c(1,2),
TargetPrompt = c("1","2"),
Event = c(1,1)
)
ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt)

magicTokenizer

Description

This function provides the number of tokens in a specified text, acting as a wrapper for an internal tokenizer function.

Usage

magicTokenizer(text)

Arguments

text

A character string: the text for which the number of tokens is required.

Value

Returns the number of tokens in the provided text.


Step4: Pre-check for token usage in experiment design.

Description

Configures experimental parameters before execution.

Usage

preCheck(
  data,
  checkToken = FALSE,
  systemPrompt = "",
  imgDetail = "auto",
  version = "2023-06-01",
  ...
)

Arguments

data

A data.frame that has been structured by the 'experimentDesign' function, containing the experimental setup.

checkToken

Whether to perform token count check, select TRUE to submit your experiment to our server's tokenizer for token count check, the default selection is FALSE (i.e., no token check will be performed, but you need to manually check if the number of tokens exceeds the model limit to avoid errors in the experiment).

systemPrompt

The system prompt text used in the chatGPT model interaction. If left empty, a space character is assumed.Note: This parameter does not work in models that do not support system prompts.

imgDetail

The image quality of the img modality is set to auto by default, with low/high as selectable options.

version

When using the Claude model, the version parameter required defaults to "2023-06-01".

...

Variable parameter lists allow you to input additional parameters supported by the model you're using, such as n=2 / logprobs=TRUE... Note: You must ensure the validity of the parameters you enter; otherwise, an error will occur.

Value

A list containing the original data and the parameters for the chatGPT model interaction, confirming that the setup has passed the token checks or indicating issues if found.

Examples

df <- data.frame(
Run = c(1,2),
Item = c(1,2),
Event =c(1,1),
Condition = c(1,2),
TargetPrompt = c("please repeat this sentence: test1","please repeat this sentence: test2")
)

ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt)

Design=experimentDesign(ExperimentItem,session=1)

gptConfig=preCheck(Design, systemPrompt="You are a participant in a psychological experiment",
                    imgDetail="low",temperature=0.7)

Run an Experiment Based on the Configuration

Description

Executes the experiment and saves the results to an Excel file.

Usage

runExperiment(gptConfig, savePath = "./output.xlsx")

Arguments

gptConfig

A list containing the configuration for the language model, including the system prompt, model specifications, and token settings.

savePath

The file path where the experiment results will be saved in Excel format. Defaults to './output.xlsx' in the current working directory.

Value

This function does not return a value but saves the experiment results to the specified Excel file. Upon completion, "Done." will be printed to the console.

Examples

## Not run: 

runExperiment(Experiment_config,"./output.xlsx")

#The first argument Experiment_config is generated by preCheck() function.

Experiment_config <- preCheck(data)

## End(Not run)

Step1: Set model's API key and url.

Description

This function allows users to set and verify an API key for data collection. You can change the default api_url for others models' API.

Usage

setKey(api_key, model, api_url = NULL, ...)

Arguments

api_key

A character string: the user's OpenAI/huggingface/gemini/claude/baichuan/other API key. Please fill 'NA' for self-deployed models.

model

A character string: specify the model version. For gemini, you could input "gemini-pro"

api_url

A character string: the API URL for the model. If not specified, the default Chat completion URL will be used based on the api_key.

...

Additional arguments to be passed (currently not used, but kept for future extensibility).

Value

Prints a message to the console indicating whether the API key setup was successful. If the setup fails, the function stops with an error message.

Examples

## Not run: 
set_key(api_key="YOUR_API_KEY", model="gpt-3.5-turbo", api_url="api.openai.com/v1/chat/completions")

## End(Not run)