En chat

mt-en

Multi-Turn Benchmark for large language models.

References:

• Data Source
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

  {
    class_path: 'ChatResponse',
    init_args: {
      eval_dataset: {
        class_path: 'ChatbotBench',
        init_args: {
          path_or_name: 'mt-en',
          ref_path_or_name: 'mt-en-ref-gpt4',
        },
      },
      metrics: [
        { class_path: 'OutputLengthStats' },
      ],
      gen_kwargs: { max_new_tokens: 1024 },
      batch_size: 4,
    },
  }
  

multi_challenge

MultiChallenge: a multi-turn conversation benchmark for LLMs (English).

Each instance is a conversation with three or more turns, where the assistant has to respond to the last user turn. The conversations are categorized into four axes:

• Inference Memory: recall and use the information given by the user in earlier turns
• Instruction Retention: keep following the instructions given in earlier turns
• Reliable Version Editing: iteratively edit a piece of work through the conversation
• Self-Coherence: stay consistent with the assistant's own earlier responses

Each conversation has a YES/NO check question (target_question) for the last assistant response. Use the multi_challenge_eval metric preset with flexeval_file to let an LLM judge the responses.

References:

• Hugging Face Dataset
• Data Source
• MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

  {
    class_path: 'ChatResponse',
    init_args: {
      eval_dataset: {
        class_path: 'OpenAIMessagesDataset',
        init_args: {
          path: 'ScaleAI/MultiChallenge',
          split: 'test',
          message_key: 'conversation',
        },
      },
      metrics: [
        { class_path: 'OutputLengthStats' },
        { class_path: 'flexeval.core.metric.repetition_n.RepetitionN' },
      ],
      batch_size: 4,
    },
  }
  

vicuna-en

Vicuna Benchmark for large language models.

References:

• Data Source

  {
    class_path: 'ChatResponse',
    init_args: {
      eval_dataset: {
        class_path: 'ChatbotBench',
        init_args: {
          path_or_name: 'vicuna-en',
          ref_path_or_name: 'vicuna-en-ref-gpt4',
        },
      },
      metrics: [
        { class_path: 'OutputLengthStats' },
      ],
      gen_kwargs: { max_new_tokens: 1024 },
      batch_size: 4,
    },
  }