Getting Started With MLFlow For LLM Evaluation

MLflow is a powerful open-source platform for managing the machine learning lifecycle. Whereas it’s traditionally used for monitoring model experiments, logging parameters, and managing deployments, MLflow has simply currently launched assist for evaluating Large Language Fashions (LLMs).

On this tutorial, we uncover straightforward strategies to make use of MLflow to guage the effectivity of an LLM—in our case, Google’s Gemini model—on a set of fact-based prompts. We’ll generate responses to fact-based prompts using Gemini and assess their prime quality using a variety of metrics supported instantly by MLflow.

Organising the dependencies

For this tutorial, we’ll be using every the OpenAI and Gemini APIs. MLflow’s built-in generative AI evaluation metrics at current rely on OpenAI fashions (e.g., GPT-4) to behave as judges for metrics like reply similarity or faithfulness, so an OpenAI API secret’s required. You presumably can pay money for:

Placing within the libraries

pip arrange mlflow openai pandas google-genai

Setting the OpenAI and Google API Keys as environment variable

import os
from getpass import getpass

os.environ["OPENAI_API_KEY"] = getpass('Enter OpenAI API Key:')
os.environ["GOOGLE_API_KEY"] = getpass('Enter Google API Key:')

Preparing Evaluation Data and Fetching Outputs from Gemini

import mlflow
import openai
import os
import pandas as pd
from google import genai

Creating the evaluation information

On this step, we define a small evaluation dataset containing factual prompts along with their applicable ground reality options. These prompts span topics equal to science, properly being, web enchancment, and programming. This structured format permits us to objectively study the Gemini-generated responses in direction of acknowledged applicable options using assorted evaluation metrics in MLflow.

eval_data = pd.DataFrame(
    {
        "inputs": [
            "Who developed the theory of general relativity?",
            "What are the primary functions of the liver in the human body?",
            "Explain what HTTP status code 404 means.",
            "What is the boiling point of water at sea level in Celsius?",
            "Name the largest planet in our solar system.",
            "What programming language is primarily used for developing iOS apps?",
        ],
        "ground_truth": [
            "Albert Einstein developed the theory of general relativity.",
            "The liver helps in detoxification, protein synthesis, and production of biochemicals necessary for digestion.",
            "HTTP 404 means 'Not Found' -- the server can't find the requested resource.",
            "The boiling point of water at sea level is 100 degrees Celsius.",
            "Jupiter is the largest planet in our solar system.",
            "Swift is the primary programming language used for iOS app development."
        ]
    }
)

eval_data

Getting Gemini Responses

This code block defines a helper carry out gemini_completion() that sends a quick to the Gemini 1.5 Flash model using the Google Generative AI SDK and returns the generated response as plain textual content material. We then apply this carry out to each quick in our evaluation dataset to generate the model’s predictions, storing them in a model new “predictions” column. These predictions will later be evaluated in direction of the underside reality options

shopper = genai.Shopper()
def gemini_completion(quick: str) -> str:
    response = shopper.fashions.generate_content(
        model="gemini-1.5-flash",
        contents=quick
    )
    return response.textual content material.strip()

eval_data["predictions"] = eval_data["inputs"].apply(gemini_completion)
eval_data

Evaluating Gemini Outputs with MLflow

On this step, we provoke an MLflow run to guage the responses generated by the Gemini model in direction of a set of factual ground-truth options. We use the mlflow.think about() method with 4 lightweight metrics: answer_similarity (measuring semantic similarity between the model’s output and the underside reality), exact_match (checking for word-for-word matches), latency (monitoring response know-how time), and token_count (logging the number of output tokens).

It’s crucial to note that the answer_similarity metric internally makes use of OpenAI’s GPT model to judge the semantic closeness between options, which is why entry to the OpenAI API is required. This setup provides an setting pleasant technique to assess LLM outputs with out relying on custom-made evaluation logic. The final word evaluation outcomes are printed and likewise saved to a CSV file for later inspection or visualization.

mlflow.set_tracking_uri("mlruns")
mlflow.set_experiment("Gemini Simple Metrics Eval")

with mlflow.start_run():
    outcomes = mlflow.think about(
        model_type="question-answering",
        information=eval_data,
        predictions="predictions",
        targets="ground_truth",
        extra_metrics=[
          mlflow.metrics.genai.answer_similarity(),
          mlflow.metrics.exact_match(),
          mlflow.metrics.latency(),
          mlflow.metrics.token_count()
      ]
    )
    print("Aggregated Metrics:")
    print(outcomes.metrics)

    # Save detailed desk
    outcomes.tables["eval_results_table"].to_csv("gemini_eval_results.csv", index=False)

To view the detailed outcomes of our evaluation, we load the saved CSV file proper right into a DataFrame and modify the present settings to verify full visibility of each response. This permits us to look at specific particular person prompts, Gemini-generated predictions, ground reality options, and the associated metric scores with out truncation, which is especially helpful in pocket e-book environments like Colab or Jupyter.

outcomes = pd.read_csv('gemini_eval_results.csv')
pd.set_option('present.max_colwidth', None)
outcomes

Check out the Codes proper right here. All credit score rating for this evaluation goes to the researchers of this endeavor. Moreover, be at liberty to adjust to us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our E-newsletter.