20-question hand-labeled AWS Bedrock RAG evaluation.
| Question | RAG answer excerpt | faithfulness | answer_relevancy | context_precision | context_recall | answer_correctness |
|---|---|---|---|---|---|---|
| What is Amazon Bedrock? | Based on the provided context, **Amazon Bedrock** is a **fully managed service** that provides **secure, enterprise-grade access to high-performing foundation models** from leading AI companies, enabling you to **build a | 1.0000 | 0.9326 | 1.0000 | 0.0000 | 0.7364 |
| Which foundation model providers are available on Amazon Bedrock? | Based on the provided context, the following foundation model providers are available on Amazon Bedrock: 1. **Amazon** (Nova models and Titan models) 2. **Anthropic** (Claude models) 3. **AI21 Labs** 4. **Cohere** 5. ** | 1.0000 | 0.9978 | 0.5000 | 1.0000 | 0.6632 |
| Does Amazon Bedrock support streaming responses? | Yes, Amazon Bedrock supports streaming responses. This is indicated by the mention of the `InvokeModelWithResponseStream` API operation and the `ConverseStream` operation, as well as the `responseStreamingSupported` fiel | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 0.5730 |
| How do you invoke a model on Amazon Bedrock? | Based on the provided context, I don't know the specific steps to invoke a model on Amazon Bedrock. The context only mentions that you should "**Submit prompts and generate responses with model inference**" operations pr | 1.0000 | 0.0000 | 0.5000 | 0.0000 | 0.5360 |
| What inference parameters can be configured when calling a Bedrock model? | Based on the provided context, I don't know based on the provided context. The context only provides an **overview/introduction** about inference request parameters and response fields for foundation models in Amazon Be | 0.6667 | 0.0000 | 0.0000 | 0.0000 | 0.1576 |
| What are Amazon Bedrock Knowledge Bases used for? | Based on the provided context, Amazon Bedrock Knowledge Bases are used to: 1. **Improve AI responses** through Retrieval Augmented Generation (RAG), a technique that uses information from data sources to improve the **r | 1.0000 | 1.0000 | 0.5000 | 1.0000 | 0.5925 |
| What are Amazon Bedrock Agents? | Based on the provided context, Amazon Bedrock Agents is a capability that allows you to **build and configure autonomous agents** in your application. These agents help end-users complete actions based on organization da | 1.0000 | 0.9005 | 0.5000 | 1.0000 | 0.7309 |
| What is the purpose of Guardrails for Amazon Bedrock? | Based on the provided context, the purpose of Amazon Bedrock Guardrails is to provide **configurable safeguards** to help build safe generative AI applications. Specifically, it: - Offers **comprehensive safety and priv | 1.0000 | 0.9644 | 0.8333 | 0.0000 | 0.6280 |
| Can you customize models on Amazon Bedrock? | Yes, you can customize models on Amazon Bedrock. Model customization is the process of providing training data to a model in order to improve its performance for specific use cases. Amazon Bedrock allows you to customize | 1.0000 | 0.9640 | 1.0000 | 0.0000 | 0.6609 |
| Can Amazon Bedrock run batch inference? | Yes, Amazon Bedrock can run batch inference. Based on the provided context, Amazon Bedrock supports batch inference jobs, including the ability to: - **Create** a batch inference job - **Monitor** batch inference jobs - | 1.0000 | 0.9436 | 1.0000 | 0.0000 | 0.5161 |
| Is Amazon Bedrock serverless? | I don't know based on the provided context. The provided context describes Amazon Bedrock as a "fully managed service" but does not specifically mention whether it is serverless or not. | 1.0000 | 0.0000 | 1.0000 | 0.0000 | 0.3789 |
| What AWS service is commonly used to store batch inference input and output data for Bedrock? | I don't know based on the provided context. | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0185 |
| What is a model ID in Amazon Bedrock? | Based on the provided context, a model ID in Amazon Bedrock is an identifier used **with the model invocation API operations**. The context mentions "a table of models and their IDs to use with the model invocation API o | 1.0000 | 0.0000 | 0.5000 | 0.0000 | 0.4269 |
| Can Bedrock Knowledge Bases retrieve context from external data for RAG? | Yes, Amazon Bedrock Knowledge Bases can retrieve context from external data sources for RAG (Retrieval Augmented Generation). According to the context, when a query is made, **"a knowledge base searches your data to find | 1.0000 | 0.9057 | 1.0000 | 1.0000 | 0.6151 |
| What does contextual grounding in Guardrails help detect? | Based on the provided context, contextual grounding checks in Guardrails help **detect hallucinations in model responses** that are either: - **Not grounded** (factually inaccurate or add new information) in the source, | 1.0000 | 0.9395 | 0.3333 | 1.0000 | 0.8593 |
| Does Amazon Bedrock require JavaScript in the browser to call InvokeModel? | Based on the provided context, the requirement for JavaScript mentioned is specifically for using the **Amazon Web Services Documentation** website, not for calling the InvokeModel API itself. The context states: > "Jav | 1.0000 | 0.0000 | 1.0000 | 1.0000 | 0.5854 |
| Does Bedrock support GPT-4o? | I don't know based on the provided context. (The context mentions a model ID "openai.gpt-oss-120b-1:0" but does not provide any specific information about GPT-4o support in Amazon Bedrock.) | 1.0000 | 0.0000 | 0.0000 | 1.0000 | 0.2239 |
| Are Amazon Bedrock Agents the same thing as ChromaDB? | No, Amazon Bedrock Agents and ChromaDB are not the same thing. Based on the provided context, Amazon Bedrock Agents is a feature within Amazon Bedrock that allows you to **build and configure autonomous agents** that orc | 0.8571 | 0.9929 | 0.3333 | 0.6667 | 0.5070 |
| Can every Bedrock model use exactly the same request body format? | Based on the provided context, **no**, not every Bedrock model uses exactly the same request body format. The context states: > "When you make inference calls to models with the model invocation API operations, you incl | 1.0000 | 0.8706 | 1.0000 | 1.0000 | 0.6020 |
| Does Amazon Bedrock automatically make private company data public? | Based on the provided context, there is no information suggesting that Amazon Bedrock automatically makes private company data public. In fact, the context indicates the opposite — it describes Amazon Bedrock as providin | 1.0000 | 0.0000 | 0.8333 | 1.0000 | 0.5574 |