[FEATURE] Adds SearchApi as WebSearchEngine and Tool (langchain4j#1216)

## Issue
Closes langchain4j#1132 

## Change
Added SearchApi as a WebSearchEngine that also can be used as a tool.
Currently using Google Search as default engine. It also allows for new
engines to be implemented using the SearchApiRequestResponseHandler
interface, and adding it to the SearchApiEngine enum so the user can
choose which one to use.


## General checklist
- [X] There are no breaking changes
- [X] I have added unit and integration tests for my change
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [X] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [X] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
* The example is in the docs, I will open a new PR to the examples repo
if it is ok

@algora-pbc /claim langchain4j#1132

docs/docs/integrations/web-search/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Web Search",
+  "position": 19,
+  "link": {
+    "type": "generated-index",
+    "description": "Web Search"
+  }
+}

docs/docs/integrations/web-search/searchapi.md

@@ -0,0 +1,109 @@
+---
+sidebar_position: 1
+---
+
+# SearchApi
+
+[SearchApi](https://www.searchapi.io/) is a real-time SERP API. You can use it to perform searches in Google, Google News, Bing, Bing News, Baidu, Google Scholar, or any other engine that returns organic results.
+
+## Usage
+
+### Dependencies setup
+
+Add the following dependencies to your project's `pom.xml`:
+```xml
+<dependency>
+  <groupId>dev.langchain4j</groupId>
+  <artifactId>langchain4j-web-search-engine-searchapi</artifactId>
+  <version>{your-version}</version> <!-- Specify langchain4j version here -->
+</dependency>
+```
+
+or project's `build.gradle`:
+
+```groovy
+implementation 'dev.langchain4j:langchain4j-web-search-engine-searchapi:{your-version}'
+```
+
+### Example code:
+
+```java
+import dev.langchain4j.memory.chat.MessageWindowChatMemory;
+import dev.langchain4j.model.chat.ChatLanguageModel;
+import dev.langchain4j.model.openai.OpenAiChatModel;
+import dev.langchain4j.model.openai.OpenAiChatModelName;
+import dev.langchain4j.service.AiServices;
+import dev.langchain4j.web.search.WebSearchTool;
+import dev.langchain4j.web.search.searchapi.SearchApiEngine;
+import dev.langchain4j.web.search.searchapi.SearchApiWebSearchEngine;
+
+public class SearchApiTool {
+
+    interface Assistant {
+        @dev.langchain4j.service.SystemMessage({
+                "You are a web search support agent.",
+                "If there is any event that has not happened yet",
+                "You MUST create a web search request with user query and",
+                "use the web search tool to search the web for organic web results.",
+                "Include the source link in your final response."
+        })
+        String answer(String userMessage);
+    }
+
+    private static final String SEARCHAPI_API_KEY = "YOUR_SEARCHAPI_KEY";
+    private static final String OPENAI_API_KEY = "YOUR_OPENAI_KEY";
+
+    public static void main(String[] args) {
+        Map<String, Object> optionalParameters = new HashMap<>();
+        optionalParameters.put("gl", "us");
+        optionalParameters.put("hl", "en");
+        optionalParameters.put("google_domain", "google.com");
+
+        SearchApiWebSearchEngine searchEngine = SearchApiWebSearchEngine.builder()
+                .apiKey(SEARCHAPI_API_KEY)
+                .engine("google")
+                .optionalParameters(optionalParameters)
+                .build();
+        ChatLanguageModel chatModel = OpenAiChatModel.builder()
+                .apiKey(OPENAI_API_KEY)
+                .modelName(OpenAiChatModelName.GPT_3_5_TURBO)
+                .logRequests(true)
+                .build();
+
+        WebSearchTool webTool = WebSearchTool.from(searchEngine);
+
+        Assistant assistant = AiServices.builder(Assistant.class)
+                .chatLanguageModel(chatModel)
+                .tools(webTool)
+                .build();
+
+        String answer = assistant.answer("My family is coming to visit me in Madrid next week, list the best tourist activities suitable for the whole family");
+        System.out.println(answer);
+        /*
+            Here are some of the best tourist activities suitable for the whole family in Madrid:
+            
+            1. **Parque del Retiro** - A beautiful public park where families can enjoy nature and various activities.
+            2. **Prado Museum** - A renowned art museum that can be fascinating for both adults and children.
+            3. **Mercado de San Miguel** - A market where you can explore and taste delicious Spanish food.
+            4. **Royal Palace** - Explore the grandeur of the Royal Palace of Madrid.
+            5. **Plaza Mayor** and **Puerta del Sol** - Historic squares with a vibrant atmosphere.
+            6. **Santiago Bernabeu Stadium** - Perfect for sports enthusiasts and soccer fans.
+            7. **Gran Via** - A famous street for shopping, entertainment, and sightseeing.
+            8. **National Archaeological Museum** - Discover Spain's rich history through archaeological artifacts.
+            9. **Templo de Debod** - An ancient Egyptian temple in the heart of Madrid.
+         */
+    }
+}
+```
+
+### Available engines in Langchain4j
+
+| SearchApi Engine                                          | Available |
+|-----------------------------------------------------------|-----------|
+| [Google Web Search](https://www.searchapi.io/docs/google) | ✅         |
+| [Google News](https://www.searchapi.io/docs/google-news)  | ✅         |
+| [Bing](https://www.searchapi.io/docs/bing)                | ✅         |
+| [Bing News](https://www.searchapi.io/docs/bing-news)      | ✅         |
+| [Baidu](https://www.searchapi.io/docs/baidu)              | ✅         |
+
+Any other engine that returns the `organic_results` array and the organic result has `title`, `link`, and `snippet` is supported by this library even if not listed above.

langchain4j-bom/pom.xml

@@ -429,6 +429,16 @@
                 <artifactId>langchain4j-web-search-engine-google-custom</artifactId>
                 <version>${project.version}</version>
             </dependency>
+            <dependency>
+                <groupId>dev.langchain4j</groupId>
+                <artifactId>langchain4j-web-search-engine-tavily</artifactId>
+                <version>${project.version}</version>
+            </dependency>
+            <dependency>
+                <groupId>dev.langchain4j</groupId>
+                <artifactId>langchain4j-web-search-engine-searchapi</artifactId>
+                <version>${project.version}</version>
+            </dependency>
 
             <!-- experimental -->
             <dependency>

langchain4j-core/src/test/java/dev/langchain4j/web/search/WebSearchEngineIT.java

@@ -11,17 +11,19 @@
  */
 public abstract class WebSearchEngineIT {
 
+    protected static Integer EXPECTED_MAX_RESULTS = 7;
+
     protected abstract WebSearchEngine searchEngine();
 
     @Test
     void should_search() {
 
         // when
-        WebSearchResults webSearchResults = searchEngine().search("LangChain4j");
+        WebSearchResults webSearchResults = searchEngine().search("What is Artificial Intelligence?");
 
         // then
         List<WebSearchOrganicResult> results = webSearchResults.results();
-        assertThat(results).hasSize(5);
+        assertThat(results).hasSizeGreaterThan(0);
 
         results.forEach(result -> {
             assertThat(result.title()).isNotBlank();
@@ -30,17 +32,17 @@ void should_search() {
             assertThat(result.content()).isNull();
         });
 
-        assertThat(results).anyMatch(result -> result.url().toString().contains("https://github.com/langchain4j"));
+        assertThat(results).anyMatch(result -> result.url().toString().contains("AI"));
     }
 
     @Test
     void should_search_with_max_results() {
 
         // given
-        int maxResults = 7;
+        int maxResults = EXPECTED_MAX_RESULTS;
 
         WebSearchRequest request = WebSearchRequest.builder()
-                .searchTerms("LangChain4j")
+                .searchTerms("What is Artificial Intelligence?")
                 .maxResults(maxResults)
                 .build();
 
@@ -49,6 +51,6 @@ void should_search_with_max_results() {
 
         // then
         List<WebSearchOrganicResult> results = webSearchResults.results();
-        assertThat(results).hasSize(maxResults);
+        assertThat(results).hasSizeLessThanOrEqualTo(maxResults);
     }
 }

pom.xml

@@ -85,6 +85,7 @@
         <!-- web search engines -->
         <module>web-search-engines/langchain4j-web-search-engine-google-custom</module>
         <module>web-search-engines/langchain4j-web-search-engine-tavily</module>
+        <module>web-search-engines/langchain4j-web-search-engine-searchapi</module>
 
         <!-- embedding store filter parsers -->
         <module>embedding-store-filter-parsers/langchain4j-embedding-store-filter-parser-sql</module>

web-search-engines/langchain4j-web-search-engine-searchapi/pom.xml

@@ -0,0 +1,94 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>dev.langchain4j</groupId>
+        <artifactId>langchain4j-parent</artifactId>
+        <version>0.34.0-SNAPSHOT</version>
+        <relativePath>../../langchain4j-parent/pom.xml</relativePath>
+    </parent>
+
+    <artifactId>langchain4j-web-search-engine-searchapi</artifactId>
+    <packaging>jar</packaging>
+
+    <name>LangChain4j :: Web Search Engine :: SearchApi</name>
+
+    <dependencies>
+
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j-core</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>com.squareup.retrofit2</groupId>
+            <artifactId>retrofit</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>com.squareup.retrofit2</groupId>
+            <artifactId>converter-jackson</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>com.squareup.okhttp3</groupId>
+            <artifactId>okhttp</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.projectlombok</groupId>
+            <artifactId>lombok</artifactId>
+            <scope>provided</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.junit.jupiter</groupId>
+            <artifactId>junit-jupiter</artifactId>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.assertj</groupId>
+            <artifactId>assertj-core</artifactId>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j</artifactId>
+            <scope>test</scope>
+        </dependency>
+
+        <!-- Visibility for WebSearchEngineIT -->
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j-core</artifactId>
+            <classifier>tests</classifier>
+            <type>test-jar</type>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j-open-ai</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <licenses>
+        <license>
+            <name>Apache-2.0</name>
+            <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
+            <distribution>repo</distribution>
+            <comments>A business-friendly OSS license</comments>
+        </license>
+    </licenses>
+
+</project>

web-search-engines/langchain4j-web-search-engine-searchapi/src/main/java/dev/langchain4j/web/search/searchapi/SearchApi.java

@@ -0,0 +1,15 @@
+package dev.langchain4j.web.search.searchapi;
+
+import retrofit2.Call;
+import retrofit2.http.GET;
+import retrofit2.http.Header;
+import retrofit2.http.QueryMap;
+
+import java.util.Map;
+
+interface SearchApi {
+
+    @GET("/api/v1/search")
+    Call<SearchApiWebSearchResponse> search(@QueryMap Map<String, Object> params,
+                                            @Header("Authorization") String bearerToken);
+}

web-search-engines/langchain4j-web-search-engine-searchapi/src/main/java/dev/langchain4j/web/search/searchapi/SearchApiClient.java

@@ -0,0 +1,75 @@
+package dev.langchain4j.web.search.searchapi;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import lombok.Builder;
+import okhttp3.OkHttpClient;
+import okhttp3.ResponseBody;
+import retrofit2.Response;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.Map;
+
+import static com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+import static dev.langchain4j.internal.ValidationUtils.ensureNotBlank;
+import static dev.langchain4j.internal.ValidationUtils.ensureNotNull;
+
+class SearchApiClient {
+
+    private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper().enable(INDENT_OUTPUT);
+
+    private final SearchApi api;
+
+    @Builder
+    SearchApiClient(Duration timeout, String baseUrl) {
+        ensureNotNull(timeout, "timeout");
+        OkHttpClient.Builder okHttpClientBuilder = new OkHttpClient.Builder()
+                .callTimeout(timeout)
+                .connectTimeout(timeout)
+                .readTimeout(timeout)
+                .writeTimeout(timeout);
+        Retrofit retrofit = new Retrofit.Builder()
+                .baseUrl(ensureNotBlank(baseUrl, "baseUrl"))
+                .client(okHttpClientBuilder.build())
+                .addConverterFactory(JacksonConverterFactory.create(OBJECT_MAPPER))
+                .build();
+        this.api = retrofit.create(SearchApi.class);
+    }
+
+    SearchApiWebSearchResponse search(SearchApiWebSearchRequest request) {
+        Map<String, Object> finalParameters = new HashMap<>(request.getFinalOptionalParameters());
+        finalParameters.put("engine", request.getEngine());
+        finalParameters.put("q", request.getQuery());
+        String bearerToken = "Bearer " + request.getApiKey();
+        try {
+            Response<SearchApiWebSearchResponse> response = api.search(finalParameters, bearerToken).execute();
+            return getBody(response);
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private SearchApiWebSearchResponse getBody(Response<SearchApiWebSearchResponse> response) throws IOException {
+        if (response.isSuccessful()) {
+            return response.body();
+        } else {
+            throw toException(response);
+        }
+    }
+
+    private static RuntimeException toException(Response<?> response) throws IOException {
+        try (ResponseBody responseBody = response.errorBody()) {
+            int code = response.code();
+            if (responseBody != null) {
+                String body = responseBody.string();
+                String errorMessage = String.format("status code: %s; body: %s", code, body);
+                return new RuntimeException(errorMessage);
+            } else {
+                return new RuntimeException(String.format("status code: %s;", code));
+            }
+        }
+    }
+}