🔭 EOSC Data Commons Search server

A server for the EOSC Data Commons project MatchMaker service, providing natural language search over open-access datasets. It exposes an HTTP POST endpoint and supports the Model Context Protocol (MCP) to help users discover datasets and tools via a Large Language Model–assisted search.

🧩 Endpoints

The HTTP API comprises 2 main endpoints:

• `/mcp`: MCP server that searches for relevant data to answer a user question using the EOSC Data Commons OpenSearch service
  • Uses Streamable HTTP transport
  • Available tools:
    • [x] Search datasets
    • [x] Get metadata for the files in a dataset (name, description, type of files)
    • [x] Search tools
    • [ ] Search citations related to datasets or tools
• `/chat`: HTTP POST endpoint (JSON) for chatting with the MCP server tools via an LLM provider (API key provided through env variable at deployment)
  • Streams Server-Sent Events (SSE) response complying with the AG-UI protocol.

> [!TIP]

>

> It can also be used just as a MCP server through the pip package.

🔌 Connect to the MCP server

The system can be used directly as a MCP server using either STDIO, or Streamable HTTP transport.

> [!WARNING]

>

> You will need access to a pre-indexed OpenSearch instance for the MCP server to work.

Follow the instructions of your client, and use the `/mcp` URL of the public server: https://matchmaker.eosc-data-commons.eu/api/search/mcp

To add a new MCP server to VSCode GitHub Copilot:

• Open the Command Palette (`ctrl+shift+p` or `cmd+shift+p`)
• Search for `MCP: Add Server...`
• Choose `HTTP`, and provide the MCP server URL: https://matchmaker.eosc-data-commons.eu/api/search/mcp

Your VSCode `mcp.json` should look like:

{
    "servers": {
        "data-commons-search-http": {
            "url": "https://matchmaker.eosc-data-commons.eu/api/search/mcp",
            "type": "http"
        }
    },
    "inputs": []
}

🛠️ Development

> [!IMPORTANT]

>

> Requirements:

>

> - [x] `uv`, to easily handle scripts and virtual environments

> - [x] docker, to deploy the database and OpenSearch service

> - [x] API key for a LLM provider: e-infra CZ, Mistral.ai, or OpenRouter

>

📥 Install dev dependencies

uv sync --all-extras

Install pre-commit hooks:

uv run --all-extras pre-commit install

Create a `keys.env` file with your LLM provider API key(s), and optionally other configurations:

CESNET_API_KEY=YOUR_API_KEY
MISTRAL_API_KEY=YOUR_API_KEY

OIDC_CLIENT_ID=
OIDC_CLIENT_SECRET=
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
POSTGRES_HOST=localhost
POSTGRES_USER=app
POSTGRES_PASSWORD=app_password

RATE_LIMITING_ENABLED=False
LOG_LEVEL=DEBUG
LOG_JSON=false

OPENSEARCH_URL=http://localhost:9200

💾 Database

The search system needs to connect to a PostgreSQL database to store authenticated users conversations.

Deploy and initialize the metadata-warehouse, in these instructions we expect the `metadata-warehouse` folder to be alongside the `data-commons-search`,in the same folder.

cd ../metadata-warehouse
docker compose up postgres

To initialize db, run from the `metadata-warehouse` repo:

uv run --directory scripts/postgres_data create_db.py --db appdb --reset

> [!IMPORTANT]

>

> For publicly available environments you will want to update the `app` user password:

>

> ```sql

> ALTER USER app WITH PASSWORD 'newpassword';

> ```

Reset db:

docker compose down --volumes --remove-orphans

Export the schema from `db.py` to the metadata-warehouse (command to run at the root of the data-commons-search repo):

uv run scripts/export_db_schema.py ../metadata-warehouse/scripts/postgres_data/create_sql/appdb/tables.sql

⚡️ Start dev server

Start the server in dev at http://localhost:8000, with MCP endpoint at http://localhost:8000/mcp pointing to a running OpenSearch instance:

uv run --all-extras uvicorn src.data_commons_search.main:app --reload

> Default `OPENSEARCH_URL=http://localhost:9200`

Customize server port through environment variable:

OPENSEARCH_URL=http://localhost:9200 SERVER_PORT=8001 uv run --all-extras uvicorn src.data_commons_search.main:app --host 0.0.0.0 --port 8001 --reload

> [!NOTE]

>

> You can deploy the `matchmaker` frontend in dev on the side pointing to this dev server:

>

> ```sh

> cd ../matchmaker

> npm run dev

> ```

> [!TIP]

>

> Example `curl` request:

>

> ```sh

> curl -X POST http://localhost:8000/chat -H "Content-Type: application/json" \

> -d '{"items": [{"type": "message", "role": "user", "content": [{"text": "Educational datasets from Switzerland covering student assessments, language competencies, and learning outcomes, including experimental or longitudinal studies on pupils or students."}]}], "model": "cesnet/agentic"}'

> ```

>

> With authenticated user access token from http://127.0.0.1:8000/auth/login:

>

> ```sh

> curl -X POST http://localhost:8000/chat -H "Content-Type: application/json" \

> -H "Cookie: access_token=$ACCESS_TOKEN" \

> -d '{"items": [{"type": "message", "role": "user", "content": [{"text": "Educational datasets from Switzerland covering student assessments, language competencies, and learning outcomes, including experimental or longitudinal studies on pupils or students."}]}], "model": "cesnet/agentic"}'

> ```

>

> Get last conversation:

>

> ```sh

> curl -X GET "http://localhost:8000/conversation/$(curl -s http://localhost:8000/conversations -H "Content-Type: application/json" -H "Cookie: access_token=$ACCESS_TOKEN" | jq -r '.[-1].thread_id')" -H "Content-Type: application/json" -H "Cookie: access_token=$ACCESS_TOKEN"

> ```

>

> Find available model from Cesnet provider:

>

> ```sh

> curl -H "Authorization: Bearer $CESNET_API_KEY" https://llm.ai.e-infra.cz/v1/models | jq ".data[].id"

> ```

>

> Recommended model: `cesnet/agentic`

🔐 Secrets Store

EGI Secret Store, get the token from aai.egi.eu/token (decode the JWT to get the actual access token)

export BASE="https://matchmaker.eosc-data-commons.eu"
curl -s "$BASE/auth/user" --cookie "access_token=$TOKEN"

curl -s -X PUT "$BASE/auth/keys/vip" --cookie "access_token=$TOKEN" \
  -H "Content-Type: application/json" -d '{"key_value":"sk-123"}'

curl -s "$BASE/auth/keys" --cookie "access_token=$TOKEN"
curl -s "$BASE/auth/keys/all" --cookie "access_token=$TOKEN"
curl -s "$BASE/auth/keys/vip" --cookie "access_token=$TOKEN"
curl -s -X DELETE "$BASE/auth/keys/vip" --cookie "access_token=$TOKEN"

🐳 Deploy with Docker

Create a `keys.env` file with the API keys (see above for complete example):

CESNET_API_KEY=YOUR_API_KEY
MISTRAL_API_KEY=YOUR_API_KEY
SEARCH_API_KEY=SECRET_KEY_YOU_CAN_USE_IN_FRONTEND_TO_AVOID_SPAM

> [!TIP]

>

> `SEARCH_API_KEY` can be used to add a layer of protection against bots that might spam the LLM, if not provided no API key will be needed to query the API.

You can use the prebuilt docker image `ghcr.io/eosc-data-commons/data-commons-search:main`

Example `compose.yml`:

services:
  mcp:
    image: ghcr.io/eosc-data-commons/data-commons-search:main
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      OPENSEARCH_URL: "http://opensearch:9200"
      CESNET_API_KEY: "${CESNET_API_KEY}"

Build and deploy the service:

docker compose up

📦 Build for production

Build package in `dist/`:

uv build

✅ Run tests

> [!CAUTION]

>

> You need to first start the server on port 8000 (see start dev server section) and PostgreSQL.

uv run pytest

Run benchmark (check success of a set of search queries):

uv run tests/benchmark.py

Run LLM jailbreak tests with `garak`:

PYTHONPATH=tests/security uv run garak --config tests/security/garak.yaml

Run stress tests (20 concurrent uses) of the API:

uv run tests/stress_api.py -c 20

🧹 Format code and type check

uvx ruff format && uvx ruff check --fix && uvx ty check

♻️ Reset the environment

Upgrade `uv`:

uv self update

Clean `uv` cache:

uv cache clean

🔧 Maintenance

Pre-compute stats for the datasets in the db to `src/data_commons_search/stats.json`:

uv run --env-file prod.env scripts/compute_stats.py

Update dependencies in `pyproject.toml`:

uvx uv-bump

🏷️ Release process

Run the release script providing the version bump: `fix`, `minor`, or `major`

.github/release.sh fix

Or an explicit version, e.g. to align with the frontend version:

.github/release.sh 0.10.0

> This will create a git tag, github release, and publish a docker image

🤝 Acknowledments

The LLM provider `cesnet` is a service provided by e-INFRA CZ and operated by CERIT-SC Masaryk University

Computational resources were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.

The authentication provider is EGI Check-in.