📈 Asset Plugin Guide

How to create a new Asset Source Provider to fetch prices from a new data source.

Base class: AssetSourceProvider (canonical path: backend/app/services/asset_sources/core.py) Plugin folder: backend/app/services/asset_source_providers/<name>/ — the code in plugin.py, beside an __init__.py holding only its docstring (see Plugin Folder and Translations) Registry: AssetProviderRegistry

Release 2 import compatibility

backend.app.services.asset_source remains a supported compatibility facade and re-exports the identical public objects. New provider code should import AssetSourceProvider and AssetSourceError from backend.app.services.asset_sources.core; the canonical thread runner and shared caches live there too.

🤖 Tip: Let an LLM scaffold the provider

Paste this entire page into an LLM with the data source's API documentation and use this prompt:

Suggested prompt

Here is the LibreFolio Asset Source Provider specification: [paste this page]

I want to add a provider for [DATA_SOURCE_NAME]. Their API works like this:
[brief description, or paste their API docs]

Write a complete Python AssetSourceProvider implementation. Follow all conventions:
@register_provider, FACurrentValue, FAHistoricalData, FAPricePoint, direct calls to
synchronous provider clients (never an internal asyncio.to_thread), test_cases, and
test_search_query if applicable. LibreFolio's service layer supplies thread isolation.

After scaffolding, implement get_current_value() and get_history_value(), add test cases, then save it as asset_source_providers/<name>/plugin.py, beside an __init__.py holding only a docstring, and restart.

🔄 Flow

The system calls provider methods in three distinct phases:

graph TD
    subgraph P1["Phase 1 — Search"]
        S1["🔎 search(query)"]
        S2["list of<br/>{identifier, name,<br/>currency, type}"]
        S1 --> S2
    end

    subgraph P2["Phase 2 — Current Price"]
        C1["💰 get_current_value(<br/>identifier, type, params)"]
        C2["FACurrentValue<br/>(price, currency, date)"]
        C1 --> C2
    end

    subgraph P3["Phase 3 — Historical Data"]
        H1["📈 get_history_value(<br/>identifier, ..., start, end)"]
        H2["FAHistoricalData<br/>(prices[], currency)"]
        H3["🔧 Core fills gaps<br/>(weekends, holidays)"]
        H1 --> H2 --> H3
    end

    P1 ~~~ P2 ~~~ P3

    style S1 fill:#e3f2fd,stroke:#1565c0
    style S2 fill:#e3f2fd,stroke:#1565c0
    style C1 fill:#e8f5e9,stroke:#2e7d32
    style C2 fill:#e8f5e9,stroke:#2e7d32
    style H1 fill:#fff3e0,stroke:#e65100
    style H2 fill:#fff3e0,stroke:#e65100
    style H3 fill:#f3e5f5,stroke:#7b1fa2

Phase 1 is optional but strongly recommended — enables users to discover and link assets without knowing the exact identifier. See Asset Search below.

Phase 2 fetches the latest price for the dashboard or manual refresh.

Phase 3 fetches historical OHLCV data. The plugin returns only actual trading days — core fills gaps (weekends, holidays) with last known value.

Plugin responsibility: Fetch raw price data from external source. Return only actual data points (trading days). Core responsibility: Gap filling (weekends/holidays → backward_filled=True), caching, database storage, currency conversion.

🧵 Provider Execution Boundary

Provider contracts are asynchronous, but the service layer treats provider implementations as an isolation boundary. Every provider I/O operation is invoked through the canonical asset_sources.core._run_provider_in_thread() helper. The helper starts a dedicated worker thread, creates a new event loop in that thread, runs the provider coroutine to completion, and enforces the caller's timeout.

Consequently, a provider that wraps a synchronous SDK or HTTP client should call that client directly inside its async def method:

async def get_current_value(self, identifier, identifier_type, provider_params):
    raw = self._sync_client.fetch(identifier)  # Direct blocking call in provider thread
    return FACurrentValue(...)

Do not add asyncio.to_thread() or run_in_executor() inside the provider. Nesting another thread boundary obscures timeout/error behavior and is unnecessary: the manager operations, metadata refresh, current/history refresh, search, and URL resolution already route provider I/O through _run_provider_in_thread(). Async-native provider clients also work because their coroutines run on the dedicated thread's event loop.

This rule applies to provider I/O only. Ordinary endpoints and service functions still must not perform blocking I/O directly on the application's main event loop.

📋 ABC Methods

✅ Required (Abstract)

Method	Signature	Description
`provider_code`	`@property → str`	Unique identifier (e.g., `"yfinance"`)
`provider_name`	`@property → str`	Display name (e.g., `"Yahoo Finance"`)
`test_cases`	`@property → list[dict]`	Test cases for automated testing (`identifier`, `identifier_type`, `provider_params`)
`test_search_query`	`@property → str \\| None`	Search query for automated tests. `None` if search not supported.
`get_current_value(identifier, type, params)`	`async → FACurrentValue`	Fetch latest price. Returns `value`, `currency`, `as_of_date`, `source`.
`get_history_value(identifier, type, params, start, end)`	`async → FAHistoricalData`	Fetch historical OHLCV data for date range. Return raw data only — no gap filling.
`validate_params(params)`	`→ None`	Validate provider-specific `provider_params`. Raise `AssetSourceError` if invalid. Use `pass` if no params needed.

💪 Strongly Recommended (Override)

Implementing search() is strongly recommended

Without search, users must know the exact asset identifier (ticker, ISIN, URL) upfront. With search, they can type a name like "Apple" and pick from results.

Method	Default	Description
`search(query)`	Raises `NOT_SUPPORTED`	Search for assets by name, ticker, or ISIN. Returns `[{identifier, identifier_type, display_name, currency, type}]`.

🔧 Optional (Override)

Method	Default	Description
`get_icon`	`None`	Provider icon URL for the UI
`supports_history`	`True`	Set `False` for providers that only support current prices (e.g., web scrapers)
`params_schema`	`[]`	List of field definitions for `provider_params`. Used by frontend to generate dynamic forms. Each field has a `key`, a `type` (`string`, `number`, `currency`, `select` with `options` / `option_labels`, or `ui_component` for a custom editor), `required` (shown with a red `*`), and optionally `default`, `placeholder`, a short `label` (the caption) and a `description` (the tooltip). The caption is the translation `assets.providerParams.<code>.label.<key>`, else `label`, else the raw `key` — declare a `label` so users do not see internal names; the tooltip is `assets.providerParams.<code>.description.<key>`, else `description`. Both translations live in the provider's own fragment: see Plugin Folder and Translations.
`get_asset_url(identifier, type, params)`	`None`	Generate URL to the provider's page for this asset (e.g., Yahoo Finance quote page)
`resolvable_url_domains`	`[]`	Domains this provider can turn a page URL back into a search-item for (opt-in). See Asset Search & Link-Finder.
`resolve_url(url)`	`None`	Inverse of `get_asset_url`: open a provider page URL → return one search-item dict, a list of dicts, or `None`. Enables the last-resort external search stack.
`accepted_identifier_types`	`[TICKER, ISIN]`	Input types accepted by this provider (shown in frontend dropdown)
`fetch_asset_metadata(identifier, type, params)`	`None`	Fetch asset metadata (type, sector, identifiers) from the provider
`provider_help_url`	`None`	URL to the provider's documentation page (served by the running instance)
`shutdown()`	no-op	Called at app teardown to clean up resources (close sessions, stop threads). Override if your provider holds persistent connections.
`generate_static_url(path)`	—	Helper to build `/api/v1/uploads/plugin/asset/{path}`

🔎 Asset Search

The search(query) method allows users to discover assets by name, ticker, or ISIN across all providers simultaneously.

⚙️ How It Works

User types a query in the UI (e.g., "Apple", "MSCI World", "IE00B4L5Y983")
Frontend calls GET /api/v1/assets/provider/search?q=Apple
Backend queries all providers in parallel via AssetSearchService.search() (asyncio.gather)
Providers that don't implement search (default raises NOT_SUPPORTED) are silently skipped
Results are aggregated and returned to the user

🔌 Provider Support

Provider	`search()`	`test_search_query`	`get_asset_url`	Notes
Yahoo Finance	✅	`"Apple"`	✅	Full ticker search with the core query cache (15 min TTL)
JustETF	✅	`"iShares Core S&P 500"`	✅	ISIN-based search across cached ETF list
CSS Scraper	❌	`None`	✅	No search — URL must be provided manually
Scheduled Investment	❌	`None`	—	Synthetic provider, no external search
Borsa Italiana	✅	`"ENEL"`	✅	Full search on borsaitaliana.it; funds price by `provider_params.codice_fondo`; `resolve_url` emits IT + EN canonical rows

supports_search detection

The list_providers endpoint checks instance.test_search_query is not None (a local property) to determine search support. This avoids cold-start HTTP calls.

🧵 Last-resort external fallback (optional)

A provider can go beyond its own on-site search. When search() returns 0 results, the orchestration can fall back to an external engine (web_link_finder, ddgs by default) to find candidate provider-domain URLs and then resolve them back into search-items. To opt in, a provider sets resolvable_url_domains and implements resolve_url(url) (see the Optional overrides list). The caller may also pass hints (extra identifiers/names, e.g. from a broker report) to narrow the external query and post-filter the results by known identifier.

resolve_url is an alternate entry point into interactive search, not into pricing. It may return:

one search-item dict;
a list of search-item dicts when one page maps to several canonical choices;
None when the URL is not recognised.

The search layer flattens lists and de-duplicates by (identifier, language). Borsa Italiana uses this to return the full IT + EN canonical set for a fund or scheda page.

This whole stack — the web_link_finder module, the two-stage "stringone", and the identifier post-filter — is documented on its own page:

🔎 Asset Search & Link-Finder — the generic three-layer resolution (on-site → link-finder → resolve_url), config, and invariants.

Never on price fetches

The external stack is interactive-only. Once an asset exists it must be priced by its stored identifier / provider_params, never by re-searching. For Borsa Italiana funds, the stored provider_params.codice_fondo is the price key; the ISIN is descriptive/identity data, not the NAV lookup key.

🧾 Alternative identifiers

Asset.identifier_other is a JSON list (list[str]) of soft or technical identifiers. Use it for alternate broker labels and provider-specific codes that do not deserve a first-class identifier column. Provider metadata patches should return a list, not a scalar; for example, Borsa Italiana fund metadata stores the internal fund code as identifier_other=["2FADB…"] while pricing uses provider_params.codice_fondo.

🌐 API Endpoints

Standard search (returns all results at once):

GET /api/v1/assets/provider/search?q=Apple&providers=yfinance,justetf

Streaming search (Server-Sent Events — results arrive as providers respond):

GET /api/v1/assets/provider/search/stream?q=Apple&providers=yfinance,justetf

Query parameters (both endpoints):

q (required): Search query, min 1 character
providers (optional): Comma-separated provider codes. Default: all providers with search support.

Response (standard):

{
  "query": "Apple",
  "total_results": 5,
  "results": [
    {
      "identifier": "AAPL",
      "display_name": "Apple Inc.",
      "provider_code": "yfinance",
      "currency": "USD",
      "asset_type": "stock",
      "provider_url": "https://finance.yahoo.com/quote/AAPL"
    }
  ],
  "providers_queried": ["yfinance", "justetf"],
  "providers_with_errors": []
}

Searches are executed in parallel — one slow provider won't block others
Provider-specific errors are logged but don't fail the entire request
Errors are reported in providers_with_errors for debugging
provider_url is generated by the provider's get_asset_url() method (if implemented)

🧪 Probe Endpoint

After a provider is configured, use the probe endpoint to dry-run test it:

POST /api/v1/assets/provider/probe

This runs selectable operations (current_price, history, metadata) without persisting anything. See Asset Architecture for details.

💻 Implementation Example

# backend/app/services/asset_source_providers/my_provider/plugin.py

from datetime import date
from decimal import Decimal
from backend.app.db import IdentifierType
from backend.app.db.models import ProviderInputType
from backend.app.services.asset_sources.core import AssetSourceError, AssetSourceProvider
from backend.app.services.provider_registry import register_provider, AssetProviderRegistry
from backend.app.schemas.assets import FACurrentValue, FAHistoricalData, FAPricePoint

@register_provider(AssetProviderRegistry)
class MyProvider(AssetSourceProvider):

    @property
    def provider_code(self) -> str:
        return "my_provider"

    @property
    def provider_name(self) -> str:
        return "My Data Provider"

    @property
    def test_cases(self) -> list[dict]:
        return [
            {"identifier": "AAPL", "identifier_type": IdentifierType.TICKER, "provider_params": None}
        ]

    @property
    def test_search_query(self) -> str | None:
        return "Apple"  # Return None if search not supported

    @property
    def params_schema(self) -> list[dict]:
        """Optional: define fields for frontend dynamic form."""
        return [
            {"key": "api_key", "type": "string", "required": True, "description": "API key for authentication"},
        ]

    def get_asset_url(self, identifier, identifier_type, provider_params=None) -> str | None:
        """Optional: URL to the provider's page for this asset."""
        return f"https://myprovider.com/asset/{identifier}"

    async def get_current_value(
        self, identifier: str, identifier_type: IdentifierType, provider_params: dict
    ) -> FACurrentValue:
        # A synchronous client call stays direct; the service runs this coroutine
        # in the dedicated provider thread.
        price = self._sync_client.fetch_price(identifier)
        return FACurrentValue(
            value=Decimal(str(price)),
            currency="USD",
            as_of_date=date.today(),
            source=self.provider_name,
        )

    async def get_history_value(
        self, identifier: str, identifier_type: IdentifierType,
        provider_params: dict | None, start_date: date, end_date: date
    ) -> FAHistoricalData:
        # Return ONLY actual trading days; do not add an internal to_thread call.
        raw_data = self._sync_client.fetch_history(identifier, start_date, end_date)
        prices = [
            FAPricePoint(date=d, close=Decimal(str(p)), open=None, high=None, low=None, volume=None, currency="USD")
            for d, p in raw_data
        ]
        return FAHistoricalData(prices=prices, currency="USD", source=self.provider_name)

    def validate_params(self, params: dict | None) -> None:
        # No params required — accept anything
        pass

🗂️ Plugin Folder and Translations

A provider is a folder of asset_source_providers/:

backend/app/services/asset_source_providers/
├── static/            ← the family's icons, served under /api/v1/uploads/plugin/asset/… (generate_static_url)
└── my_provider/
    ├── __init__.py    ← its docstring and nothing else
    ├── plugin.py      ← the code above
    └── i18n/          ← optional: en.json, it.json, fr.json, es.json

Restart the app: AssetProviderRegistry loads plugin.py as backend.app.services.asset_source_providers.my_provider.plugin. Import and patch the provider's names from that module: the package re-exports nothing. A helper only this provider uses sits beside plugin.py as _*.py. Discovery, the layout and the gate that keeps it (./dev.py test utils gate-plugin-layout) are in Plugin Folder Layout.

Translations. A provider's space is assets.providerParams.<provider_code>: the code provider_code returns, which need not be the folder's name — the yahoo_finance/ folder registers yfinance. Its fragment holds the captions and tooltips of its params_schema fields, label.<key> and description.<key>; Borsa Italiana's (borsa_italiana/i18n/) is the only one today. Add the keys in the four languages, naming the folder with --into:

./dev.py i18n add "assets.providerParams.my_provider.label.api_key" --en "API key" --it "Chiave API" --fr "Clé API" --es "Clave API" --into asset_source_providers/my_provider
./dev.py i18n add "assets.providerParams.my_provider.description.api_key" --en "API key for authentication" --it "…" --fr "…" --es "…" --into asset_source_providers/my_provider
./dev.py i18n where "assets.providerParams.my_provider.label.api_key"   # the file that holds the key

Without --into, add refuses a new provider's first key, because the keys under assets.providerParams belong to plugins only. ./dev.py test utils gate-plugin-i18n-ownership then keeps every key under the provider's space in its fragment (A Plugin's Translations).

🏗️ Asset Architecture — Provider interface, caching, refresh logic
🔌 Asset Providers — All providers (Yahoo Finance, JustETF, CSS Scraper, Scheduled Investment)
📦 Providers Overview — Summary table of all providers
🔄 Registry Pattern Overview — How the plugin system works