Vai al contenuto

🇮🇹 Borsa Italiana Provider (borsa_italiana)

The Borsa Italiana provider fetches financial data from borsaitaliana.it using the borsa-italiana-scraping library. It supports stocks, bonds (including BTP), ETFs listed on Borsa Italiana markets (MTA, MOT, ETFPlus), and funds/SICAVs exposed on Borsa fund-detail pages.

⚙️ How it Works

  1. Identifier: An ISIN code (e.g., IT0003128367 for ENEL S.p.A.) for listed instruments.
  2. Identifier Types: ISIN for listed instruments. Mutual funds are priced by a Borsa internal code carried in provider_params.codice_fondo (see Mutual funds below); the asset identifier is still the real ISIN when the fund page exposes one, otherwise an OTHER identifier is used.
  3. provider_params:
    • language — optional ("en" or "it", default "en"). Controls the language of asset names, metadata descriptions, and the provider URL.
    • codice_fondo — optional Borsa internal fund code (e.g. 2FADB602822). When present, current/historical value use the fund NAV path instead of the market API.
    • mic — optional market MIC (e.g. ETLX for EuroTLX, MOTX for MOT). Auto-filled from the search result's own link; routes the instrument page (get_asset_url) and metadata to the right market.
    • platform — optional trading platform (e.g. TLX for EuroTLX). Required by some markets so the universal search/scheda.html URL redirects to the real market page.
    • url — not a form field (absent from params_schema): set automatically to the instrument's own page address — the site's search link, or the page resolve_url actually loaded (scheda.url_pagina). get_asset_url and metadata prefer it, so a EuroTLX link keeps working even when platform is missing (e.g. an asset created while the site search had the ISIN de-indexed).

🧭 Market routing (mic / platform)

Borsa Italiana lists the same instrument families across several markets (MTA, MOT, ExtraMOT, ETFplus, SeDeX, MIV, EuroTLX, GEM). Most resolve through the universal search/scheda.html?code={ISIN} URL, but EuroTLX does not — it requires mic=ETLX&platform=TLX or the URL stays on a dead generic page. The provider takes mic/platform from the site's own search link (never a hardcoded map), stores them in provider_params, and reuses them everywhere:

  • Search results carry provider_params: {language, mic, platform?, url} (url = the site's own link) — so the URL and pricing are correct by construction for any market, present or future.
  • Dead-result filter: a non-fund result with no parseable mic in its link is skipped (never emit a result whose URL would be the unresolved /search/ page). Indices (Indice/Index) are excluded — they are benchmarks, not purchasable instruments.
  • History auto-discovery (library ≥ 0.3.0): the chart API only knows XMIL/ETLX; if an ISIN is unknown to XMIL and no exchange is passed, the library discovers the MIC via the site search and retries. This self-heals legacy assets saved before mic was propagated.
  • UNSUPPORTED_PAGE: when even mic/platform can't resolve a parseable market page (a family we don't handle yet), the provider raises UNSUPPORTED_PAGE with a message inviting the user to open a GitHub issue — instead of a cryptic parse error.

💱 Currency

A listed instrument is priced in its trading currency on Borsa Italiana: EUR for most listings, but e.g. USD for FX-denominated EuroTLX bonds such as the US Treasury US912810TU25. One source serves both sides: the metadata currency is the currency the prices carry.

  • Prices (get_current_value, get_history_value): the currency of the grafici.borsaitaliana.it chart-API reply (ottieni_prezzo_corrente / ottieni_storico), EUR when the reply carries none. History goes through _storico_con_mic_retry: default route first, then, on StrumentoNonTrovato, the stored mic as the chart-API exchange.
  • Metadata and resolve_url (listed instruments): the currency is not read from the scheda page. ETF/ETC scheda pages often show only the fund's denomination currency («Valuta di Denominazione USD»), while the instrument trades — and is priced — in EUR; read off the scheda, it gave USD metadata over EUR prices (e.g. Invesco Physical Gold ETC IE00B579F325, iShares Core MSCI World IE00B4L5Y983, iShares Core S&P 500 IE00B5BMR087). The module helper _price_api_currency(identifier, mic) asks the chart API instead: the same _storico_con_mic_retry call, the same EUR default.
  • Currency unknown: when that chart-API call fails (any exception), fetch_asset_metadata leaves currency unset in the returned FAAssetPatchItem (patch contract: present, even if None → update/blank; absent → keep), so a metadata refresh keeps the stored asset currency instead of blanking it. resolve_url rows carry currency: None.
  • Search: rows for listed instruments carry currency: None — the cerca payload has no currency; the metadata fetch fills it.
  • Funds (codice_fondo): unchanged — the currency comes from the fund page (DatiFondo.valuta, default EUR).

Note (library ≥ 0.3.2): the scheda parser no longer takes the denomination row for the trading currency — «Valuta» / «Currency» match only as an exact label — and exposes the denomination separately as SchedaStrumento.valuta_denominazione, which the plugin does not use. The fix still matters for current prices: ottieni_prezzo_corrente falls back to scraping the scheda when its chart-API read fails.

💰 Current Value (get_current_value)

  • Funds (when provider_params.codice_fondo is set): returns the fund NAV only if the published NAV is dated today. A fund NAV is published once per day with a lag, so exposing a stale NAV as the "current" value would misstate the portfolio. When the NAV date ≠ today, the provider raises NO_DATA: no price is stored for today, and valuation carries the latest observation forward through the unified price resolver (backend/app/services/financial_math/price_marks.py) — the last NAV point, or a more recent trade-derived mark (LAST_TRADE_PRICE). There is no separate last-buy fallback.
  • Listed instruments: uses ottieni_prezzo_corrente(isin, mic=…, platform=…) from the scraping library. If it raises StrumentoNonRisolto / StrumentoNonTrovato and a mic is stored, it retries once with exchange=mic (EuroTLX answers under its own exchange code, equal to the MIC).
  • Returns FACurrentValue with price, date, currency, and source ("Borsa Italiana").

📈 Historical Data (get_history_value)

  • Funds (when provider_params.codice_fondo is set): the fund page exposes only the latest NAV (no series), dated to the day it actually refers to (never today). The provider returns a single price point at that real NAV date; the core backward-fills the gaps up to the requested range. Out-of-range NAV dates yield no points.
  • Listed instruments: uses ottieni_storico(isin, periodo) from the grafici.borsaitaliana.it JSON API.
    • Returns full OHLCV data (open, high, low, close, volume) for each trading day.
    • Period selection: The API uses fixed windows (1M, 3M, 6M, 1Y, 3Y, 5Y, MAX). The provider automatically selects the smallest period that covers the requested start_date..end_date range.
    • Results are filtered in-memory to the exact requested date range.
    • The core handles gap filling (weekends, holidays) — the plugin returns only actual trading days.

🧾 Mutual funds (NAV by internal code)

Mutual funds / SICAVs are not on the XMIL market API, so ottieni_storico / ottieni_prezzo_corrente cannot price them. Their only public NAV source is the fund detail page /borsa/fondi/dettaglio/{code}.html, addressed by a Borsa internal code (2FADB…) that is not the ISIN. The scraping library exposes:

  • ottieni_dati_fondo(codice, sessione) — fetch NAV, currency, NAV date (dd/mm/yy), name, and the real ISIN extracted from the page (DatiFondo). DatiFondo also carries the descriptive sections scraped from the page — caratteristiche, societa_gestione, costi (each a label → value dict, N.D. placeholders dropped) — consumed by fetch_asset_metadata to enrich the fund description.
  • ottieni_dati_fondo_da_url(url, sessione) / estrai_codice_da_url(url) — the URL variant used by resolve_url.

The provider captures the internal code into provider_params.codice_fondo at asset creation (from search or resolve_url); pricing afterwards uses that stored code with no external search. Price fetches never hit any web search — that invariant is what keeps automated syncs deterministic.

  • Uses cerca(query) — the internal JSON search engine of borsaitaliana.it.
  • Searches across all instrument types: stocks, bonds, ETFs, ETC/ETN, and funds.
  • Fund names are indexed with Borsa abbreviations (for example Obbligaz. instead of Obbligazionaria), so the provider retries common full-word variants with abbreviated terms.
  • Funds: cerca returns the Borsa internal code in the isin field (not a real ISIN). The provider fetches each fund page once per code (in-search cache) via ottieni_dati_fondo to recover the real ISIN, sets it as the identifier (ISIN), and carries the internal code in provider_params.codice_fondo. If the page can't be fetched or no ISIN is present, it falls back to the code as an OTHER identifier — still priceable by NAV.
  • Dual-language results: emits two entries per instrument (🇮🇹 Italiano + 🇬🇧 English) with flag emojis in display_name. Each result carries provider_params: {language: "en"|"it"} so the user's selection is propagated on assignment.
  • Result ordering: results are grouped by the cerca-level identifier (the internal code for funds, the real ISIN otherwise), preserving first-seen order, and within each group Italian is emitted before English. This keeps sibling funds — whose internal codes may differ by a single letter — from interleaving across languages. Grouping also de-duplicates per (identifier, language).
  • Last-resort external fallback: when cerca returns nothing, the search orchestration calls the LibreFolio web_link_finder stack to turn the query into a candidate Borsa page URL, then resolve_urls it. The caller can pass hints (report-extracted ISIN + names) which drive a two-stage "stringone" query and an identifier post-filter that narrows the resolved siblings to the exact ISIN when it is known — all handled centrally, see Asset Search & Link-Finder. Best-effort, interactive-only, never on price fetches.

📋 Metadata (fetch_asset_metadata)

  • Listed instruments (ISIN path): uses ottieni_scheda(isin, lingua) — scrapes the instrument detail page in the configured language, routed by the stored mic/platform. If that raises StrumentoNonRisolto and provider_params.url is stored, the scheda is read straight from that page (url_diretto).
  • Extracts:
    • Name: from <h1> tag on the page, appended with language flag emoji (e.g., "ENEL S.p.A. 🇬🇧").
    • Type: mapped from instrument type field (e.g., obbligazione → BOND, azione → STOCK, etf → ETF). etc/etn also maps to ETF, so ETFs, ETCs and ETNs all arrive as plain ETF — the frontend provider-data comparison treats a stored ETF subtype (e.g. ETF_STOCK) as agreeing with it (isFamilyOnlyProposal in frontend/src/lib/utils/assetTypes.ts). Fund labels (fondi comuni, fondo chiuso, closed-end fund, …) → FUND; other obbligazione… / bond… labels fall back to BOND by prefix; unknown labels → no type.
    • Currency: from the chart API that prices the asset, not from the scheda; left unset when unknown — see Currency.
    • Description: assembled from page description, market, issuer, maturity date, coupon rate, structure, tipology, coupon frequency.
    • Ticker: if available on the page (mainly for stocks).
    • Geographic Area: inferred from issuer name via _ISSUER_TO_COUNTRY (exact match, then substring), e.g. "Republic of Italy" → ITA, "United States of America" → USA.
    • Sector: inferred from settore (stocks) or tipologia (bonds) fields. Bond tipologia goes through _TIPOLOGIA_TO_SECTOR (EN + IT labels): government paper → Government Bonds, corporate → Corporate Bonds, supranational → Financials; an unmapped tipologia yields no sector. The mapping only shapes the metadata offered: values already stored on an asset change only through the provider-data comparison.
  • Bond-specific fields available in the raw data: cedola_annua, scadenza, emittente, rendimento_lordo, struttura_bond, frequenza_cedola.
  • Funds (when provider_params.codice_fondo is set): funds are not on the XMIL scheda, so metadata is built from the fund detail page by internal code instead. The short_description is assembled from the fund name, the real ISIN, and the non-N.D. entries scraped from the page's Caratteristiche, Società di Gestione and Costi sections (e.g. … | ISIN: LU2178929613 | Classe: P | Grado di Rischio: 3 | Categoria Assogestioni: Bilanciati | Costi — Gestione: 1.3 …). The internal code is persisted in identifier_other as a JSON-list entry (["2FADB…"]). The section fields are read defensively, so metadata degrades to just name + ISIN if the installed scraping library predates the enriched DatiFondo.

🔗 get_asset_url

Precedence, first match wins:

  1. Funds with provider_params.codice_fondo → https://www.borsaitaliana.it/borsa/fondi/dettaglio/{codice_fondo}.html?lang={language}, the NAV page keyed by the internal fund code (the generic scheda page does not resolve a fund by ISIN).
  2. A stored provider_params.url → that page, with lang={language} appended when it is missing. It is a working page by construction.
  3. Otherwise → https://www.borsaitaliana.it/borsa/search/scheda.html?code={ISIN}&lang={language}, plus &mic={mic} and &platform={platform} when stored. Without them the bare URL still resolves for MTA, MOT, ETFplus, SeDeX and MIV, but not for EuroTLX.

The lang parameter follows the user's provider_params.language selection (default en).

🔁 resolve_url (inverse of get_asset_url)

The provider opts into the generic URL → search-item capability:

  • resolvable_url_domains = ["borsaitaliana.it"] → supports_url_resolution is True.
  • resolve_url(url) recognises two public page families:
    • Fund detail pages (/borsa/fondi/dettaglio/{code}.html) — fetches the page and returns the full canonical set a normal search would emit for that fund: the IT + EN pair (Italian first, with flag in display_name), each {identifier: <ISIN from page> or code, identifier_type, display_name, currency, type: "FUND", provider_params: {codice_fondo, language}}.
    • Stock / bond / ETF scheda pages (…/scheda/{ISIN}[-{MIC}].html) — returns the same IT + EN canonical set, priced by ISIN. The rows' currency comes from one chart-API call shared by every row (None if it fails), not from the scheda.

It is only a different entry point — the orchestration flattens the list and de-dupes by (identifier, language). Anything that is not a recognisable Borsa page (off-domain, no extractable fund code or ISIN path) returns None. Best-effort: fetch/parse errors return None, never raise.

This lets an externally discovered page URL (e.g. found via web_link_finder, or pasted by the user in a future UI) be turned into a ready-to-create asset with the correct pricing params.

Central reference

The link-finder is generic. This section keeps only the Borsa-Italiana-relevant summary — the full contract (three layers, hints, identifier filter, config, invariants) lives on Asset Search & Link-Finder, which is the source of truth.

backend/app/services/web_link_finder.py is a LibreFolio-level module (deliberately not inside the scraping library, so the choice of external engine stays a LibreFolio concern). It turns a free-text query/ISIN into candidate provider-domain URLs, used only at search time as a last resort when a provider's on-site search yields nothing.

  • Best-effort: any failure (rate-limit, anomaly page, network/parse error) yields [] and is never fatal — the search always completes and emits done.
  • Async-safe: sync I/O runs in asyncio.to_thread; short timeout, small TTL cache, domain-filtered hits, structured logs.
  • Pluggable engine: default DdgsEngine wraps the ddgs metasearch library (bing/brave/google/DuckDuckGo/… aggregated via backend="auto") with an ApiKeyEngine seam for a paid API (Brave/Bing/SerpAPI) later; searxng reserved for a deferred Fase B.
  • Config (all optional env vars): LIBREFOLIO_WEB_LINK_FINDER_ENABLED (1/0, default 1), _ENGINE (ddgs|apikey), _API_KEY, _DDGS_REGION (default wt-wt), _DDGS_BACKEND (default auto), _TIMEOUT (s), _MAX.

Invariant

External search is only ever hit during interactive asset search. Price fetches (frequent, automated) must never call web_link_finder — once an asset exists it is priced by its stored provider_params (a fund's codice_fondo), not by search.

🔌 Technical Details

HTTP Session

The provider maintains a shared Sessione instance across all calls:

  • WAF handling: The library manages Imperva WAF cookies automatically.
  • JWT token: Extracted from the interactive chart page (required for the grafici.borsaitaliana.it API).
  • Rate limiting: Built-in minimum pause between requests (0.5s default).
  • Shutdown: The session is closed via shutdown() at app teardown.

params_schema (Dynamic UI Form)

The provider exposes optional parameters via params_schema:

Key Type Options Default Description
language select en (🇬🇧 English), it (🇮🇹 Italiano) en Language for names and metadata
codice_fondo string — — Borsa internal fund code (e.g. 2FADB602822); when set, current/history use the fund NAV path
mic string — — Market MIC (auto-filled from search, e.g. ETLX for EuroTLX); routes the instrument page and metadata
platform string — — Trading platform (auto-filled from search, e.g. TLX for EuroTLX); needed by some markets to resolve the page

Uses option_labels for human-readable display in the frontend dropdown. Each field also carries a short label — the form caption (Language, Fund internal code, Market MIC, Platform) — and a longer description shown as its tooltip; the frontend translates both under assets.providerParams.borsa_italiana.

Error Handling

Library Exception Mapped Error Code Meaning
StrumentoNonRisolto UNSUPPORTED_PAGE Instrument found but its market page can't be resolved/read (unhandled market family) — user is invited to open a GitHub issue
StrumentoNonTrovato NOT_FOUND ISIN not recognized
DatiNonDisponibili NO_DATA No data available (market closed, delisted, or empty window for an illiquid instrument)
RicercaNonDisponibile FETCH_ERROR Search endpoint down
BorsaItalianaErrore FETCH_ERROR Generic library error

Note (library ≥ 0.3.0): an empty history window (e.g. an illiquid ExtraMOT bond over 1M) now maps to NO_DATA, never NOT_FOUND — the library distinguishes "unknown instrument" from "known but no trades in the window" via a MAX-period probe.

Dependencies

borsa-italiana-scraping @ git+https://github.com/Librefolio/borsaItaliana-scraping.git

Transitive: httpx, beautifulsoup4, lxml.

🧪 Test Cases

Identifier provider_params Description
IT0003128367 — ENEL S.p.A. (stock, MTA)
LU2178929613 {codice_fondo: "2FADB602822"} Eurizon Next 2.0 Alloc. Divers. 40 P (fund, NAV by code)
US912810TU25 {mic: "ETLX", platform: "TLX"} US T-Bond 4.375% Aug43 (EuroTLX bond, USD-denominated, issuer → USA)

Search test query: "ENEL" (listed) / "EURIZON NEXT 2.0 DIVERSIFICATO 40 P" (fund by report name). resolve_url test: https://www.borsaitaliana.it/borsa/fondi/dettaglio/2FADB602822.html → IT + EN fund search-items with ISIN LU2178929613 + codice_fondo.