macrosynergy.download.jpmaqs#

JPMaQS Download Interface

deconstruct_expression(expression)[source]#

Deconstruct an expression into a list of cid, xcat, and metric. Achieves the inverse of construct_expressions(). For non-JPMaQS expressions, the returned list will be [expression, expression, ‘value’]. The metric is set to ‘value’ to ensure the reported metric is consistent with the standard JPMaQS metrics (JPMaQSDownload.valid_metrics).

Parameters:

expression (Union[str, List[str]]) – expression to deconstruct. If a list is provided, each element will be deconstructed and returned as a list of lists.

Return <list[str]>:

list of cid, xcat, and metric.

Raises:

  • TypeError – if expression is not a string or a list of strings.

  • ValueError – if expression is an empty list.

Return type:

Union[List[str], List[List[str]]]

construct_expressions(tickers=None, cids=None, xcats=None, metrics=None)[source]#

Construct expressions from the provided arguments.

Parameters:
Return <list[str]>:

list of expressions.

Return type:

List[str]

get_expression_from_qdf(df)[source]#
Return type:

List[str]

get_expression_from_wide_df(df)[source]#
Return type:

List[str]

timeseries_to_qdf(timeseries)[source]#

Converts a dictionary of time series to a QuantamentalDataFrame.

Parameters:

timeseries (Dict[str, Any]) – A dictionary of time series.

Return <QuantamentalDataFrame>:

The converted DataFrame.

Return type:

QuantamentalDataFrame

concat_single_metric_qdfs(df_list, errors='ignore')[source]#

Combines a list of Quantamental DataFrames into a single DataFrame.

Parameters:

  • df_list (List[QuantamentalDataFrame]) – A list of Quantamental DataFrames.

  • errors (str) – The error handling method to use. If ‘raise’, then invalid items in the list will raise an error. If ‘ignore’, then invalid items will be ignored. Default is ‘ignore’.

Return <QuantamentalDataFrame>:

The combined DataFrame.

Return type:

QuantamentalDataFrame

timeseries_to_column(timeseries, errors='ignore')[source]#

Converts a dictionary of time series to a DataFrame with a single column.

Parameters:

  • timeseries (Dict[str, Any]) – A dictionary of time series.

  • errors (str) – The error handling method to use. If ‘raise’, then invalid items in the list will raise an error. If ‘ignore’, then invalid items will be ignored. Default is ‘ignore’.

Return <pd.DataFrame>:

The converted DataFrame.

Return type:

DataFrame

concat_column_dfs(df_list, errors='ignore')[source]#

Concatenates a list of DataFrames into a single DataFrame.

Parameters:

  • df_list (List[DataFrame]) – A list of DataFrames.

  • errors (str) – The error handling method to use. If ‘raise’, then invalid items in the list will raise an error. If ‘ignore’, then invalid items will be ignored. Default is ‘ignore’.

Return <pd.DataFrame>:

The concatenated DataFrame.

Return type:

DataFrame

validate_downloaded_df(data_df, expected_expressions, found_expressions, start_date=None, end_date=None, verbose=True)[source]#

Validate the downloaded data in the provided dataframe.

Parameters:

  • data_df (DataFrame) – dataframe containing the downloaded data.

  • expected_expressions (List[str]) – list of expressions that were expected to be downloaded.

  • found_expressions (List[str]) – list of expressions that were actually downloaded.

  • start_date (Optional[str]) – start date of the downloaded data.

  • end_date (Optional[str]) – end date of the downloaded data.

  • verbose (bool) – whether to print the validation results.

Return <bool>:

True if the downloaded data is valid, False otherwise.

Raises:

<TypeError> – if data_df is not a dataframe.

Return type:

bool

class JPMaQSDownload(oauth=True, client_id=None, client_secret=None, crt=None, key=None, username=None, password=None, check_connection=True, proxy=None, suppress_warning=True, debug=False, print_debug_data=False, dq_download_kwargs={}, *args, **kwargs)[source]#

Bases: DataQueryInterface

JPMaQSDownload Object. This object is used to download JPMaQS data via the DataQuery API. It can be extended to include the use of proxies, and even request generic DataQuery expressions.

Parameters:

  • oauth (bool) – True if using oauth, False if using username/password with crt/key.

  • client_id (Optional[str]) – oauth client_id, required if oauth=True.

  • client_secret (Optional[str]) – oauth client_secret, required if oauth=True.

  • crt (Optional[str]) – path to crt file.

  • key (Optional[str]) – path to key file.

  • username (Optional[str]) – username for certificate based authentication.

  • password (Optional[str]) – paired with username for certificate

  • debug (bool) – True if debug mode, False if not.

  • suppress_warning (bool) – True if suppressing warnings, False if not.

  • check_connection (bool) – True if the interface should check the connection to the server before sending requests, False if not. False by default.

  • proxy (Optional[Dict]) – proxy to use for requests, None if not using proxy (default).

  • print_debug_data (bool) – True if debug data should be printed, False if not (default).

  • dq_kwargs – additional arguments to pass to the DataQuery API object such calender and frequency for the DataQuery API. For more fine-grained usage, initialize the DataQueryInterface object explicitly.

  • kwargs – any other keyword arguments.

Return <JPMaQSDownload>:

JPMaQSDownload object

Raises:

  • <TypeError> – if provided arguments are not of the correct type.

  • <ValueError> – if provided arguments are invalid or semantically incorrect.

validate_download_args(tickers, cids, xcats, metrics, start_date, end_date, get_catalogue, expressions, show_progress, as_dataframe, dataframe_format, report_time_taken)[source]#

Validate the arguments passed to the download function.

Params:

– see macrosynergy.download.jpmaqs.JPMaQSDownload.download() :params: – see macrosynergy.download.jpmaqs.JPMaQSDownload.download().

Return <bool>:

True if valid.

Raises:

  • <TypeError> – If any of the arguments are not of the correct type.

  • <ValueError> – If any of the arguments are semantically incorrect.

Return type:

bool

filter_expressions_from_catalogue(expressions, verbose=True)[source]#

Method to filter a list of expressions against the JPMaQS catalogue. This avoids requesting data for expressions that are not in the catalogue, and provides the user wuth the complete list of expressions that are in the catalogue.

Parameters:

tickers – list of tickers to filter.

Return <List[str]>:

list of tickers that are in the JPMaQS catalogue.

Return type:

List[str]

download_data(*args, **kwargs)[source]#

Download data from the DataQuery API.

Parameters:

  • expressions – list of expressions to download.

  • start_date – start date for the data in the ISO-8601 format (YYYY-MM-DD).

  • end_date – end date for the data in the ISO-8601 format (YYYY-MM-DD).

  • show_progress – whether to show a progress bar for the download.

  • endpoint – endpoint to use for the download.

  • calender – calendar setting to use for the download.

  • frequency – frequency of data points to use for the download.

  • conversion – conversion setting to use for the download.

  • nan_treatment – NaN treatment setting to use for the download.

  • reference_data – reference data to pass to the API kwargs.

  • retry_counter – number of times the download has been retried.

  • delay_param – delay between requests to the API.

Return <List[Dict]>:

list of dictionaries containing the response data.

Raises:

  • <ValueError> – if any arguments are invalid or semantically incorrect (see validate_download_args()).

  • <DownloadError> – if the download fails.

  • <ConnectionError(HeartbeatError)> – if the heartbeat fails.

  • <Exception> – other exceptions may be raised by underlying functions.

download_all_to_disk(path, expressions=None, as_dataframe=True, dataframe_format='qdf', show_progress=True, delay_param=0.2, batch_size=None, retry=3, overwrite=True, *args, **kwargs)[source]#

Downloads all JPMaQS data to disk.

Return type:

None

download(tickers=None, cids=None, xcats=None, metrics=['value'], start_date='2000-01-01', end_date=None, expressions=None, get_catalogue=False, show_progress=False, debug=False, suppress_warning=False, as_dataframe=True, dataframe_format='qdf', report_time_taken=False, *args, **kwargs)[source]#

Driver function to download data from JPMaQS via the DataQuery API. Timeseries data can be requested using tickers with metrics, or passing formed DataQuery expressions. cids and xcats (along with metrics) are used to construct expressions, which are ultimately passed to the DataQuery Interface.

Parameters:

  • tickers (Optional[List[str]]) – list of tickers.

  • cids (Optional[List[str]]) – list of cids.

  • xcats (Optional[List[str]]) – list of xcats.

  • metrics (List[str]) – list of metrics, one of “value” (default), “grading”, “eop_lag”, “mop_lag”. “all” is also accepted.

  • start_date (str) – start date of the data to download, in the ISO format - YYYY-MM-DD.

  • end_date (Optional[str]) – end date of the data to download in the ISO format - YYYY-MM-DD.

  • expressions (Optional[List[str]]) – list of DataQuery expressions.

  • get_catalogue (bool) – If True, the JPMaQS catalogue is downloaded and used to filter the list of tickers. Default is False.

  • show_progress (bool) – True if progress bar should be shown, False if not (default).

  • suppress_warning (bool) – True if suppressing warnings. Default is True.

  • debug (bool) – Override the debug behaviour of the JPMaQSDownload class. If True, debug mode is enabled.

  • print_debug_data – True if debug data should be printed, False if not (default). If debug=True, this is set to True.

  • as_dataframe (bool) – Return a dataframe if True (default), a list of dictionaries if False.

  • dataframe_format (str) – Format of the dataframe to return, one of “qdf” or “wide”. QDF is the Quantamental Dataframe format, and wide is the wide format with each expression as a column, and a single date column.

  • report_time_taken (bool) – If True, the time taken to download and apply data transformations is reported.

Return <pd.DataFrame|list[Dict]>:

dataframe of data if as_dataframe is True, list of dictionaries if False.

Raises:

<ValueError> – if provided arguments are invalid or semantically incorrect (see macrosynergy.download.jpmaqs.JPMaQSDownload.validate_download_args()).

Return type:

Union[DataFrame, List[Dict]]