macrosynergy.download.dataquery#

Interface for downloading data from the JPMorgan DataQuery API. This module is not intended to be used directly, but rather through macrosynergy.download.jpmaqs.py. However, for a use cases independent of JPMaQS, this module can be used directly to download data from the JPMorgan DataQuery API.

validate_response(response, user_id)[source]#

Validates a response from the API. Raises an exception if the response is invalid (e.g. if the response is not a 200 status code).

Parameters:

response (Response) – response object from requests.request().

Return <dict>:

response as a dictionary. If the response is not valid, this function will raise an exception.

Raises:

  • <InvalidResponseError> – if the response is not valid.

  • <AuthenticationError> – if the response is a 401 status code.

  • <KeyboardInterrupt> – if the user interrupts the download.

Return type:

dict

request_wrapper(url, headers=None, params=None, method='get', tracking_id=None, proxy=None, cert=None, **kwargs)[source]#

Wrapper for requests.request() that handles retries and logging. All parameters and kwargs are passed to requests.request().

Parameters:

  • url (str) – URL to request.

  • headers (Optional[Dict]) – headers to pass to requests.request().

  • params (Optional[Dict]) – params to pass to requests.request().

  • method (str) – HTTP method to use. Must be one of “get” or “post”. Defaults to “get”.

  • kwargs – kwargs to pass to requests.request().

  • tracking_id (Optional[str]) – default None, unique tracking ID of request.

  • proxy (Optional[Dict]) – default None, dictionary of proxy settings for request.

  • cert (Optional[Tuple[str, str]]) – default None, tuple of string for filename of certificate and key.

Return <dict>:

response as a dictionary.

Raises:

  • <InvalidResponseError> – if the response is not valid.

  • <AuthenticationError> – if the response is a 401 status code.

  • <DownloadError> – if the request fails after retrying.

  • <KeyboardInterrupt> – if the user interrupts the download.

  • <ValueError> – if the method is not one of “get” or “post”.

  • <Exception> – other exceptions may be raised by requests.request().

Return type:

dict

class OAuth(client_id, client_secret, proxy=None, token_url='https://authe.jpmchase.com/as/token.oauth2', dq_resource_id='JPMC:URI:RS-06785-DataQueryExternalApi-PROD')[source]#

Bases: object

Class for handling OAuth authentication for the DataQuery API.

Parameters:

  • client_id (str) – client ID for the OAuth application.

  • client_secret (str) – client secret for the OAuth application.

  • proxy (Optional[dict]) – proxy to use for requests. Defaults to None.

  • token_url (str) – URL for getting OAuth tokens.

  • dq_resource_id (str) – resource ID for the JPMaQS Application.

Return <OAuth>:

OAuth object.

Raises:

  • <ValueError> – if any of the parameters are semantically incorrect.

  • <TypeError> – if any of the parameters are of the wrong type.

  • <Exception> – other exceptions may be raised by underlying functions.

get_auth()[source]#

Returns a dictionary with the authentication information, in the same format as the macrosynergy.download.dataquery.CertAuth.get_auth() method.

Return type:

Dict[str, Union[str, Tuple[str, str], None]]

class CertAuth(username, password, crt, key, proxy=None)[source]#

Bases: object

Class for handling certificate based authentication for the DataQuery API.

Parameters:

  • username (str) – username for the DataQuery API.

  • password (str) – password for the DataQuery API.

  • crt (str) – path to the certificate file.

  • key (str) – path to the key file.

Return <CertAuth>:

CertAuth object.

Raises:

  • <AssertionError> – if any of the parameters are of the wrong type.

  • <FileNotFoundError> – if certificate or key file is missing from filesystem.

  • <Exception> – other exceptions may be raised by underlying functions.

get_auth()[source]#

Returns a dictionary with the authentication information, in the same format as the macrosynergy.download.dataquery.OAuth.get_auth() method.

Return type:

Dict[str, Union[str, Tuple[str, str], None]]

validate_download_args(expressions, start_date, end_date, show_progress, endpoint, calender, frequency, conversion, nan_treatment, reference_data, retry_counter, delay_param, batch_size)[source]#

Validate the arguments passed to the download_data() method.

:params : – see download_data() method :params : – see download_data() method.

Return <bool>:

True if all arguments are valid.

Raises:

  • <TypeError> – if any of the arguments are of the wrong type.

  • <ValueError> – if any of the arguments are semantically incorrect.

class DataQueryInterface(client_id=None, client_secret=None, crt=None, key=None, username=None, password=None, proxy=None, oauth=True, debug=False, batch_size=20, check_connection=True, base_url='https://api-developer.jpmorgan.com/research/dataquery-authe/api/v2', token_url='https://authe.jpmchase.com/as/token.oauth2', suppress_warning=True, custom_auth=None)[source]#

Bases: object

High level interface for the DataQuery API.

When using OAuth authentication:

Parameters:

  • client_id (Optional[str]) – client ID for the OAuth application.

  • client_secret (Optional[str]) – client secret for the OAuth application.

When using certificate authentication:

Parameters:

  • crt (Optional[str]) – path to the certificate file.

  • key (Optional[str]) – path to the key file.

  • username (Optional[str]) – username for the DataQuery API.

  • password (Optional[str]) – password for the DataQuery API.

  • oauth (bool) – whether to use OAuth authentication. Defaults to True.

  • debug (bool) – whether to print debug messages. Defaults to False.

  • concurrent – whether to use concurrent requests. Defaults to True.

  • batch_size (int) – default 20, number of expressions to send in a single request. Must be a number between 1 and 20 (both included).

  • check_connection (bool) – whether to send a check_connection request. Defaults to True.

  • base_url (str) – base URL for the DataQuery API. Defaults to OAUTH_BASE_URL if oauth is True, CERT_BASE_URL otherwise.

  • token_url (str) – token URL for the DataQuery API. Defaults to OAUTH_TOKEN_URL.

  • suppress_warnings – whether to suppress warnings. Defaults to True.

  • custom_auth – custom authentication object. When specified oauth must be False :param custom_auth: custom authentication object. When specified oauth must be False and the object must have a get_auth method. Defaults to None.

Return <DataQueryInterface>:

DataQueryInterface object.

Raises:

  • <TypeError> – if any of the parameters are of the wrong type.

  • <ValueError> – if any of the parameters are semantically incorrect.

  • <InvalidResponseError> – if the response from the server is not valid.

  • <DownloadError> – if the download fails to complete after a number of retries.

  • <HeartbeatError> – if the heartbeat (check connection) fails.

  • <Exception> – other exceptions may be raised by underlying functions.

check_connection(verbose=False, raise_error=False)[source]#

Check the connection to the DataQuery API using the Heartbeat endpoint.

Parameters:

verbose – whether to print a message if the heartbeat is successful. Useful for debugging. Defaults to False.

Return <bool>:

True if the connection is successful, False otherwise.

Raises:

<HeartbeatError> – if the heartbeat fails.

Return type:

bool

get_catalogue(group_id='JPMAQS', page_size=1000, verbose=True)[source]#

Method to get the JPMaQS catalogue. Queries the DataQuery API’s Groups/Search endpoint to get the list of tickers in the JPMaQS group. The group ID can be changed to fetch a different group’s catalogue.

Parameters:

  • group_id (str) – the group ID to fetch the catalogue for. Defaults to “JPMAQS”.

  • page_size (int) – the number of tickers to fetch in a single request. Defaults to 1000 (maximum allowed by the API).

Return <List[str]>:

list of tickers in the requested group.

Raises:

<ValueError> – if the response from the server is not valid.

Return type:

List[str]

download_data(expressions, start_date='2000-01-01', end_date=None, show_progress=False, endpoint='/expressions/time-series', calender='CAL_ALLDAYS', frequency='FREQ_DAY', conversion='CONV_LASTBUS_ABS', nan_treatment='NA_NOTHING', reference_data='NO_REFERENCE_DATA', retry_counter=0, delay_param=0.2, batch_size=None, *args, **kwargs)[source]#

Download data from the DataQuery API.

Parameters:

  • expressions (List[str]) – list of expressions to download.

  • start_date (str) – start date for the data in the ISO-8601 format (YYYY-MM-DD).

  • end_date (str) – end date for the data in the ISO-8601 format (YYYY-MM-DD).

  • show_progress (bool) – whether to show a progress bar for the download.

  • endpoint (str) – endpoint to use for the download.

  • calender (str) – calendar setting to use for the download.

  • frequency (str) – frequency of data points to use for the download.

  • conversion (str) – conversion setting to use for the download.

  • nan_treatment (str) – NaN treatment setting to use for the download.

  • reference_data (str) – reference data to pass to the API kwargs.

  • retry_counter (int) – number of times the download has been retried.

  • delay_param (float) – delay between requests to the API.

Return <List[Dict]>:

list of dictionaries containing the response data.

Raises:

  • <ValueError> – if any arguments are invalid or semantically incorrect (see validate_download_args()).

  • <DownloadError> – if the download fails.

  • <ConnectionError(HeartbeatError)> – if the heartbeat fails.

  • <Exception> – other exceptions may be raised by underlying functions.

Return type:

List[Dict]