Version 2 Upgrade Guide

QuantRocket version 2 is a major product upgrade from version 1, with many new features and additional data and brokers. Due to the scale of changes, there are many breaking API changes that version 1 users should be aware of. This document outlines your options for upgrading.

Overview of changes

The version 1 series was tightly coupled to Interactive Brokers (IBKR) for broker and data functions, and IBKR conids were the security IDs used throughout QuantRocket. In version 2, IBKR is one broker and one data provider among several. This has necessitated two major categories of changes:

  1. Security IDs: IBKR conids are no longer used as the security IDs, except internally to communicate with the IBKR API. In version 2, security IDs are called sids (short for Security ID) and are based on FIGI identifiers. This change allows important benefits including the ability to mix and match data from different providers, improved continuity of data across ticker changes and other corporate actions, and support for survivorship-bias free data. However, changing security IDs is a major change that touches nearly all parts of the software, including database structures.

  2. Numerous commands and functions for working with Interactive Brokers that were generically named in version 1 are renamed in version 2 to reflect their association to IBKR. For example, the function create_db is now create_ibkr_db to distinguish it from other, similar functions for other data providers. There is no longer a function simply called create_db. A full list of API changes is provided below.

In addition, underlying libraries in the Anaconda distribution have been updated, including Pandas. Library updates always create a risk of breakages in user code, especially in the case of Pandas.

A new subscription is required

Version 1 subscriptions are not valid for version 2. To use version 2 beyond the evaluation features, a version 2 subscription is required.

To view the version 2 pricing, please log in. When you are ready to purchase, please contact us first so we can apply a pro-rated credit for your version 1 subscription which will partially offset your purchase price. (This pro-ration step is not handled automatically by the website for v1 to v2 upgrades.)

How to try before you buy

You can try version 2 using the free evaluation license before deciding whether to upgrade. We recommend doing a fresh installation of version 2 onto a different computer (or cloud server) from the one running your version 1 deployment. Simply follow the standard installation instructions.

(While it is possible to run version 1 and 2 on the same local computer by specifying a different --project-name in the docker-compose command and by changing the houston port, this is more advanced and can be confusing and lead to problems if you're not careful. Therefore we don't recommend it for most users.)

Deployment strategies

If you decide to upgrade to version 2, there are several options for deploying version 2. You can start over with version 2, migrate version 1 to version 2, or run version 1 and version 2 as two separate deployments.

Option 1: Start over with v2

The easiest approach, which may be suitable for users who have not invested a substantial amount of time in version 1, is simply to start over with version 2.

For Mac and Windows users, the easiest way to completely start over is to uninstall Docker Desktop, which will delete all QuantRocket images and containers as well as any data you've collected or code you've created inside JupyterLab. (Back up anything you want to keep before doing this.) Then use the installation guide to install version 2.

Option 2: Migrate v1 to v2

The second option is to migrate your version 1 deployment to version 2. With this approach, you run a single deployment, replacing version 1 with version 2. There is a migration kit on GitHub to migrate your databases and highlight API changes that should be made in your code. The kit shows you the steps to follow to make sure you can revert back to version 1 if anything goes wrong.

To undertake this approach:

  • Ensure you are running version 1.9 or 1.8. If you are running an earlier version, please update to version 1.9 before migrating to 2.0.
  • Choose a weekend or some other time when your deployment can be "offline" (that is, not trading, collecting data, etc.) and when you can allot some time to work through the API changes.
  • On the computer running version 1, visit the software update page and follow the steps to update your software to version 2.
  • Open JupyterLab in the version 2 deployment and clone the migration kit which will guide your through the migration steps:
$ quantrocket codeload clone 'migrate-v1-to-v2'

Option 3: Separate v1 and v2 deployments

The third option is to continue running your legacy code unchanged on version 1 and run version 2 as a new, completely separate deployment.

This option may be appealing if you are running trading strategies under version 1 and are risk averse to making changes to the environment they run under. You can use version 2 for new research and strategies while leaving your old strategies unchanged.

If you are interested in this option, you can receive complimentary continued access to version 1 with your version 2 upgrade. We will keep your version 1 subscription active but disable future charges. Only your version 2 subscription will be billed. Just reach out and tell us this is what you want to do.

Note the following downsides of running two separate deployments:

  • 2 separate computers or cloud servers are required.
  • For any given IBKR login credentials, you can only run IB Gateway on one computer at a time. Thus, if you need to run IB Gateway on both your old and new deployments, you will need to stop IB Gateway on one deployment before starting it on another.
  • It may be more difficult to "merge" your two deployments later compared to an up-front migration. Each deployment's databases will accrue their own parallel history as the deployments run side-by-side. The database migration scripts in the migration kit are designed for use with a new v2 deployment, and we have not tested whether they would work for use at a later time. Therefore, if you choose option 2 but plan to eventually cease using version 1, you should probably plan on eventually "abandoning" your version 1 databases rather than hoping to eventually merge their history into your version 2 databases.

Can I keep using version 1 without upgrading?

We hope you will want to take advantage of the many new features and benefits of version 2. However, if you choose not to upgrade to version 2, you can continue using version 1 with your existing subscription.

API changes

Below are the version 1 APIs that have changed in version 2. (New version 2 functionality is not documented here.)

The migration kit will help you find any places in your code where you are using the old version 1 APIs.

Due to the volume of changes and the fact that version 2 is major new release, the usual practice of providing temporary backward compatibility for deprecated functions has been omitted. The version 1 APIs shown below have been removed and will not work in version 2.

Renamed functions

With the addition of new data and brokers, the following commands and functions have been renamed to be more specific:

Python API

Version 1Version 2
quantrocket.fundamental.collect_shortable_sharesquantrocket.fundamental.collect_ibkr_shortable_shares
quantrocket.fundamental.download_shortable_sharesquantrocket.fundamental.download_ibkr_shortable_shares
quantrocket.fundamental.get_shortable_shares_reindexed_likequantrocket.fundamental.get_ibkr_shortable_shares_reindexed_like
quantrocket.fundamental.collect_borrow_feesquantrocket.fundamental.collect_ibkr_borrow_fees
quantrocket.fundamental.download_borrow_feesquantrocket.fundamental.download_ibkr_borrow_fees
quantrocket.fundamental.get_borrow_fees_reindexed_likequantrocket.fundamental.get_ibkr_borrow_fees_reindexed_like
quantrocket.history.create_dbquantrocket.history.create_ibkr_db
quantrocket.launchpad.get_credentialsquantrocket.ibg.get_credentials
quantrocket.launchpad.set_credentialsquantrocket.ibg.set_credentials
quantrocket.launchpad.list_gateway_statusesquantrocket.ibg.list_gateway_statuses
quantrocket.launchpad.start_gatewaysquantrocket.ibg.start_gateways
quantrocket.launchpad.stop_gatewaysquantrocket.ibg.stop_gateways
quantrocket.launchpad.load_launchpad_configquantrocket.ibg.load_ibg_config
quantrocket.launchpad.get_launchpad_configquantrocket.ibg.get_ibg_config
quantrocket.master.collect_listingsquantrocket.master.collect_ibkr_listings
quantrocket.master.collect_option_chainsquantrocket.master.collect_ibkr_option_chains
quantrocket.master.diff_securitiesquantrocket.master.diff_ibkr_securities
quantrocket.master.delist_securityquantrocket.master.delist_ibkr_security
quantrocket.master.create_comboquantrocket.master.create_ibkr_combo
quantrocket.master.collect_calendarquantrocket.master.collect_ibkr_calendar
moonshot.commission.PercentageCommission.IB_COMMISSION_RATEmoonshot.commission.PercentageCommission.BROKER_COMMISSION_RATE
moonshot.commission.PercentageCommission.IB_COMMISSION_RATE_TIER_2moonshot.commission.PercentageCommission.BROKER_COMMISSION_RATE_TIER_2
moonshot.commission.FuturesCommission.IB_COMMISSION_PER_CONTRACTmoonshot.commission.FuturesCommission.BROKER_COMMISSION_PER_CONTRACT
moonshot.commission.PerShareCommission.IB_COMMISSION_PER_SHAREmoonshot.commission.PerShareCommission.BROKER_COMMISSION_PER_SHARE
moonshot.commission.PerShareCommission.IB_COMMISSION_PER_SHARE_TIER_2moonshot.commission.PerShareCommission.BROKER_COMMISSION_PER_SHARE_TIER_2
moonshot.slippage.BorrowFeesmoonshot.slippage.IBKRBorrowFees
quantrocket.realtime.create_tick_dbquantrocket.realtime.create_ibkr_tick_db
quantrocket.zipline.run_algorithmquantrocket.zipline.backtest
quantrocket.zipline.ingest_bundlequantrocket.zipline.create_db_bundle and quantrocket.zipline.ingest_bundle (see Zipline usage guide)
quantrocket.zipline.clean_bundlesquantrocket.zipline.drop_bundle
ib_trading_calendarsquantrocket_trading_calendars

CLI

Version 1Version 2
quantrocket fundamental collect-financialsquantrocket fundamental collect-reuters-financials
quantrocket fundamental collect-estimatesquantrocket fundamental collect-reuters-estimates
quantrocket fundamental collect-shortsharesquantrocket fundamental collect-ibkr-shortshares
quantrocket fundamental collect-shortfeesquantrocket fundamental collect-ibkr-borrowfees
quantrocket fundamental collect-sharadarquantrocket fundamental collect-sharadar-fundamentals
quantrocket fundamental financialsquantrocket fundamental reuters-financials
quantrocket fundamental codesquantrocket fundamental reuters-codes
quantrocket fundamental estimatesquantrocket fundamental reuters-estimates
quantrocket fundamental shortsharesquantrocket fundamental ibkr-shortshares
quantrocket fundamental shortfeesquantrocket fundamental ibkr-borrowfees
quantrocket fundamental sharadarquantrocket fundamental sharadar-fundamentals
quantrocket history create-dbquantrocket history create-ibkr-db
quantrocket launchpad credentialsquantrocket ibg credentials
quantrocket launchpad statusquantrocket ibg status
quantrocket launchpad startquantrocket ibg start
quantrocket launchpad stopquantrocket ibg stop
quantrocket launchpad configquantrocket ibg config
quantrocket master exchangesquantrocket master list-ibkr-exchanges
quantrocket master collectquantrocket master collect-ibkr
quantrocket master optionsquantrocket master collect-ibkr-options
quantrocket master diffquantrocket master diff-ibkr
quantrocket master delistquantrocket master delist-ibkr
quantrocket master create-comboquantrocket master create-ibkr-combo
quantrocket master collect-calendarquantrocket master collect-ibkr-calendar
quantrocket realtime create-tick-dbquantrocket realtime create-ibkr-tick-db
quantrocket zipline runquantrocket zipline backtest
quantrocket zipline ingestquantrocket zipline create-db-bundle and quantrocket zipline ingest (see Zipline usage guide)
quantrocket zipline bundlesquantrocket zipline list-bundles
quantrocket zipline cleanquantrocket zipline drop-bundle

Other name changes

The IB Gateway permission configuration file quantrocket.launchpad.permissions.yml, if used, should be renamed quantrocket.ibg.permissions.yml.

Conids are replaced by Sids

The following functions and commands which previously accepted conids (conids/--conids) now accept sids instead (sids/--sids):

  • quantrocket.account.download_account_portfolio / quantrocket account portfolio
  • quantrocket.blotter.cancel_orders / quantrocket blotter cancel
  • quantrocket.blotter.download_order_statuses / quantrocket blotter status
  • quantrocket.blotter.download_positions / quantrocket blotter positions
  • quantrocket.blotter.list_positions
  • quantrocket.blotter.close_positions / quantrocket blotter close
  • quantrocket.blotter.download_executions / quantrocket blotter executions
  • quantrocket.blotter.download_pnl / quantrocket blotter pnl
  • quantrocket.fundamental.collect_reuters_financials / quantrocket fundamental collect-reuters-financials
  • quantrocket.fundamental.collect_reuters_estimates / quantrocket fundamental collect-reuters-estimates
  • quantrocket.fundamental.collect_wsh_earnings_dates / quantrocket fundamental collect-wsh
  • quantrocket.fundamental.download_reuters_financials / quantrocket fundamental reuters-financials
  • quantrocket.fundamental.download_reuters_estimates / quantrocket fundamental reuters-estimates
  • quantrocket.fundamental.download_wsh_earnings_dates / quantrocket fundamental wsh
  • quantrocket.fundamental.download_sharadar_fundamentals / quantrocket fundamental sharadar-fundamentals
  • quantrocket.fundamental.download_ibkr_shortable_shares / quantrocket fundamental ibkr-shortshares
  • quantrocket.fundamental.download_ibkr_borrow_fees / quantrocket fundamental ibkr-borrowfees
  • quantrocket.history.create_ibkr_db / quantrocket history create-ibkr-db
  • quantrocket.history.collect_history / quantrocket history collect
  • quantrocket.history.download_history_file / quantrocket history get
  • quantrocket.master.collect_ibkr_listings / quantrocket master collect-ibkr
  • quantrocket.master.collect_ibkr_option_chains / quantrocket master collect-ibkr-options
  • quantrocket.master.diff_ibkr_securities / quantrocket master diff-ibkr
  • quantrocket.master.download_master_file / quantrocket master get
  • quantrocket.master.delist_ibkr_security / quantrocket master delist-ibkr
  • quantrocket.get_prices
  • quantrocket.realtime.create_ibkr_tick_db / quantrocket realtime create-ibkr-tick-db
  • quantrocket.realtime.collect_market_data / quantrocket realtime collect
  • quantrocket.realtime.cancel_market_data / quantrocket realtime cancel
  • quantrocket.realtime.download_market_data_file / quantrocket realtime get
  • quantrocket realtime stream

Similarly, functions and commands which accepted --exclude-conids/exclude_conids now accept --exclude-sids/exclude_sids:

  • quantrocket.fundamental.download_reuters_financials / quantrocket fundamental reuters-financials
  • quantrocket.fundamental.download_reuters_estimates / quantrocket fundamental reuters-estimates
  • quantrocket.fundamental.download_wsh_earnings_dates / quantrocket fundamental wsh
  • quantrocket.fundamental.download_sharadar_fundamentals / quantrocket fundamental sharadar-fundamentals
  • quantrocket.fundamental.download_ibkr_shortable_shares / quantrocket fundamental ibkr-shortshares
  • quantrocket.fundamental.download_ibkr_borrow_fees / quantrocket fundamental ibkr-borrowfees
  • quantrocket.history.download_history_file / quantrocket history get
  • quantrocket.master.download_master_file / quantrocket master get
  • quantrocket.get_prices
  • quantrocket.realtime.download_market_data_file / quantrocket realtime get
  • quantrocket realtime stream

The following Moonshot class attributes have changed:

  • CONIDS is now SIDS
  • EXCLUDE_CONIDS is now EXCLUDE_SIDS
  • BENCHMARK, if used, should reference a sid, not a conid

Elimination of "domains"

In version 1, there were separate securities master "domains" for IBKR and Sharadar records. In version 2, all vendors are consolidated in a single securities master database allowing data to be seamlessly mixed and matched. Therefore, the "domain" concept is no longer applicable, and accordingly the domain parameter has been removed from the following functions and commands:

  • quantrocket.master.download_master_file / quantrocket master get
  • quantrocket.master.get_securities_reindexed_like
  • quantrocket.master.list_universes / quantrocket master list-universes
  • quantrocket.master.create_universe / quantrocket master universe
  • quantrocket.master.delete_universe / quantrocket master delete-universe
  • quantrocket.fundamental.download_sharadar_fundamentals / quantrocket fundamental sharadar-fundamentals
  • quantrocket.fundamental.get_sharadar_fundamentals_reindexed_like

Securities master fields

With the advent of multiple data vendors, many IBKR-specific fields in the securities master file are now prefixed with ibkr_:

Version 1Version 2
ConIdibkr_ConId
PrimaryExchangeibkr_PrimaryExchange
TradingClassibkr_TradingClass
MarketNameibkr_MarketName
LongNameibkr_LongName
ValidExchangesibkr_ValidExchanges
AggGroupibkr_AggGroup
Sectoribkr_Sector
Industryibkr_Industry
Categoryibkr_Category
MinTickibkr_MinTick
MdSizeMultiplieribkr_MdSizeMultiplier
ContractMonthibkr_ContractMonth
RealExpirationDateibkr_RealExpirationDate
UnderConIdibkr_UnderConId
UnderSymbolibkr_UnderSymbol
UnderSecTypeibkr_UnderSecType
MarketRuleIdsibkr_MarketRuleIds
Strikeibkr_Strike
Rightibkr_Right
ComboLegsibkr_ComboLegs
EvRuleibkr_EvRule
EvMultiplieribkr_EvMultiplier

In version 2, the Symbol field in the securities master file for IBKR securities returns the version 1 LocalSymbol field if available, otherwise the version 1 Symbol field. The version 1 Symbol and LocalSymbol fields were usually identical for stocks but for futures LocalSymbol is more specific. To preserve version 1 functionality, perform the following replacements:

Version 1Version 2
Symbolibkr_Symbol
LocalSymbolibkr_LocalSymbol

In version1, ISINs were stored in the Cusip field with a prefix of "ISIN:". Now they are stored in the ibkr_Isin field, without any prefix.

Package versions

In version 2, the Anaconda version has been updated to Anaconda 2020.02. Version 1 was based on Anaconda 5.2.0. Major updates include:

Removal of 1.x deprecations

In addition to the API changes above, all API parameters and function names which were deprecated in previous releases have been fully removed in version 2. Please refer to the 1.x release notes for a list of these deprecations.