#396: Mistral’s New Mixture-of-Experts Model, Mixtral 8x7B, Delivers on Both Performance and Efficiency, & More

Last week, Mistral AI released^[i] Mixtral 8x7B, a new open-source Mixture-of-Experts (MoE) large language model (LLM) that includes 8 small neural networks, each of which is finetuned in a specific area of expertise. Also included in GPT-4, the MoE structure enables higher efficiency than in other LLMs, calling on relevant expert models to answer different queries. During inference, Mixtral activates fewer than 30% of its 46.7 billion parameters and achieves the speed and cost associated with much smaller models while outperforming Llama 2's 70 billion active parameters on the Massive Multitask Language Understanding MMLU benchmark, as shown below.

Source: ARK Investment Management LLC, 2023, based on data from Mistral^[ii] as of 12/1123 and Yi Open-Source^[iii] as of 11/05/23. For informational purposes only and should not be considered investment advice or a recommendation to buy, sell, or hold any particular security. Past performance is not indicative of future results.

Mistral plans to increase Mixtral’s efficiency further as it prepares to launch an AI assistant capable of running on a 16GB laptop^[iv] by early next year.

Last Thursday,^[v] hackers attacked the infrastructure of Ledger, a Paris-based crypto self-custody wallet, by inserting malicious code into the Github library of the Connect Kit software that connects the wallet to several decentralized finance (DeFi)^[vi] protocols. The hackers stole $484,000 USD worth of cryptoassets and impacted Ledger’s platform users, including Lido, Metamask, and Coinbase.

Accomplished through a phishing attack on one of its employees, the hack is the third scandal that Ledger has faced in the last four years. In 2020, the “Ledger leak”^[vii] exposed clients’ residential addresses and various phone numbers online. Then, in May 2023, Ledger suffered through a public-relations fiasco^[viii] after it announced firmware to enable a “backdoor” once thought impossible that would have allowed client seed phrases—the set of 24 random words that contains the data needed to recover the user’s private keys—to be separated from client hardware wallets.

Last week’s hack could hamper the adoption of crypto self-custody wallets, as the public at large, even if mistakenly, may mistrust the mantra "not your keys, not your coins" given recent events. In our view, the solution to counterparty risk will highlight the benefit of open-source self-custody firms like Coinkite with publicly verifiable^[ix] and auditable source code. As bitcoin becomes more integrated into the traditional financial ecosystem, especially if the spot ETFs are approved, access to practical, transparent self-custody hardware and software will become critical to preserving the open ethos of the digital assets industry.

Last week, a jury agreed^[x] with Epic Games that Google Play, Google’s app store, engaged in anticompetitive behavior, a verdict contrary to the outcome of Epic’s court battle against Apple’s App store. District Judge James Donato soon will announce the penalties that Google will face. In 2021, Epic lost in its District Court^[xi] antitrust suit against Apple, a decision largely upheld^[xii] by the Ninth Circuit Court of Appeals in 2023. Now, the Supreme Court is deciding whether to review^[xiii] the Apple and Epic appeals of the District Court and Circuit Court cases.

Interestingly, a judge—not a jury—ruled in favor of Apple in the Apple-Epic case, while a jury decided against Google, perhaps a reflection of the public’s negative perception of big tech’s dominance and power. The Justice Department^[xiv] also is focused increasingly on Google’s monopoly position in the search engine space.

Apple and Google rely heavily on their app stores, which accounted for 22%^[xv] and 15%^[xvi] of revenues, respectively, in 2022. If Google’s loss to Epic signals a regulatory push to support third-party app stores and payment methods, the hit to app store revenues could be in early days, leading to lower app store take rates, more volume for global payment facilitators like Adyen, Braintree, and Stripe, and more startups targeting app developer tools than otherwise would be the case.

[i] Mistral. 2023. “Mixtral of experts | Mistral AI | Open source models.”

[ii] Ibid.

[iii] Yi Open-Source. 2023.

[iv] Smith, T. and Lewin, A. 2023. “See the pitch memo that raised €105m for four-week-old startup Mistral.” Sifted.

[v] Knight, O. 2023. “Ledger Exploit Drained $484K, Upended DeFi; Former Staffer Linked to Malicious Code.” CoinDesk.

[vi]“Decentralized finance.“ Wikipedia.

[vii] Powers, B. 2020. “SIM Swaps to Physical Threats: Ledger Leak Has Dire Consequences.” CoinDesk.

[viii] Kessler, S. 2023. “Ledger Recover Fiasco Exposes Gap Between Bitcoin's Self-Custody Ideals and Technical Reality." CoinDesk.

[ix] Coldcard/Firmware. 2023.

[x] Perez, S. 2023. “Epic Games won its antitrust battle with Google. But what comes next?” TechCrunch.

[xi] Robertson, A. 2021. “Breaking down the Epic v. Apple Fortnite trial ruling.” The Verge.

[xii] Leswing, K. 2023 “Apple declares victory after decision reached in Epic Games appeal.” CNBC.

[xiii] Owen, M. 2023. “Epic vs. Apple: Fortnite, history, court ruling, appeal results.” Apple Insider.

[xiv] Robertson, A. 2023. “US v. Google antitrust trial: updates.” The Verge

[xv] Curry, D. 2023. “Apple Store Statistics.” Business of Apps.

[xvi] Larson, S. 2023. “Google Play Store Revenue, Ratings & Subscription Stats 2023.” Priori Data.