Backend Biweekly #1: Important AI Developments and ML Learning Resources
Updates on Apple, Mistral, and Microsoft and resources to build the GPT tokenizer, use MLX to train models, and benchmark LLMs
Welcome to the first edition of the Backend Biweekly! Backend Biweekly’s contain:
The AI developments of the past two weeks that you should know about and why they’re important.
The best recent ML educational resources and guides.
The complete list of tools and services I use to learn about, write about, and build with ML.
Free subscribers will have access to the first 5 or 6 resources. The full list of all resources (~25 this time) is only available to supporters. Remember, you can support Society's Backend for just $1/mo for the first year and there’s a free 7-day trial:
Let's dig in! I've included a headline, blurb, and link to the original article/post for each.
AI Developments
Tumblr and Midjourney content deal
Tumblr's owner, Automattic, is reportedly in discussions with AI companies Midjourney and OpenAI to provide them with training data derived from users' posts on Tumblr. This information comes from a report by 404 Media, which cites an anonymous source within the company. The deals between Automattic and the AI firms are described as 'imminent.' This development follows rumors on Tumblr about a potential deal with Midjourney, which could open up a new revenue stream for the platform.
This follows Google making a deal with Reddit to use the data users post on Reddit as training data for LLMs. This an interesting area to keep an eye on because it highlights data privacy concerns, ethical considerations for using user data, and the legal ramifications of using data for training LLMs.
Apple cancels work on electric car, ending decadelong effort
Apple has decided to wind down its decade-long project to build an electric car, known as Titan. The company will be shifting employees from some car teams to Apple's AI division.
Keep tabs on this. Apple seems to be going all-in on AI even though they aren't as outright about it as other tech companies. There are rumors of an LLM-based Siri releasing this summer, Apple has put a considerable amount of effort into MLX (a ML framework for Apple silicon), and Apple is releasing a Copilot-like feature for Xcode.
Microsoft partners with Mistral
Microsoft has entered into a multi-year partnership with Mistral AI. This collaboration is part of Microsoft's strategy to diversify its AI offerings on Azure, extending beyond its significant investment in OpenAI. As part of the agreement, Microsoft will make Mistral AI's models available on Azure and has also taken a minority stake in the company.
This is an excellent move from Microsoft. Everyone thinks of Microsoft whenever they think of OpenAI to the point where Microsoft has been listed as a codefendant on lawsuits against OpenAI. This not only diversifies Microsoft's AI offerings, it increases their global reach, and solidifies a commitment to open-source.
Mistral launches Mistral Large
Mistral AI has launched a new large language model called Mistral Large, which is designed to compete with other top-tier models such as GPT-4 and Claude 2. The model is proficient in code and mathematics, multilingual (supporting English, French, German, Spanish, and Italian), and is available through Mistral AI's API platform and Microsoft's Azure AI platform.
More models are always good—we need competition to push the state of the art. Unlike Mistral's other models, Large is proprietary showing Mistral's desire to commercialize its models.
Genie: Generative Interactive Environments
Google released Genie: a groundbreaking generative interactive environment created through unsupervised training from unlabelled Internet videos. It creates a "controllable virtual world" at 11B parameters. It consists of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model.
This has implications for any type of virtual environment. This means simulations, gaming, and virtual training scenarios. It currently generates an environment at 1 FPS and is only available for research purposes, but is a major step forward for the world of AI. I've always said video games are in a prime position to take advantage of AI and this is giving us a taste of how they will.
Stable Diffusion 3 launch
Keep reading with a 7-day free trial
Subscribe to Society's Backend to keep reading this post and get 7 days of free access to the full post archives.