xAI has opened the source code of the Grok large language model
xAI
xAI, a company owned by Elon Musk, has published on GitHub the source code of its large language model Grok-1 with 314 billion parameters under the open Apache 2.0 licence.
Here's What We Know
The open source release includes the model's base weights, network architecture, and other components. As noted, Grok-1 has not been specially optimized or censored for specific tasks like dialogue systems.
░W░E░I░G░H░T░S░I░N░B░I░O░
- Grok (@grok) March 17, 2024
The model was trained over four months by October 2023 and was designed to generate code, texts and answer questions. Opening up the code will allow third-party researchers and developers to work with the model.
Despite the discovery of scales and other components, Grok-1 comes without training code and reproducible datasets.
Grok weights are out under Apache 2.0: https://t.co/9K4IfarqXK
- Sebastian Raschka (@rasbt) March 17, 2024
It's more open source than other open weights models, which usually come with usage restrictions.
It's less open source than Pythia, Bloom, and OLMo, which come with training code and reproducible datasets. https://t.co/kxu2anrNiP pic.twitter.com/UeNew30Lzn
Flashback
Since buying Twitter (X), Ilon Musk has repeatedly criticised the closed source code of competitors' AI models, such as OpenAI. Now he's suing OpenAI, accusing the company of violating its open source agreement.
Publishing source code is a common practice to get feedback and improve models. However, major players prefer to keep the code closed or grant limited licences, as in the case of Meta's Llama 2.
After launching the Grok chatbot based on this model, it previously required a paid X subscription to access it. But in preliminary testing, it failed to stand out against ChatGPT, Gemini and other more advanced systems.
