Results from our first test
We compressed Gemma 4 26B from 51.6 GB to 8.7 GB and tested it against the original and against six other compressed versions: Google's own 4-bit version and five community builds. This was a test of the method, not a product.
- Smaller than the original
- 5.9×
- Size on disk
- 8.7 GB
- Less memory than Google's 4-bit
- 44%
Against the original
Uluka is at most 4.8 points below the original Gemma 4 26B on any test, and within 1.5 on three of the five. It is one-sixth the size.
| Test | Original 51.6 GB | Uluka 26B 8.7 GB | AWQ INT4 17.2 GB | UD-IQ3_XXS 12.6 GB |
|---|---|---|---|---|
| CodeHumanEval+ | 93.3 | 92.1 | 90.9 | 91.5 |
| MathMATH-500 | 85.8 | 82.0 | 82.0 | 84.0 |
| Math with thinkingMATH-500, Google's recommended settings | 84.8 | 83.4 | 73.8 | 84.6 |
| KnowledgeMMLU-Pro | 80.0 | 75.2 | 78.0 | 78.4 |
| Long documentsRULER, 8k and 32k tokens | 90.4 | 89.6 | 85.8 | 89.7 |
Against other compressed versions
Four standard tests, each scored out of 100, with the same questions and settings for every model. Smallest first.
| Version | MMLU knowledge | HellaSwag common sense | GSM8K maths | IFEval instructions | Average |
|---|---|---|---|---|---|
| Uluka 26BUluka, 8.7 GB | 59.0 | 45.1 | 87.8 | 87.8 | 69.9 |
| UD-IQ2_Munsloth, community, 11.2 GB | 57.9 | 43.9 | 88.8 | 88.0 | 69.6 |
| UD-IQ3_XXSunsloth, community, 12.6 GB | 61.3 | 43.5 | 86.5 | 88.0 | 69.8 |
| UD-Q3_K_Munsloth, community, 13.9 GB | 51.6 | 44.8 | 86.0 | 88.5 | 67.7 |
| Q3_K_Mbartowski, community, 14.2 GB | 54.4 | 43.0 | 90.0 | 90.2 | 69.4 |
| QAT Q4_0Google, 15.6 GB | 61.4 | 47.3 | 84.5 | 88.5 | 70.4 |
| AWQ INT4cyankiwi, community, 17.2 GB | 58.8 | 47.2 | 87.2 | 87.5 | 70.2 |
Uluka is the smallest of the seven. Its 69.9 average is within 0.5 of Google's 70.4 and beats UD-Q3_K_M. Google's version scores higher on MMLU and HellaSwag, and Uluka scores higher on GSM8K. Some community builds win single tests, but all are bigger. We did not measure speed in these tests.