Llama’s Leap: Updates of Heurist Testnet Points

Image Credit: https://twitter.com/than0s_ai

At Heurist, our core commitment is to maximize decentralization by removing any barriers to entry in AI mining. We have eliminated the need for whitelist requirements, KYC processes, and there is no necessity to hold any form of NFT or license key to participate.

Our miner software is transparently open-sourced on GitHub, ensuring everyone has the opportunity to contribute. This approach, while enhancing flexibility and inclusivity, also introduces certain complexities to our mining rewards mechanism which we are eager to refine.

Let us briefly describe two incidents that disrupted the fairness of🦙Llama Points distribution, and remedy plans.

Part 1

On 4/10, some community miners started to notice the number of LLM requests received became scarce, and sequencer server went down frequently. As a consequence, many LLM miners earned very few points from 4/10 to 4/13.

Example point history of a miner with RTX 4090

This is caused by multiple reasons:

1. Some miners increased num_child_process parameter to 20~50 in config.toml and resulted in a large number of concurrent requests sent to the server and took over most pending requests.
2. Under heavy loads of requests, the sequencer server became unstable, randomly dropping connections, rejecting miner requests and even going offline for hours.

We have implemented the following fixes:

1. Upgraded the sequencer server to a much more powerful bare-metal machine that can resist at least 100x more traffic.
2. Added a grouping mechanism that allows only a subset of all active miners to receive job requests at any given time. It’s only activated when there are too many miners and too few requests. The groups are randomly shuffled to ensure no one consistently gets an advantage in receiving more requests.
3. Implemented a mechanism to dynamically adjust concurrency in LLM miner. (will roll out in the next miner release)

Compensation plan:

For all LLM miners during 4/10 12AM UTC ~ 4/13 12AM UTC, if Llama Points earned each day is lower than 80% of the previous 3-day average (4/8 ~4/10), we increase the Llama Points of the selected 4 days to 1.5x of previous 3-day average.

Part 2

We observed a growing concern that A6000 did not earn as many points as RTX 4090, although it is supposed to run Mixtral 8x7b model which is more capable than Mistral 7b in terms of reasoning, knowledge, comprehension and skillsets, and therefore has much more industry use cases. However, the larger model comes at the cost of lower inference speed.

Research shows Mixtral 8x7b model outcompetes Mistral 7b in all dimensions (source https://mistral.ai/news/mixtral-of-experts/)

Based on our evaluation, 4090 running Mistral 7b (full precision) produces 50 tokens/s. A6000 running Mixtral 8x7b (int4 quantization) produces 20 tokens/s. If the point reward rate for this two models were the same, Mistral 7b would earn 2.5x as Mixtral 8x7b.

Our goal is to make the reward rate of Mistral 7b be 50% of Mixtral 8x7. This number is reasonable because (1) A6000 price is almost 2x as that of 4090 (2) A6000 has 48GB VRAM which is twice of that of 4090. This indicates that we should cut down Llama Points / token for Mistral 7b to be 0.2x of Mixtral 8x7b.

To understand the difference between these two widely used GPU models without the technical jargons, a community miner shared an analogy:

The 4090 is like a Hyundai Genesis sedan packed with the latest technology, while the A6000 feels like an older model BMW or Benz. That’s how I see it.

We have decided to recalibrate the reward rate as:

Compensation plan:

For all LLM miners, we will evaluate the request history from April 1 12:00 AM UTC to April 15 12:00 AM UTC. If there are M requests for openhermes-mixtral-8x7b-gptq + openhermes-2-yi-34b-gptq and N total requests, we increase the Llama Points by 0.2*M/N.

Conclusion

As we embark on this several-month testnet phase (lasting at least three months), let’s get prepared for an evolving process. Changes, improvements, and even occasional disruptions are expected as we diligently work to stress-test our system and identify any edge cases. This dynamic phase is crucial for us to enhance the robustness and fairness of our mining system ahead of the mainnet launch. We appreciate your understanding and active participation in this important stage, as your feedback will be instrumental in shaping a reliable and equitable platform.

Heurist team will develop the necessary software tools to analyze the data and execute the change in a fully auditable and transparant way in 1~2 weeks. The incentivized testnet campaign will go on as usual, with more equitable reward mechanisms and more robust distributed software system. Remember, this is a bold journey towards decentralized AGI. We are on this miraculous journey with you together.

Special thanks to our partner Chasm for their role in discovering the first issue and reviewing this article. You can learn more about Chasm at https://chasm.net