• 3 Posts
  • 26 Comments
Joined 1 year ago
cake
Cake day: July 22nd, 2023

help-circle
  • I tried Mistral Nemo 12B instruct this morning. It’s actually quite good. I’d say it’s close to dolphin mistral 8x7B which is a monster in size and very smart, about 45 or 50GB. So I’d say Arli is a good deal Mistral Nemo 12B for 4 or $5 per month and privacy so they claim.

    If you don’t mind logging for some questions, you can get access to very good or if not the best models at lmsys.org without monetary cost. Just go to the “Arena”. This is where you contribute with your blind evaluation by voting which of two is better. I often get models like 4o and sonnet 3.5 by Anthropic, google’s best, etc., and at other times many good 70B models. You see two answers at once and vote your favorite between the two. In return, you get “free” access.

    Be careful with AMD GPUs as they are not as well supported for local AI. However, support is gaining ground. Some people are doing it but it takes effort and hassle, from what I’ve read.


  • I know that people are using P40 and P100 GPUs. These are outdated but still work with some software stacks / applications. The P40 GPU, once very cheap for the amount of VRAM, is no longer as cheap as it was probably because folks have been picking them up for inference.

    I’m getting a lot done with an NVidia GTX 1080 which only has 8GB VRAM. I can run a quant of dolphin Mixtral 7x8B and it works well enough. It takes minutes to load, almost too long for me, but after that I get 3-5 TPS with some acceptable delay between questions.

    I can even run Miqu quants at 2 or 3 bits. It’s super smart even at these low quant levels.

    llama 3.1 8B runs great with this 1080 8BG GPU at 4_K_M and also 5 or 6_K_M. I believe I can run gemma 9B f16 at 8 bpw.



  • Interactive Brokers is also my next choice. Although, beware that you have to install Java runtime from Oracle in order to be able to log in to they servers. Java runtime environment has seen many beaches of security in the past, particularly the Internet was still in adolescence. Oracle claims to have solved those but this needs to be verified.

    I’m waiting for Schwab mainly because, as it turns out, there is magic there. Namely, our assets are protected from online fraud. I’m sure there are limits to that protection. And that protection has applied to their normal online accounts. Will it apply to API accounts? We will have to reread the fine print when it’s final.




  • The community can only read the source code, as of yet. All of the source code has been provided by a set of internal developers.

    The fact that it is open source means that, if somehow two malware elements have made it into the source code, then someone will eventually report it. But this doesn’t mean that two malware elements cannot be there right now.

    These two malware hits on total virus scan should be communicated to the developers.









  • Take the free ones. Ignore the discounted ones, don’t buy them.

    There is too much concentration in your livelihood when you invest in your employer. For example, and I know too many examples of this, if your employer starts doing badly, you can not only lose your job, but they might move out of town leaving your home in a state where you may need to sell it in a depressed market. Often the shares you would have invested in the company are worth too little to sell. Your assets, your job, your home, all take a hit at the same crazy time. Not worth it.

    Instead, invest in broad-market index funds. Go to Bogleheads where they discuss this and ask there. If you like momentum, arguably the greatest investor that has ever lived, Warren Buffett recommends a split between 90% SPY or IVV (S&P500) and 10% cash. The S&P500 is something like a momentum fund of the top winners of the US economy, and constantly changing.

    Your employer is only trying to tie you down and have real skin in the game so that you’ll work harder. Ignore the tendency.

    Best of luck.