My intuition:

• There’re “genuine” instances of hapax legomena which probably have some semantic sense, e.g. a rare concept, a wordplay, an artistic invention, an ancient inside joke.
• There’s various noise because somebody let their cat on the keyboard, because OCR software failed in one small spot, because somebody was copying data using a noisy channel without error correction, because somebody had a headache and couldn’t be bothered, because whatever.
• Once a dataset is too big to be manually reviewed by experts, the amount of general noise is far far far larger than what you’re looking for. At the same time you can’t differentiate between the two using statistics alone. And if it was manually reviewed, the experts have probably published their findings, or at least told a few colleagues.
• Transformers are VERY data-hungry. They need enormous datasets.

So I don’t think this approach will help you a lot even for finding words and phrases. And everything I’ve said can be extended to semantic noise too, so your extended question also seems a hopeless endeavour when approached specifically with LLMs or big data analysis of text.

For Tolkien’s work, there is the twelve volume “The Complete History of Middle Earth” which is about as inside baseball as you can get for Tolkien.

I’d replace HoME with Parma Eldalamberon, Vinyar Tengwar and other journals publishing his early materials here.

Recommending Italo Calvino’s Six Memos for the Next Millennium, the lectures he has been preparing shortly before his death.

Not an assembly guide for a work of literature, but it’ll help your own process if it’s already ongoing and you want to improve.

The lectures also have some comments on what Calvino himself was doing here and there and why.

For me specifically, if spoilers hurt a book, it probably wasn’t worth reading in the first place. I love when authors demonstrate mastery of language and narration, and no amount of spoilers can overshadow the direct experience of witnessing it enacted.

ChatMusician isn’t exactly new and the underlying dataset isn’t particularly diverse, but it’s one of the few models made specifically for classical music.

Are there any others, by the way?

The Phoebus cartel strikes again!

I enjoy xenharmonic music and modern academic music the most, but I’m not familiar with everything there, so any recommendations are welcome if you, reader, have something in your mind.

Because we have tons of ground-level sensors, but not a lot in the upper layers of the atmosphere, I think?

Why is this important? Weather processes are usually modelled as a set of differential equations, and you want to know the border conditions in order to solve them and obtain the state of the entire atmosphere. The atmosphere has two boundaries: the lower, which is the planet’s surface, and the upper, which is where the atmosphere ends. And since we don’t seem to have a lot of data from the upper layers, it reduces the quality of all predictions.

Yeah, while tripping on acid.

Interessen-Gemeinschaft Matte.

Given the fact that there was an unintentional DDOS when federated Lemmy instances were requesting the same preview around the same time, it must be one of LW’s servers, not anything on your side.

The only sure way to get rid of this effect is to use an instance entirely hosted on servers in anglophone countries, I think.

I know Google likes to localise their websites based on IP addresses. Perhaps the preview was requested from a Russian IP? (not necessarily yours, could be a VPN if you use one or one of LW’s servers)

This is not to say that Jung wasn’t a genius. Jung was THE BOMB DIGGIDITY (which, by the way, I wish was an official term in the Oxford dictionary).

If they love Jung so much (which I agree they should because Jung was amaaaaazing), why don’t they honor him by using the spelling he actually used?

Love etymological articles with unreliable narrators.

“These hills are being bombed”?

No IPA notation? ⸨I’m somewhat disappointed⸩

It would. But it’s a good option when you have computationally heavy tasks and communication is relatively light.

TOTP can be backed up and used on several devices at least.

Once configured, Tor Hidden Services also just work (you may need to use some fresh bridges in certain countries if ISPs block Tor there though). You don’t have to trust any specific third party in this case.

Discounting temporary tech issues, I haven’t browsed internet without an adblocker for a single day in my entire life. Nobody is entitled to abuse my attention; no guilt, no exceptions.

If config prompt = system prompt, its hijacking works more often than not. The creators of a prompt injection game (https://tensortrust.ai/) have discovered that system/user roles don’t matter too much in determining the final behaviour: see appendix H in https://arxiv.org/abs/2311.01011.