Done: categories can now be selected on the search page. Plus you can run a search without entering a search term to get all the discussions in a category. If it doesn't work maybe you need to clear your cached files.

(Unless of course the "dump" or rawest form of database storage you can access has some field or notation for category, in which case it'd be even easier!) — Outlander

Yep. I've done it already. :up:

(not live yet)

The one big improvement I could make to the archive is to include the categories. For some reason Plush gave me an export without categories so I just went ahead and built the archive without them. Now I've got them so I could potentially incorporate them somehow. Maybe just in the search and your-posts pages.

I'm not migrating any posts to the new site, so the current compromise is fine. As far as I recall it's only me and @hypericin who ever cared anyway.

OK, I'll leave them. I've found a decent compromise, which is that they won't be indexed by search engines but anyone can visit those pages, given the URLs (or find them from the archive's search page or whatever).

The above issue alerted me to another one: the possibility of SEO penalization for duplicate content. So I've now blocked crawlers with meta tags, a robots.txt file, and also in the web server config, thus preventing search engines from indexing any more pages on tpfarchive.com. I'll remove those blocks when we close the Plush site.

Google has already begun to index tpfarchive.com, no doubt because we've linked to it here.

I forgot to remove the short stories. On the one hand they should be archived, on the other hand some of us (like me and @hypericin) don't want our stories to be publicly available, i.e., published, on the web.

Approach, I said.

New topic if you like.

All very interesting but we've gone too far off-topic. Since this is an important topic I'd like to keep it pure.

Next, we preprocessed the posts by generating embeddings. Using the MiniLM-L6-v2 model, each post was converted into a 384-dimensional numerical vector capturing its semantic meaning. These vectors were stored locally as embeddings.npy . To enable fast similarity search, we built a FAISS index from the embeddings, allowing the bot to retrieve only the most relevant posts for a user query rather than scanning all 29,918 posts each time.

We then integrated the BannoBot script, which takes a user’s question, converts it into an embedding, searches the FAISS index for top-k relevant posts, and constructs a prompt including these excerpts. This prompt is passed to a local LLM (Orca-Mini), which generates a natural-language answer in the style and content of your posts. All processing—embedding, search, and LLM inference—occurs on your laptop, ensuring privacy and avoiding cloud APIs.

Cool. I wonder though...

I'll tempt you to do something like this with the entire data file... a master philosophy forum bot... — Banno

Like an amalgam of everyone on TPF? Tempting though it is, my instinct is that the wider the range of data, the more it would approach a regular LLM like ChatGPT and lose its own point of view.

In my present immoral state, I'll tempt you to do something like this with the entire data file... a master philosophy forum bot... — Banno

I vaguely recall we were warned about this around 1818. Something to do with an unconventional Swiss scientist.

Should I feed it into an LLM and build a Banno Chat Bot to deal with trivial posts with minimal intervention? — Banno

I notice you didn't wait for an answer and just went ahead and did it. So preoccupied with whether or not you could, you didn't stop for more than a moment to think if you should (to paraphrase Michael Crichton).

As it stands it is only using the top 4 posts. I'll have a play and see if it can do more without being too slow. — Banno

So it's reading all your posts before it comes up with an answer each time? I guess the next step is to build your own LLM model that incorporates the words of Banno in its training data.

Out of curiosity, I am wondering whether Discourse was the only option able to accommodate the new laws. Were other options also capable? — Leontiskos

I think the alternatives (NodeBB, XenForo, etc) would have enabled us to accommodate the laws. I chose Discourse over and above that requirement because I liked the way it worked.

If you're interested, the main software requirements to accommodate the new laws are more configuration, crucially including the configuration of the sign-up form, and more moderation tools, crucially including the flagging and moderation of direct messages.

I don't suppose there's any chance of uploaded images being rehabilitated, on the archive site? — bongo fury

No, any images that survive in the archive will be those hosted externally. Images uploaded to this site will not be in the archive. And I can't restore any of the images I had to remove when we were running out of disk space.

If not, is there a deadline for replacing them with linked ones? — bongo fury

You could edit your posts and replace the images, including broken ones, with images hosted externally (see the help page on images), and you'll have until we shut down this site to do that. Around February or March.

Yes, the script I write will achieve the same thing. :up:

I'll be using JavaScript or Python, and I won't have much control over the web server redirects on the new site, so as I say I'll be putting the redirects on Cloudflare, so the requests won't even get to Discourse.

Preferably also sending a 301 Moved Permanently header. Provided you or another staff member created the archive site from scratch (not using a pre-boxed framework or library) your knowledge in such fields seems sufficient enough to do so easily (and most importantly: properly or safely). — Outlander

Yep. Well I've already created it, and it doesn't use a framework. It's just HTML, Javascript and JSON, and CSS. The build is done locally by running a python script, written from scratch (I used AI to speed up the process, while I played the role of its micro-managing project manager).

tpfarchive.com

Yeah, I was just thinking about that. The comment URLs on this site don't contain the dicussion IDs, so I can't just use a basic mapping. But I don't want to include an explicit individual redirect for every one of the million or whatever comments.

This would mean having the comment ID fed to a script that pulls up the discussion ID and then redirects the user to the relevant discussion URL prefixed with a hash anchor containing the comment ID. — Outlander

Yeah, I could probably use Cloudflare workers for that, combined with a key-value store to map the comment IDs to the discussion IDs, which I'll only need to produce once, based on the Plush export.

If your bookmarks are of URLs such as...

https://thephilosophyforum.com/discussion/comment/1026204

(the share link for your last comment)

... then I can probably set up redirects to the comment in the archive:

https://tpfarchive.com/discussions/16281-a-new-home-for-tpf.html#comment-1026204

So when you request the former you'll be redirected to the latter, where you should be able to see the comment in context. I'll make a note to set this up.

Note: the latter URL doesn't work right now because the data on the archive site is a few days old.

If you work in a restaurant, you try to separate stored goods from actual food production. And the idea is central to other means of production. So, I applied a pedestrian truism to a current situation. Not expecting a Pulitzer prize for that observation. — Paine

No no, I'm sure it's brilliant.

Let me see if I understand. The stored goods represent the content of TPF, which we can store independently of its use in ongoing discussion, the latter represented by meal preparation, i.e., "grille"?

Stored food = TPF archived content
The preparation of meals = TPF discussion

I was confused by two things. One, it seemed like you were saying "why don't you do this, it's a no-brainer" (and yet I didn't know what it was you were suggesting); two, I didn't understand "grille," but now I'm thinking you meant the grill as in the place in the kitchen where food is cooked, maybe.

I hope you appreciate the time I've taken over this analogy :wink:

Some of you might be wondering about redirects. Probably this is the best way:

1. All discussion URLs, i.e., all links to specific discussions/threads/OPs from the old site (this one), will be redirected to the relevant pages of the archive. This means threads which are indexed on Google and linked to from other sites will not leave a trail of broken links behind them. I'm not sure if it's worth doing this for individual comments too.

2. All other URLs pointing to this site, like categories, login, etc., will go to the new site. This will happen by default anyway, since the new site will have the domain.

The one thing I'm concerned about transitioning to another platform is that this website will just deteriorate over time without maintenance and the whole archive will be lost, something that's been happening to the rest of the internet with broken sourced hyperlinks everywhere. — Saphsin

This website, the one we're on right now, will not deteriorate---it will be completely closed early next year. The content, however, will live on, hosted by me on a static website (tpfarchive.com), which is read-only. I will maintain this indefinitely (though it won't need much maintenance).

If you have a better plan to preserve the archive, let me know. Maybe you want to put up the money to make a physical book?

Thanks Bret!

Looking at the last page of those discussion results, it seems The Philosophy Forum has been around for just over 10 years, or since October 20th of 2015. That's curious and wonderful the website is a decade old. — Bret Bernhoft

TPF: Curious and wonderful since 2015.

In fact, this site is a kind of continuation of forums.philosophyforums.com, which started probably in the early 2000s but collapsed around the time when this one started.

I don't know what you mean.

The new site should be good too.

Well I was only talking about the archive site, where it's less critical.

:up:

It's a little too slow from where I am. Maybe I'll paginate the results on the "your posts" page (you can get them all in a single file in the download).

But you can have a look at the archive site, built on a data export from a few days ago.

https://tpfarchive.com

I had the temporary URL of the new site showing here for a moment there but I was forgetting that since the legal stuff isn't in place I can't approve any new members yet.

Fine, carry on.

How do you mean? — Christoffer

I meant that I'm not on board with your rhetoric. Anyway I only came in here to shed some light on the political leanings of TPF. In any case, the conservatives on the staff are no more ignorant of the facts than the others, as far as I can see. Facts have nothing to do with it, and in general the accusation that the other side ignores the facts is thrown from both sides. It's a nothing argument, on its own.

Yet, there's also a point to be made that many arguments that rely on facts tend to be called leftist. There's far more climate science deniers or general deniers of scientific results on the conservative right... so if a human's best attempt to reach a truth based on facts and scientific data is considered "leftist", then I guess that tells more about the political spectrum than that this forum is "unbalanced". — Christoffer

I don't agree with any of this, and I'm very left.

On the staff we have between 2 and 2.5 Marxists, 2 left-liberals, and 2 conservatives. I guess that's more left than right.

Why is it better for me to turn it off globally than for you to ignore it?

I can think of reasonable answers to that—I just want to understand the opposition.

As for downloading, you can just do Ctrl+p and then save as PDF. This will work on the new site (you can try it here on another Discourse forum) and also on the archive site.

It sounds like you're talking about ongoing discussions, but all the discussions on the archive site will be finished, static, frozen in time. So I'm confused about what you're looking for. You can't "enter the fray" on the archive site.

As it happens I built a first version of the archive site yesterday and the way it works is it shows the full discussion on one page.

The new site doesn't have pagination at all. Every discussion will be infinite scroll. There's an AI "Summarize this topic" button too, but I haven't tried it, and it can be turned off.

The archive will exist as a website, where you will be able to see all the discussions and posts. It will have some kind of search functionality, and it will have a "Your posts" page where you can view and download your posts (or anybody else's: you'll just enter the username). You'll be able to access that download while you're offline, if you keep it on your device (obviously).

If you need to access everything offline, and not just your own posts...well, it's possible for me to provide a package you can run locally, like a private website, but I find myself wondering why you would need to.

Since the old site will be preserved, I wonder if the "advanced search" functions that work now will work there — Paine

The content will be preserved but it won't look or work like the existing site. It depends how much time I spend on building it. Searching by author or discussion title are easy enough — not sure about a full text search.

Thanks Jamal, the projected new forum sounds great. Is there any way of exporting one's posts from the old forum as a word file? — Janus

I'll have all the data and I'll be able to provide members with a posts export on request. In fact I might provide it at the archive site.