Next, we preprocessed the posts by generating embeddings. Using the MiniLM-L6-v2 model, each post was converted into a 384-dimensional numerical vector capturing its semantic meaning. These vectors were stored locally as embeddings.npy . To enable fast similarity search, we built a FAISS index from the embeddings, allowing the bot to retrieve only the most relevant posts for a user query rather than scanning all 29,918 posts each time.

We then integrated the BannoBot script, which takes a user’s question, converts it into an embedding, searches the FAISS index for top-k relevant posts, and constructs a prompt including these excerpts. This prompt is passed to a local LLM (Orca-Mini), which generates a natural-language answer in the style and content of your posts. All processing—embedding, search, and LLM inference—occurs on your laptop, ensuring privacy and avoiding cloud APIs.

Cool. I wonder though...

I'll tempt you to do something like this with the entire data file... a master philosophy forum bot... — Banno

Like an amalgam of everyone on TPF? Tempting though it is, my instinct is that the wider the range of data, the more it would approach a regular LLM like ChatGPT and lose its own point of view.

In my present immoral state, I'll tempt you to do something like this with the entire data file... a master philosophy forum bot... — Banno

I vaguely recall we were warned about this around 1818. Something to do with an unconventional Swiss scientist.

Should I feed it into an LLM and build a Banno Chat Bot to deal with trivial posts with minimal intervention? — Banno

I notice you didn't wait for an answer and just went ahead and did it. So preoccupied with whether or not you could, you didn't stop for more than a moment to think if you should (to paraphrase Michael Crichton).

As it stands it is only using the top 4 posts. I'll have a play and see if it can do more without being too slow. — Banno

So it's reading all your posts before it comes up with an answer each time? I guess the next step is to build your own LLM model that incorporates the words of Banno in its training data.

Out of curiosity, I am wondering whether Discourse was the only option able to accommodate the new laws. Were other options also capable? — Leontiskos

I think the alternatives (NodeBB, XenForo, etc) would have enabled us to accommodate the laws. I chose Discourse over and above that requirement because I liked the way it worked.

If you're interested, the main software requirements to accommodate the new laws are more configuration, crucially including the configuration of the sign-up form, and more moderation tools, crucially including the flagging and moderation of direct messages.

I don't suppose there's any chance of uploaded images being rehabilitated, on the archive site? — bongo fury

No, any images that survive in the archive will be those hosted externally. Images uploaded to this site will not be in the archive. And I can't restore any of the images I had to remove when we were running out of disk space.

If not, is there a deadline for replacing them with linked ones? — bongo fury

You could edit your posts and replace the images, including broken ones, with images hosted externally (see the help page on images), and you'll have until we shut down this site to do that. Around February or March.

Yes, the script I write will achieve the same thing. :up:

I'll be using JavaScript or Python, and I won't have much control over the web server redirects on the new site, so as I say I'll be putting the redirects on Cloudflare, so the requests won't even get to Discourse.

Preferably also sending a 301 Moved Permanently header. Provided you or another staff member created the archive site from scratch (not using a pre-boxed framework or library) your knowledge in such fields seems sufficient enough to do so easily (and most importantly: properly or safely). — Outlander

Yep. Well I've already created it, and it doesn't use a framework. It's just HTML, Javascript and JSON, and CSS. The build is done locally by running a python script, written from scratch (I used AI to speed up the process, while I played the role of its micro-managing project manager).

tpfarchive.com

Yeah, I was just thinking about that. The comment URLs on this site don't contain the dicussion IDs, so I can't just use a basic mapping. But I don't want to include an explicit individual redirect for every one of the million or whatever comments.

This would mean having the comment ID fed to a script that pulls up the discussion ID and then redirects the user to the relevant discussion URL prefixed with a hash anchor containing the comment ID. — Outlander

Yeah, I could probably use Cloudflare workers for that, combined with a key-value store to map the comment IDs to the discussion IDs, which I'll only need to produce once, based on the Plush export.

If your bookmarks are of URLs such as...

https://thephilosophyforum.com/discussion/comment/1026204

(the share link for your last comment)

... then I can probably set up redirects to the comment in the archive:

https://tpfarchive.com/discussions/16281-a-new-home-for-tpf.html#comment-1026204

So when you request the former you'll be redirected to the latter, where you should be able to see the comment in context. I'll make a note to set this up.

Note: the latter URL doesn't work right now because the data on the archive site is a few days old.

If you work in a restaurant, you try to separate stored goods from actual food production. And the idea is central to other means of production. So, I applied a pedestrian truism to a current situation. Not expecting a Pulitzer prize for that observation. — Paine

No no, I'm sure it's brilliant.

Let me see if I understand. The stored goods represent the content of TPF, which we can store independently of its use in ongoing discussion, the latter represented by meal preparation, i.e., "grille"?

Stored food = TPF archived content
The preparation of meals = TPF discussion

I was confused by two things. One, it seemed like you were saying "why don't you do this, it's a no-brainer" (and yet I didn't know what it was you were suggesting); two, I didn't understand "grille," but now I'm thinking you meant the grill as in the place in the kitchen where food is cooked, maybe.

I hope you appreciate the time I've taken over this analogy :wink:

Some of you might be wondering about redirects. Probably this is the best way:

1. All discussion URLs, i.e., all links to specific discussions/threads/OPs from the old site (this one), will be redirected to the relevant pages of the archive. This means threads which are indexed on Google and linked to from other sites will not leave a trail of broken links behind them. I'm not sure if it's worth doing this for individual comments too.

2. All other URLs pointing to this site, like categories, login, etc., will go to the new site. This will happen by default anyway, since the new site will have the domain.

The one thing I'm concerned about transitioning to another platform is that this website will just deteriorate over time without maintenance and the whole archive will be lost, something that's been happening to the rest of the internet with broken sourced hyperlinks everywhere. — Saphsin

This website, the one we're on right now, will not deteriorate---it will be completely closed early next year. The content, however, will live on, hosted by me on a static website (tpfarchive.com), which is read-only. I will maintain this indefinitely (though it won't need much maintenance).

If you have a better plan to preserve the archive, let me know. Maybe you want to put up the money to make a physical book?

Thanks Bret!

Looking at the last page of those discussion results, it seems The Philosophy Forum has been around for just over 10 years, or since October 20th of 2015. That's curious and wonderful the website is a decade old. — Bret Bernhoft

TPF: Curious and wonderful since 2015.

In fact, this site is a kind of continuation of forums.philosophyforums.com, which started probably in the early 2000s but collapsed around the time when this one started.

I don't know what you mean.

The new site should be good too.

Well I was only talking about the archive site, where it's less critical.

:up:

It's a little too slow from where I am. Maybe I'll paginate the results on the "your posts" page (you can get them all in a single file in the download).

But you can have a look at the archive site, built on a data export from a few days ago.

https://tpfarchive.com

I had the temporary URL of the new site showing here for a moment there but I was forgetting that since the legal stuff isn't in place I can't approve any new members yet.

Fine, carry on.

How do you mean? — Christoffer

I meant that I'm not on board with your rhetoric. Anyway I only came in here to shed some light on the political leanings of TPF. In any case, the conservatives on the staff are no more ignorant of the facts than the others, as far as I can see. Facts have nothing to do with it, and in general the accusation that the other side ignores the facts is thrown from both sides. It's a nothing argument, on its own.

Yet, there's also a point to be made that many arguments that rely on facts tend to be called leftist. There's far more climate science deniers or general deniers of scientific results on the conservative right... so if a human's best attempt to reach a truth based on facts and scientific data is considered "leftist", then I guess that tells more about the political spectrum than that this forum is "unbalanced". — Christoffer

I don't agree with any of this, and I'm very left.

On the staff we have between 2 and 2.5 Marxists, 2 left-liberals, and 2 conservatives. I guess that's more left than right.

Why is it better for me to turn it off globally than for you to ignore it?

I can think of reasonable answers to that—I just want to understand the opposition.

As for downloading, you can just do Ctrl+p and then save as PDF. This will work on the new site (you can try it here on another Discourse forum) and also on the archive site.

It sounds like you're talking about ongoing discussions, but all the discussions on the archive site will be finished, static, frozen in time. So I'm confused about what you're looking for. You can't "enter the fray" on the archive site.

As it happens I built a first version of the archive site yesterday and the way it works is it shows the full discussion on one page.

The new site doesn't have pagination at all. Every discussion will be infinite scroll. There's an AI "Summarize this topic" button too, but I haven't tried it, and it can be turned off.

The archive will exist as a website, where you will be able to see all the discussions and posts. It will have some kind of search functionality, and it will have a "Your posts" page where you can view and download your posts (or anybody else's: you'll just enter the username). You'll be able to access that download while you're offline, if you keep it on your device (obviously).

If you need to access everything offline, and not just your own posts...well, it's possible for me to provide a package you can run locally, like a private website, but I find myself wondering why you would need to.

Since the old site will be preserved, I wonder if the "advanced search" functions that work now will work there — Paine

The content will be preserved but it won't look or work like the existing site. It depends how much time I spend on building it. Searching by author or discussion title are easy enough — not sure about a full text search.

Thanks Jamal, the projected new forum sounds great. Is there any way of exporting one's posts from the old forum as a word file? — Janus

I'll have all the data and I'll be able to provide members with a posts export on request. In fact I might provide it at the archive site.

All reality is subjective private mental state — Corvus

It proves that Banno has never read Hume or Kant — Corvus

Kant wrote his massive tome to show this is wrong.

Mind in Life — Wayfarer

Sent me to sleep, that one. I may try again one day.

Yeah that's more like it :grin:

reason is just a tool — Tom Storm

As Wayfarer might be about to point out, this is the problematic Enlightenment notion of reason which is in question.

On the other hand, could you tell me if there will be a way to fine-tune the settings to hide topics I don't want to see (in case I want to create an echo chamber and not know what people think about certain things?) — Astorre

Yes, on the new platform you'll be able to mute topics and whole categories, so that they don't appear on the /latest page.

I also wanted to suggest, if appropriate, adding more sections—for example, metaepistemology or axiology—so that I could narrow my choices a bit more — Astorre

We can always make subcategories but I don't think those ones would be used enough to merit that.

I also thought the layout from the old site was better in certain ways (although it had countless bugs and unreliability problems) because it showed the categories and the posts by recency by each category and not just everything at once. What happens under our current system is that if 10 people come up with new religion posts (for example), the main board is overwhelmed with that and it looks like that's the ony thing being discussed. If posts are divided by category, that doesn't happen because you can just not pay attention to those categories you're not interested in. I don't know if the new software addresses that or not though. — Hanover

I've made the home page of the new site show topics from all categories ordered by most recent, just like here. But unlike here, if you go to the "All categories" page the categories are ordered by most recent—so that page would fit what you'd like to see as the home page (Philosophy of Religion would show at the top but wouldn't overwhelm the page). Personally I wouldn't want that as the home page. Maybe I'll do a poll.

On the old site I always went to /latest or whatever it was, which was the same as our home page here.

Anyway in Discourse it's all configurable.

Had some problem receiving emails from this site when changing passwords. If the move over to the new site requires an email invite for the current account, what happens if it fails? — Christoffer

It won't be invitation-only and you'll be able to just go to the site and sign up yourself. I'll be making an announcement when it's open for new sign-ups. As I say, around March.

Wouldn't there be an email list for all current members? So that taking that list into approved members for the new site will work and when signing up the email is already registered on an approved list there.

Meaning, using the email you registered on this site will let you into the door of the new site when registering. — Christoffer

It's essential that all users of the new site have read and explicitly agreed to the new Acceptable Use Policy. New users confirm that when they sign up. So there won't be any pre-approved accounts.

Also, what happens to stuff hidden on this site from people not logged in? Like the short stories? If there's stuff like that disappearing from view, maybe that should be moved over to the new site? — Christoffer

Since I have to write some code that turns the existing site into a static site (based on the data export), I can make everything visible (obviously not private messages).

EDIT: Actually you'll need to verify your email when you sign up. If emails from infoatthephilosophyforumdotcom are not reaching you, that will be an issue.

Even if they're on old-fashioned software I think we should celebrate the continuing existence of independent discussion forums. Not everyone wants to discuss everything on Reddit.

Yes, I had a look at that when I was researching NodeBB. To be fair—and because I'm reflexively argumentative—dull isn't necessarily bad for a forum, and NodeBB can be customized extensively. What matters more to me is how smoothly everything works. Discourse in my opinion just runs better, always looks nicer due to better all-round design and compatibility of themes and colour schemes, and setup is less of a struggle.