Edit robots.txt in WeWeb

How do you edit the robots.txt file in WeWeb? I tried adding one manually to my custom domain and via the “Files” in WeWeb but it doesn’t change. There is a default WeWeb one and I need to amend it. Thanks.

Hi @seo :wave:

This is currently not possible. It’s on the roadmap but I don’t have an ETA yet. Can you tell me more about your use case and the timing within which you’d need the feature so I can share with the tech team?

Thanks for the response, @Joyce! I just don’t want my app indexed in search engines. That might sound strange but I am hosting the Vue app from WeWeb on a subdomain of my Wordpress website and I’ll be creating all the pages I would need for SEO on the www. Rather than random pages from my app being arbitrarily indexed in Google (most of the app is behind a login) I’d rather just tell bots not to crawl the app for now and amending the robots.txt is the best way to do that.

1 Like

Got it!

For now, I think you need to do it page by page:

1 Like

Thanks for the info. That will add a “noindex” HTTP header to each page it’s enabled on which is OK for the time-being. Appreciate the follow-up and hopefully we’ll be able to customize the robots.txt file in the future :slight_smile:

You can customize the robots.txt if you self-host. Its some work but hours not days. The “weweb-server” component is not required in the vast majority of cases: it’s just a matter of hosting your html/css/js elsewhere.

We’ve had a number of State Change members who did this for various advanced hosting needs like the one you describe.

1 Like

Thanks very much @raydeck, that’s probably the only solution right now. Will bear that in mind as an option.